QCon is one of my favorite conferences.  Although it’s a relatively small gathering compared to TechEd or JavaOne, it always has an awesome selection of sessions and speakers.  Last year was amazing, and I had the honor of presenting a session (Patterns for Cloud Computing) in one of the many great tracks.

This year (San Francisco, Nov 3 – 5, 2010), I’ll be participating in a panel to talk about data in the cloud.  For most organizations, I think this is one of the most important aspects of moving applications to the cloud.  For example: How should organizations store data in the cloud?  What type of storage is best (relational, non-relational, structured, blob, something else)?  How do organizations manage encrypted data in the cloud?  All in all, I’m really looking forward to an exciting discussion.  Fellow panelists include Amir Awadallah (CTO of Cloudera), Damien Katz (Creator of CouchDB), and Roger Bodamer (SVP of MongoDB).

If you are planning on attending this year, I look forward to seeing you there.  If you are not, and would like to get a discount on your registration, feel free to use the following promotional code: GUES100.  By using the code, you’ll receive $100 off your registration price and the organizers of the event will donate $100 to the St. Anthony Foundation in San Francisco. 

But wait!  There’s more!  If you use the registration code I’ll donate an extra $100 to St. Anthony’s for the first ten attendees.  So that’s a great conference, $100 off the ticket price, plus $200 on your behalf to a great cause!  Hope to see you there!

Many applications store data using the file system (e.g. read / write a file to d:dataconfig.xml).  Moving these types of applications to the cloud can be difficult as the storage mechanisms in the cloud are somewhat different. 

To help address this problem, in Feb 2010 we released v1.1 of the Windows Azure SDK which supports a new feature call Cloud Drives.  A Cloud Drive is the ability to mount a virtual drive in the cloud so that applications can write to it as they would a local drive.  This cloud drive is persisted so that even if the machine goes down, the data is not lost.

What many people don’t realize however is that the format that the Cloud Drive uses is interchangeable with the VHD (Virtual Hard Drive) format used by virtualization products such as Hyper-V, Virtual PC, Windows Server, and Windows 7.  What this means is that you can create a single VHD, use it locally on a Windows PC, attach it to a virtualized image, and now copy it to the cloud and access it there also.  As you can imagine, for anyone looking to migrate applications to the cloud, this opens up a world of possibilities.

In this post, I’ll walk through a series of steps that show creating a VHD from scratch, copying it to the cloud, and accessing it from a web role running in Azure.

Before you begin

You’ll of course need an Azure account.  If you don’t have one, you can go here to sign up.

Once you have your account, you’ll need to setup two new services. 

You’ll need to create a storage account – this is where the VHDs will be stored – and a hosted service to run your application from.  When you set them up, make sure that they are created in the same geography (i.e. don’t have your storage account in Asia, and your hosted service in the US as the drive mounting has to be in the same region).

Once you have your storage account and service created, you are ready to create and upload your VHD. 

Step 1.  You’ll need a VHD

If you already have a VHD file that you would like to use, you can skip to Step 2.  You’ll be copying this VHD to the cloud, so you may want to choose a smaller VHD file for test purposes.

If you are running Windows 7, creating a new VHD is easy. 

Firstly, go to the start menu and type disk management. 

From the start menu click on Create and format hard disk partitions

This will open up the disk management tool.  From the Action menu, select Create VHD.

Enter a location for the VHD file and a size.  The minimum is 16Mb and the maximum is likely the amount of free disk space on your machine. 

When your disk appears in the management tool, right click on the disk name and select Initialize Disk.  Select the default (MBR) and select OK.

After initialization, right click on the disk and create a New Simple Volume.  Walk through the wizard, accepting the defaults and assign a drive letter to the VHD.  Finally, format the disk using NTFS (a quick format is fine).  If a dialog box appears asking you to format the disk (sometimes the disk gets detected just before formatting has begun), just hit cancel.

After this is done, you can open the drive and access it as you would any other drive. Add some files and folders, etc.

When you are ready to upload to the cloud, you’ll need to detach the drive.  To do this, go back in to the disk management tool, right click on the disk name and select Detach VHD.

(Do not select the Delete the virtual hard disk file checkbox – otherwise, they’ll be nothing to upload!)

Step 2.  Upload it to Azure Storage

The next step is to upload the VHD to the cloud.  The VHD will be stored in something called Blob (Binary Large OBject) storage.  You may have heard of blob storage if you’ve seen any of the introductory Azure presentations.  It’s the type of storage that is used for binary data such as images, videos, and the like.

In February we enabled support for Page Blobs.  Page Blobs are binary objects stored in the cloud that support random read/writes through pages.  This makes them ideal for dealing with VHD type of access, and this is the format that we’ll be using to upload our VHD to the cloud.

The one caveat of this is that you must use a tool that supports page blobs (most of the tools available today only support regular blobs).  The tool I would recommend is Cloud Storage Studio by Cerebrata as it supports page blobs through a nice interface.

Open the tool and access the storage account that you created in the “Before you begin” section earlier in this post. 

I would recommend creating a new container for your VHD images to keep them separate from other content.  As shown in the screenshot above, I’ve created a container called vhd.

Select the container, and click on the Upload Page Blob button.  There are two upload buttons – you need the one with the icon that includes the page and the up arrow.

Point to the location of your VHD file and click on the Upload button.

This might take a little way depending on the size of your VHD and the speed of your network.

Once you have uploaded the file, validate that it exists by hitting the refresh button.

With the VHD uploaded, we can now move on to creating the cloud application that accesses it.

Step 3.  Accessing the VHD from Windows Azure

I’m going to assume that you have some knowledge of creating an ASP.NET application and uploading it to Windows Azure.  If you haven’t done this before, I would recommend checking out the Windows Azure Training Kit.

To access the VHD in the cloud, we create a webrole and use the following code.

Add reference to the Microsoft.WindowsAzure.CloudDrive DLL that can be found in the GAC (assuming that you have the v1.1 SDK installed).  Then, in your webrole, you’ll need to add the following using statements:

using Microsoft.WindowsAzure;
using Microsoft.WindowsAzure.ServiceRuntime;
using Microsoft.WindowsAzure.StorageClient;

Here is the code that you should run in your web role.  (You can place this behind a button or something…)

// Setup the account details
String accountName = "";  // insert your storage account name here
String accountKey = "”;  // insert your storage account primary key here

StorageCredentialsAccountAndKey credentials = new StorageCredentialsAccountAndKey(accountName, accountKey);

CloudStorageAccount storageAccount = new CloudStorageAccount(credentials, new Uri(String.Format("http://{0}.blob.core.windows.net",accountName)),
                                                                            new Uri(String.Format("http://{0}.queue.core.windows.net", accountName)),
                                                                            new Uri(String.Format("http://{0}.table.core.windows.net", accountName))
                                                                            );

// initialize the local cache - need to trim the end backslash otherwise, HRESULT=D0000033
LocalResource localCache = RoleEnvironment.GetLocalResource("LocalCache");
CloudDrive.InitializeCache(localCache.RootPath.TrimEnd('\'), localCache.MaximumSizeInMegabytes);

// Create and mount the drive
CloudDrive c = storageAccount.CreateCloudDrive("vhd/myvhd.vhd");
String driveLetter = c.Mount(localCache.MaximumSizeInMegabytes, DriveMountOptions.None);

// do regular System.IO.File and System.IO.Directory operations here against driveLetter
Response.Write(System.IO.Directory.GetFiles(driveLetter + "\").Count() + " files found on the root of the mounted drive.");

// unmount the drive
c.Unmount();

There are five main pieces to the code.

Firstly, the storage account reference is created using the account name and key that you retrieve when you setup your Azure account.

After that we need to initialize a local cache used by the cloud drive.  To create this, insert the following code into your ServiceDefinition.csdef file on your webrole.

<LocalResources>
  <LocalStorage name="LocalCache" cleanOnRoleRecycle="false" sizeInMB="100" />
</LocalResources>

We then create and mount the drive using the CreateCloudDrive and Mount methods.  Make sure that the path and name to your VHD matches what you uploaded in Step 2.

Once the drive is mounted we get a drive letter (as a string – e.g. “f:\”).  You can then make calls to that drive letter using the regular calls (e.g. System.IO.File and System.IO.Directory).  In a worker role you can even launch applications from the VHD by spawning a new process. 

Finally, to unmount the drive, just call the Unmount method to release the VHD image.

There are a lot of details behind the scenes that I’m not going to go into here (e.g. read and read/write access to the VHD)  – if you want to learn more, I would recommend checking out this PDC session by Brad Calder

Appendix.  Running this in a local development environment.

You should note that uploading a VHD only works using a live deployment in the cloud.  In the local development fabric, you’ll find that you cannot access a remote VHD (you’ll get a HRESULT = 80070003 error).  Also, you can’t upload a page blob to local development storage.

However, you can still duplicate the same functionality in the local development environment by using this code:

// Setup the account details
CloudStorageAccount storageAccount = CloudStorageAccount.DevelopmentStorageAccount;
// initialize the local cache - need to trim the end backslash otherwise, HRESULT=D0000033
LocalResource localCache = RoleEnvironment.GetLocalResource("LocalCache");
CloudDrive.InitializeCache(localCache.RootPath.TrimEnd('\'), localCache.MaximumSizeInMegabytes);

// Create and mount the drive
CloudDrive c = storageAccount.CreateCloudDrive("vhd/myvhd.vhd");

// try creating the drive if it doesn't exist
try
{
        c.Create(16);
}
        catch (Exception)
{
         // ignore - VHD folder already exists
}

String driveLetter = c.Mount(localCache.MaximumSizeInMegabytes, DriveMountOptions.None);

// do regular System.IO.File and System.IO.Directory operations here against driveLetter
Response.Write(System.IO.Directory.GetFiles(driveLetter + "\").Count() + " files found on the root of the mounted drive.");

// unmount the drive
c.Unmount();

There are a couple of differences from the live environment.

Firstly, instead of using the public account details, we can just make a call to CloudStorageAccount.DevelopmentStorageAccount. 

Secondly, just before we mount the drive we actually try to create it using the c.Create(16) command.  If it already exists, then it throws an exception that we just ignore.

It’s important to know that this doesn’t actually create a VHD file. Instead, it creates a new directory in this folder:

c:users[username]AppDataLocaldftmpwadddevstoreaccount1vhd

You can now access this folder through Windows Explorer, go into the .vhd folder and add/remove files as you wish.  Once you are ready to move to a production environment you simply copy the files from this directory in to a new VHD (as created in Step 1), upload, and switch to the production code.

http://channel9.msdn.com/ShowPost.aspx?PostID=275794

http://msdn2.microsoft.com/en-us/architecture/ms954605

I can't believe it's been a year since I was at JavaOne last year – it seems like only yesterday I was getting my picture at Duke's 10 year birthday!  (only kidding, of course )

Although I'm not in San Francisco for this year's conference, Microsoft do have a couple of sessions and a booth on the trade floor.  I've been following Mohammed's excellent blog on the in's and out's of the event.  If you in the Moscone Center, be sure to track down the team!

New features include support for .NET Fx 2.0 and Visual Studio 2005, together with partial classes, nullable types, SOAP 1.2, generics and a host of other goodies.

1. Uses WSDL.  A Web Service should expose its service contract using WSDL.  If it can’t give you a WSDL document, it’s probably just XML over HTTP…

2. Uses SOAP.  All messages sent from and received by the Web Service must use SOAP formatting.  If it’s not using SOAP it’s probably just XML over HTTP…

3. Uses XSD.  All data types in the SOAP payload must be XSD compliant.  No platform native types are allowed.  If it’s not using XSD it’s probably just XML over HTTP…

4. Uses XML.  The underlying messages should of course be formatted using XML.

5. No Arbitrary Binary Data.  The message payload should 7 bit ASCII and should contain no embedded binary blobs.  Any binary data passed over a Web Service should be sent using either SwA, DIME or MTOM (preferably MTOM).

6. Transport is likely to be HTTP.  Although not a requirement, the majority of Web Services today use HTTP as the transport.  Compliant Web Services should definitely work over HTTP.

7. Discovery can be through UDDI.  Again although not a requirement, it should be possible to host the Web Service endpoint using UDDI.

8. Agreed Versions of Specifications.  The versions of the above specifications (WSDL, SOAP, XSD, XML, HTTP, UDDI) should be in line with the latest version of the WS-I Basic Profile (http://www.ws-i.org) – to ensure Web Service compliance between vendors.

9. Operations should be Document Style.  Operations to/from a Web Service should be Document/Message Style (e.g. SendOrder(order o)).  RPC style should be avoided (e.g. SetOrderLine1(orderId id)).

10. Should be compliant with WS-*.  Compliant Web Services should be able to accept WS-* payloads and extensions for Security, Reliability and Transactions (although not all stacks today support these yet).

I came back from vacation today to find I had won a "JavaOne 2005 Speaker Award" for being in the top 20 of speakers at JavaOne this year . 

Here's a list of the top 20 sessions, and if you'd like to watch the session (I co-presented with Raghavan Srinivas from Sun), you can find it here.