Cloud Service Fundamentals in Windows Azure

The "Cloud Service Fundamentals" application, referred to as "CSFundamentals," demonstrates how to build database-backed Azure services.  This includes a description of the scenario, implementation architecture and reusable components for logging, configuration and data access.  The code base is intended to be a tangible exploration of best practices for delivering scalable, available services on Azure based on production deployments by the Windows Azure Customer Advisory Team.  

Creators:  Michael Thomassy, Rafael Fernandez, Mark Simms, Christian Martinez, Ewan Fairweather, James Podgorski, Paolo Salvatori, Rama Ramani, Shaun Tinline-Jones, Silvano Coriani, Valery Mizonov, Gus Apostol

Note: The complete version of this "Description" page is contained in the Microsoft Word document titled "Exploring CSFundamentals in WA.docx" as part of the "C# Download" linked above. The purpose of the rest of this page is to highlight each section in the document and describe the topics covered in those sections.

Details of indiviudal components are also documented on the Cloud Service Fundamentals wiki collection.

Scenario Description

Many modern cloud services integrate social experiences, based on a foundation of registering and managing per-user information.  The intent of the CSFundamentals code reference is to demonstrate several best practices and experiences for implementing modern data-driven applications on Azure in context.  The code base implements the following functionality:

• MVC4 web application providing scalable user registration and login against a sharded database with distributed cache integration
• Multi-cloud service deployment, using Application Request Routing (ARR) to transparently leverage multiple cloud services for additional scale and reliability
• Queryable operational data store, with scheduled tasks for collecting and integrating application and server performance/health metrics.

Prerequisites

The reference implementation was developed using Visual Studio 2012 Ultimate; however, there shouldn't be issues with other editions.  The following items need to be installed:

• Windows Azure SDK v2.0. Recommended to install through the Web Platform Installer. 
• Microsoft® SQL Server® 2012 Express (for the sqlcmd.exe utility).
• For SQL Reporting, install Microsoft SQL Server Data Tools - Business Intelligence for Visual Studio 2012.

Visual Studio Project Layout

The CSFundamentals applications incorporates several projects in one solution and three cloud service definitions for use by developers.  This section provides an overview of these artifacts, the directory structure and third party libraries.

Visual Studio Solution

The code base is contained in one Visual Studio solution, CloudServiceFundamentals.sln.  Open this Visual Studio solution to work with the application code, reusable libraries, and publish the application services to Azure.

Visual Studio Projects

The overall solution consists of several projects; with reusable and scenario specific code separated.

• Common:  Reusable components.
  • Microsoft.AzureCat.Patterns.Common.
    Reusable baseline aspects of the codebase, including configuration, logging, serialization and helper base classes.
  • Microsoft.AzureCat.Patterns.Web.
    An implementation of the MVC4 providers and configuration utilities designed to work against a scale-out relational data service
• Data:  Libraries for working with data stores.
  • Microsoft.AzureCat.Patterns.CacheClient.AzureCaching.
    Provides a wrapper around the Azure Caching client SDK, including the use of a pluggable binary serializer (in this case protobuf-net)
  • Microsoft.AzureCat.Patterns.Data.Common.
    Helper classes for working with data storage, relational and non-relational, including retry and telemetry logic for working with Windows Azure SQL Database.
  • Microsoft.AzureCat.Patterns.Data.SqlAzureDalSharded.
    Helper classes for implementing a scale-out relational database solution, including partitioning,  connection affinity and metadata management.
• Database:  Database schema definitions.
  • Microsoft.AzureCat.Patterns.Data.SqlAzure.OpsStatsDB.
    The schema for the operations database, used to centralize telemetry information from compute, storage and SQL.
  • Microsoft.AzureCat.Patterns.Data.SqlAzure.RootDB.
    The database project for the user metadata and shard management databases.
  • Microsoft.AzureCat.Patterns.Data.SqlAzure.UserProfileDB.
    The database project for the user profile and information databases.
• Logic:
  • CSFundamentals.Logic. Application specific logic and data schema.
• Roles:  Cloud service web and worker roles.
  • CSFundamentalsCaching.WorkerRole.
    Hosting the Azure Caching web roles providing caching for the CSFundamentalsService.WebRole.
  • CSFundamentalsService.WebRole.
    The public web site exposed by the MVC front-end web application as an Azure Web Role.
  • RouterService.WebRole.
    The worker role hosting the cookie based application request routing (ARR).
  • SchedulerService.WorkerRole.
    The worker role hosting the Quartz.NET scheduler
• Router:
  • Microsoft.AzureCat.Router.LoadBalanceThenCustomCookieProvider.
    Load Balance requests based on the cookie contents routing traffic to multiple
    cloud services (multiple CSFundamentalsServices)
• SQLReporting: Telemetry Reports
  • OpsStatsDBReporting.
  • Parameterized reports (.rdl) using SQL Reporting from the OpsStatsDB database. The reports show query stats, database size & growth, wait times and errors across all sharded databases.
• Tasks:
  • Microsoft.AzureCat.Patterns.Tasks.SqlAzureMgmt.
    Collection of scheduled tasks using the Quartz scheduler to collect and load
    telemetry data from different sources into a Windows Azure SQL Database for
    exploration and charting.
• CSFundamentalsService: Azure web service hosting the front-end CSFundamentalsService.WebRole
  and CSFundamentalsCaching.WorkerRole.
• RouterService: Azure web service hosting the RouterService.WebRole.
• SchedulerService: Azure web service hosting the SchedulerService.WorkerRole.

Shared Libraries

The reference implementation makes use of multiple shared libraries, managed through NuGet packages. The Microsoft libraries distributed through NuGet used by this solution include:

• Reactive Extensions (library for supporting asynchronous and push-based operations)
• Enterprise Library’s Transient Fault Handling Application Block

The non-Microsoft libraries used by this solution:

• Dapper (micro-ORM, used in the database access code)
• Quartz.NET (scheduler library, used for managing recurring tasks)
• Common.Logging (used by Quartz.NET)
• NLog (application logging library)
• Protobuf-net (binary serializer)

Reusable Components

There are several reusable aspects bundled with the CSFundamentals application.  This document provided describes the components, explains their design, configuration and potential applicability to your own projects. These components include:

• Configuration to integrate managing multiple configuration files
• Logging using NLog to capture application diagnostic information
• Data Access to sharded databases using Windows Azure SQL Database
• Caching using the Azure Caching in Windows Azure SDK 1.8
• Scheduling using the Quartz.NET scheduler to manage Telemetry collection
• Application request routing to set affinitization to a hosted service
• Reports on Telemetry data stored in the OpsStats database using Windows Azure SQL Reporting of the application Logging information collected by the Scheduler

Social Scenario Implementation

The CSFundamentals application implements a custom membership provider against a scaled-out relational store based on Windows Azure SQL Database.  For simplicity, the CSFundamentals code sample does not focus on functional capabilities but does provide four very simple social networking features in the MVC Web UI for purposes of demonstrating the key Azure implementation concepts.

1. User Registration
2. User Login
3. Add Comments
4. Retrieving Comments

 The focus of the implementation is on these core concepts:

• Database Design: a scaled-out relational data store based describing the design, partitioning model and scale-out aware data access layer implemented by the CSFundamentals application.
• Shard Map: the application's data access layer supports sets of sharded databases called the "Root" databases and the "UserProfile" databases with the shard map contained in the file “RootUsersShardMap.xml".
• Implemented Features: a description of the implementation for registration, login, add comment and get comment using the scale-out database design, shard map and code snippets from the data access layer.
• Telemetry Monitor: details of the task-driven telemetry collection and transformation service and how the SchedulerService aggregates data from multiple sources (Windows Azure Diagnostics, the Root and UserProfile sharded Windows Azure SQL Database, etc.) into a consolidated destination database.

Configuration and Deployment

In the Configuration and Deployment section, a description is provided for each step to configure, build and deploy the code sample to Windows Azure.  These steps include:

1. Get an Azure subscription and create services for the CSFundamentals services: Hosted Services in Windows Azure, Azure SQL Database server and databases, Windows Azure Storage account
2. Download the CloudServiceFundamentals solution and C# source code from the link above
3. Create Certificates - management certificates, certificate for Remote Desktop usage and creating SSL certificate (the SSL certificate will be added to the RouterService project)
4. Change the placeholder “csfundamentals_” settings in the appropriate project files to your Windows Azure service values including modifying the QuartzJobs.xml and the RootUsersShardMap.xml
5. Package and deploy the 3 hosted services in the CloudServiceFundamentals Visual Studio solution:
   1. CSFundamentalsService
   2. RouterService
   3. SchedulerService

For these steps, refer to the word document provided. This details the configuration files and project files that require modification and customization for your subscription's services.

Building the CloudServiceFundamentals Solution

To build the solution, the CloudServiceFundamentals.sln file relies on enabling NuGet packages to be automatically downloaded by Visual Studio. These packages include Microsoft and 3rd party libraries previously listed. Typically, the Release BUILD configuration is selected which is recommended to use the provide Powershell scripts for configuration and deployment.

Configuration and Deployment from Visual Studio

The configuration and deployment sections describe how to modify the files to enable the solution and services to be deployed either directly by Visual Studio or through the use of Powershell scripts using the Windows Azure cmdlets. When using Visual Studio for deployment, there 9 files that require manual changes before the projects can be built and the service packages created.  The Powershell scripts help simplify modifying these files programmatically.

Configuration and Deployment from Powershell

The Powershell scripts provided for configuring and deploying the CloudServiceFundamentals solution to Windows Azure are located under the “PowerShellDeployment” directory. The scripts are organized and numbered in the order they should be executed along with a description of the steps.  There are manual steps involved too that must be followed in the order described which include uploading the management certificate to the Azure portal, modifying the RoutingService project (selecting your SSL certificate) and creating the deployment packages for each service from Visual Studio.

More Resources

For additional reading and background, please refer to the following resources:

• Building Big: Lessons learned from Windows Azure customers - Part I (Scale)
• Building Big: Lessons learned from Windows Azure customers - Part II (Availability)
• Best Practices for the Design of Large-Scale Services on Windows Azure Cloud Services
• Failsafe: Guidance for Resilient Cloud Architectures

Version History

• v1.10: April 12, 2013
  • Renamed the code gallery project from "ContosoSocial" to "Cloud Service Fundamentals" with short names CSFundamentals and CloudServiceFundamentals.
  • Added SQL Reporting (.rdl) reports project.  These reports are provided for use against the OpsStatsDB Telemetry database.

• v1.20: July 30, 2013

  • Updated projects to support Windows Azure SDK 2.0
  • Updated retry logic support using Enterprise Library’s Transient Fault Handling Application Block