I’ve been learning about using Queues in Azure and thought the best way to learn was to start writing an application to test this. My example site will probably only run on single instances; but I decided from the start to write something that will be scalable so came up with the following principle
The client/website can only READ from the database. Anything that requires updating the database must be done via the back end. The client/website must be able to cope with the fact that the requested update will not happen immediately.
The upshot of this, I came up with a job system that uses Azure queues to pass work to the back end worker role. For the moment, I decided to have one queue which handled multiple types of jobs. This lead to me creating a job base class which will cover the common things that the different jobs required. I also decided to use an XML format to describe the job. I need to make sure though that the jobs do not grow bigger than the size allowed for an Azure Queue job (8k).
Below is the base class which is inherited in the actual job classes:
Imports Microsoft.WindowsAzure
Imports Microsoft.WindowsAzure.StorageClient
Imports Microsoft.WindowsAzure.ServiceRuntime
Public MustInherit Class tjob
' Enable database access
Protected Friend _db As New TyranntDB
' A place holder for the user information
Protected Friend _usr As tUser
' setup variables to allow for access to Azure Queues
Protected Friend _storageAccount As CloudStorageAccount
Protected Friend _jobQueue As CloudQueue
' A sample of a job entry in the Queue
Private sample = <job type="email" user="102">
<type/>
</job>
' the user ID which is used to pull the user information
Private _userID As Int64
' Initialise the job from the XML String
Public Sub New(jobStr As String)
Dim jobXML As XElement = XElement.Parse(jobStr)
_JobType = jobXML.@type
Dim usrstr As String = jobXML.@userid
UserID = Convert.ToInt64(usrstr)
End Sub
' Create a blank job, this is used for creating a job to
' put onto the queue.
Public Sub New()
_JobType = ""
_userID = -1
End Sub
' Job type. Used to create the correct object.
Public Property JobType As String
' The user ID. If this is being set then it
' will look up the user from the database
Public Property UserID As Integer
Get
Return _userID
End Get
Set(value As Integer)
_userID = value
If _userID > 0 Then
GetUserDetails()
End If
End Set
End Property
' This is the code that "Processes" the job. Each job type must
' implement this code.
Public MustOverride Function Process() As Boolean
' A general variable for storing any errors that
' occur. If it's empty then no errors are assumed.
Public Property ErrorMessage As String
' This will generate an XML element that describes the job.
Public MustOverride Function ToXML() As XElement
' This will generate a string version of the XML
' which describes this job.
Public Overrides Function ToString() As String
Return ToXML.ToString
End Function
' This routine will pull the user information from the
' database and store the user detals in the _usr object.
Protected Friend Sub GetUserDetails()
Dim q = From u In _db.users
Where u.ID = _userID
Select u
If q.Count > 0 Then
_usr = q.Single
End If
End Sub
' If the job is being created. This function will add the job
' to the Azure Queue.
Public Sub AddJobToQueue()
' Get the azure storage account object.
_storageAccount = CloudStorageAccount.Parse( RoleEnvironment.GetConfigurationSettingValue(TyranntSupport.Constants.STORAGE_CONNECTION))
' Now get the queue client.
Dim client As CloudQueueClient = _storageAccount.CreateCloudQueueClient
_jobQueue = client.GetQueueReference(Constants.QUEUE_JOBS)
' Create the queue if it doesn't exist.
_jobQueue.CreateIfNotExist()
Try
' Now add the job details to the queue.
Dim msg As New CloudQueueMessage(Me.ToString)
_jobQueue.AddMessage(msg)
Catch ex As Exception
_ErrorMessage = ex.Message
End Try
End Sub
End Class
The TyranntDB class is an Entity Framework code first class that allows access to the SQL Azure database. Any class prefixed with a t (E.g. tUser) is a database table class. You will notice a Sample variable using XML Literals. This is just as an example to show how that job is formed in XML. Each inherited class will have it’s own sample.
All job classes need to be able to add themselves to the Azure Queue. They also need to be able to access the SQL Azure database. These features were written into the base class. All job classes must also have a way of being “Processed” which meant adding a MustOverride function called Process. They must also be able to export themselves as an XML document which is why the MustOverride function called ToXML is added.
The website uses Forms Authentication to enable it to validate users. But I also have a site specific user table to add extra information too. As this involves updating the database, this is the first “Job” that needs creating:
Public Class tjNewUser
Inherits tjob
' an example of a new user job
Private sample = <job type="newuser" user="-1">
<user name="{username}" email="{user)@{domain}">Full Name</user>
</job>
' Extra data required by this class
Public Property userName As String
Public Property email As String
Public Property fullName As String
Public Sub New(jobStr As String)
' initialise basic information
MyBase.New(jobStr)
Dim jobXML As XElement = XElement.Parse(jobStr)
' now initialise new user information
_userName = jobXML...<user>.@name
_email = jobXML...<user>.@email
_fullName = jobXML...<user>.Value
End Sub
Public Sub New()
' initialise the base information
MyBase.New()
JobType = "newuser"
_userName = ""
_email = ""
_fullName = ""
End Sub
' Create the new user in the database
Public Overrides Function Process() As Boolean
' first check to see if the user already exists
Dim r = From u In _db.users
Where u.username = _userName
Select u
If r.Count > 0 Then
' User already exists so do not continue
' return true in this case as request
' has been processed more than one.
Return True
End If
' create a new user
Dim usr As New tUser
' populate the generic information
usr.username = _userName
usr.email = _email
usr.fullname = _fullName
' now set the user group to be member
Try
Dim grp As tUserGroup = _db.GetUserGroup("member")
If IsNothing(grp) Then
_db.CreateBaseGroups()
usr.usergroup = _db.GetUserGroup("member")
Else
usr.usergroup = grp
End If
Catch ex As Exception
ErrorMessage = ex.Message
Return False
End Try
' now save the user
Try
_db.users.Add(usr)
_db.SaveChanges()
Catch ex As Exception
ErrorMessage = ex.Message
Return False
End Try
' Now that the user was sucessfully created,
' generate a new user email job
Dim jb As New tjEmail
jb.EmailType = "newuser"
jb.From = "mail@me.uk"
jb.UserID = usr.ID
' Add the job to the Azure job queue
jb.AddJobToQueue()
If jb.ErrorMessage = "" Then
Return True
Else
ErrorMessage = jb.ErrorMessage
Return False
End If
End Function
Public Overrides Function ToXML() As XElement
Return <job type="newuser" userid=<%= UserID %>>
<user name=<%= _userName %> email=<%= _email %>><%= _fullName %></user>
</job>
End Function
End Class
I’ve added the extra properties required for adding a new user and the extraction of these properties from the XML. The process function is also created which will create the user row in the users table. Hopefully the comments in the code should explain the process to do this. This routine also makes use of XML Literals which is a VB only thing at time of writing. (For example used in the ToXML and New functions.
As you can see at the end of the processing, we need to send a confirmation email to the user who has created the account. This kind of thing is also ideal for the back end to deal with hence being handled by the job queue system:
Imports System.Net.Mail
Public Class tjEmail
Inherits tjob
' a sample email job
Private sample = <job type="email" user="102">
<email from="mail@me.uk" type="newuser"/>
</job>
' setup extra information required by this job
Private _from As String
Private _emailType As String
' The is the from email address
Public WriteOnly Property From As String
Set(value As String)
_from = value
End Set
End Property
' This will be the email type e.g. newuser
Public WriteOnly Property EmailType As String
Set(value As String)
_emailType = value
End Set
End Property
' If the job XML already exists this will set up
' the information automatically
Public Sub New(jobStr As String)
MyBase.new(jobStr)
Dim jobXML As XElement = XElement.Parse(jobStr)
_from = jobXML...<email>.@from
_emailType = jobXML...<email>.@type
End Sub
' Create an empty email job if creating a new job
Public Sub New()
MyBase.New()
JobType = "email"
_from = ""
_emailType = ""
End Sub
' Send the email
Public Overrides Function Process() As Boolean
Dim email As MailMessage
' Generate the correct body of the email
Select Case _emailType
Case "newuser"
email = GenerateNewUserEmail()
Case Else
ErrorMessage = String.Format("Email Type [{0}] not recognised", _emailType)
Return False
End Select
' Now set up the SMTP server client to send the email.
Dim smtp As New SmtpClient(My.Resources.smtpServer, Integer.Parse(My.Resources.smtpPort))
' Pull the smtp login details.
smtp.Credentials = New Net.NetworkCredential(My.Resources.smtpUser, My.Resources.smtpPass)
Try
smtp.Send(email)
Catch ex As Exception
ErrorMessage = ex.Message
Return False
End Try
Return True
End Function
' This will generate the subject and body of the newuser email
Private Function GenerateNewUserEmail() As MailMessage
Dim email As New MailMessage(_from, _usr.email)
email.Subject = My.Resources.Resources.TyranntAccountCreated
email.BodyEncoding = System.Text.Encoding.Unicode
email.IsBodyHtml = False
email.Body = String.Format(My.Resources.newUserEmail, _usr.username)
Return email
End Function
Public Overrides Function ToXML() As XElement
Return <job type="email" userid=<%= UserID %>>
<email from=<%= _from %> type=<%= _emailType %>/>
</job>
End Function
End Class
The process function in this job will generate an email and pull the smtp server details out of a resource file and depending on the type of email, will take the email body from resources too.
Now these job classes are created, they can be added to the job queue by using {job}.AddJobToQueue() method as shown in the tjNewUser class. But how will they be processed. This is where the WorkerRole comes into play. As all the work is done by the job classes themselves, only a very simple piece of code is required to process the queues:
Public Overrides Sub Run()
Trace.WriteLine("TyranntDogsbody entry point called.", "Information")
' Loop forever
While (True)
' Get the next message from the queue
Dim msg As CloudQueueMessage = Nothing
msg = _jobQueue.GetMessage(TimeSpan.FromSeconds(30))
If IsNothing(msg) Then
' If message doesn't exist then seep for 1 minute
Thread.Sleep(60000)
Else
' Message exists so process the message
ProcessMessage(msg)
End If
End While
End Sub
Private Sub ProcessMessage(msg As CloudQueueMessage)
Try
' Turn the message into an XML element
Dim xmlMsg As XElement = XElement.Parse(msg.AsString)
' Extract the message type from the element
Dim type As String = xmlMsg.@type
' Now we create a job
Dim job As tjob
Select Case type
' Use the message type to see what kind of job is required
Case "newuser"
job = New tjNewUser(msg.AsString)
Case "email"
job = New tjEmail(msg.AsString)
Case Else
Exit Sub
End Select
' Process the job.
If job.Process() = True Then
' The job succeeded so write a trace message to say this and
' delete the message from the queue.
Trace.WriteLine(String.Format("{0} succeeded", type), "Information")
_jobQueue.DeleteMessage(msg)
Else
' The job failed so write a trace error message saying why the job failed.
' This will leave the job on the queue to be processed again.
Trace.WriteLine(String.Format("{0} failed: {1} ", type, job.ErrorMessage), "Error")
End If
Catch ex As Exception
' something big has gone wrong so write this out as an error trace message.
Trace.WriteLine(String.Format("Failed to parse xml message: [{0}]", msg.AsString), "Error")
Exit Sub
End Try
End Sub
As you can see from the above code. There very little extra code required to process the job as all the work is done inside the job class.
This may be refined in the future but I hope it’s helpful for some people.
Jas
VB Magic 