Building an Active/Standby HA Architecture with Queue-Based Microservices using Azure Functions

In recent years, the microservices architecture has gained traction owing to its remarkable scalability and adaptability. Azure Functions, a serverless compute service, facilitates the development of dynamic, scalable microservices. When designing microservices for enterprise systems, achieving high availability (HA) is crucial. This blog post delves into a strategy for constructing a HA architecture that incorporates queue-based microservices through the use of Azure Functions.

Challenges with Queue-Based Microservices

Creating HA queue-based microservices using Azure Functions comes with some design challenges. One of them is deciding between an Active/Active or Active/Standby architecture. While having an Active/Active architecture is trivial, it carries a cost from inter-region traffic and compute resources. Additionally, there may be scenarios where multiple consumers for the same queue are not necessary or desired, making Active/Standby a better option. 

But there is a problem with Active/Standby for queue consumers, if more than one consumer is processing from a queue, they will both receive messages in a competing consumer mode. This makes it difficult to use Active/Standby in a failover mode where only one consumer is active, and the other is on standby. Unlike HTTP services where a load balancer like Azure Front Door can be used to solve the problem, a queue-based microservices architecture requires building custom mechanics to handle this scenario.

Approach to Building an Active/Standby HA Architecture

To address the challenges of building a HA architecture with queue-based microservices, let’s create a multi-region design that uses Azure Function Apps, a Health Check endpoint, and a Logic App. In my particular scenario, I have a multi-region setup with a primary and secondary region. The microservices are running by default in the primary one. In the event of a failure where the microservice dies or is unable to work in the primary region, the system needs to automatically start the service in the secondary region. After the primary region recovers its health, then it will stop or shut down the secondary region service and keep the primary region as the only active one. 

The first step is to provide a Health Check endpoint, for this we will create an Azure Function App that exposes two functions. The first function is the one that consumes and processes messages from the queue (Queue Storage Trigger). The second function is a Health Check function (HTTP Trigger) that exposes the health of the service.

The second step is to provide auto-failover capabilities, we will build a Logic App that periodically monitors the Health Check endpoint for both copies of the consumer function. The Logic App will then start or stop the functions as needed. In the event of a failure, the Logic App can stop the primary function and activate the standby function. The logic app will detect when the primary region is available again and then switch over to the primary again, and shut down the secondary.


Putting it all together 

Conclusion

Building a high-availability architecture with queue-based microservices using Azure Functions is a good way to ensure your service remains highly available and scalable. By adopting a multi-region design that involves the use of Azure Function Apps, a Health Check endpoint, and a Logic App, we can create a robust architecture that can seamlessly process messages from queues, with Active/StandBy HA and auto-failover. 

Comments

Post a Comment

Popular posts from this blog

Get-ChildItem vs Dir in PowerShell

Image
Batch as usual Recently I was in the need of modifying a huge amount of files across a network share. After installing Serviio media streaming service I noticed that it will crash "randomly". After checking the logs it was clear that ffmpeg was crashing when trying to open subtitle files that were not in the encoding indicated by the Serviio console. I needed a quick way to update all subtitle files to a common encoding, so I decided to convert all files to UTF8. Since I use a windows environment I went directly to PowerShell and wrote this.  Get-ChildItem -path .\ -filter *.srt -file  | ForEach-Object { (get-content $_.FullName) | Out-File $_.FullName -encoding utf8 } This worked almost right except for files with "[" or "]" (among other symbols) on the name or path. To solve it, just added the "-LiteralPath" switch to tell powershell not to consider any wildcards on the path name and just use it exactly as it is.  Get-ChildIt
Read more

The case for a 4 week sprint

Image
Every time we talk about SCRUM, the length of the sprint is stated as a period of "two to four weeks", rarely, however, have I seen people going outside of the 2 weeks. This is not written on stone of course, and while 3 weeks might seems odd, there is a benefit to using a longer period for your sprints.  Image is taken from scrum.org Without any hard data to back that decision, the answer I get from asking about this decision seems to be related to "delivery speed", we assume that because we deliver value to the business every 2 weeks, we are in a more "agile" environment. Well, that might not be so true in every sense or for every project. Let's break some arguments: 1. Sprint length is not equal to the speed of delivery There is more to the speed of development and delivery than the timeframe of the delivery cycle, in a world with DevOps and full CI/CD implemented for your project, you could (should?) actually deliver features every single day all t
Read more

NODE.js fs.readFile and the BOM marker

Working on a ReactJS project, got a small glitch while trying to read a JSON file and parse it as a javascript object. getDataObject() { var dataString = fs.readFileSync("./json/data.json", "utf8"); return JSON.parse(dataString); } output:   SyntaxError: Unexpected token   After checking, double and triple checking again that the file was correct I went deeper and realized that "fs.readFileSync" was returning me the BOM for the UTF file at the beginning of the string. So we just need to strip it out. getDataObject() { var dataString = fs.readFileSync("./json/data.json", "utf8"); return JSON.parse(dataString.replace(/^\uFEFF/, "")); } I then checked and there are "packages" for this, however I don't think is proper to just import a new dependency just to fix something this small. The big issue here is why is this considered correct. The BOM is not
Read more