Building Event-Driven Apps on Azure

Introduction

In today's highly dynamic and interconnected digital landscape, traditional request-response architectures often struggle to meet the demands of real-time processing, scalability, and loose coupling. Event-driven architectures (EDA) offer a powerful alternative, enabling systems to react to changes and events as they occur, fostering greater agility and responsiveness. Azure provides a rich ecosystem of services specifically designed to facilitate the construction of robust and scalable event-driven applications.

This article delves into the principles and practicalities of building event-driven applications on Azure, leveraging key services like Azure Event Grid and Azure Functions. It is intended for cloud architects, developers, and infrastructure engineers who are looking to design, implement, or optimize their applications for event-driven paradigms within the Microsoft Azure ecosystem, aiming for enhanced scalability, resilience, and operational efficiency.

Why this matters

Embracing event-driven architectures on Azure brings significant business and technical advantages. From a business perspective, EDAs enable organizations to react in real-time to customer actions, IoT telemetry, or financial transactions, leading to improved customer experiences and faster decision-making. This agility can translate directly into enhanced productivity and competitive advantage. For example, an e-commerce platform can instantly process an order, trigger inventory updates, and send shipping notifications, all in a highly decoupled and scalable manner.

Technically, EDAs significantly improve system resilience and observability. By decoupling components, failures in one part of the system are less likely to cascade, improving overall fault tolerance. Services like Azure Event Grid offer built-in reliability features such as dead-lettering and retry policies, further enhancing system robustness. Furthermore, the consumption-based billing models of services like Azure Functions align well with the sporadic nature of events, leading to significant cost optimizations. Developers can focus on business logic rather than infrastructure concerns, boosting productivity. Compliance requirements can often be met more easily through auditable event logs and clear separation of concerns, thereby reducing risk.

Key concepts

• Event: A record of a discrete action or occurrence in a system. Events are immutable, fact-based, and typically contain information about what happened, when it happened, and the source. They do not contain commands or instructions.
• Event Producer: The component or service that generates and publishes events. Examples include Azure Storage publishing "File Created" events, or an IoT device sending telemetry events.
• Event Consumer (Handler): The component or service that subscribes to and processes events. Consumers react to events without direct knowledge of the producer.
• Event Broker: A service that acts as an intermediary, receiving events from producers and delivering them to interested consumers. It handles routing, filtering, and often provides reliability features.
• Azure Event Grid: A fully managed event routing service that enables uniform event consumption using a publish-subscribe model. It provides near real-time delivery of events from various sources (Azure services, custom applications) to various destinations.
• Azure Functions: A serverless compute service that allows you to run event-driven code without provisioning or managing infrastructure. Functions can be triggered by events from Event Grid, HTTP requests, timers, and many other Azure services.
• Loose Coupling: A design principle where components of a system have minimal dependencies on each other. This improves modularity, flexibility, and scalability.

Step-by-step implementation

Let's walk through an example of configuring an Azure Function to process events from an Azure Storage Account via Event Grid.

1. Create an Azure Storage Account:

Navigate to the Azure portal. Search for "Storage accounts" and click "Create". Choose a resource group, provide a unique storage account name, and select appropriate region and performance tiers.

1. Create an Azure Function App:

In the Azure portal, search for "Function App" and click "Create". Select the same resource group, provide a unique Function App name, choose a runtime stack (e.g., .NET, Node.js), and select a consumption plan for cost efficiency.

1. Develop the Azure Function:

Within your Azure Function App, create a new function. Choose the "Azure Event Grid trigger" template. This will scaffold a function that can receive Event Grid events. The key to this is the integration provided by the serverless framework.

1. Configure the Event Grid Subscription:

Navigate back to your Azure Storage Account in the Azure portal. In the left-hand menu, under "Eventing", select "Events". Click the "+ Event Subscription" button. Topic type: "Storage Accounts" Source resource: Select your storage account. Event types: Choose "Blob Created" and "Blob Deleted". Endpoint type: Select "Azure Function". Endpoint: Click "Select an endpoint" and choose your Function App and the specific Event Grid triggered function you created. Click "Create".

1. Test the setup:

Upload a blob to your storage account. After a short delay, you should see logs in your Azure Function indicating that it processed the "Blob Created" event. Delete the blob, and observe the "Blob Deleted" event processing.

# Step 1: Create a Resource Group
New-AzResourceGroup -Name "myEventDrivenRG" -Location "EastUS"

# Step 2: Create an Azure Storage Account
# Note: Storage account names must be globally unique
$storageAccountName = "myeventstor" + (Get-Random -Maximum 99999)
New-AzStorageAccount -ResourceGroupName "myEventDrivenRG" `
    -Name $storageAccountName `
    -Location "EastUS" `
    -SkuName Standard_LRS `
    -Kind StorageV2

# Step 3: Create an Azure Function App
# Note: Function App names must be globally unique
$functionAppName = "myeventfunc" + (Get-Random -Maximum 99999)
New-AzFunctionApp -ResourceGroupName "myEventDrivenRG" `
    -Name $functionAppName `
    -Location "EastUS" `
    -Runtime "dotnet" `
    -ConsumptionPlanEnabled $true `
    -StorageAccountName $storageAccountName # Link to the storage account created above

Write-Host "Azure Storage Account '$storageAccountName' and Azure Function App '$functionAppName' created successfully."
Write-Host "Next, create an Event Grid triggered function in '$functionAppName' and then the Event Grid subscription from the storage account pointing to that function."

Example configuration

Here's an example of an Azure Bicep template that deploys an Azure Storage Account, an Azure Function App, and an Event Grid subscription to connect Blob Created events from the storage account to the function.

param location string = resourceGroup().location
param storageAccountName string = 'stg${uniqueString(resourceGroup().id)}'
param functionAppName string = 'func${uniqueString(resourceGroup().id)}'
param functionPlanName string = 'plan${uniqueString(resourceGroup().id)}'
param eventGridFunctionAuthLevel string = 'function' // Can be 'function' or 'anonymous'

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-01-01' = {
  name: storageAccountName
  location: location
  sku: {
    name: 'Standard_LRS'
  }
  kind: 'StorageV2'
}

resource functionAppPlan 'Microsoft.Web/serverfarms@2022-09-01' = {
  name: functionPlanName
  location: location
  sku: {
    name: 'Y1' // Consumption plan
    tier: 'Dynamic'
  }
}

resource functionApp 'Microsoft.Web/sites@2022-09-01' = {
  name: functionAppName
  location: location
  kind: 'functionapp'
  properties: {
    serverFarmId: functionAppPlan.id
    httpsOnly: true
    siteConfig: {
      appSettings: [
        {
          name: 'FUNCTIONS_WORKER_RUNTIME'
          value: 'dotnet' // Or 'node', 'python', etc.
        }
        {
          name: 'AzureWebJobsStorage'
          value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccount.name};AccountKey=${storageAccount.listKeys().keys[0].value};EndpointSuffix=${environment().suffixes.storage}'
        }
        {
          name: 'WEBSITE_CONTENTAZUREFILECONNECTIONSTRING'
          value: 'DefaultEndpointsProtocol=https;AccountName=${storageAccount.name};AccountKey=${storageAccount.listKeys().keys[0].value};EndpointSuffix=${environment().suffixes.storage}'
        }
        {
          name: 'WEBSITE_CONTENTSHARE'
          value: toLower(functionAppName)
        }
      ]
    }
  }
}

resource eventGridSubscription 'Microsoft.EventGrid/eventSubscriptions@2023-12-15-preview' = {
  scope: storageAccount
  name: 'storageBlobCreatedSubscription'
  properties: {
    topic: storageAccount.id
    destination: {
      endpointType: 'AzureFunction'
      properties: {
        functionId: '/subscriptions/${subscription().id}/resourceGroups/${resourceGroup().name}/providers/Microsoft.Web/sites/${functionAppName}/functions/BlobTriggerFunction' // Replace BlobTriggerFunction with your actual function name
        // Optional: Include a code property if your function has a function-level key
        // code: listKeys(functionApp.id, '2022-09-01').functionKeys.BlobTriggerFunction 
        // Note: Retrieving specific Function Keys dynamically can be complex and might require custom deployment scripts
      }
    }
    filter: {
      includedEventTypes: [
        'Microsoft.Storage.BlobCreated'
        'Microsoft.Storage.BlobDeleted'
      ]
      // subjectBeginsWith: '/blobServices/default/containers/images/' // Example filter for specific container
    }
  }
}

output storageAccountId string = storageAccount.id
output functionAppId string = functionApp.id
output eventGridSubscriptionName string = eventGridSubscription.name

Common pitfalls

• Over-relying on Polling: Many developers default to polling mechanisms instead of event-driven approaches. Polling is inefficient, wasteful of resources, and introduces latency, whereas event-driven systems react immediately.
• Ignoring Idempotency: Event consumers might receive duplicate events due to retry mechanisms or distributed system complexities. Failing to design consumers to be idempotent can lead to incorrect data or unintended side effects.
• Lack of Dead-Lettering: Not configuring dead-letter queues for Event Grid subscriptions means lost events when handlers fail persistently. This leads to data loss and debugging challenges.
• Insufficient Event Filtering: Subscribing to all event types or too broad a scope can overwhelm consumers with irrelevant events, leading to unnecessary processing and increased costs.
• Tight Coupling of Producer and Consumer: Designing events that expose implementation details or requiring consumers to know too much about the producer defeats the purpose of loose coupling, making systems brittle.
• Ignoring Event Schema Evolution: Events are facts, and their schemas can evolve. Not planning for schema versioning or backwards compatibility can break older consumers as event producers change.

Best practices

• Design for Idempotency: Per the Well-Architected Framework's reliability pillar, ensure your event consumers can process the same event multiple times without adverse effects. Implement idempotency keys or check for previous processing.
• Leverage Dead-Letter Queues: Always configure dead-letter queues for Event Grid subscriptions. This is crucial for reliability, allowing you to inspect and reprocess failed events, aligning with the Azure Well-Architected Framework's operational excellence guidance.
• Fine-Grained Event Filtering: Use Event Grid's advanced filtering capabilities (prefix, suffix, event types) to ensure consumers only receive events relevant to their business logic, optimizing performance and cost.
• Domain-Driven Event Design: Design events to be business-centric and technology-agnostic. Events should represent facts about what happened in a domain, avoiding internal implementation details, promoting loose coupling.
• Monitor and Alert: Implement robust monitoring for Event Grid metrics (e.g., delivered events, failed deliveries) and Azure Function executions. Set up alerts for anomalies to quickly identify and address issues, a core aspect of the Cloud Adoption Framework's operational readiness.
• Secure Endpoints with Managed Identities: Whenever possible, use Managed Identities for Azure Functions to authenticate with other Azure services. This eliminates the need for managing secrets, aligning with Zero Trust principles and reducing the attack surface.
• Plan for Event Schema Versioning: Treat event schemas as part of your API. Implement a strategy for evolving schemas (e.g., adding optional fields, using major/minor versions) to ensure backward compatibility and allow consumers to gracefully handle changes.

Further reading

• What is Azure Event Grid?
• Introduction to Azure Functions
• Event-driven architecture style
• Choose an Azure messaging service
• Reliability in Azure Event Grid