Cloud performance and scalability. Azure Service Bus and Azure Durable functions - part 2

This is the continuation of an article I have written last month you can find it here: https://www.ideliversoft.com/post/cloud-performance-and-scalability-1

I have measured the performance of the Azure durable functions for these configurations:

1. Creating an app service scale out plan - manual scale.

2. Creating an app service scale out plan - custom scale.

3. Configuring the durable functions parameters for the task hub.

4. Running it on a consumption plan.

5. Running it on a premium plan.

and using the following loads:

𝚂𝚖𝚊𝚕𝚕 𝚕𝚘𝚊𝚍:

• 𝟷𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜 𝚎𝚊𝚌𝚑 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚢

𝙼𝚎𝚍𝚒𝚞𝚖 𝚕𝚘𝚊𝚍:

• 𝟷𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜 𝚎𝚊𝚌𝚑 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚢 𝚘𝚛

• 𝟷𝟶𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜 𝚙𝚎𝚛 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚢

𝙷𝚒𝚐𝚑 𝚕𝚘𝚊𝚍:

• 𝟷𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶𝟶𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜 𝚎𝚊𝚌𝚑 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚢 𝚘𝚛

• 𝟷𝟶𝟶𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜 𝚙𝚎𝚛 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚢

Automatic scaling - scale out the App Service Plan

As with all Azure Functions running in the Consumption and Elastic Premium plans, Durable Functions supports auto-scale via the Azure Functions scale controller.

The Scale Controller monitors how long messages and tasks have to wait before they are processed. Based on these latencies it can decide whether to add or remove workers.

You can read more about it here:

https://learn.microsoft.com/en-us/azure/azure-functions/event-driven-scaling

Scaling the number of VM instances

The app service plan can be scaled out by increasing the number of VM machines instances on which the functions executes.

The instance count slider can be set to a value between 1 to 30 for the p2v2 app service plan.

Custom scaling plan using metrics for the Azure durable function's storage account's queues

You can create a custom autoscale plan and define a number of rules for each control or worker-item queue.

Each rule will increase or decrease the VM instances count based on several metric thresholds.

The number of scaled out / in instances during processing the load.

Findings

1)

𝙰 𝚋𝚊𝚜𝚎𝚕𝚒𝚗𝚎 𝚎𝚡𝚎𝚌𝚞𝚝𝚒𝚘𝚗, 𝚛𝚞𝚗𝚗𝚒𝚗𝚐 𝚠𝚒𝚝𝚑 𝚍𝚎𝚏𝚊𝚞𝚕𝚝 𝚟𝚊𝚕𝚞𝚎𝚜 𝚒𝚗 𝚝𝚑𝚎 𝚑𝚘𝚜𝚝.𝚓𝚜𝚘𝚗 𝚏𝚒𝚕𝚎 𝚊𝚗𝚍 𝚗𝚘 𝚜𝚌𝚊𝚕𝚒𝚗𝚐 𝚍𝚘𝚗𝚎 𝚏𝚘𝚛 𝚝𝚑𝚎 𝚊𝚙𝚙 𝚜𝚎𝚛𝚟𝚒𝚌𝚎 𝚙𝚕𝚊𝚗 𝚙𝚛𝚘𝚍𝚞𝚌𝚎𝚜 𝚝𝚑𝚎 𝚏𝚘𝚕𝚕𝚘𝚠𝚒𝚗𝚐 𝚛𝚎𝚜𝚞𝚕𝚝𝚜:

Consumption plan

• 𝚜𝚖𝚊𝚕𝚕 𝚕𝚘𝚊𝚍 - 𝟷:𝟹𝟹𝚖𝚒𝚗𝚜

• 𝚖𝚎𝚍𝚒𝚞𝚖 𝚕𝚘𝚊𝚍

  • 𝚝𝚎𝚜𝚝 𝟷 (𝟷𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜) - 𝟻:𝟸𝟸 𝚖𝚒𝚗𝚜

  • 𝚝𝚎𝚜𝚝 𝟸 (𝟷𝟶𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜) - 𝟸:𝟻𝟻 𝚖𝚒𝚗𝚜

• 𝚑𝚒𝚐𝚑 𝚕𝚘𝚊𝚍

  • 𝚝𝚎𝚜𝚝 𝟷 - 𝟷𝟼:𝟸𝟿 𝚖𝚒𝚗𝚜

  • 𝚝𝚎𝚜𝚝 𝟸 - 𝟾:𝟶𝟸 𝚖𝚒𝚗𝚜

2)

𝚆𝚑𝚎𝚗 𝚛𝚞𝚗𝚗𝚒𝚗𝚐 𝚒𝚗 𝙲𝚘𝚗𝚜𝚞𝚖𝚙𝚝𝚒𝚘𝚗 𝚙𝚕𝚊𝚗, 𝚒𝚗𝚌𝚛𝚎𝚊𝚜𝚒𝚗𝚐 𝚝𝚑𝚎 𝚖𝚊𝚡𝙲𝚘𝚗𝚌𝚞𝚛𝚛𝚎𝚗𝚝𝙰𝚌𝚝𝚒𝚟𝚒𝚝𝚢𝙵𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚊𝚗𝚍 𝚖𝚊𝚡𝙲𝚘𝚗𝚌𝚞𝚛𝚛𝚎𝚗𝚝𝙾𝚛𝚌𝚑𝚎𝚜𝚝𝚛𝚊𝚝𝚘𝚛𝙵𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚟𝚊𝚕𝚞𝚎𝚜 𝚏𝚛𝚘𝚖𝟷𝟶 𝚝𝚘 𝟷𝟶𝟶 𝚝𝚘 𝟷𝟶𝟶𝟶 𝖎𝖓 𝖙𝖍𝖊 𝖍𝖔𝖘𝖙.𝖏𝖘𝖔𝖓 𝖋𝖎𝖑𝖊, 𝚍𝚘𝚎𝚜 𝚗𝚘𝚝 𝚙𝚛𝚘𝚟𝚒𝚍𝚎 𝚊 𝚙𝚛𝚘𝚙𝚘𝚛𝚝𝚒𝚘𝚗𝚊𝚕 𝚎𝚡𝚎𝚌𝚞𝚝𝚒𝚘𝚗 𝚝𝚒𝚖𝚎 𝚒𝚖𝚙𝚛𝚘𝚟𝚎𝚖𝚎𝚗𝚝 𝚖𝚘𝚜𝚝 𝚕𝚒𝚔𝚎𝚕𝚢 𝚍𝚞𝚎 𝚝𝚘 𝚝𝚑𝚎 𝚏𝚊𝚌𝚝 𝚝𝚑𝚊𝚝 𝚝𝚑𝚎 𝚗𝚞𝚖𝚋𝚎𝚛 𝚘𝚏 𝚌𝚘𝚗𝚌𝚞𝚛𝚛𝚎𝚗𝚝 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚊𝚗𝚍 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚊𝚛𝚎 𝚏𝚒𝚡𝚎𝚍 (𝚜𝚎𝚎 𝚝𝚑𝚎 𝚙𝚛𝚎𝚟𝚒𝚘𝚞𝚜 𝚊𝚛𝚝𝚒𝚌𝚕𝚎).

3)

𝚆𝚑𝚎𝚗 𝚛𝚞𝚗𝚗𝚒𝚗𝚐 𝚒𝚗 𝚝𝚑𝚎 𝙳𝚎𝚍𝚒𝚌𝚊𝚝𝚎𝚍 𝚘𝚛 𝙿𝚛𝚎𝚖𝚒𝚞𝚖 𝙿𝚕𝚊𝚗 𝚜𝚎𝚝𝚝𝚒𝚗𝚐 𝚞𝚙 𝚝𝚑𝚎 𝚙𝚊𝚛𝚊𝚖𝚎𝚝𝚎𝚛𝚜 𝚒𝚗 𝚝𝚑𝚎 𝚑𝚘𝚜𝚝.𝚓𝚜𝚘𝚗 𝚏𝚒𝚕𝚎 𝚘𝚛 𝚜𝚌𝚊𝚕𝚒𝚗𝚐 𝚘𝚞𝚝 𝚝𝚑𝚎 𝚊𝚙𝚙 𝚜𝚎𝚛𝚟𝚒𝚌𝚎 𝚙𝚕𝚊𝚗 𝚊𝚛𝚎 𝚙𝚛𝚘𝚟𝚒𝚍𝚒𝚗𝚐 𝚊 𝚙𝚛𝚘𝚙𝚘𝚛𝚝𝚒𝚘𝚗𝚊𝚕 𝚝𝚒𝚖𝚎 𝚒𝚖𝚙𝚛𝚘𝚟𝚎𝚖𝚎𝚗𝚝.

𝙵𝚘𝚛 𝚊 premium plan baseline 𝚎𝚡𝚎𝚌𝚞𝚝𝚒𝚘𝚗 𝚝𝚑𝚘𝚞𝚐𝚑, 𝚛𝚞𝚗𝚗𝚒𝚗𝚐 𝚠𝚒𝚝𝚑 𝚍𝚎𝚏𝚊𝚞𝚕𝚝 𝚟𝚊𝚕𝚞𝚎𝚜 𝚒𝚗 𝚝𝚑𝚎 𝚑𝚘𝚜𝚝.𝚓𝚜𝚘𝚗 𝚏𝚒𝚕𝚎 𝚊𝚗𝚍 𝚊 𝚜𝚒𝚗𝚐𝚕𝚎 𝚅𝙼 𝚒𝚗𝚜𝚝𝚊𝚗𝚌𝚎, 𝚝𝚑𝚎 𝚛𝚎𝚜𝚞𝚕𝚝𝚜 𝚏𝚘𝚛 𝚊 𝚑𝚒𝚐𝚑 𝚕𝚘𝚊𝚍 𝚊𝚛𝚎 𝚛𝚎𝚊𝚕𝚕𝚢 𝚋𝚊𝚍:

• 𝚜𝚖𝚊𝚕𝚕 𝚕𝚘𝚊𝚍 : 𝟷:𝟹𝟽 𝚖𝚒𝚗𝚜

• 𝚖𝚎𝚍𝚒𝚞𝚖 𝚕𝚘𝚊𝚍: 𝚝𝚎𝚜𝚝 𝟷 - 𝟷𝟼:𝟺𝟻 𝚖𝚒𝚗𝚜

• 𝚖𝚎𝚍𝚒𝚞𝚖 𝚕𝚘𝚊𝚍: 𝚝𝚎𝚜𝚝 𝟸 - 𝟷𝟼:𝟹𝟺 𝚖𝚒𝚗𝚜

• 𝚑𝚒𝚐𝚑 𝚕𝚘𝚊𝚍: 𝚝𝚎𝚜𝚝 𝟷 - 𝟻𝟻:𝟹𝟸 𝚖𝚒𝚗𝚜

• 𝚑𝚒𝚐𝚑 𝚕𝚘𝚊𝚍: 𝚝𝚎𝚜𝚝 𝟸 - 𝟹𝟿𝟻 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚌𝚘𝚖𝚙𝚕𝚎𝚝𝚎𝚍 𝚘𝚞𝚝 𝚘𝚏 𝟷𝟶𝟶𝟶 𝚒𝚗 𝟼𝟻 𝚖𝚒𝚗𝚜

4)

𝚆𝚑𝚎𝚗 𝚛𝚞𝚗𝚗𝚒𝚗𝚐 𝚠𝚒𝚝𝚑 𝚊 𝚌𝚘𝚖𝚋𝚒𝚗𝚎𝚍 𝚌𝚘𝚗𝚏𝚒𝚐𝚞𝚛𝚊𝚝𝚒𝚘𝚗 - 𝚌𝚞𝚜𝚝𝚘𝚖 𝚜𝚌𝚊𝚕𝚒𝚗𝚐 𝚜𝚎𝚝 𝚞𝚙 𝚘𝚗 𝚊 𝙿𝚛𝚎𝚖𝚒𝚞𝚖 𝙿𝚕𝚊𝚗 𝚠𝚒𝚝𝚑 𝚌𝚞𝚜𝚝𝚘𝚖 𝚙𝚊𝚛𝚊𝚖𝚎𝚝𝚎𝚛𝚜 𝚌𝚘𝚗𝚏𝚒𝚐𝚞𝚛𝚎𝚍 𝚒𝚗 𝚝𝚑𝚎 𝚑𝚘𝚜𝚝.𝚓𝚜𝚘𝚗 - 𝚝𝚑𝚎 𝚛𝚎𝚜𝚞𝚕𝚝𝚜 𝚏𝚘𝚛 𝚑𝚒𝚐𝚑 𝚕𝚘𝚊𝚍 𝚎𝚡𝚌𝚎𝚎𝚍 𝚖𝚢 𝚎𝚡𝚙𝚎𝚌𝚝𝚊𝚝𝚒𝚘𝚗𝚜 𝚙𝚛𝚘𝚟𝚒𝚍𝚒𝚗𝚐 𝚝𝚑𝚎 𝚋𝚎𝚜𝚝 𝚛𝚎𝚜𝚞𝚕𝚝𝚜, 𝚠𝚒𝚝𝚑𝚒𝚗 𝟷 𝚖𝚒𝚗𝚞𝚝𝚎 𝚘𝚛 𝚕𝚎𝚜𝚜 𝚏𝚘𝚛 𝚝𝚑𝚎 𝚠𝚑𝚘𝚕𝚎 𝚙𝚛𝚘𝚌𝚎𝚜𝚜 (𝚝𝚑𝚊𝚝 𝚒𝚜, 𝚛𝚞𝚗𝚗𝚒𝚗𝚐 𝟷𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶𝟶𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜 𝚎𝚊𝚌𝚑 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚢 𝚘𝚛 𝟷𝟶𝟶𝟶 𝚏𝚞𝚗𝚌𝚝𝚒𝚘𝚗𝚜 𝚡 𝟷𝟶 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚒𝚎𝚜 𝚡 𝟷𝟶 𝚜𝚎𝚌𝚘𝚗𝚍𝚜 𝚙𝚎𝚛 𝚊𝚌𝚝𝚒𝚟𝚒𝚝𝚢).

Function timeout

One of the reasons the functions needs to be well tuned and have a fast execution is because they have a limited duration span.

The whole process should run in less than 10 minutes for the Consumption Plan and less than 60 mins on the Premium Plan.

In the next article, that will be the last part of this series, I will provide a few tips recommended by Microsoft for configuring the Azure Durable functions values in the host.json file.