AWS Lambda: Asynchronous Invocation Observations

AWS Lambda is great solution for a lot of problems but one might hit certain issues when pushing it to its limits. Let’s look at some interesting observations about it which can help us to work better with it.

A user can invoke a Lambda function either synchronously or asynchronously and the limits applied to the function depend on how a function is invoked, among other reasons. Some interesting aspects for asynchronous invocations are:

Internal invocation queue

Lambda internally uses an invocation queue (one per function) to manage the asynchronous invocations. Once invoked, Lambda queues the event and then executes it at a concurrency level that is compatible with the account, region and user limits.

For a scenario where you have a bug in the invoked function and you want to stop it, these features can be problematic. Some workarounds are:

• Pre-step: Throttle the function.

One can throttle the function concurrency to 0, if necessary, to Immediately stop the execution of the queued events.

• Contact AWS support to clear the invocation queue

This is the cleanest way to purge the invocation queue though AWS might roll out a way for customers to view and edit the invocation queue themselves. The downside is of course the turnaround time.

• Delete the function!

By deleting the queued function, a user can clear the invocation queue regardless of how many invocations are in the queue. It’s a quick and dirty way to get out of a messy situation but remember that all invocations that happen when the function is in deleted state are lost. Ensure you have a way to re-trigger your asynchronous functions.

• Just wait!

AWS deletes all queued events in the invocation queue after 4 days (unofficial information, no guarantees about it and it might vary depending upon your account specifics).

“Pause, Update and Continue”

If you find a bug in an asynchronously invoked function and you only care about the new invocations, an approach that could help is:

• Throttle the concurrency of the asynchronous function to 0
• Update the function code with a bug fix
• Increase the concurrency limit to 1 or more and continue from where you paused

Retries

Lambda can retry a function execution for a number of reasons. Being aware of this is important for asynchronous invocations since the only feedback that you get is from CloudWatch (no synchronous responses!). Ensure that the side-effects of your functions are idempotent for best results.

Multi-level fan-out

Asynchronous invocations are a great way to quickly scale up the concurrency level of the invocations and also, get around the execution time limitation of 300 seconds per function. An approach would look like this.

• Invoke function_1
• function_1 invokes function_2 x times
• Each function_2 invocation in turn invokes function_3 y times

Such a setup lets you invoke function_3 x * y times very quickly.

e.g., let’s say you need to invoke function_3 10000 times and time needed to invoke this function is 10ms.

Sequential invocation

Total time = 10000 * 10ms = 100 seconds

Fan-out invocation

Level 1: 1 * 10ms (function_1 invoked)

Level 2: 100 * 10ms (function_2 invoked 100 times by function_1)

Level 3: 100 * 10ms (function_3 invoked 100 times by every invocation of function_2)

Total time = 10ms + 1000ms + 1000ms = 2.1 seconds

The result of the fan-out method of invocation is two orders of magnitude faster and it works because Lambda functions support a high level of concurrency.

Certain caveats with this are:

• Ensure your concurrency limits are high enough for this or request for an increase in concurrency limits
• function_3 execution happens with very high concurrency. This limits that options that this function can perform to other highly concurrent applications. e.g., you might not be able to update a RDBMS database. Additionally, you may get throttled by applications that usually support concurrent access, but cannot support the sudden spike of traffic.

Speed of invocation

Time required to invoke a Lambda function asynchronously is in the neighborhood of 10ms to 25ms (tested with Golang. It might change with the programming language and might improve in the future).

Summary

Asynchronous invocation can be a powerful tool in a serverless programmer’s toolbelt. Be aware of the pitfalls and the performance considerations for best results.