Serverless Testing

Serverless Architecture Conference
October 14 – 16, 2019 | Berlin
29
Aug

Adapting Testing for Serverless Applications

There’s no doubt about it, testing serverless applications is a difficult job. Although the fundamental principles you’re familiar with from traditional architectures remain the same, the fact that your application exists entirely in the cloud and consists of a whole host of managed services tied together by your code, requires an entirely new testing paradigm.

The new testing pyramid – it’s all about the middle

In the traditional testing pyramid we’re all familiar with, unit tests sit at the bottom, component testing in the middle, and user interface testing at the top. The reason for this is simple: most emphasis is placed on the cheapest, least complex, and most easily automatable tests.

Unit tests are simplest to write and perform, so by focusing on unit tests we’re able to get the best possible testing coverage for our software.
Component – or Integration – testing sits in the middle and focuses on the interaction between the various component parts of our application.
User interface testing is the most difficult of the three types of tests to automate (though there are many tools for the task, these types of tests are the first to break after any small change in the software), and typically has been performed by a separate QA or testing department.
So, how do things change in the world of serverless? You’ll be glad to know we still have a pyramid with the same familiar parts. The difference now is that we need to place much more emphasis on component testing than before. Our pyramid has become slightly misshapen.

This is a shift we already saw in the move to microservices, but it’s become even more critical in serverless applications (that many refer to as nano services), which are typically comprised of a complex web of ephemeral functions and third-party services such as DynamoDB, Kinesis, S3 and so on.
Not only is serverless even more granular than microservices, but since your application exists entirely on the cloud, and you do not own all the code that makes it up, it becomes difficult to test locally.

Want more great articles about Serverless Architecture? Subscribe to our newsletter!

The limits of local testing in serverless

Unit testing for serverless applications can be (and is) conducted locally with the same ease as for other architectures. So, let’s skip over our pyramid’s foundation and focus on the more interesting question: is it good practice to perform component testing on your local machine?

After all, it’s easier to test locally, where you have visibility over your entire environment, not to mention cheaper, since you don’t have to pay your cloud service provider for a testing environment.
Tools, such as Lambda-local and LocalStack, do allow you to do this, not to mention two of the most popular deployment tools, AWS SAM and Serverless Framework. There are also mocks for some of the most commonly used managed services. But while it’s technically possible to test your application locally, there are some major disadvantages to doing so. The first and most obvious issue is that while official and unofficial mocks exist for many of the managed services we use in production, there are notable absences such as Kinesis and SQS, as well as non-AWS services like Auth0.

And even for those that do exist, mocks may not have all the features that the most updated live version of the services have (the open source mocks are not updated at the pace of the AWS managed services). But one of the biggest reasons to test in the cloud is the importance of configuration to the smooth running of a serverless application. Many of the issues we run into come down to improper configuration of the managed services held together by our Lambdas. When testing against local mocks everything may appear fine, but in production the interaction between various other components may cause issues we didn’t foresee

 

Serverless component testing should be done in the cloud

Testing in the cloud is the only way to ensure that everything is configured correctly. It’s also the only way to test against the various limitations set by the cloud service providers.
There are limitations set by AWS on the number of concurrent functions (this is currently set to a maximum of 1,000 concurrent executions per account) and on how long a Lambda function can run (at the moment the timeout limit is 900 seconds).

 

Testing in the cloud is the only way to ensure that everything is configured correctly. It’s also the only way to test against the various limitations set by the cloud service providers.
There are limitations set by AWS on the number of concurrent functions (this is currently set to a maximum of 1,000 concurrent executions per account) and on how long a Lambda function can run (at the moment the timeout limit is 900 seconds).

Efficiency and cost can also only be tested properly in the cloud. We pay based on the amount of memory we allocate to a function and the duration of execution. Assigning more memory can lead to much faster execution (as the number of cores AWS assign to the task is based on the amount of memory requested), and the trick is to find the right balance to ascertain the most cost-effective allocation. We can only deduce this by testing in the cloud environment.

While testing in the cloud is more expensive, because every developer will need their own account specifically for testing, it isn’t prohibitively so, even for larger teams.
The main problem is with the extra time that it takes to deploy everything to the cloud for testing. At Lumigo we use a simple bash script to automatically check whether any changes have been made to the application. If so, we deploy the whole lot to the cloud for testing. If not, we only need to deploy the function that we wish to test.

Summary

  • While we can apply the traditional testing pyramid to serverless application testing, much more emphasis needs to be placed on component testing
  • Local testing still suffices for unit tests but for proper component testing it comes with too many limitations.
  • Testing in the cloud introduces additional expense and complication but is the only way to ensure our application is configured correctly and operates within the limits set in the cloud environment.
Stay tuned!
Learn more about Serverless
Architecture Conference 2019

Behind the Tracks

Software Architecture & Design
Software innovation & more
Microservices
Architecture structure & more
Agile & Communication
Methodologies & more
Emerging Technologies
Everything about the latest technologies
DevOps & Continuous Delivery
Delivery Pipelines, Testing & more
Cloud & Modern Infrastructure
Everything about new tools and platforms
Big Data & Machine Learning
Saving, processing & more