The new testing pyramid – it’s all about the middle

In the traditional testing pyramid we’re all familiar with, unit tests sit at the bottom, component testing in the middle, and user interface testing at the top. The reason for this is simple: most emphasis is placed on the cheapest, least complex, and most easily automatable tests.

Unit tests are simplest to write and perform, so by focusing on unit tests we’re able to get the best possible testing coverage for our software.
Component – or Integration – testing sits in the middle and focuses on the interaction between the various component parts of our application.
User interface testing is the most difficult of the three types of tests to automate (though there are many tools for the task, these types of tests are the first to break after any small change in the software), and typically has been performed by a separate QA or testing department.
So, how do things change in the world of serverless? You’ll be glad to know we still have a pyramid with the same familiar parts. The difference now is that we need to place much more emphasis on component testing than before. Our pyramid has become slightly misshapen.

This is a shift we already saw in the move to microservices, but it’s become even more critical in serverless applications (that many refer to as nano services), which are typically comprised of a complex web of ephemeral functions and third-party services such as DynamoDB, Kinesis, S3 and so on.
Not only is serverless even more granular than microservices, but since your application exists entirely on the cloud, and you do not own all the code that makes it up, it becomes difficult to test locally.

The limits of local testing in serverless

Unit testing for serverless applications can be (and is) conducted locally with the same ease as for other architectures. So, let’s skip over our pyramid’s foundation and focus on the more interesting question: is it good practice to perform component testing on your local machine?

After all, it’s easier to test locally, where you have visibility over your entire environment, not to mention cheaper, since you don’t have to pay your cloud service provider for a testing environment.
Tools, such as Lambda-local and LocalStack, do allow you to do this, not to mention two of the most popular deployment tools, AWS SAM and Serverless Framework. There are also mocks for some of the most commonly used managed services. But while it’s technically possible to test your application locally, there are some major disadvantages to doing so. The first and most obvious issue is that while official and unofficial mocks exist for many of the managed services we use in production, there are notable absences such as Kinesis and SQS, as well as non-AWS services like Auth0.

And even for those that do exist, mocks may not have all the features that the most updated live version of the services have (the open source mocks are not updated at the pace of the AWS managed services). But one of the biggest reasons to test in the cloud is the importance of configuration to the smooth running of a serverless application. Many of the issues we run into come down to improper configuration of the managed services held together by our Lambdas. When testing against local mocks everything may appear fine, but in production the interaction between various other components may cause issues we didn’t foresee

Serverless component testing should be done in the cloud

Testing in the cloud is the only way to ensure that everything is configured correctly. It’s also the only way to test against the various limitations set by the cloud service providers.
There are limitations set by AWS on the number of concurrent functions (this is currently set to a maximum of 1,000 concurrent executions per account) and on how long a Lambda function can run (at the moment the timeout limit is 900 seconds).

Testing in the cloud is the only way to ensure that everything is configured correctly. It’s also the only way to test against the various limitations set by the cloud service providers.
There are limitations set by AWS on the number of concurrent functions (this is currently set to a maximum of 1,000 concurrent executions per account) and on how long a Lambda function can run (at the moment the timeout limit is 900 seconds).

Efficiency and cost can also only be tested properly in the cloud. We pay based on the amount of memory we allocate to a function and the duration of execution. Assigning more memory can lead to much faster execution (as the number of cores AWS assign to the task is based on the amount of memory requested), and the trick is to find the right balance to ascertain the most cost-effective allocation. We can only deduce this by testing in the cloud environment.

While testing in the cloud is more expensive, because every developer will need their own account specifically for testing, it isn’t prohibitively so, even for larger teams.
The main problem is with the extra time that it takes to deploy everything to the cloud for testing. At Lumigo we use a simple bash script to automatically check whether any changes have been made to the application. If so, we deploy the whole lot to the cloud for testing. If not, we only need to deploy the function that we wish to test.