(Cross-post) End to end serverless testing with the Driver Pattern

(Cross-posted from https://blog.freetrade.io/end-to-end-serverless-testing-with-the-driver-pattern-bcf2d22bf73a)

In September 2019, Freetrade’s test code consisted of a large number of fast unit tests and a smaller number of end-to-end API tests. These make real HTTP requests against the platform and assert on the results.

The unit tests confirm that individual classes are behaving correctly and improve the design of interfaces, while the API tests confirm that higher-level business logic is working correctly. This is not a bad combination and at the time already gave us decent test coverage of a large proportion of our platform.

However, the API tests could be flaky and were giving us a lot of false positives about failures. That was due to the way they make real network requests and then wait for asynchronous operations to complete.

As a result, the integration tests were difficult to maintain, as a given test can’t determine if a failure is due to an asynchronous operation in the platform taking a little longer than expected, or due to a genuine bug. Running the tests also requires deployment to a real Google Cloud Platform project, which adds friction to the development iteration cycle.

While we were finding ways to improve the integration tests and make them more effective, we also decided to experiment with another testing approach: swapping out Firebase as a “driver”.

The Driver Pattern is related to the Bridge Pattern, Facade Pattern and Proxy Pattern. I’m not a fan of getting obsessed with patterns, so I don’t think a super-distinct definition is that helpful. The key idea is to access an area of functionality through a single point of entry (the driver), which allows you to easily swap the underlying implementation from a single place. Whether you call this a driver, bridge, facade, proxy, gateway or something else, it’s the seamless swapping in and out of implementations that is important.

In our testing strategies, we created an in-memory, single-process implementation of the key Firebase behaviours that we relied on in our platform. This implementation is an ongoing work-in-progress and has been open-sourced here.

In production, pre-prod and integration test environments, the driver uses the real Firebase platform. In “driver tests”, as we’ve started calling them, Firebase is swapped out for the in-memory, single-process implementation.

This has several benefits:

• The tests are fast as there is no network involved, and the simulated database state is just stored as a simple object in memory.
• You can run the application in a debugger, looking at the database state as you step through the code; this makes it easier to identify bugs.
• Because the test and application run in a single process, you can mock out other dependencies for an end-to-end style test (for example, we use nock to intercept HTTP requests).
• No deployment or other set up is required to run the tests; you can change code and run it immediately.
• It’s easy to parallelise the tests as they don’t share any state between processes.

Some of the tradeoffs with this approach are:

• Up-front development cost of implementing the in-process version of the database (and this was only possible at all because the Firebase database is relatively simple).
• We can’t rely on the in-process version behaving identically to the real thing, meaning we still need some system tests for new flows that haven’t otherwise been proven with the real database.
• We’ve encountered versioning and type-system issues trying to keep the interfaces consistent across different projects with different versions of Firebase SDKs.

Overall the driver-testing has been successful and we expect to continue using our in-process Firebase tool to test a lot of our platform.