In September 2019, Freetrade’s test code consisted of a large number of fast
unit tests and a smaller number of end-to-end API tests. These make real HTTP
requests against the platform and assert on the results.
The unit tests confirm that individual classes are behaving correctly and
improve the design of interfaces, while the API tests confirm that higher-level
business logic is working correctly. This is not a bad combination and at the
time already gave us decent test coverage of a large proportion of our platform.
However, the API tests could be flaky and were giving us a lot of false
positives about failures. That was due to the way they make real network
requests and then wait for asynchronous operations to complete.
As a result, the integration tests were difficult to maintain, as a given test
can’t determine if a failure is due to an asynchronous operation in the platform
taking a little longer than expected, or due to a genuine bug. Running the tests
also requires deployment to a real Google Cloud Platform project, which adds
friction to the development iteration cycle.
While we were finding ways to improve the integration tests and make them more
effective, we also decided to experiment with another testing approach: swapping
out Firebase as a “driver”.
The Driver Pattern is related to the Bridge
Pattern and Proxy
Pattern. I’m not a fan of getting
obsessed with patterns, so I
don’t think a super-distinct definition is that helpful. The key idea is to
access an area of functionality through a single point of entry (the driver),
which allows you to easily swap the underlying implementation from a single
place. Whether you call this a driver, bridge, facade, proxy, gateway or
something else, it’s the seamless swapping in and out of implementations that is
In our testing strategies, we created an in-memory, single-process
implementation of the key Firebase behaviours that we relied on in our platform.
This implementation is an ongoing work-in-progress and has been open-sourced
In production, pre-prod and integration test environments, the driver uses the
real Firebase platform. In “driver tests”, as we’ve started calling them,
Firebase is swapped out for the in-memory, single-process implementation.
This has several benefits:
- The tests are fast as there is no network involved, and the simulated
database state is just stored as a simple object in memory.
- You can run the application in a debugger, looking at the database state as
you step through the code; this makes it easier to identify bugs.
- Because the test and application run in a single process, you can mock out
other dependencies for an end-to-end style test (for example, we use nock to
intercept HTTP requests).
- No deployment or other set up is required to run the tests; you can change
code and run it immediately.
- It’s easy to parallelise the tests as they don’t share any state between
Some of the tradeoffs with this approach are:
- Up-front development cost of implementing the in-process version of the
database (and this was only possible at all because the Firebase database is
- We can’t rely on the in-process version behaving identically to the real
thing, meaning we still need some system tests for new flows that haven’t
otherwise been proven with the real database.
- We’ve encountered versioning and type-system issues trying to keep the
interfaces consistent across different projects with different versions of
Overall the driver-testing has been successful and we expect to continue using
our in-process Firebase tool to test a lot of our platform.
(Cross-post) End to end serverless testing with the Driver Pattern