(Cross-post) Why we use the Firebase Realtime Database at Freetrade

(Cross-posted from https://blog.freetrade.io/coming-to-firebases-nosql-realtime-database-from-a-sql-background-b605b2910c4c)

We use the Firebase Realtime Database at Freetrade, as part of our infrastructure on Google Cloud.

The Firebase Realtime Database is a cloud-hosted NoSQL database that lets you store and sync data between your users in realtime.

Firebase Realtime Database has a markedly different development philosophy to the more traditional SQL databases that many developers are used to. Here’s a rundown of what it’s like coming to Firebase Realtime Database from an SQL background. How the Firebase Realtime Database is different

A good bit of advice when dealing with databases is to let the database do its job. With SQL databases like MySQL and Postgres, designing the data schema and constraints up front and then letting the database ensure those are enforced correctly is a big part of this. Changing the schema and constraints later requires more planning to ensure the migration goes smoothly. Otherwise, you might end up locking the orders table for an hour while it migrates, which would prevent Freetrade customers being able to place orders.

With Firebase Realtime Database, the database’s circle of competence is different. It excels at speed, scalability and real-time synchronization of data between clients.

The schema is fluid, and there is no concept of tables and rows. The entire database is more like a single nested object, with an arbitrary structure at every level. The application can read and write large amounts of data in any structure to the database without trouble, but it’s also up to the application logic to ensure that data is well-kept and to maintain integrity.

In the example of migrating an orders table, it would be possible to start writing the new structure to the Realtime Database immediately without problems. Client code would need to handle the possibilities of the old and new structures, but the database can easily deal with having both present in a collection, and handle large volumes of data. It also makes it easy to keep a real-time view of the data available on clients with live updates.

An RDBMS is designed for joining related data across tables, and those tables should be designed to be as normalised as possible. This means that data should not be duplicated and there should be as little redundancy as possible in the schema, with the relationships between data exactly defined. This is powerful and allows highly complex queries to be executed, but is not easy to get right, both in terms of design and in maintaining performance.

Firebase Realtime Database takes a different approach to this by encouraging denormalisation. If similar data is needed in multiple places then it should exist in all those places at once. If a complex relationship needs to be queried, then a map of that relationship should be written to make the queries simple. Google Cloud Platform makes it easy to maintain this kind of data redundancy with event functions that listen and respond to changing data. The result is that reading data is simple and fast even with large volumes. The caveat is that you do still have to think upfront about getting the right indexes on your data to keep your use cases efficient, as you would with a SQL database product. The trade-off

There is a trade-off between focusing your development effort on scaling or focusing it on data consistency. With a SQL database, once you’ve got it handling the scale you need, you get a lot of data consistency work for free. With the Realtime Database, once you’ve got data consistency correct, you get a lot of scaling for free (Google certainly charges for it, but you don’t spend as much extra development effort on it).

We’ve been able to scale to tens of thousands of customer accounts without hitting database scaling issues while keeping database response times in single digit milliseconds (as reported by the profiling tool for Realtime Database). At the same time, we’ve been able to get real-time updates in client apps without spending extra development time implementing and maintaining that. Why we decided to use Firebase Realtime Database

Both types of database can be scalable and ensure data consistency, but the work required to achieve those goals is different. We’ve decided to use Firebase Realtime Database because it gives a better trade-off in development work as we scale to many more users making many more trades.

(Cross-posted from https://blog.freetrade.io/coming-to-firebases-nosql-realtime-database-from-a-sql-background-b605b2910c4c )