Linear's sync engine architecture Page

In this talk, Tuomas Artman discusses Linear's sync engine architecture.

While this is a very simplified version of sync engine implementations, it covers the fundamental building blocks well. This approach is likely what future local-first frameworks will have in common.

Linear's journey hasn't been without challenges as they've scaled. Tuomas talks about the evolution of their sync engine architecture in his next talk.

Linear's sync engine uses an object graph, similar to what you'd find in typical modern applications with a normalized data store in memory. You are probably already familiar with this setup. Linear uses MobX to build the in-memory layer, with an event-driven system for updating remote data.

Write Path

The frontend can directly manipulate the object graph, not even the object pool. This triggers an update to the object pool, which then creates a transaction in the queue to update the backend. The transaction queue is persisted to IndexedDB.

Read Path

When the remote data receives updates from a frontend, it publishes an event through a web socket. The receiving frontend fetches the updated data and reflects the changes to the object pool and IndexedDB. It then dispatches an event to update the object graph, maintaining synchronization across the system with MobX's reactivity.

Benefits of this Architecture

This setup offers several advantages:

The app works offline for users. Users can not only browse but also make changes that will sync to the remote server when the device comes back online.
For developers: The frontend does not have to manually make network requests. The sync layer encapsulates when and how to make requests, error handling, and optimistic cache updates. The entire process is just one method call away:
```
user.name = 'updated name';
user.save();
```
By calling the save method on the object, you trigger the entire sync cycle. This also means the network layer is not required for the application to function, allowing for quicker iteration. You can start developing a new feature in the frontend, and after finalizing the data modeling, you can then synchronize with the backend.
For the company, it's cost-efficient to run the service. As services scale, cloud costs increase significantly. But since data is persisted on local devices, you don't pay for database reads as often as in a typical 3-tier application. Tuomas discusses this in more detail in his talk at the local-first conference in Berlin.

In his follow-up talk, Tuomas discusses challenges the Linear team encountered, particularly cases where the read path or write path become very outdated and when/how to load data into memory when it gets huge. This is an interesting evolution of their system as it scaled.

I'm curious about the shape of the transaction queue. As the complexity of mutations increases, this could potentially become tricky to manage.

Another thing to note is that Linear didn't use CRDTs until recently. Even now, it only uses them for issue descriptions. In a podcast, Tuomas mentioned that conflicts are actually not that common in Linear. This makes sense when you think about it - for certain systems, conflict resolution isn't an urgent issue. Linear's approach shows that there are other viable strategies, depending on the specific needs of your application. In this case, the LWW (Last-Write-Win) system is enough. (Related: CRDTs for Mortals by James Long)

Lastly, it's a beautiful implementation to use decorators to hook up this system. All you need to do is define a model using the decorator. If you have experience with MobX or any ORMs in the backend, you're very familiar with this syntax.

As we continue to explore local-first and offline-first applications, approaches like this will shape the future of web development.