Do we need the database at all? – Event Sourcing!

In my previous blog post I reasoned about how the database could be hidden behind an ORM tool. And logic could now be written in lambdas (java or scala) instead of SQL. The ORM tool will transform the data model (residing in the database) to an object model (consisting of  java/scala objects).
I pinpointed  some big advantages of this, but I also mentioned that there was an even bigger advantage with the object model that I will address in this post.
Event Sourcing, let me explain what it is with a simple example.

Example: The dry cleaner

The dry cleaner is a place where you can leave your clothes to get them cleaned. In return for your clothes you receive a receipt. When the clothes are cleaned, you can get them back by showing the receipt.
A simple object model for this could be:

(more about how the DryCleaner trait could be implemented later)
The dry cleaner trait has methods for leaving and retrieving clothes as well as a method for looking at the dry cleaners current inventory of clothes.
A simple example of using this model:

The implementation of the DryCleaner could be backed by a database using an ORM tool, but let’s consider another alternative right now. What if all modifying methods of the DryCleaner implementation wrote their arguments to a log file? (the leave() and retrieve() methods modify the object model while the inventory method is a read only method).

The Event Log

What if all (modifying) operations on the object model resulted in a log entry. If all modifications was stored as “events” in a log.
A log created from the example above could yield the following log:

Note: the receipt number need not be present on “leave” rows, but increases the readability

This log contains all the information contained in the method calls of the example plus a serial number of the call and a time stamp. (The timestamps in this example has been modified to improve readability, the code example above would of course execute instantly).
Could you recreate the object model from the information contained in this log?
Yes!
How could the object model be recreated?
By using the information on each log row to call the right method on the DryCleaner object.
Does this work for complex object models?
Yes, the object model can be as complex as you need, below I will describe how to automate the process of reading/writing this log.
In that case, can you do without a database?
Yes, you can choose to store this log in a text file if you would like.

This is called Event Sourcing, and here are some benefits of Event Sourcing:

  • The object model will be portable using a simple text file
  • Human readable, if your events have good names, the resulting log is easy to understand for a human.
  • The log itself is a very elaborate audit log.
  • No database data model needed. Even if you store this log in a database, you do not need to model the entities in a data model.
  • A time machine, you know everything that has happened and when (unlike a data model in a relational database).

The time machine

What if I need to know the inventory of the dry cleaner at a specific point in time?
If I want to know the inventory of the dry cleaner at 2015-08-28 at 09:00, I simply load all events that has a timestamp less than this time. Thus we will have recreated the exact state of the dry cleaner at this moment!
In fact, we can recreate the state of all involved objects for any time we choose!
This is an extremely useful feature of event sourcing. You can return to any point in history to see exactly what happened.

Additional benefits of event sourcing

  • External systems can listen to the events (collect statistics or react to events)
  • Extreme performance. All operations on the object model will be in memory (no need to go to a database). This will be extremely fast.
  • Performance for persistence. All events are immutable and are always added, never removed. You can use an append-only operation for storing them.
  • Simplification. The events are simple. No complex domain object or graphs of objects need to be saved. The object model can of course be as complex as needed, the complexity is not transferred to the persistence mechanism.
  • Testing. It is easy to set up test data, a pre recorded event log can be used to set up the test data. No messy mocking needed.
  • Flexibility of design. The events does not describe the domain objects. So the object model can be redesigned without any need to migrate existing data to the new design.

And some Cons

  • You might need to support ‘old’ events for a long time. You can easily add new types of events, but removing old events is harder. Because you still have to be able to read old event logs. There is ways to remedy this (by making snapshots for example). But still, it could be a pain.
  • Event Sourcing is a less known technology compared to RDBMS etc. It will pose a learning curve to most software developers. And it might be harder to sell to an organization.

Automate the reading and writing of the event log

How do you implement this? Do you have to implement logging to the event log in every (modifying) method? And to parse and apply the states again, do you need to implement this for every class?
No, the implementation could be done through the reflection mechanism. Once and for all for all entities. You could use a naming convention or annotations to mark which methods that modify the object model.
In our bookkeeping application we implement the logging to the event log through a single class of 100 lines of code. All our 120 entities use this single class for all event writing and reading. This is an extremely big win, only 100 lines of code to handle all persistence for a very big application. All the slick code (see my last blog post) could also be removed! We can now concentrate on writing application logic!

Application of Event Sourcing at Speedledger

Our main product at Speedledger is our bookkeeping application. And bookkeeping seems to be the perfect match for event sourcing as described above.

  • We can restore any customers bookkeeping to any point in time. And thus we can let our customers do this in some cases (however, there are some legal aspects what you can do with your bookkeeping).
  • We can easily transfer data for one customer from one system to another. For example, we can transfer customer data from our production environment to the system used by our support personnel simply by exporting the event log as a text file. The support person can then view an exact copy of the customers data in their system while helping the customer.
  • Our support personnel can use the event log itself when helping customers, they see exactly what the customers has done in their bookkeeping. The support personnel has no specific technical training but can still read and use the event log since it only describes bookkeeping events (and not in a very technical way).

You need a database!

Just because it is possible to store the events in a text file without the database entirely, I do not recommend this. A database has many other advantages (transactions for instance). The point is that you do not need a relational database or SQL. You could use something much more lightweight, like a document oriented database. And the data model can be extremely simple, you just need a place to store the event log.

100% consistent object model and no bugs (next blog post)

What about database constraints? The database helps you perserve data integrity by using constraints. If you do not use a real data model (or a database at all), how can you preserve data integrity?

In my next blog post I will describe how to preserve data integrety in an object model with Event Sourcing. And in a much more complete way than you can hope to acheive by database constraints. And this will in turn lead to a manner of coding that makes bugs much less likely.

Share on FacebookTweet about this on TwitterEmail this to someoneShare on Google+Share on TumblrShare on LinkedIn

Comments

  1. Along with every little thing that appears to be developing within this specific subject material, all your points of view are very refreshing. Having said that, I appologize, because I do not subscribe to your entire plan, all be it radical none the less. It appears to me that your comments are not completely validated and in simple fact you are generally your self not completely confident of your argument. In any event I did take pleasure in looking at it.

Leave a Reply