Should I Event Source?

If the 2020s have showcased anything in tech, the word “blockchain” has got to be up there in a myriad of positive and negative interpretations. If we take the “good” or “useful” aspects of a blockchain, they’re often connected with leveraging the advantages of its immutable nature. History is history.

So, you have an idea and think “blockchain would be perfect for this”, but you also have to acknowledge the elephant in the room: web3, blockchains, and crypto technology are difficult technologies to integrate. If you come from a traditional approach in software engineering, good news! There is a pattern of data design that has been around for quite some time (particularly in fields like Finance) that has a similar approach to tracing the historical state of data. In this article, we’re going to explore Event Sourcing.

The Origins of Event Sourcing

All Together Now: Data is the new OilEvent Sourcing as a concept was born out of the need to transactionally log data and what transformations would happen to data over time. Institutions such as banks would need to be able to see a ‘snapshot’ of what an entity of sorts looked like at a precise time. Using the concept of Domain-Driven-Design (DDD), the more formal birth of Event Sourcing happened with the launch of Command Query Responsibility Segregation (or CQRS), which is commonly used in Microsoft-based environments. This segregation focuses not on data but on the actions that happen when its state changes. These actions are recorded as immutable entities, and it is this pattern that defines Event Sourcing.

A Typical Application in Finance

It's a Bull Market, but how do you audit it?Take, for example, something simple such as an account balance. In a traditional setup, you’d have something like a relational database, and an application pulling data out of that account with a prepared SQL statement. The balance figure could be obtained either if transactions hold the account’s balance in state on each transaction, or if the application attempts to aggregate everything. This is where Event Sourcing comes in: what if you have millions of rows, and you want to see the status of a balance at an exact time in the past? You have to hope that the transactional data is all just correct, and that a migration downstream hasn’t happened, or an outage has occurred just at the exact point of attempting to carry a total balance figure in a transaction record.

With Event Sourcing, you “replay history” because the data that is held is about the changes to the state of the data. So, to apply this, you’d create some custom Event models named appropriately, such as:

class AccountCreatedEvent extends Event

class AccountDepositEvent extends Event

class AccountWithdrawEvent extends Event

This series of events is called a stream. The way to handle getting the balance out is with an aggregator, which is named a projector. A projector class would take an argument, then read in the events (queried previously by a primary key such as the Account ID) passed into it to work out the state:

class AccountBalanceProjector {

    public static int projectBalance(List<Event> events) {

        int balance = 0;
        boolean accountCreated = false;

        for (Event event : events) {
            if (event instanceof AccountCreatedEvent) {
                balance = ((AccountCreatedEvent) event).initialBalance;
                accountCreated = true;
            } else if (accountCreated && event instanceof AccountDepositEvent) {
                balance += ((AccountDepositEvent) event).amount;
            } else if (accountCreated && event instanceof AccountWithdrawEvent) {
                balance -= ((AccountWithdrawEvent) event).amount;
            }
        }
        return balance;
    }
}

And there we have it. You would have a method in your Object-Relational Mapper (ORM) that would retrieve all of the events for a specific accountId, then pass in those events to AccountBalanceProjector.projectBalance(events), and back comes your balance. It’s not pulled out the balance that has been pre-created somewhere, it’s recreated it.

Using Event Sourcing with Vonage Data

Controlling Data Can Be TrickyLet’s look at an example using data generated when using Vonage. We’re going to be looking at the Voice API, which generates an event status webhook every time an action occurs in the duration of a voice call. According to the Voice API Webhooks documentation, the following states can be given:

• started

• ringing

• answered

• busy

• cancelled

• unanswered

• disconnected

• rejected

• failed

• human/machine

• timeout

• completed

• record

• input

• transfer

If I feed these states into AI and ask the prompt to generate my classes, I can get a little magic automated sprinkle to generate my list of events: 

public class CallStarted extends BaseVoiceEvent {
    public CallStarted(String callId) { super(callId); }

    @Override public String getName() { return "started"; }
}

public class CallRinging extends BaseVoiceEvent {
    public CallRinging(String callId) { super(callId); }

    @Override public String getName() { return "ringing"; }
}

public class CallAnswered extends BaseVoiceEvent {
    private String answerType;

    public CallAnswered(String callId, String answerType) {
        super(callId);

        this.answerType = answerType;
    }

    @Override public String getName() { return "answered"; }

    @Override public Map<String, Object> toMap() {
        Map<String, Object> map = super.toMap();
        map.put("answer_type", answerType);
        return map;
    }
}

public class CallBusy extends BaseVoiceEvent {
    public CallBusy(String callId) { super(callId); }

    @Override public String getName() { return "busy"; }
}

public class CallCancelled extends BaseVoiceEvent {
    public CallCancelled(String callId) { super(callId); }

    @Override public String getName() { return "cancelled"; }
}

public class CallUnanswered extends BaseVoiceEvent {
    public CallUnanswered(String callId) { super(callId); }

    @Override public String getName() { return "unanswered"; }
}

public class CallDisconnected extends BaseVoiceEvent {
    public CallDisconnected(String callId) { super(callId); }

    @Override public String getName() { return "disconnected"; }
}

public class CallRejected extends BaseVoiceEvent {
    public CallRejected(String callId) { super(callId); }

    @Override public String getName() { return "rejected"; }
}

public class CallFailed extends BaseVoiceEvent {
    public CallFailed(String callId) { super(callId); }

    @Override public String getName() { return "failed"; }
}

public class CallHumanMachineDetected extends BaseVoiceEvent {
    private String detectionResult;

    public CallHumanMachineDetected(String callId, String detectionResult) {
        super(callId);

        this.detectionResult = detectionResult;
    }

    @Override public String getName() { return "human_machine"; }

    @Override public Map<String, Object> toMap() {
        Map<String, Object> map = super.toMap();
        map.put("detection_result", detectionResult);

        return map;
    }
}

public class CallTimeout extends BaseVoiceEvent {
    public CallTimeout(String callId) { super(callId); }

    @Override public String getName() { return "timeout"; }
}

public class CallCompleted extends BaseVoiceEvent {
    public CallCompleted(String callId) { super(callId); }

    @Override public String getName() { return "completed"; }
}

// etc etc, truncated for length reasons

You’ll notice that these Events extend off a base class, which we define as abstract:

import java.time.Instant;

import java.util.HashMap;

import java.util.Map;

public abstract class BaseVoiceEvent {
    protected String callId;
    protected Instant timestamp;

    public BaseVoiceEvent(String callId) {
        this.callId = callId;

        this.timestamp = Instant.now();
    }

    public BaseVoiceEvent(String callId, Instant timestamp) {
        this.callId = callId;

        this.timestamp = timestamp;
    }

    public abstract String getName();

    public Map<String, Object> toMap() {
        Map<String, Object> map = new HashMap<>();
        map.put("event", getName());
        map.put("call_id", callId);
        map.put("timestamp", timestamp.toString());

        return map;
    }
}

You now have the beginning structure of an Event Sourcing architecture. The finishing steps would be to have controllers to handle event creation at an HTTP endpoint to your application, and then a Call Status Projector. That projector, similar to the example we have previously, would be able to take a stream of events, take an optional timestamp, which then works out the correct status of the callID to the timestamp you have given (or to the most recent by default).

What Are The Advantages of Event Sourcing

• Event-driven architecture can be refactored. Because the system is decoupled from the state data that comes in, you can re-run your new controllers with new logic (for instance, to handle a new event type) and rebuild your data streams for projection.

• Full audit trails regardless of time: this is very important for highly regulated industries, as we have seen in something like Finance and Healthcare.

• Separation of Reads vs. Writes: using the aforementioned CQRS, how data is accessed and written is split. When tuning an application requiring a high Input/Output of data, you can scale separate aspects of your architecture accordingly.

What Are The Disadvantages of Event Sourcing

I wouldn’t be able to call myself a sensible engineer without stating what the drawbacks are, and they’re pretty big

• Backwards Compatibility: I mentioned how quick it can be to push new changes into your models and re-run your event handlers, but not about when something becomes deprecated. Thought needs to be put into Event handling, Semantic Versioning, and behaviours, and quite possibly leading to sizable collections of old data needing to go into storage while having to do complex ETL procedures

• System Complexity: This is the biggest hurdle to overcome. I have worked on Event-Driven systems that were scaling up in the past, and the biggest part of this complexity is similar to microservices. If you haven’t architected your event system correctly right from the start, then I can assure you that a changing enum is just waiting to trip up your whole system

• Debugging: One of the biggest frustrations with anything in software design that I have found is when the right debugging tools are not available. I have often spoken about my dismay at not having Derick Rethan’s xdebug as part of a PHP stack; I just don’t see how anyone can code without it, rather than killing runtime and dumping out individual arrays or strings. Similarly, debugging microservices, which leads me onto Event Sourcing environments: to debug streams accurately, you will probably need to write extra custom tooling on top of your system for when things go wrong

• Storage: This one might be a little obvious, but instead of having one source of truth (in our example, it was storing the Vonage Voice API Webhooks), you now have several extra tables that have a lot of additional data being created on top of the source data

Doing It Your Way

From what we've seen, the vanilla structure of doing this is quite complex. The good news is that libraries are out there to help us implement it. Here are my recommendations for each backend language:

• Java: Axon

• PHP: Verbs (Laravel), EventSauce (generic/Symfony)

• NodeJS: Kurrent

• NET/C#:EventFlow

• Python: eventsourcing

• Ruby: Rails Event Store

• Elixir: Commanded. Want to know why I’ve thrown Elixir in the mix here? Have you even heard of Elixir? Check out my article on a Brief History of Erlang.

Conclusion

As with every new engineering pattern that emerges, I am often cynical about how useful these things are in practice. I’ve seen talks about hexagonal architecture, and inverse-inheritance pattern React frontend applications, and I just cannot fathom why developers would want this level of complexity. However, in the case of Event Sourcing, I feel it’s a very different situation. Yes, it’s difficult to implement and architect, but as with everything, what is your use case? If you're writing in a vertical market that requires high levels of data scrutiny, then setting out with Event Sourcing from the get-go is going to save your business from potentially big headaches down the road.