Architecture shift
The platform needed to become a distributed system.
The previous version could subscribe to two exchanges and create a volatility surface in real time, but it was not going to scale cleanly from that point.
A few more exchanges could probably have been held together with extra instances, more caching, and enough operational glue. That would not have solved the core problem: everything was coupled together.
Why coupling hurt
One market-data update could trigger the whole object graph.
A Deribit update arrived, mutated a collection of Python objects, triggered recalculation of smiles, updated surfaces, refreshed the API, and pushed updates to the frontend.
That is workable while everything lives in one process. It becomes painful when multiple exchanges, currencies, and consumers need to share the same market-data stream.
- Should every exchange maintain its own copy of the surface?
- How should market data be combined across venues?
- How does another process consume the same stream?
- What happens if one component falls behind?
- How do you replay historical data after changing the fitting algorithm?
Kafka pipeline
Every significant state change became a message.
The exchange connectors became deliberately simple. They no longer knew anything about volatility surfaces, databases, dashboards, or API consumers. Their job was to normalize exchange-specific websocket messages into an internal format and publish them.
From that point on, the system became a pipeline. No service needed to know who consumed its output or where its input came from. Each service transformed one message into another.
Exchange
-> Market Data Topic
-> Order Book Worker
-> Order Book Topic
-> Volatility Worker
-> Volatility Topic
-> Surface Worker
-> Surface Topic
-> API
-> Dashboard
-> Historical Storage
-> Analytics
Kubernetes
Independent workers made scaling operationally simple.
Each worker can be deployed independently. If implied-volatility calculation becomes the bottleneck, increase the number of volatility workers. If another exchange is added, deploy another connector. If another currency is introduced, feed it through the same pipeline.
Kubernetes handles failed worker restarts, rolling deployments, and scaling replicas when required. The architecture scales horizontally instead of vertically.
Event contracts
The interface moved from implementation to message contracts.
The codebase did not stop being object-oriented. Each worker can still use sensible classes internally. The difference is that those classes are now local implementation details.
Communication between services happens through immutable events, and those events can be replayed. Each service behaves like a transformation from input event to output event.
Input Event -> Transformation -> Output Event
Result
The surface platform became easier to extend.
If the implied-volatility calculation is rewritten in Rust or C++, downstream services do not need to change as long as the new worker consumes the same input message and produces the same output message.
Need another exchange? Deploy another connector. Need another currency? Feed it into the same pipeline. Need more throughput? Increase worker replicas. Need another consumer? Subscribe to the topic.
The important shift was bigger than using Kafka or running on Kubernetes. The platform stopped being a single application with a large object graph and became a collection of small, independent services connected by replayable events.
Links