Architecture shift
The platform needed to become a distributed system.
The previous version could subscribe to two exchanges and create a volatility surface in real time, but it was not going to scale cleanly from that point.
A few more exchanges could probably have been held together with extra instances, more caching, and enough operational glue. That would not have solved the core problem: everything was coupled together.
Why coupling hurt
One market-data update could trigger the whole object graph.
A Deribit update arrived, mutated a collection of Python objects, triggered recalculation of smiles, updated surfaces, refreshed the API, and pushed updates to the frontend.
That is workable while everything lives in one process. It becomes painful when multiple exchanges, currencies, and consumers need to share the same market-data stream.
- Should every exchange maintain its own copy of the surface?
- How should market data be combined across venues?
- How does another process consume the same stream?
- What happens if one component falls behind?
- How do you replay historical data after changing the fitting algorithm?
Kafka pipeline
Every significant state change became a message.
The exchange connectors became deliberately simple. They no longer knew anything about volatility surfaces, databases, dashboards, or API consumers. Their job was to normalize exchange-specific websocket messages into an internal format and publish them.
From that point on, the system became a pipeline. No service needed to know who consumed its output or where its input came from. Each service transformed one message into another.
Exchange
-> Market Data Topic
-> Order Book Worker
-> Order Book Topic
-> Volatility Worker
-> Volatility Topic
-> Surface Worker
-> Surface Topic
-> API
-> Dashboard
-> Historical Storage
-> Analytics
Topic model
Topics should follow market-state ownership, not just service names.
A useful Kafka design is not one topic per code module. The topic model should describe the state transitions the volatility product actually needs: raw venue events, normalized books, expiry forwards, implied-volatility points, fitted smiles, surface snapshots, risk nodes, diagnostics, and dashboard patches.
That shape keeps replay useful. If SVI weighting changes, the surface worker can replay normalized quote and forward topics without re-consuming raw exchange payloads. If a dashboard bug appears, the websocket patch topic can be inspected without rerunning calibration.
- Keep raw venue events immutable for audit and incident replay.
- Use normalized quote-state topics as the boundary between exchange connectors and volatility workers.
- Publish VolState and SmileState separately so IV inversion and SVI calibration can scale independently.
- Keep dashboard patches compact and derived from accepted surface state, not directly from book deltas.
raw_exchange_events -> normalized_quote_state -> expiry_forward_state -> implied_vol_state -> svi_smile_state -> surface_snapshot -> risk_node_state -> dashboard_patch
Partitioning
Partition keys need to preserve expiry ordering while allowing horizontal scale.
Crypto options data has natural boundaries: currency, venue, expiry, and sometimes instrument. The wrong partition key can either serialize too much work or split one expiry across workers that then race to publish inconsistent smiles.
Derivasys needs the ordering guarantee where the state is fit together. Quote updates can be highly parallel, but smile preparation and SVI fitting need a coherent expiry view before risk reversals, flies, and fixed-tenor rows are published.
- Partition raw venue messages by venue and instrument when reconstructing books.
- Partition normalized quote state by currency and expiry before IV and fit preparation.
- Keep all quotes for one fitted expiry on an ordered path into the smile worker.
- Publish final surface snapshots by currency so dashboard and API consumers receive coherent state.
Kubernetes
Independent workers made scaling operationally simple.
Each worker can be deployed independently. If implied-volatility calculation becomes the bottleneck, increase the number of volatility workers. If another exchange is added, deploy another connector. If another currency is introduced, feed it through the same pipeline.
Kubernetes handles failed worker restarts, rolling deployments, and scaling replicas when required. The architecture scales horizontally instead of vertically.
Lag and readiness
Consumer lag and readiness checks have to be market-aware.
A worker can be alive and still not be publishing a trustworthy surface. Kubernetes readiness therefore needs more than process health, and Kafka lag needs more than a single consumer-group number.
The useful signal is market-aware: which currency is behind, which expiry is stale, which risk nodes were carried forward, and whether the latest accepted SVI fit is still good enough for dashboard and API consumers.
- Fail readiness when a worker cannot reach Kafka, cannot load config, or cannot publish accepted surface state.
- Expose lag by topic, partition, currency, expiry, and worker type.
- Downgrade stale expiries without blanking the entire dashboard when other expiries are healthy.
- Keep previous accepted snapshots available for graceful degradation while lag recovers.
readiness = kafka_connected && config_version_loaded && latest_input_lag < threshold && latest_accepted_surface_age < threshold && reject_rate_within_limit
Event contracts
The interface moved from implementation to message contracts.
The codebase did not stop being object-oriented. Each worker can still use sensible classes internally. The difference is that those classes are now local implementation details.
Communication between services happens through immutable events, and those events can be replayed. Each service behaves like a transformation from input event to output event.
Input Event -> Transformation -> Output Event
Schema evolution
Worker contracts need versions before the Rust migration starts.
The service boundary is only useful if the message contract can evolve without breaking replay or downstream consumers. QuoteState, VolState, SmileState, SurfaceSnapshot, and DashboardPatch events need explicit versions and compatibility rules.
That prevents a common migration failure: the new worker is faster, but it changes field names, timing semantics, reject reasons, or units in a way that silently breaks monitoring and dashboard consumers.
- Version every event type and keep deprecated fields readable through at least one migration window.
- Document units for IV, total variance, risk reversals, flies, timestamps, and latency fields.
- Treat reject reasons and quality flags as API-like contracts, not incidental log strings.
- Replay historical events through new schemas before promoting a worker.
VolState v2 {
schema_version,
quote_state_id,
expiry,
strike,
option_type,
implied_vol,
total_variance,
solver_path,
quality_flags,
calculated_at
}Rust migration
A Rust migration works only if the worker contract is already stable.
The tempting version of a Rust migration is to rewrite the slowest code first. The safer version is to make the service boundary explicit first: what event the volatility worker consumes, what surface state it publishes, and which diagnostics must stay identical after the implementation changes.
For Derivasys, the useful migration boundary is the implied-volatility and fit-preparation worker. That worker can become Rust, C++, or a mixed native extension without forcing the dashboard, API, Kafka topics, or surface consumers to know how the calculation is implemented.
- Freeze the input event schema before changing the implementation language.
- Keep output fields stable: IV, total variance, residuals, quality flags, timing, and reject reasons.
- Replay historical quote-state events through both workers before promoting the native path.
- Compare risk reversals, flies, SVI parameters, and fixed-tenor rows, not only raw IV speed.
- Roll out by currency, venue, or expiry bucket so the dashboard can fall back cleanly.
QuoteState topic -> python_vol_worker -> VolState topic -> rust_vol_worker -> VolState shadow topic -> compare risk nodes, residuals, timings -> promote rust worker after surface outputs match
Shadow replay
The native worker should prove equivalence before it owns production output.
The practical Rust migration path is a shadow worker. It consumes the same QuoteState events as the Python worker, publishes to a shadow VolState topic, and compares output against the current production path before any dashboard or API consumer depends on it.
The comparison should be domain-specific. Matching raw IV speed is not enough. The rollout needs to compare fitted SVI parameters, residuals, risk reversals, flies, fixed-tenor rows, reject reasons, and dashboard patch timing.
- Run the native worker behind the same Kafka consumer contract before replacing production output.
- Compare outputs by currency, venue, expiry, moneyness bucket, and solver path.
- Promote gradually by currency or expiry group instead of switching the whole surface at once.
- Keep a rollback path that returns consumers to the previous VolState topic without changing dashboard code.
Dashboard contract
Kafka and Kubernetes should not change what the browser has to understand.
The architecture can become distributed without making the frontend understand Kafka topics, worker lag, or replay mechanics. The dashboard still needs a compact surface snapshot and small patches that explain what changed.
Keeping that contract stable protects users from backend migration risk. A trader should see fresher smiles, clearer diagnostics, and more reliable risk nodes, not a new mental model for how the pipeline is deployed.
- Publish accepted surface snapshots and websocket patches from one dashboard-facing boundary.
- Include snapshot age, source topic offset, and diagnostic status for observability.
- Keep API and dashboard consumers aligned so they do not display different surface states.
- Expose worker health in diagnostics panels without coupling UI rendering to infrastructure internals.
Result
The surface platform became easier to extend.
If the implied-volatility calculation is rewritten in Rust or C++, downstream services do not need to change as long as the new worker consumes the same input message and produces the same output message.
Need another exchange? Deploy another connector. Need another currency? Feed it into the same pipeline. Need more throughput? Increase worker replicas. Need another consumer? Subscribe to the topic.
The important shift was bigger than using Kafka or running on Kubernetes. The platform stopped being a single application with a large object graph and became a collection of small, independent services connected by replayable events.
Prerequisites and product context
Place this step in the surface pipeline.
The roadmap separates the currently deployed dashboard contract from planned replayable worker boundaries.
Links