Architecture
Learn more about how sepp is structured under the hood.
A sepp server is a single binary: a gRPC front backed by an embedded fjall LSM store, with every state change funneled through one committer thread that applies it in memory and fsyncs it to disk in order. Here is a visual diagram:
The storage engine
Sepp uses fjall, a pure Rust embedded LSM store that is similar to RocksDB, as its storage engine. LSM based stores are great for implementing queues because they offer great write performance and fast scans for "smallest key in a range" which is exactly what you need to find the next ready job in a queue.
The tombstone trap
LSM stores don't delete in place. Instead, a delete writes a tombstone marking the key gone, reclaimed later during compaction. A high-churn queue (every reserve and ack deletes a key) piles these up and a range scan then has to step over the whole graveyard to reach a live job. Repeated for each of the n jobs, draining the queue that way degrades to O(n2).
So sepp never scans the store for ready jobs at runtime. It serves them from an in-memory index — updated on every commit, rebuilt from the store on boot — leaving the store to handle only writes and cheap point-reads by id.
The committer
Sepp uses a single writer architecture, where all of the state changes are funneled through a single committer thread. This means we don't have to worry about lock contention or consistency issues that arise from having multiple writers, at the cost of some write throughput when running in buffer persist mode.
In a durable system, the fsync cost almost always dominates. Every time we write something to disk, we must call fsync to ensure that it is actually written and doesn't sit in some OS buffer somewhere. Because we only have a single writer, we can batch multiple state changes into a single fsync call, amortizing the time cost per change. Thanks to this sepp can achieve a very high throughput as well as stay fully durable at the cost of some additional latency.
How aggressively the committer flushes is configurable:
| mode | flush | durability | throughput |
|---|---|---|---|
sync_all (default) | fsync — data + metadata | a response means it's on disk | baseline |
sync_data | fdatasync — data only | a response means it's on disk | slightly better than baseline |
buffer | left to the OS | recent writes can be lost on a crash | ~4× the sync modes |
Full durability
In the sync modes, once a client receives a response the effect is already on disk. There's no window where a crash between responding and writing loses an acknowledged change, which is a weakness of systems that only fsync in the background.
See also
- Job lifecycle & delivery: the states a job moves through and the at-least-once guarantee.
- Queues & payloads: named queues and what jobs carry.
- Configuration: persist modes, durability and storage tuning.