Observability overview

Sepp is developed with observability in mind, including distributed tracing between the clients and server out of the box. It exposes metrics, traces and logs via OpenTelemetry. Prometheus metrics are also available via a separate endpoint. The admin UI provides a live view of the server state, including queue depths and rates, and can be used to inspect jobs in real-time.

What sepp exposes

Sepp gives you four signals:

The admin UI dashboard is the zero-setup way to look at the server right now. It streams queue depths and per-second rates over SSE once a second.

For retention, history and alerting you will something more heavyweight like Prometheus and Grafana. Sepp can push metrics and traces to an OpenTelemetry Collector, which can then export to your backend of choice. The Prometheus scrape endpoint is also available if you prefer that model for metrics.

Recommended stack

A production setup might look something like this:

• Collector — run an OpenTelemetry Collector as a sidecar or daemon. Give it an OTLP receiver on 0.0.0.0:4317 and exporters for your backend. Both tracing.otlp_endpoint and metrics.otlp_endpoint default to http://localhost:4317, matching the collector's gRPC receiver out of the box.
• Prometheus — scrape metrics.prometheus_listen_addr (default 0.0.0.0:9464) at GET /metrics. The endpoint serves the standard Prometheus exposition format and requires no authentication. Bind it to a private interface when the host is multi-homed.
• Grafana — point it at Prometheus and visualize the counters and gauges sepp publishes. The metrics catalogue lists every metric with its labels.
• Log aggregation — switch logging.format to "json" and ship stdout to Loki, Elasticsearch or your aggregator of choice.

Logging

Logging is configured under [logging] and requires a restart to change. Two knobs:

[logging]
level = "info,lsm_tree=warn,fjall=warn"
format = "text"

Level directives

The level field accepts an EnvFilter directive string. A bare level like "warn" applies everywhere. A directive like "info,sepp=debug" sets the default to info but traces the sepp target at debug. The default filters out the LSM storage engine's internal chatter while keeping sepp's own output at info.

RUST_LOG takes precedence when set in the environment, so prefer the config file for stability and an env var for one-off debugging:

RUST_LOG=debug ./sepp

Formats

Audit logs

Admin actions are logged to the same output as the rest of the server with the target field set to sepp::audit, so you can filter them out or route them to a separate sink in your aggregator. The same actions are also stored durably and browsable in the admin UI; see the audit log.

Health checks

Sepp registers the gRPC health checking protocol on the same port as the QueueService. The service name is the fully qualified gRPC symbol. A client calls grpc.health.v1.Health/Check:

grpcurl -plaintext localhost:50051 grpc.health.v1.Health/Check
# { "status": "SERVING" }

This is the endpoint to wire into your orchestration probes:

• Liveness — call Health/Check. If it responds SERVING the gRPC server is accepting connections. A failure means the process is unhealthy and should be restarted.
• Readiness — same call. Sepp reports SERVING as soon as the server binds and the committer starts. There is no separate "not ready" state: once the listener is up, the server can accept and queue requests.

The health endpoint is served alongside QueueService on the same gRPC server, behind the same TLS if enabled but the health service itself is not gated by the API-key interceptor — probes work without sending an authorization header.

Wiring it together

Putting the pieces side by side, here is how the observability surface maps to each listener:

Each listener/export is independent so enable only what you need. A minimal production setup enables the admin UI on a private bind, Prometheus for dashboards and the gRPC health check for your orchestrator.

Where to next