Metrics

Sepp publishes metrics through two independent channels: OTLP push to a collector and a Prometheus scrape endpoint. You can enable either, both or neither.

The [metrics] section requires a restart to apply. All fields, with their defaults:

[metrics]
enabled = false
otlp_endpoint = "http://localhost:4317"
export_interval_ms = 5000
prometheus_enabled = false
prometheus_listen_addr = "0.0.0.0:9464"

Enabling OTLP metrics

Set enabled = true and sepp will push metrics to the OpenTelemetry collector at otlp_endpoint every export_interval_ms milliseconds. Both the counter and histogram instruments and the observable gauges are exported. The collector sees the same catalogue described below regardless of whether it arrives via OTLP or Prometheus.

The otlp_endpoint defaults match the collector's standard gRPC receiver, so if you run the collector locally on the default port no further wiring is needed. Point it elsewhere to talk to a sidecar or a hosted backend that accepts OTLP.

Prometheus scrape endpoint

Set prometheus_enabled = true. Sepp opens a plain HTTP/1.1 listener on prometheus_listen_addr and serves GET /metrics in the standard Prometheus exposition format.

The endpoint carries no authentication. Bind it to a private interface (e.g. 127.0.0.1:9464) when the host is reachable from multiple networks and configure Prometheus to scrape it:

scrape_configs:
  - job_name: sepp
    static_configs:
      - targets: ["localhost:9464"]

OTLP and Prometheus are independent. Enabling both is fine — the Prometheus reader and the OTLP periodic reader coexist inside the same meter provider and both see every instrument.

Metric catalogue

Every metric is prefixed sepp. and tagged with labels where shown. Counters are cumulative since the process started; histograms record milliseconds; gauges are point-in-time snapshots polled by the exporter.

Job lifecycle counters

These track jobs as they move through the queue. All are labeled with queue.

Reserve and sweep counters

Request metrics

Queue-depth gauges

Each gauge emits a data point per queue with a queue label.

Commit and command-queue metrics

How cycle metrics flush

Job counts aren't incremented one-by-one on the hot path. The committer accumulates per-cycle deltas — how many enqueues, reserves, acks, etc. happened in one commit — and flushes them to the OTel instruments in a batch after the cycle finishes. This means counters update at the cadence of the committer (typically tens of milliseconds), not every individual operation and a single cycle's worth of data is atomic from the collector's perspective.

Queue-depth gauges are refreshed the same way: the committer snapshots the in-memory indexes after each cycle and stores the snapshot behind an ArcSwap. The gauge callbacks read the latest snapshot at export time, so the gauges always reflect the state as of the last completed commit.

Building a dashboard

The counters and gauges above give you enough to build the essential panels:

• Job throughput — rate of sepp.jobs.enqueued and sepp.jobs.acked stacked, broken down by queue. A gap between enqueue and ack rate is your backlog growing.
• Queue health — sepp.queue.ready as a timeseries per queue, overlaid with sepp.queue.inflight. Inflight climbing while ready stays flat means jobs are taking too long to process.
• Error signals — sepp.jobs.rejected by reason (a spike in queue_full means you hit a depth cap) and sepp.jobs.dead_lettered by cause (a spike in lease_expired suggests workers are too slow or crashing).
• Latency — sepp.request.duration p50/p95/p99 per method and sepp.commit.duration p95 to spot storage contention.
• Backpressure — sepp.command_queue.depth. A healthy server hovers near zero. Sustained elevation means the committer can't keep up with write load.

The admin UI dashboard gives you the live state without any setup. Prometheus and Grafana give you history, retention and alert rules. See the observability overview for wiring them together.