seppv0.1.0
Concepts

Retries & dead-letter

Retry directives on nack, attempt exhaustion and the three ways a job is dead-lettered.

When a job fails, the worker nacks it. A nack is more than "this failed": it tells sepp what happens next, a retry on the queue's schedule, a retry after a delay you pick or a trip straight to the dead-letter store. Together with a per-job attempt limit, those directives decide whether a job comes back around and the three ways it can end up dead-lettered.

Nack retry strategies

A nack carries a retry directive that picks one of three outcomes:

  • Retry (Default): another attempt as long as attempts remain, timed by the queue's retry policy. Immediate unless the queue configures a delay.
  • Retry after a delay (After): the job waits in Scheduled and becomes Ready once the delay has elapsed.
  • Dead-letter (DeadLetter): the job skips any remaining attempts and is dead-lettered right away.

nack returns a boolean that is true when the job was dead-lettered, either because you asked for DeadLetter or because the failing attempt hit the limit, so you can log or alert on a terminal failure. If you drive jobs through the Worker instead of nack directly, its HandlerError variants map onto these: retry, retry-after and permanent (see processing jobs).

use std::time::Duration;
use sepp_rs::client::RetryDirective;

// Pick a directive based on why the job failed.
let directive = if err.is_transient() {
    RetryDirective::Default                          // retry now
} else if err.is_rate_limited() {
    RetryDirective::After(Duration::from_secs(60))   // retry, but not yet
} else {
    RetryDirective::DeadLetter                        // unrecoverable, skip the rest
};

// True if this nack ended the job (DeadLetter, or the attempt limit was hit).
let dead_lettered = client.nack(&job.ctx, directive, err.to_string()).await?;

Backoff and delay

The Default directive follows the queue's retry policy. Out of the box that policy is retry immediately (retry_delay_ms = 0), so a job that fails the instant it is reserved is retried just as fast. Give the queue a base delay and every directive-less nack waits before its next attempt. Because the policy lives on the server it also covers workers that never set a directive, so one outdated worker nacking everything cannot burn a job's attempts in milliseconds:

[limits]
retry_delay_ms = 5000          # first retry waits 5s
retry_backoff = "exponential"  # then 10s, 20s, 40s, ...
retry_delay_max_ms = 3600000   # never more than an hour

[[queues]]
name = "emails"
retry_delay_ms = 30000         # this queue backs off harder

With retry_backoff = "none" every retry waits the base delay; "exponential" doubles it each attempt, capped at retry_delay_max_ms. The server subtracts up to 25% jitter, derived from the job id, so a batch that fails together does not retry together, even after their delays have all hit the cap. An explicit After directive overrides the policy for that one nack, computing the delay yourself.

The policy times nacks only. A lease that expires because a worker crashed or stalled requeues the job immediately, so those attempts pace at the lease duration rather than the retry policy.

A delayed retry, from the policy or from After, takes the same path as a scheduled job: it waits in Scheduled and is promoted to Ready when the delay is up, so the delay is the earliest the retry runs and not an exact time. The delay is capped at the queue's schedule horizon, the same limit as scheduled_at, so a very long backoff is clamped rather than rejected.

use std::time::Duration;
use sepp_rs::client::RetryDirective;

// Override the queue policy for this nack with a delay you compute.
let backoff = Duration::from_secs(2u64.pow(job.ctx.attempt));
client.nack(&job.ctx, RetryDirective::After(backoff), "retrying with backoff").await?;

Attempts, default_max_attempts, and the ceiling

Every delivery carries an attempt number starting at 1, incremented on each redelivery whether the previous one ended in a nack-retry or a lease expiry. A job is delivered up to max_attempts times; once the failing attempt reaches max_attempts, sepp dead-letters it instead of retrying.

A job's max_attempts is resolved in order: the value the producer set on the job, otherwise the queue's default_max_attempts (global default 3) and in all cases clamped to max_attempts_ceiling (global default 100). The ceiling is a guardrail so no producer can ask for an unbounded number of retries; a request above it is clamped down rather than rejected. default_max_attempts itself must not exceed the ceiling, which is checked when the config loads.

[limits]
default_max_attempts = 3    # used when a job does not set its own
max_attempts_ceiling = 100  # hard cap on what any job may request

[[queues]]
name = "emails"
default_max_attempts = 5    # this queue retries a little more before giving up

Override the queue default on a single job at enqueue time:

use sepp_rs::EnqueueRequest;

// Override the queue default for this one job.
let request = EnqueueRequest::new("emails", "welcome")?.with_max_attempts(5);
client.enqueue(request).await?;

Both attempt and max_attempts ride on the job's ctx, so a handler can tell when it is on its last try and act differently rather than just failing again:

// Inside a handler: ctx carries this delivery's place in the sequence.
if ctx.attempt == ctx.max_attempts {
    // Last try; a failure now dead-letters the job, so degrade gracefully.
}

Dead-letter causes

A job leaves for the dead-letter store in exactly one of three ways, recorded as the record's cause:

  • attempts_exhausted: the job ran out of attempts across its nacks and redeliveries, with the final failing attempt reaching max_attempts.
  • rejected: a worker nacked with the DeadLetter directive, skipping any remaining attempts.
  • lease_expired: the lease expired while the job was on its final attempt, so there was no attempt left to redeliver into.

The first two arrive through a nack; the third comes from a worker that simply went away on the last attempt (see leasing & redelivery). Dead-lettered jobs are terminal and are never redelivered.

Retention and replay

By default the dead-letter store is off (dead_letter_retention_ms = 0): a dead-lettered job is dropped on the spot and nothing is kept. Set a positive retention and sepp keeps each dead-lettered job for that long, sweeping it away once the window passes. While a job is retained you can inspect it and replay the ones worth another go.

[storage]
dead_letter_retention_ms = 604800000  # keep dead jobs 7 days; 0 (the default) disables the store

You read them with drain_dead_letters, which returns up to max records (oldest-first, optionally for a single queue) and removes them in the same call. That makes it destructive: a dropped response loses exactly that batch, so the clients do not retry it for you. An empty result is ambiguous, it can mean nothing matched or that retention is off, so check dead_letter_retention_enabled in the server capabilities first.

Draining is destructive

drain_dead_letters removes the records it returns in the same transaction. There is no second copy, so persist or act on a batch before you move on; if the response is lost, that batch is gone.

Each record is a snapshot of the dead job (its cause, the last nack reason, the final attempt and the original fields) with a helper that rebuilds an enqueue for the same queue. A replay is a fresh job: the server assigns a new id and resets the attempt counter to 1.

// Drain up to 100 dead-lettered jobs from "emails" (this removes them).
let records = client.drain_dead_letters(Some("emails"), 100).await?;
for record in records {
    eprintln!(
        "{} died on attempt {} ({:?}): {}",
        record.job_id,
        record.final_attempt,
        record.cause,
        record.last_reason.as_deref().unwrap_or("no reason"),
    );

    // Replay the ones worth another go as a fresh job.
    client.enqueue(record.to_enqueue_request()).await?;
}

Draining is the programmatic path. The admin UI reads the same store without consuming it: it lists a queue's dead letters with their cause and last reason, and can requeue or delete the ones you select.

See also

Priority & scheduling

Priority 0-9 dequeue order and scheduled/delayed jobs with the schedule horizon.

Producing jobs

The producer API end to end: build an enqueue request field by field, schedule and batch jobs, and stay within server limits.

On this page

Nack retry strategiesBackoff and delayAttempts, default_max_attempts, and the ceilingDead-letter causesRetention and replaySee also