relata/docs/decisions/0004-leased-job-queue-in-mongodb.md

4. A leased job queue in MongoDB, not a message broker

Status: accepted

Context

Processing takes tens of seconds per item and must survive restarts and crashes. Options: a message broker (RabbitMQ, NATS, Kafka), a dedicated job library backed by Redis, or a queue in the database Relata already has.

Decision

A leased queue in MongoDB. Claiming a job atomically sets an owner and an expiry; the worker renews the lease while it runs; an expired lease returns the job to the pending pool.

Rationale

Relata already requires MongoDB. A broker would add a fourth service to deploy, monitor, back up and secure, for a queue that peaks at a few hundred jobs.

More importantly, the queue needs properties brokers make awkward:

• Jobs are inspectable. The queue page shows every job, its attempts, its last error and a retry button. That is a database query, not a management API.
• Idempotency is enforced by the store. A partial unique index on idempotency_key for pending jobs means a duplicate enqueue is a no-op while completed jobs remain as history.
• Failure history is part of the record. Each attempt appends to the job, so "why did this item never process?" is answerable from the job itself.

Leases rather than locks: a lock needs someone to release it, and a crashed worker releases nothing. A lease expires on its own, so crash recovery needs no external coordination. Losing a lease also cancels the worker's context, so a worker that was slow rather than dead abandons its attempt instead of writing a result another worker has superseded.

Consequences

Good: no extra infrastructure. Jobs are queryable, inspectable and retryable through the ordinary interface. Crash recovery is automatic and testable. Multiple instances against one MongoDB would already cooperate correctly.

Bad: polling rather than push, so there is up to one poll interval of latency. MongoDB carries the write load of claims and heartbeats. No fan-out or routing — but Relata's pipeline is a linear chain, so neither is needed.

Bug worth recording: the first implementation deadlocked. execute waited for the heartbeat goroutine to exit, while that goroutine only exited when the job context was cancelled — which happened via defer after execute returned. Every worker wedged after one job. It was caught by running the application, not by the unit tests, and TestWorkerCompletesJobsAndDoesNotDeadlock now queues more jobs than there are workers so a regression fails immediately.