relata/docs/ARCHITECTURE.md

Architecture

Relata is a modular monolith with in-process background workers: one binary, one process, three external data stores. That choice is deliberate and is documented in decisions/0001-modular-monolith.md.

---

Package layout

cmd/relata/                 serve · createadmin · seed · version

internal/
  domain/                   types + port interfaces. Standard library only.
  config/                   environment configuration, validated at startup
  ids/                      UUIDs, session tokens, deterministic dedupe keys
  observability/            structured logging, correlation IDs, metrics

  parsing/                  text · markdown · json · pdf · docx → Document
  chunking/                 structure-aware chunking
  entities/                 normalisation, deterministic extraction, extraction service
  analysis/                 Analyser port, Ollama adapter, strict model-output validation
  embeddings/               Embedder port, Ollama adapter, bounded retry
  relationships/            candidate discovery, versioned scoring, human review
  search/                   hybrid retrieval, answer generation
  ranking/                  rank fusion and result explanation
  ingest/                   ingestion service and pipeline stages
  jobs/                     leased job queue worker pool

  storage/mongo/            repositories implementing the domain ports
  storage/qdrant/           vector store implementing the domain port

  auth/                     Argon2id, sessions, CSRF, login rate limiting
  audit/                    append-only audit recording
  api/                      /api/v1 REST, middleware, router
  web/                      server-rendered HTML, templates, static assets

tests/                      integration and security tests (build tag: integration)
migrations/                 note on why there are none; indexes are applied at startup
docs/                       this documentation, plus openapi.yaml

---

The dependency rule

internal/domain imports only the standard library. No MongoDB driver, no HTTP types, no Qdrant client. It declares:

• the domain types (Item, Chunk, Entity, Claim, Relationship, Job, …);
• the port interfaces the service layer depends on (ItemRepository, VectorStore, Embedder, Analyser, …).

Adapters implement those ports:

      ┌───────────────┐
      │    domain     │  types + ports, stdlib only
      └───────▲───────┘
              │ implements                     │ depends on
   ┌──────────┴──────────┐         ┌───────────▼──────────┐
   │ storage/mongo       │         │ ingest · search      │
   │ storage/qdrant      │         │ relationships        │
   │ embeddings (Ollama) │         │ entities · analysis  │
   │ analysis  (Ollama)  │         └───────────┬──────────┘
   └─────────────────────┘                     │
                                    ┌──────────▼──────────┐
                                    │      api · web      │
                                    └─────────────────────┘

Practical consequences:

• Swapping Ollama for an OpenAI-compatible API means adding one file in embeddings/ and one in analysis/. Nothing else changes. See AI_PROVIDER_INTERFACE.md.
• The scoring model can be unit-tested without a database — scoring_test.go runs in milliseconds and needs nothing running.
• api and web depend on services, never on storage directly.

api and web have a one-way relationship: web imports api for the principal/session context helpers, and the router accepts the web handlers through the api.WebRoutes interface, so api never imports web.

---

Data model

MongoDB is the authoritative store. Qdrant holds vectors and the minimum needed to resolve a hit back to MongoDB.

Collections

Collection	Contents	Key indexes
workspaces	The data boundary	slug (unique)
users	Local accounts, Argon2id hashes	username (unique)
sessions	Server-side sessions	token_hash (unique), TTL on absolute_expiry
items	Source items and their originals	(workspace, ingested_at), (workspace, content_hash), text index
chunks	Retrievable sections	(workspace, item, sequence), (workspace, embedded, model), text index
entities	Extracted subjects	(workspace, normalised_key) unique, (workspace, item_ids)
entity_aliases	Candidate name links	(workspace, entity, normalised) unique
claims	Statements items make	(workspace, subject), (workspace, dedupe_key) unique
relationships	Scored, evidenced links	(workspace, dedupe_key) unique, (workspace, source), (workspace, review_status, score)
jobs	The persistent queue	(status, run_after, priority), (status, lease_expires), partial unique on idempotency_key
processing_runs	Per-item processing history	(workspace, item, started_at)
model_versions	Models that produced stored data	(workspace, kind, name) unique
audit_events	Append-only action log	(workspace, at), (workspace, action, at)
api_tokens	Bearer credentials; hashes only, never secrets	token_hash unique, (workspace, user, created_at), TTL on expires_at

Identifiers

Every record uses a string UUID _id. Chunk IDs are used verbatim as Qdrant point IDs, which Qdrant accepts in UUID form. That removes an entire class of ID-mapping bugs and makes "delete this item's vectors" a filter rather than a join.

Chunk IDs are deterministic — derived from (workspace, item, sequence, content_hash) — so re-chunking an unchanged item reuses the same point IDs and overwrites the vectors rather than orphaning them.

The three unique indexes that carry the weight

These are not merely constraints; they are what make the pipeline idempotent under concurrency.

• entities (workspace, normalised_key) — one entity per normalised name and type. Concurrent workers extracting the same company from different documents converge on one record instead of racing to create two. Similar-but-different names never collide here; they produce a candidate alias for a person to adjudicate, and even a confirmed alias leaves both records in place.
• relationships (workspace, dedupe_key) — the key is derived from the endpoint pair, so reprocessing updates an existing relationship rather than adding a parallel one. Crucially, the upsert refuses to overwrite a human review decision: if someone has confirmed, rejected or edited it, only the evidence is refreshed.
• jobs (idempotency_key) where status = "pending" — a partial index, so at most one pending job may exist per key while completed jobs remain as history. Re-enqueuing after a crash is a no-op rather than duplicate work.

Workspace enforcement

Every user-content record carries workspace_id, and every repository method takes one. The helper wsFilter is the only sanctioned way to begin a filter and returns an error on an empty workspace, so a missing principal cannot silently widen a query.

There is deliberately no GetByID(id) that skips the workspace. Cross-workspace access returns not_found, never forbidden, so probing cannot confirm that an ID exists elsewhere.

Qdrant payloads

Only what retrieval needs:

workspace_id · item_id · chunk_id · source_type · created_at · embedding_model

No text, no titles, no relationship data. Keeping the payload thin is what stops Qdrant drifting into a second, unversioned copy of the corpus. Every payload field is indexed, so a workspace-filtered search stays a filtered search rather than degrading into a scan.

---

Request flow

A search

GET /api/v1/search?q=…
   │
   ├─ RequestID · Recover · Logging · SecurityHeaders · RequireAuth
   │
   ├─ searchQueryFromRequest — typed values only; unknown enums rejected
   │
   └─ search.Service.Search
        ├─ items.TextSearch      (MongoDB $text, workspace-filtered)
        ├─ chunks.TextSearch     (MongoDB $text, for excerpts)
        ├─ embedder.Embed(query) → vectors.Search  (Qdrant, workspace-filtered)
        ├─ entities.FindByNormalised  (query n-grams → entity keys)
        ├─ items.GetMany         (re-checks workspace; drops anything not returned)
        ├─ rels.ListForItems     (expansion; rejected links excluded)
        └─ ranking.Rank          (reciprocal-rank fusion + explanations)

Note the second workspace check after GetMany: candidates whose items are not returned are dropped. Qdrant's filter is already workspace-scoped, but the authoritative check is the MongoDB read.

An ingestion

POST /api/v1/items
   │
   └─ ingest.Service.Ingest
        ├─ admission semaphore (bounded concurrent ingestion)
        ├─ parsing.Sniff        — type from CONTENT, not the filename or header
        ├─ validate title, metadata, size, creation date
        ├─ SHA-256 → FindByContentHash → exact-duplicate check
        ├─ store original bytes (binary) or inline text
        ├─ items.Create
        └─ jobs.Enqueue(parse_item)
              │
              └─ parse → chunk → embed → entities → claims → relationships

Each stage is a separate job that enqueues the next. See INGESTION_PIPELINE.md.

---

Background workers

A bounded pool draining a Mongo-backed queue. The design goals are recoverability and boundedness.

Leases, not locks. A claimed job records its owner and an expiry. A worker renews the lease while it runs. A worker that dies stops renewing; the lease expires; ReleaseExpired returns the job to the pending pool. Nothing external is needed to detect the crash.

Losing a lease cancels the work. If a heartbeat finds the lease gone — another worker has taken over — the job's context is cancelled so the original worker abandons its attempt instead of writing a stale result.

Two independent limits. Worker count bounds concurrent jobs; a separate semaphore bounds concurrent AI calls. A small local model is not swamped just because the worker count was raised.

Only transient failures retry. Invalid model output, an unparseable document, a missing handler and a deleted item are permanent — repeating them wastes the queue. Transient failures retry with bounded exponential backoff and full jitter, within an attempt budget.

A panicking handler does not take the worker down. Handlers run guarded; a panic becomes an ordinary job failure.

Shutdown drains. The HTTP server stops accepting first, then workers are given until the shutdown deadline. A job that does not finish keeps its lease, which expires and returns it to the queue.

---

Idempotency

Reprocessing must not multiply data. Each derived record type has its own mechanism:

Records	Mechanism
Chunks	ReplaceForItem — delete then insert, so one generation exists
Vectors	Deterministic chunk IDs; DeleteByItem before re-chunking
Claims	ReplaceForItem, plus a unique dedupe_key
Entities	UpsertByKey on (workspace, normalised_key); observations accumulate
Relationships	UpsertByDedupeKey; human decisions are never overwritten
Jobs	Partial unique index on idempotency_key for pending jobs
Items	Partial unique index on (workspace_id, content_hash) for live, non-deliberate items — duplicate detection is a read then a write, so the index is what actually enforces it (ADR 6)
Users	Partial unique index on (oauth_provider, oauth_subject) — first sign-in is a read then a write, and two concurrent ones must not provision two workspaces for one person (ADR 8)
API tokens	Unique index on token_hash; authentication is a lookup on it and nothing else

Processing runs are the exception: each reprocess adds a run, because the point is to retain what previous parser and model versions produced.

---

Security architecture

Detailed in THREAT_MODEL.md. The structural points:

• Content-based type detection. A browser's Content-Type and the filename are hints; the magic bytes decide. A zip renamed .docx is rejected unless it genuinely contains word/document.xml.
• Stored paths never derive from user input. An uploaded file is stored under the item's own UUID with an extension from a fixed allow-list. The user's filename is retained for display only.
• Escaping by construction. All HTML goes through html/template's contextual escaping, and the web package deliberately contains no helper that turns a stored string into template.HTML.
• A CSP that makes XSS inert. default-src 'none', script-src 'self', no unsafe-inline, no remote origins. Relata ships its own CSS and JavaScript as files, so nothing inline is needed.
• Typed queries only. Filters are assembled in the storage layer from Go values. A client string becomes an operand, never a key or an operator.
• Model output is untrusted input. Strict schema decoding with unknown fields rejected, enum validation, reference checking against what was actually supplied, and no regex-based repair of malformed JSON.
• Uploaded content is data, never instruction. Retrieved text reaches a model only inside delimited blocks whose terminator the content cannot forge.

---

Testing strategy

Level	Location	Needs
Unit	internal/*/[!i]*_test.go	nothing
Fuzz	parsers, model-output decoding, fences, hashes	nothing
HTTP	internal/api/http_test.go	nothing
Integration	tests/ (tag integration)	Docker: MongoDB + Qdrant
Security regression	across packages	nothing

Unit tests assert behaviour and design invariants, not coverage. Several encode the central claims of the design and will fail if a change breaks them:

• similarity alone cannot produce a strong relationship score;
• a maximally confident model opinion cannot reach the storage threshold;
• observed contradiction outranks a model's opinion when inferring a type;
• chronology alone never yields a causal or updating relationship;
• a cross-workspace access is indistinguishable from a missing record;
• a fabricated citation cannot resolve to a real item.