andy/relata
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Code sync, security fixes

Browse Source
This commit is contained in:
Andy Dixon
2026-08-11 10:40:10 +01:00
parent 4e00deedf8
commit 53dbc28383
141 changed files with 36194 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+304
View File
@@ -0,0 +1,304 @@
# Architecture
Relata is a **modular monolith with in-process background workers**: one binary, one process,
three external data stores. That choice is deliberate and is documented in
[decisions/0001-modular-monolith.md](decisions/0001-modular-monolith.md).
---
## Package layout
```
cmd/relata/ serve · createadmin · seed · version
internal/
domain/ types + port interfaces. Standard library only.
config/ environment configuration, validated at startup
ids/ UUIDs, session tokens, deterministic dedupe keys
observability/ structured logging, correlation IDs, metrics
parsing/ text · markdown · json · pdf · docx → Document
chunking/ structure-aware chunking
entities/ normalisation, deterministic extraction, extraction service
analysis/ Analyser port, Ollama adapter, strict model-output validation
embeddings/ Embedder port, Ollama adapter, bounded retry
relationships/ candidate discovery, versioned scoring, human review
search/ hybrid retrieval, answer generation
ranking/ rank fusion and result explanation
ingest/ ingestion service and pipeline stages
jobs/ leased job queue worker pool
storage/mongo/ repositories implementing the domain ports
storage/qdrant/ vector store implementing the domain port
auth/ Argon2id, sessions, CSRF, login rate limiting
audit/ append-only audit recording
api/ /api/v1 REST, middleware, router
web/ server-rendered HTML, templates, static assets
tests/ integration and security tests (build tag: integration)
migrations/ note on why there are none; indexes are applied at startup
docs/ this documentation, plus openapi.yaml
```
---
## The dependency rule
**`internal/domain` imports only the standard library.** No MongoDB driver, no HTTP types, no
Qdrant client. It declares:
- the domain types (`Item`, `Chunk`, `Entity`, `Claim`, `Relationship`, `Job`, …);
- the **port interfaces** the service layer depends on (`ItemRepository`, `VectorStore`,
`Embedder`, `Analyser`, …).
Adapters implement those ports:
```
┌───────────────┐
│ domain │ types + ports, stdlib only
└───────▲───────┘
│ implements │ depends on
┌──────────┴──────────┐ ┌───────────▼──────────┐
│ storage/mongo │ │ ingest · search │
│ storage/qdrant │ │ relationships │
│ embeddings (Ollama) │ │ entities · analysis │
│ analysis (Ollama) │ └───────────┬──────────┘
└─────────────────────┘ │
┌──────────▼──────────┐
│ api · web │
└─────────────────────┘
```
Practical consequences:
- Swapping Ollama for an OpenAI-compatible API means adding one file in `embeddings/` and one
in `analysis/`. Nothing else changes. See [AI_PROVIDER_INTERFACE.md](AI_PROVIDER_INTERFACE.md).
- The scoring model can be unit-tested without a database — `scoring_test.go` runs in
milliseconds and needs nothing running.
- `api` and `web` depend on services, never on storage directly.
`api` and `web` have a one-way relationship: `web` imports `api` for the principal/session
context helpers, and the router accepts the web handlers through the `api.WebRoutes`
interface, so `api` never imports `web`.
---
## Data model
MongoDB is the **authoritative store**. Qdrant holds vectors and the minimum needed to
resolve a hit back to MongoDB.
### Collections
| Collection | Contents | Key indexes |
|---|---|---|
| `workspaces` | The data boundary | `slug` (unique) |
| `users` | Local accounts, Argon2id hashes | `username` (unique) |
| `sessions` | Server-side sessions | `token_hash` (unique), TTL on `absolute_expiry` |
| `items` | Source items and their originals | `(workspace, ingested_at)`, `(workspace, content_hash)`, text index |
| `chunks` | Retrievable sections | `(workspace, item, sequence)`, `(workspace, embedded, model)`, text index |
| `entities` | Extracted subjects | **`(workspace, normalised_key)` unique**, `(workspace, item_ids)` |
| `entity_aliases` | Candidate name links | `(workspace, entity, normalised)` unique |
| `claims` | Statements items make | `(workspace, subject)`, `(workspace, dedupe_key)` unique |
| `relationships` | Scored, evidenced links | **`(workspace, dedupe_key)` unique**, `(workspace, source)`, `(workspace, review_status, score)` |
| `jobs` | The persistent queue | `(status, run_after, priority)`, `(status, lease_expires)`, partial unique on `idempotency_key` |
| `processing_runs` | Per-item processing history | `(workspace, item, started_at)` |
| `model_versions` | Models that produced stored data | `(workspace, kind, name)` unique |
| `audit_events` | Append-only action log | `(workspace, at)`, `(workspace, action, at)` |
| `api_tokens` | Bearer credentials; hashes only, never secrets | `token_hash` unique, `(workspace, user, created_at)`, TTL on `expires_at` |
### Identifiers
Every record uses a **string UUID** `_id`. Chunk IDs are used verbatim as Qdrant point IDs,
which Qdrant accepts in UUID form. That removes an entire class of ID-mapping bugs and makes
"delete this item's vectors" a filter rather than a join.
Chunk IDs are **deterministic** — derived from `(workspace, item, sequence, content_hash)` —
so re-chunking an unchanged item reuses the same point IDs and overwrites the vectors rather
than orphaning them.
### The three unique indexes that carry the weight
These are not merely constraints; they are what make the pipeline idempotent under
concurrency.
- **`entities (workspace, normalised_key)`** — one entity per normalised name and type.
Concurrent workers extracting the same company from different documents converge on one
record instead of racing to create two. Similar-but-different names never collide here;
they produce a candidate alias for a person to adjudicate, and even a confirmed alias
leaves both records in place.
- **`relationships (workspace, dedupe_key)`** — the key is derived from the endpoint pair, so
reprocessing updates an existing relationship rather than adding a parallel one. Crucially,
the upsert **refuses to overwrite a human review decision**: if someone has confirmed,
rejected or edited it, only the evidence is refreshed.
- **`jobs (idempotency_key)` where `status = "pending"`** — a partial index, so at most one
*pending* job may exist per key while completed jobs remain as history. Re-enqueuing after
a crash is a no-op rather than duplicate work.
### Workspace enforcement
Every user-content record carries `workspace_id`, and every repository method takes one. The
helper `wsFilter` is the only sanctioned way to begin a filter and **returns an error on an
empty workspace**, so a missing principal cannot silently widen a query.
There is deliberately no `GetByID(id)` that skips the workspace. Cross-workspace access
returns `not_found`, never `forbidden`, so probing cannot confirm that an ID exists elsewhere.
### Qdrant payloads
Only what retrieval needs:
```
workspace_id · item_id · chunk_id · source_type · created_at · embedding_model
```
No text, no titles, no relationship data. Keeping the payload thin is what stops Qdrant
drifting into a second, unversioned copy of the corpus. Every payload field is indexed, so a
workspace-filtered search stays a filtered search rather than degrading into a scan.
---
## Request flow
### A search
```
GET /api/v1/search?q=…
│
├─ RequestID · Recover · Logging · SecurityHeaders · RequireAuth
│
├─ searchQueryFromRequest — typed values only; unknown enums rejected
│
└─ search.Service.Search
├─ items.TextSearch (MongoDB $text, workspace-filtered)
├─ chunks.TextSearch (MongoDB $text, for excerpts)
├─ embedder.Embed(query) → vectors.Search (Qdrant, workspace-filtered)
├─ entities.FindByNormalised (query n-grams → entity keys)
├─ items.GetMany (re-checks workspace; drops anything not returned)
├─ rels.ListForItems (expansion; rejected links excluded)
└─ ranking.Rank (reciprocal-rank fusion + explanations)
```
Note the second workspace check after `GetMany`: candidates whose items are not returned are
dropped. Qdrant's filter is already workspace-scoped, but the authoritative check is the
MongoDB read.
### An ingestion
```
POST /api/v1/items
│
└─ ingest.Service.Ingest
├─ admission semaphore (bounded concurrent ingestion)
├─ parsing.Sniff — type from CONTENT, not the filename or header
├─ validate title, metadata, size, creation date
├─ SHA-256 → FindByContentHash → exact-duplicate check
├─ store original bytes (binary) or inline text
├─ items.Create
└─ jobs.Enqueue(parse_item)
│
└─ parse → chunk → embed → entities → claims → relationships
```
Each stage is a separate job that enqueues the next. See
[INGESTION_PIPELINE.md](INGESTION_PIPELINE.md).
---
## Background workers
A bounded pool draining a Mongo-backed queue. The design goals are recoverability and
boundedness.
**Leases, not locks.** A claimed job records its owner and an expiry. A worker renews the
lease while it runs. A worker that dies stops renewing; the lease expires; `ReleaseExpired`
returns the job to the pending pool. Nothing external is needed to detect the crash.
**Losing a lease cancels the work.** If a heartbeat finds the lease gone — another worker has
taken over — the job's context is cancelled so the original worker abandons its attempt
instead of writing a stale result.
**Two independent limits.** Worker count bounds concurrent jobs; a separate semaphore bounds
concurrent AI calls. A small local model is not swamped just because the worker count was
raised.
**Only transient failures retry.** Invalid model output, an unparseable document, a missing
handler and a deleted item are permanent — repeating them wastes the queue. Transient
failures retry with bounded exponential backoff and full jitter, within an attempt budget.
**A panicking handler does not take the worker down.** Handlers run guarded; a panic becomes
an ordinary job failure.
**Shutdown drains.** The HTTP server stops accepting first, then workers are given until the
shutdown deadline. A job that does not finish keeps its lease, which expires and returns it
to the queue.
---
## Idempotency
Reprocessing must not multiply data. Each derived record type has its own mechanism:
| Records | Mechanism |
|---|---|
| Chunks | `ReplaceForItem` — delete then insert, so one generation exists |
| Vectors | Deterministic chunk IDs; `DeleteByItem` before re-chunking |
| Claims | `ReplaceForItem`, plus a unique `dedupe_key` |
| Entities | `UpsertByKey` on `(workspace, normalised_key)`; observations accumulate |
| Relationships | `UpsertByDedupeKey`; **human decisions are never overwritten** |
| Jobs | Partial unique index on `idempotency_key` for pending jobs |
| Items | Partial unique index on `(workspace_id, content_hash)` for live, non-deliberate items — duplicate detection is a read then a write, so the index is what actually enforces it ([ADR 6](decisions/0006-duplicate-detection-enforced-by-index.md)) |
| Users | Partial unique index on `(oauth_provider, oauth_subject)` — first sign-in is a read then a write, and two concurrent ones must not provision two workspaces for one person ([ADR 8](decisions/0008-federated-identity-private-workspaces.md)) |
| API tokens | Unique index on `token_hash`; authentication is a lookup on it and nothing else |
Processing runs are the exception: each reprocess **adds** a run, because the point is to
retain what previous parser and model versions produced.
---
## Security architecture
Detailed in [THREAT_MODEL.md](THREAT_MODEL.md). The structural points:
- **Content-based type detection.** A browser's `Content-Type` and the filename are hints; the
magic bytes decide. A zip renamed `.docx` is rejected unless it genuinely contains
`word/document.xml`.
- **Stored paths never derive from user input.** An uploaded file is stored under the item's
own UUID with an extension from a fixed allow-list. The user's filename is retained for
display only.
- **Escaping by construction.** All HTML goes through `html/template`'s contextual escaping,
and the `web` package deliberately contains **no** helper that turns a stored string into
`template.HTML`.
- **A CSP that makes XSS inert.** `default-src 'none'`, `script-src 'self'`, no
`unsafe-inline`, no remote origins. Relata ships its own CSS and JavaScript as files, so
nothing inline is needed.
- **Typed queries only.** Filters are assembled in the storage layer from Go values. A client
string becomes an *operand*, never a key or an operator.
- **Model output is untrusted input.** Strict schema decoding with unknown fields rejected,
enum validation, reference checking against what was actually supplied, and no regex-based
repair of malformed JSON.
- **Uploaded content is data, never instruction.** Retrieved text reaches a model only inside
delimited blocks whose terminator the content cannot forge.
---
## Testing strategy
| Level | Location | Needs |
|---|---|---|
| Unit | `internal/*/[!i]*_test.go` | nothing |
| Fuzz | parsers, model-output decoding, fences, hashes | nothing |
| HTTP | `internal/api/http_test.go` | nothing |
| Integration | `tests/` (tag `integration`) | Docker: MongoDB + Qdrant |
| Security regression | across packages | nothing |
Unit tests assert **behaviour and design invariants**, not coverage. Several encode the
central claims of the design and will fail if a change breaks them:
- similarity alone cannot produce a strong relationship score;
- a maximally confident model opinion cannot reach the storage threshold;
- observed contradiction outranks a model's opinion when inferring a type;
- chronology alone never yields a causal or updating relationship;
- a cross-workspace access is indistinguishable from a missing record;
- a fabricated citation cannot resolve to a real item.