Code sync, security fixes

This commit is contained in:
Andy Dixon
2026-08-11 10:40:10 +01:00
parent 4e00deedf8
commit 53dbc28383
141 changed files with 36194 additions and 0 deletions
+47
View File
@@ -0,0 +1,47 @@
# 1. A modular monolith with in-process workers
**Status:** accepted
## Context
Relata has several distinct concerns — ingestion, parsing, embedding, analysis, search — and
background work that can take tens of seconds per item. That shape invites splitting into
services: an ingestion service, an analysis worker, a search API.
## Decision
One binary, one process. Modules are Go packages with enforced dependency boundaries, and
background workers run in-process against a job queue stored in MongoDB.
## Rationale
The workload does not justify distribution. Relata targets a personal or small-team corpus;
the bottleneck is the language model, and that is not helped by splitting the application
around it.
Distribution would add: service discovery, inter-service authentication, distributed tracing
to answer questions a single stack trace answers now, partial-failure handling on every call
that is currently a function call, and a deployment story requiring an orchestrator.
The boundaries that matter are enforced without processes. `internal/domain` imports only the
standard library; storage and AI adapters implement its ports. That gives the testability and
replaceability people usually go to microservices for — the scoring model is unit-tested with
nothing running — while keeping a stack trace that spans the whole request.
The job queue is the one piece that looks like distributed infrastructure, because it must
survive a crash. It is a leased queue in MongoDB rather than a message broker: Relata already
requires MongoDB, and adding Kafka or RabbitMQ for a queue that peaks at a few hundred jobs
would be infrastructure without a purpose.
## Consequences
**Good:** one process to deploy, monitor and debug. Refactoring across modules is a compiler
problem rather than a migration. Local development needs no orchestrator.
**Bad:** no independent scaling — a heavy analysis load competes with request handling for the
same process, though the worker pool and AI semaphore bound that. No independent deployment.
Horizontal scaling would need work, though the leased queue means multiple instances against
one MongoDB would already cooperate correctly.
**If this needs revisiting:** the boundary to extract first is the worker pool, since it
already communicates only through the queue.
@@ -0,0 +1,47 @@
# 2. Relationship strength is computed from signals, not asked of a model
**Status:** accepted
## Context
Relata must decide how strongly two items are related. The obvious approach is to show a
language model both documents and ask for a confidence percentage.
## Decision
Strength is a weighted sum of named, individually-stored signals under a versioned scoring
model. A language model may classify the relationship *type* or describe the connection; its
opinion is **one signal**, weighted 0.15 — below the storage threshold, so it cannot create a
relationship on its own.
## Rationale
A model's percentage cannot be explained, reproduced or debugged. Asked twice it may answer
differently; asked why, it produces a plausible rationalisation rather than the reason. A user
seeing "87%" has no way to judge it, and no recourse when it is wrong.
A weighted signal sum is inspectable: the interface shows every signal, its strength, its
weight and its contribution, and the total is arithmetic the user can check. When a
relationship looks wrong, the reason is visible — and often it is a signal firing incorrectly,
which is fixable.
It is also *stable*. The same evidence yields the same score, so a change in score means a
change in evidence, not a change in the model's mood.
The weights encode judgements that would otherwise be invisible: similarity alone tops out
below the storage threshold; a maximally confident model opinion cannot reach it either; human
confirmation dominates everything. Those are the design's central claims, and as weights they
are both explicit and testable — `scoring_test.go` asserts each one.
## Consequences
**Good:** every score is explainable and reproducible. Changing behaviour means changing a
weight, visibly. The system works without a language model at all.
**Bad:** the weights are judgement, not calibration — which is exactly why the output is called
a *heuristic strength* and never a probability. Signals must be hand-built; a relationship
whose evidence has no corresponding signal is invisible.
**Rejected alternative:** learning weights from user confirmations. Tempting, but it would make
scores unstable, unexplainable, and dependent on a training set that does not exist on first
run. Confirmations influence ranking directly instead.
@@ -0,0 +1,54 @@
# 3. Entities merge only on an exact normalised key
**Status:** accepted
## Context
"Example Ltd", "Example Limited", "EXAMPLE LTD" and "Example" may all refer to one company.
Merging them makes shared-entity evidence stronger and the entity browser tidier.
## Decision
Entities are keyed on `(workspace, type, normalised_name)` and merge only on an **exact** key
match. Normalisation is shallow: case folding, whitespace, punctuation, apostrophes,
diacritics. It deliberately does **not** strip legal suffixes or expand abbreviations.
Names that look alike produce a **candidate alias** for a human to confirm. Nothing merges
automatically.
## Rationale
The failure modes are not symmetric.
Failing to merge costs a little evidence: two entity records where one would do, and a
slightly weaker shared-entity signal. Visible, harmless, fixable.
Merging wrongly is **silent data corruption**. If "Example Ltd" and "Example Holdings Ltd"
become one entity, every relationship built on that shared entity is now built on a false
premise, and nothing in the interface reveals it. Those are genuinely different companies far
more often than they are variant spellings of each other — parent and subsidiary, or two
unrelated firms with a common word.
The same asymmetry applies to people. "Dr A Chen" and "Alice Chen" are probably the same
person; "J Smith" and "John Smith" in a corpus of thousands are a coin toss.
So Relata proposes and a human disposes. The alias machinery uses a deliberately *looser* key —
suffixes stripped, plus token overlap, edit distance and acronym detection — precisely because
that key is unsafe for identity but useful for suggestion.
## Consequences
**Good:** no silent corruption of the entity graph. Shared-entity evidence means what it says.
The alias list makes near-matches visible rather than hiding them behind a merge.
**Bad:** more entity records than strictly necessary. Shared-entity signals are weaker until
someone confirms aliases. Confirmation is manual work.
**Note:** entity type is part of the key, so the place "Washington" and the person "Washington"
never collide.
**Confirming an alias does not merge anything.** It records that a person judged two names to
refer to the same thing, and that judgement becomes a `alias_entity` signal weighted at 0.10 —
below an exact name match at 0.25. The two entity records remain separate and independently
inspectable, so the decision is visible and reversible. An *unconfirmed* candidate contributes
nothing at all: treating a proposal as evidence would defeat the purpose of proposing it.
@@ -0,0 +1,50 @@
# 4. A leased job queue in MongoDB, not a message broker
**Status:** accepted
## Context
Processing takes tens of seconds per item and must survive restarts and crashes. Options: a
message broker (RabbitMQ, NATS, Kafka), a dedicated job library backed by Redis, or a queue
in the database Relata already has.
## Decision
A leased queue in MongoDB. Claiming a job atomically sets an owner and an expiry; the worker
renews the lease while it runs; an expired lease returns the job to the pending pool.
## Rationale
Relata already requires MongoDB. A broker would add a fourth service to deploy, monitor, back
up and secure, for a queue that peaks at a few hundred jobs.
More importantly, the queue needs properties brokers make awkward:
- **Jobs are inspectable.** The queue page shows every job, its attempts, its last error and a
retry button. That is a database query, not a management API.
- **Idempotency is enforced by the store.** A partial unique index on `idempotency_key` for
*pending* jobs means a duplicate enqueue is a no-op while completed jobs remain as history.
- **Failure history is part of the record.** Each attempt appends to the job, so "why did this
item never process?" is answerable from the job itself.
Leases rather than locks: a lock needs someone to release it, and a crashed worker releases
nothing. A lease expires on its own, so crash recovery needs no external coordination. Losing
a lease also cancels the worker's context, so a worker that was slow rather than dead abandons
its attempt instead of writing a result another worker has superseded.
## Consequences
**Good:** no extra infrastructure. Jobs are queryable, inspectable and retryable through the
ordinary interface. Crash recovery is automatic and testable. Multiple instances against one
MongoDB would already cooperate correctly.
**Bad:** polling rather than push, so there is up to one poll interval of latency. MongoDB
carries the write load of claims and heartbeats. No fan-out or routing — but Relata's pipeline
is a linear chain, so neither is needed.
**Bug worth recording:** the first implementation deadlocked. `execute` waited for the
heartbeat goroutine to exit, while that goroutine only exited when the job context was
cancelled — which happened via `defer` after `execute` returned. Every worker wedged after one
job. It was caught by running the application, not by the unit tests, and
`TestWorkerCompletesJobsAndDoesNotDeadlock` now queues more jobs than there are workers so a
regression fails immediately.
@@ -0,0 +1,59 @@
# 5. Model output is untrusted input
**Status:** accepted
## Context
Relata asks a language model for structured data: entities, claims, relationship
classifications, answers. The convenient assumption is that a model asked for JSON returns
usable JSON.
## Decision
Model output is validated exactly as strictly as an HTTP request body from an anonymous
client: strict decoding with unknown fields rejected, trailing content refused, size bounded,
enum values checked, referenced IDs verified against what was actually supplied, and quoted
text confirmed to exist in the source. Invalid output is recorded and discarded — never
repaired, never blindly retried.
## Rationale
Three distinct failure modes, and the third is the interesting one.
**Ordinary unreliability.** Models emit trailing prose, wrap JSON in fences, invent fields and
occasionally return something that is not JSON at all. Any of these silently corrupts stored
data if accepted.
**Hallucinated references.** A model asked to cite sources will sometimes cite one that does
not exist. If citations were trusted, an answer could point at an item that has nothing to do
with it — and look authoritative doing so. So the model is shown opaque handles (`S1`, `S2`)
rather than item IDs: a fabricated citation cannot resolve to a real record, by construction.
The same reasoning applies to claim quotations: a claim whose quote cannot be located in the
source is discarded, which is what actually guarantees "a claim never loses its link to the
original source text".
**Compromise.** If the provider were malicious — or if a prompt injection succeeded — the
response is attacker-controlled. Strict schema validation means the worst it can do is
populate typed fields. It cannot invent a relationship type outside the enum, reference an
item it was not shown, or reach anything else, because the model has no tools, no database
access and no credentials.
Deliberately **no regular expressions** are used to find or repair JSON. Regex repair is how
malformed output becomes accepted-but-wrong data. A complete markdown fence is unwrapped —
a structural operation on a delimiter — and if what remains is not valid JSON it is still
rejected.
Invalid output is **not retried**, because a model that cannot follow a schema will not start
on the third attempt; it would only burn the queue. It is recorded on
`relata_model_invalid_output_total{task,reason}` so a systematically failing model is visible.
## Consequences
**Good:** a model cannot corrupt stored data by being unreliable. A fabricated citation cannot
resolve. Rejection rates are measurable, which turns "the model is bad" into a number.
**Bad:** a strict schema rejects output that a lenient parser would have salvaged, so smaller
models yield less. That is the intended trade: less data rather than wrong data.
**Consequence worth stating:** because rejection is per-row where possible — one invalid entity
does not discard the whole response — a partly-usable response is partly used.
@@ -0,0 +1,72 @@
# 6. Duplicate detection is enforced by an index, not by a check
**Status:** accepted
## Context
Ingestion hashes the incoming content, looks for an existing item with that hash, and reports
a duplicate if it finds one. Detection was therefore a read followed by a write, with only a
non-unique index underneath.
That is a time-of-check-to-time-of-use gap. Concurrent requests carrying identical content all
perform the read before any of them performs the write, so all of them find nothing and all of
them store an item. Eight concurrent uploads of the same document produced four items, each
having reported itself as the only copy.
The gap is not exotic. A retried upload, a double-clicked form, or two workers fed from the
same source hit it without anyone trying.
## Decision
A partial unique index on `(workspace_id, content_hash)`, covering only items where
`duplicate_of = ""` and `deleted = false`. The insert's duplicate-key rejection is mapped to
`domain.ErrDuplicate` and reported as exactly the same `DuplicateError` the pre-check produces.
The pre-check stays. It is what makes the common case a clean, cheap answer that names the
existing item; the index is what makes that answer *true*.
## Rationale
A uniqueness rule that a race can defeat is not a rule. The database is the only place where
"at most one" can be decided, because it is the only place where concurrent writers are
serialised. Anything above it is advisory.
The index is **partial** for two reasons, both of which a blanket unique index would break:
- A user may deliberately keep a second copy (`allow_duplicate: true`). Those items carry a
non-empty `duplicate_of` and fall outside the filter, so any number of them can coexist —
the link to the original is recorded rather than the copy being refused.
- A tombstoned item still occupies its row until it is hard-deleted. Excluding
`deleted = true` means removing an item frees its content hash immediately, rather than
blocking re-ingestion for the length of the deletion job.
Losing the race is handled according to what the caller asked for, not according to which
code path noticed. A request with `allow_duplicate: true` that loses the insert has still
asked for a second copy to be kept: the race only revealed *which* item it duplicates, so the
link is recorded and the insert retried, which succeeds because the item is now outside the
partial filter. Refusing it would make the flag mean "unless you were unlucky".
Reporting the same error from both paths matters more than it appears. A caller that sees
`DuplicateError` from the check and an opaque write failure from the race would have to handle
two outcomes for one condition, and the seed command, the API and the web upload form would
each get it subtly differently.
`duplicate_of` is stored unconditionally rather than with `omitempty`, because a partial index
filter matches an empty string but not an absent field. Items written before that change need
a one-off backfill, which `EnsureIndexes` performs itself; it is idempotent and logs what it
touched.
## Consequences
**Good:** the guarantee holds under concurrency, and the enforcement lives next to the data
rather than in whichever caller remembered to check. Deliberate duplicates are unaffected.
**Bad:** an existing deployment whose data already violates the constraint cannot build the
index, and startup fails. That is deliberate — silently proceeding would leave the guarantee
untrue while claiming otherwise — but it means the failure message has to name the fix, and
`migrations/README.md` has to carry the query that finds the offending rows. Nothing is
deleted automatically: which copy is canonical is not a decision Relata should make for you.
**Rejected alternative:** a transaction around check-and-insert. It would work, but it makes
every ingest pay for a case that is rare, requires a replica set, and still leaves the
invariant expressed in application code where a future caller can bypass it.
@@ -0,0 +1,62 @@
# 7. Unusable model output degrades the answer; it does not fail the request
**Status:** accepted
## Context
Model output is validated strictly: strict JSON decoding, enum checks, citations resolved
against the references actually supplied, quotations grounded in the source. Anything that
fails is rejected rather than repaired.
Adding degeneracy validation — rejecting free text that is a schema field name, a placeholder,
or too short to carry meaning — made a pre-existing weakness visible. `Answer` returned
`nil, err` for *any* generation failure. A model that satisfied the schema without answering
therefore produced no sources, no citations and no conflicts: the retrieval was discarded
along with the response.
`llama3.2:3b` returns `{"answer": "sufficient", "sufficient": true, "citations": []}` often
enough that this was the normal experience on a small model, not an edge case.
## Decision
Output rejected as invalid produces an answer with no prose, `sufficient = false`, the
retrieved sources intact, the recorded contradictions between them surfaced, and the rejection
reason stated in the uncertainty section.
Transport failures — a timeout, an unreachable provider — still return an error.
## Rationale
Retrieval and contradiction detection are deterministic and do not involve the model at all.
Discarding them because the model misbehaved throws away the part of the system that worked,
in precisely the situation where the user most needs to read the sources themselves.
It also matches what the surrounding code already does. "Nothing was retrieved" and "the
retrieved items had no readable text" both return an answer explaining the situation rather
than an error. A useless model response is the third case of the same kind, and it was the
only one that behaved differently.
The distinction between an invalid response and a transport failure is the distinction between
a bad answer and a broken dependency. The first is a normal outcome of asking a small model a
question, and the user can work around it by reading the sources. The second is an operational
condition: it needs to reach the operator as a failure, with the 504 that names the remedy,
because no amount of reading sources will fix Ollama being down.
Stating the rejection reason matters. Silently omitting the prose would look like the model
had nothing to say about material that plainly addresses the question.
## Consequences
**Good:** a weak model degrades the experience instead of removing it. The guarantees that do
not depend on the model — sources shown, contradictions surfaced independently of what the
model said — hold in exactly the case where the model is least trustworthy.
**Bad:** a caller can no longer treat "no error" as "an answer was generated". `Sufficient` and
the uncertainty list have to be read, which the API documents and the interface renders, but
it is more to get right than a bare error would be.
**Rejected alternative:** retrying on rejection. A permanently invalid response does not become
valid by repetition, and retrying spends the user's time — 100 seconds per attempt on CPU — to
reach the same place. Rejections are counted on
`relata_model_invalid_output_total{task,reason}` so a model that does this constantly is
visible, which is the actionable signal.
@@ -0,0 +1,85 @@
# 8. Federated identity, keyed on the subject, with a private workspace each
**Status:** accepted
## Context
Relata needed to sit behind an organisation's identity provider, with a public description of
the product in front of it. Two questions had to be answered before any code: what identifies
an account, and what does a new person get.
## Decision
OAuth 2.0 Authorization Code with PKCE, upgraded to OpenID Connect when the provider offers an
issuer. Accounts are keyed on `(provider, subject)`. Each person who signs in gets a **private
workspace of their own**, in which they are admin.
Local password login remains, disabled whenever an identity provider is configured, as a way
back in when that provider is misconfigured or unreachable.
## Rationale
### The subject, never the email
`sub` is the only identifier a provider undertakes not to reassign. Email addresses are
recycled constantly — a person leaves, their address is given to someone new — and an account
keyed on one hands the new holder the previous holder's documents. That is not a subtle
failure: it is a silent, total disclosure of somebody's corpus to a stranger, and it looks like
a successful login.
The email is still stored, for display and for the optional domain allowlist. It is never
looked up.
### A private workspace each
Relata already enforces a workspace boundary in every storage query, from the first commit.
Giving each person their own reuses that boundary rather than inventing a second notion of
separation on top of it — and a mechanism exercised by every query is worth far more than one
added for this feature alone.
The cost is real and worth stating rather than discovering: **relationship discovery never
spans users.** Two people who upload documents about the same contract get no link between
them, because as far as Relata is concerned they are separate corpora. The product's central
value — finding connections you did not know about — stops at the workspace edge. Anyone
choosing this model should choose it knowing that.
Making them admin of their own workspace follows from it. There is nobody else in there to
protect anything from, and a person who cannot review the relationships in their own corpus
cannot use the system at all.
### Provisioning must be idempotent
First sign-in creates a workspace and an account. Two clicks, two tabs, or a retried callback
mean two simultaneous first sign-ins, and creating two workspaces for one person would split
their corpus in a way nothing would later reconcile. A unique index on
`(oauth_provider, oauth_subject)` decides it: the loser re-reads the winner's account.
A workspace created by a losing racer is abandoned rather than deleted. Deleting it would race
with the winner, and an empty unreferenced workspace costs nothing.
### Local login stays, and defaults to off
An identity provider is a single point of failure for getting in at all. A wrong client secret,
an expired registration, or an outage would otherwise mean editing the database to recover.
It defaults to on while no provider is configured and off once one is, so a fresh installation
works out of the box and a federated one does not keep a second front door open by accident.
Setting the variable explicitly always wins. A federated account has no password hash and is
refused by the local path explicitly, rather than relying on hash verification to fail on an
empty string — the guarantee should not depend on a parser's behaviour.
## Consequences
**Good:** identity is stable across email changes; isolation reuses a boundary already
enforced everywhere; a broken provider is recoverable without database surgery.
**Bad:** no shared corpora, so cross-user discovery is impossible by construction. No account
linking either — a local account and an OAuth account with the same email are two people with
two workspaces, because merging them is a decision to make deliberately, not to infer from a
matching address.
**Rejected alternative:** one shared workspace with roles mapped from provider groups. It would
make discovery work across everyone's documents, which is where Relata is most useful — but it
means every person who signs in can read everything anyone has uploaded, and group-to-role
mapping is a poor place to discover you got the boundary wrong. If shared workspaces are added
later, claim-based role mapping belongs with them.
@@ -0,0 +1,80 @@
# 9. API tokens are hashed, scoped, and bounded by their owner's role
**Status:** accepted
## Context
Programs need to ingest and query without a browser session. Sessions are the wrong shape for
that: they expire on idleness, rotate, and are bound to a cookie and a CSRF token.
## Decision
Bearer tokens a person creates for themselves. Only a SHA-256 hash is stored. Each token
carries explicit scopes (`ingest`, `query`, `review`), and its effective permission is the
intersection of those scopes and its owner's **current** role, re-read on every request.
Tokens authenticate `/api/v1` only, and cannot manage tokens.
## Rationale
### Hash, not encrypt
An encrypted token is recoverable by whoever holds the key, which is the same process that
holds the database connection. Hashing means a database read, a backup, a dump or a stolen
snapshot yields nothing that authenticates. The cost — the secret cannot ever be shown again —
is the point rather than a limitation, and the interface says so at the moment of creation.
Authentication is a lookup on a unique index over the whole hash. There is no comparison in
application code for an attacker to time, and no way to probe for a prefix. The 12-character
display prefix is kept in clear so a person can tell their tokens apart; it is far too short to
be a usable fraction of a 32-byte secret.
### Intersection, not assignment
Scopes could have been a fixed grant checked on its own. Making permission the *intersection*
with the owner's live role means one thing that a fixed grant cannot give: **demoting or
disabling a person immediately narrows or kills every token they hold**, without anyone having
to find those tokens. The user record is re-read per request precisely to make that true; a
role cached on the token would mean the change took effect at expiry, which is exactly when it
does not matter.
### Tokens cannot manage tokens
The single most valuable thing a leaked credential can do is make itself permanent. Requiring
an interactive session for creation, listing and revocation means a leaked token has a fixed
expiry it cannot extend, cannot mint replacements, and cannot revoke the tokens that would
replace it.
### The CSRF exemption is about *how*, not *whether*
Bearer requests carry no CSRF token, and should not have to: CSRF defends against a
third-party page spending *ambient* credentials, and an `Authorization` header is not ambient —
a browser never attaches one by itself.
The dangerous way to implement that is to skip CSRF when an `Authorization` header is present,
because then any cookie-authenticated request can disable the check by adding one. So the
exemption keys off how the request actually authenticated, and authentication resolves the
session cookie **first**. A request holding a cookie is never token-authenticated, so the
header cannot switch anything off. This was verified against a running instance, not only
reasoned about.
### Rejected: PATs that never expire
Convenient, and the reason so many leaked credentials stay live for years. Every token has an
expiry, capped by configuration, and stops working with no further action.
### Rejected: tokens in query parameters
A token in a URL ends up in access logs, browser history, referrer headers and screenshots.
Only the `Authorization` header is read.
## Consequences
**Good:** a stolen database yields no usable credential; a role change takes effect instantly
across every token; a leaked token is bounded in scope, in lifetime and in what it can escalate
to.
**Bad:** a lost token cannot be recovered, only revoked and replaced. Scopes are coarse — three
of them, checked per route, with no per-item restriction. Creating a token through the
interface puts its value in the browser's history for one page load, which the page warns
about; the API route returns it in a response body and avoids that entirely.
+19
View File
@@ -0,0 +1,19 @@
# Architectural decision records
One file per decision that would be expensive to reverse or surprising to encounter without
explanation. Each records the context, what was decided, why, and what it costs — including
the alternative that was rejected, since that is usually the part a later reader wants.
| # | Decision | Turns on |
|---|---|---|
| [1](0001-modular-monolith.md) | Modular monolith with in-process workers | One binary and a real dependency rule beat a distributed system for a corpus this size |
| [2](0002-scoring-not-model-percentage.md) | Relationship strength is computed from signals, not asked of a model | A model's percentage cannot be explained, reproduced or debugged |
| [3](0003-conservative-entity-resolution.md) | Entities merge only on an exact normalised key; similar names become candidate aliases | "Example Ltd" and "Example Holdings Ltd" are different companies more often than they are typos |
| [4](0004-leased-job-queue-in-mongodb.md) | A leased job queue in MongoDB rather than a broker | A lease expires by itself; a lock held by a dead worker does not |
| [5](0005-untrusted-model-output.md) | Model output is parsed as untrusted input and never repaired | A regex that fixes malformed JSON is inventing content the model did not produce |
| [6](0006-duplicate-detection-enforced-by-index.md) | Duplicate detection is enforced by a partial unique index, not by a check | A uniqueness rule that a race can defeat is not a rule |
| [7](0007-unusable-model-output-degrades.md) | Unusable model output degrades the answer; it does not fail the request | Retrieval and contradiction detection do not involve the model, so a bad response should not discard them |
| [8](0008-federated-identity-private-workspaces.md) | Federated identity keyed on the provider subject, with a private workspace each | Email addresses get reassigned; keying on one hands a recycled address the previous holder's documents |
| [9](0009-api-tokens-hashed-and-scoped.md) | API tokens are hashed, scoped, and bounded by their owner's live role | A stolen database should yield no usable credential, and demoting someone should narrow every token they hold |
Superseding a decision means adding a record that says so, not editing the old one.