Code sync, security fixes
This commit is contained in:
@@ -0,0 +1,47 @@
|
||||
# 1. A modular monolith with in-process workers
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
Relata has several distinct concerns — ingestion, parsing, embedding, analysis, search — and
|
||||
background work that can take tens of seconds per item. That shape invites splitting into
|
||||
services: an ingestion service, an analysis worker, a search API.
|
||||
|
||||
## Decision
|
||||
|
||||
One binary, one process. Modules are Go packages with enforced dependency boundaries, and
|
||||
background workers run in-process against a job queue stored in MongoDB.
|
||||
|
||||
## Rationale
|
||||
|
||||
The workload does not justify distribution. Relata targets a personal or small-team corpus;
|
||||
the bottleneck is the language model, and that is not helped by splitting the application
|
||||
around it.
|
||||
|
||||
Distribution would add: service discovery, inter-service authentication, distributed tracing
|
||||
to answer questions a single stack trace answers now, partial-failure handling on every call
|
||||
that is currently a function call, and a deployment story requiring an orchestrator.
|
||||
|
||||
The boundaries that matter are enforced without processes. `internal/domain` imports only the
|
||||
standard library; storage and AI adapters implement its ports. That gives the testability and
|
||||
replaceability people usually go to microservices for — the scoring model is unit-tested with
|
||||
nothing running — while keeping a stack trace that spans the whole request.
|
||||
|
||||
The job queue is the one piece that looks like distributed infrastructure, because it must
|
||||
survive a crash. It is a leased queue in MongoDB rather than a message broker: Relata already
|
||||
requires MongoDB, and adding Kafka or RabbitMQ for a queue that peaks at a few hundred jobs
|
||||
would be infrastructure without a purpose.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** one process to deploy, monitor and debug. Refactoring across modules is a compiler
|
||||
problem rather than a migration. Local development needs no orchestrator.
|
||||
|
||||
**Bad:** no independent scaling — a heavy analysis load competes with request handling for the
|
||||
same process, though the worker pool and AI semaphore bound that. No independent deployment.
|
||||
Horizontal scaling would need work, though the leased queue means multiple instances against
|
||||
one MongoDB would already cooperate correctly.
|
||||
|
||||
**If this needs revisiting:** the boundary to extract first is the worker pool, since it
|
||||
already communicates only through the queue.
|
||||
@@ -0,0 +1,47 @@
|
||||
# 2. Relationship strength is computed from signals, not asked of a model
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
Relata must decide how strongly two items are related. The obvious approach is to show a
|
||||
language model both documents and ask for a confidence percentage.
|
||||
|
||||
## Decision
|
||||
|
||||
Strength is a weighted sum of named, individually-stored signals under a versioned scoring
|
||||
model. A language model may classify the relationship *type* or describe the connection; its
|
||||
opinion is **one signal**, weighted 0.15 — below the storage threshold, so it cannot create a
|
||||
relationship on its own.
|
||||
|
||||
## Rationale
|
||||
|
||||
A model's percentage cannot be explained, reproduced or debugged. Asked twice it may answer
|
||||
differently; asked why, it produces a plausible rationalisation rather than the reason. A user
|
||||
seeing "87%" has no way to judge it, and no recourse when it is wrong.
|
||||
|
||||
A weighted signal sum is inspectable: the interface shows every signal, its strength, its
|
||||
weight and its contribution, and the total is arithmetic the user can check. When a
|
||||
relationship looks wrong, the reason is visible — and often it is a signal firing incorrectly,
|
||||
which is fixable.
|
||||
|
||||
It is also *stable*. The same evidence yields the same score, so a change in score means a
|
||||
change in evidence, not a change in the model's mood.
|
||||
|
||||
The weights encode judgements that would otherwise be invisible: similarity alone tops out
|
||||
below the storage threshold; a maximally confident model opinion cannot reach it either; human
|
||||
confirmation dominates everything. Those are the design's central claims, and as weights they
|
||||
are both explicit and testable — `scoring_test.go` asserts each one.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** every score is explainable and reproducible. Changing behaviour means changing a
|
||||
weight, visibly. The system works without a language model at all.
|
||||
|
||||
**Bad:** the weights are judgement, not calibration — which is exactly why the output is called
|
||||
a *heuristic strength* and never a probability. Signals must be hand-built; a relationship
|
||||
whose evidence has no corresponding signal is invisible.
|
||||
|
||||
**Rejected alternative:** learning weights from user confirmations. Tempting, but it would make
|
||||
scores unstable, unexplainable, and dependent on a training set that does not exist on first
|
||||
run. Confirmations influence ranking directly instead.
|
||||
@@ -0,0 +1,54 @@
|
||||
# 3. Entities merge only on an exact normalised key
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
"Example Ltd", "Example Limited", "EXAMPLE LTD" and "Example" may all refer to one company.
|
||||
Merging them makes shared-entity evidence stronger and the entity browser tidier.
|
||||
|
||||
## Decision
|
||||
|
||||
Entities are keyed on `(workspace, type, normalised_name)` and merge only on an **exact** key
|
||||
match. Normalisation is shallow: case folding, whitespace, punctuation, apostrophes,
|
||||
diacritics. It deliberately does **not** strip legal suffixes or expand abbreviations.
|
||||
|
||||
Names that look alike produce a **candidate alias** for a human to confirm. Nothing merges
|
||||
automatically.
|
||||
|
||||
## Rationale
|
||||
|
||||
The failure modes are not symmetric.
|
||||
|
||||
Failing to merge costs a little evidence: two entity records where one would do, and a
|
||||
slightly weaker shared-entity signal. Visible, harmless, fixable.
|
||||
|
||||
Merging wrongly is **silent data corruption**. If "Example Ltd" and "Example Holdings Ltd"
|
||||
become one entity, every relationship built on that shared entity is now built on a false
|
||||
premise, and nothing in the interface reveals it. Those are genuinely different companies far
|
||||
more often than they are variant spellings of each other — parent and subsidiary, or two
|
||||
unrelated firms with a common word.
|
||||
|
||||
The same asymmetry applies to people. "Dr A Chen" and "Alice Chen" are probably the same
|
||||
person; "J Smith" and "John Smith" in a corpus of thousands are a coin toss.
|
||||
|
||||
So Relata proposes and a human disposes. The alias machinery uses a deliberately *looser* key —
|
||||
suffixes stripped, plus token overlap, edit distance and acronym detection — precisely because
|
||||
that key is unsafe for identity but useful for suggestion.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** no silent corruption of the entity graph. Shared-entity evidence means what it says.
|
||||
The alias list makes near-matches visible rather than hiding them behind a merge.
|
||||
|
||||
**Bad:** more entity records than strictly necessary. Shared-entity signals are weaker until
|
||||
someone confirms aliases. Confirmation is manual work.
|
||||
|
||||
**Note:** entity type is part of the key, so the place "Washington" and the person "Washington"
|
||||
never collide.
|
||||
|
||||
**Confirming an alias does not merge anything.** It records that a person judged two names to
|
||||
refer to the same thing, and that judgement becomes a `alias_entity` signal weighted at 0.10 —
|
||||
below an exact name match at 0.25. The two entity records remain separate and independently
|
||||
inspectable, so the decision is visible and reversible. An *unconfirmed* candidate contributes
|
||||
nothing at all: treating a proposal as evidence would defeat the purpose of proposing it.
|
||||
@@ -0,0 +1,50 @@
|
||||
# 4. A leased job queue in MongoDB, not a message broker
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
Processing takes tens of seconds per item and must survive restarts and crashes. Options: a
|
||||
message broker (RabbitMQ, NATS, Kafka), a dedicated job library backed by Redis, or a queue
|
||||
in the database Relata already has.
|
||||
|
||||
## Decision
|
||||
|
||||
A leased queue in MongoDB. Claiming a job atomically sets an owner and an expiry; the worker
|
||||
renews the lease while it runs; an expired lease returns the job to the pending pool.
|
||||
|
||||
## Rationale
|
||||
|
||||
Relata already requires MongoDB. A broker would add a fourth service to deploy, monitor, back
|
||||
up and secure, for a queue that peaks at a few hundred jobs.
|
||||
|
||||
More importantly, the queue needs properties brokers make awkward:
|
||||
|
||||
- **Jobs are inspectable.** The queue page shows every job, its attempts, its last error and a
|
||||
retry button. That is a database query, not a management API.
|
||||
- **Idempotency is enforced by the store.** A partial unique index on `idempotency_key` for
|
||||
*pending* jobs means a duplicate enqueue is a no-op while completed jobs remain as history.
|
||||
- **Failure history is part of the record.** Each attempt appends to the job, so "why did this
|
||||
item never process?" is answerable from the job itself.
|
||||
|
||||
Leases rather than locks: a lock needs someone to release it, and a crashed worker releases
|
||||
nothing. A lease expires on its own, so crash recovery needs no external coordination. Losing
|
||||
a lease also cancels the worker's context, so a worker that was slow rather than dead abandons
|
||||
its attempt instead of writing a result another worker has superseded.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** no extra infrastructure. Jobs are queryable, inspectable and retryable through the
|
||||
ordinary interface. Crash recovery is automatic and testable. Multiple instances against one
|
||||
MongoDB would already cooperate correctly.
|
||||
|
||||
**Bad:** polling rather than push, so there is up to one poll interval of latency. MongoDB
|
||||
carries the write load of claims and heartbeats. No fan-out or routing — but Relata's pipeline
|
||||
is a linear chain, so neither is needed.
|
||||
|
||||
**Bug worth recording:** the first implementation deadlocked. `execute` waited for the
|
||||
heartbeat goroutine to exit, while that goroutine only exited when the job context was
|
||||
cancelled — which happened via `defer` after `execute` returned. Every worker wedged after one
|
||||
job. It was caught by running the application, not by the unit tests, and
|
||||
`TestWorkerCompletesJobsAndDoesNotDeadlock` now queues more jobs than there are workers so a
|
||||
regression fails immediately.
|
||||
@@ -0,0 +1,59 @@
|
||||
# 5. Model output is untrusted input
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
Relata asks a language model for structured data: entities, claims, relationship
|
||||
classifications, answers. The convenient assumption is that a model asked for JSON returns
|
||||
usable JSON.
|
||||
|
||||
## Decision
|
||||
|
||||
Model output is validated exactly as strictly as an HTTP request body from an anonymous
|
||||
client: strict decoding with unknown fields rejected, trailing content refused, size bounded,
|
||||
enum values checked, referenced IDs verified against what was actually supplied, and quoted
|
||||
text confirmed to exist in the source. Invalid output is recorded and discarded — never
|
||||
repaired, never blindly retried.
|
||||
|
||||
## Rationale
|
||||
|
||||
Three distinct failure modes, and the third is the interesting one.
|
||||
|
||||
**Ordinary unreliability.** Models emit trailing prose, wrap JSON in fences, invent fields and
|
||||
occasionally return something that is not JSON at all. Any of these silently corrupts stored
|
||||
data if accepted.
|
||||
|
||||
**Hallucinated references.** A model asked to cite sources will sometimes cite one that does
|
||||
not exist. If citations were trusted, an answer could point at an item that has nothing to do
|
||||
with it — and look authoritative doing so. So the model is shown opaque handles (`S1`, `S2`)
|
||||
rather than item IDs: a fabricated citation cannot resolve to a real record, by construction.
|
||||
The same reasoning applies to claim quotations: a claim whose quote cannot be located in the
|
||||
source is discarded, which is what actually guarantees "a claim never loses its link to the
|
||||
original source text".
|
||||
|
||||
**Compromise.** If the provider were malicious — or if a prompt injection succeeded — the
|
||||
response is attacker-controlled. Strict schema validation means the worst it can do is
|
||||
populate typed fields. It cannot invent a relationship type outside the enum, reference an
|
||||
item it was not shown, or reach anything else, because the model has no tools, no database
|
||||
access and no credentials.
|
||||
|
||||
Deliberately **no regular expressions** are used to find or repair JSON. Regex repair is how
|
||||
malformed output becomes accepted-but-wrong data. A complete markdown fence is unwrapped —
|
||||
a structural operation on a delimiter — and if what remains is not valid JSON it is still
|
||||
rejected.
|
||||
|
||||
Invalid output is **not retried**, because a model that cannot follow a schema will not start
|
||||
on the third attempt; it would only burn the queue. It is recorded on
|
||||
`relata_model_invalid_output_total{task,reason}` so a systematically failing model is visible.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** a model cannot corrupt stored data by being unreliable. A fabricated citation cannot
|
||||
resolve. Rejection rates are measurable, which turns "the model is bad" into a number.
|
||||
|
||||
**Bad:** a strict schema rejects output that a lenient parser would have salvaged, so smaller
|
||||
models yield less. That is the intended trade: less data rather than wrong data.
|
||||
|
||||
**Consequence worth stating:** because rejection is per-row where possible — one invalid entity
|
||||
does not discard the whole response — a partly-usable response is partly used.
|
||||
@@ -0,0 +1,72 @@
|
||||
# 6. Duplicate detection is enforced by an index, not by a check
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
Ingestion hashes the incoming content, looks for an existing item with that hash, and reports
|
||||
a duplicate if it finds one. Detection was therefore a read followed by a write, with only a
|
||||
non-unique index underneath.
|
||||
|
||||
That is a time-of-check-to-time-of-use gap. Concurrent requests carrying identical content all
|
||||
perform the read before any of them performs the write, so all of them find nothing and all of
|
||||
them store an item. Eight concurrent uploads of the same document produced four items, each
|
||||
having reported itself as the only copy.
|
||||
|
||||
The gap is not exotic. A retried upload, a double-clicked form, or two workers fed from the
|
||||
same source hit it without anyone trying.
|
||||
|
||||
## Decision
|
||||
|
||||
A partial unique index on `(workspace_id, content_hash)`, covering only items where
|
||||
`duplicate_of = ""` and `deleted = false`. The insert's duplicate-key rejection is mapped to
|
||||
`domain.ErrDuplicate` and reported as exactly the same `DuplicateError` the pre-check produces.
|
||||
|
||||
The pre-check stays. It is what makes the common case a clean, cheap answer that names the
|
||||
existing item; the index is what makes that answer *true*.
|
||||
|
||||
## Rationale
|
||||
|
||||
A uniqueness rule that a race can defeat is not a rule. The database is the only place where
|
||||
"at most one" can be decided, because it is the only place where concurrent writers are
|
||||
serialised. Anything above it is advisory.
|
||||
|
||||
The index is **partial** for two reasons, both of which a blanket unique index would break:
|
||||
|
||||
- A user may deliberately keep a second copy (`allow_duplicate: true`). Those items carry a
|
||||
non-empty `duplicate_of` and fall outside the filter, so any number of them can coexist —
|
||||
the link to the original is recorded rather than the copy being refused.
|
||||
- A tombstoned item still occupies its row until it is hard-deleted. Excluding
|
||||
`deleted = true` means removing an item frees its content hash immediately, rather than
|
||||
blocking re-ingestion for the length of the deletion job.
|
||||
|
||||
Losing the race is handled according to what the caller asked for, not according to which
|
||||
code path noticed. A request with `allow_duplicate: true` that loses the insert has still
|
||||
asked for a second copy to be kept: the race only revealed *which* item it duplicates, so the
|
||||
link is recorded and the insert retried, which succeeds because the item is now outside the
|
||||
partial filter. Refusing it would make the flag mean "unless you were unlucky".
|
||||
|
||||
Reporting the same error from both paths matters more than it appears. A caller that sees
|
||||
`DuplicateError` from the check and an opaque write failure from the race would have to handle
|
||||
two outcomes for one condition, and the seed command, the API and the web upload form would
|
||||
each get it subtly differently.
|
||||
|
||||
`duplicate_of` is stored unconditionally rather than with `omitempty`, because a partial index
|
||||
filter matches an empty string but not an absent field. Items written before that change need
|
||||
a one-off backfill, which `EnsureIndexes` performs itself; it is idempotent and logs what it
|
||||
touched.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** the guarantee holds under concurrency, and the enforcement lives next to the data
|
||||
rather than in whichever caller remembered to check. Deliberate duplicates are unaffected.
|
||||
|
||||
**Bad:** an existing deployment whose data already violates the constraint cannot build the
|
||||
index, and startup fails. That is deliberate — silently proceeding would leave the guarantee
|
||||
untrue while claiming otherwise — but it means the failure message has to name the fix, and
|
||||
`migrations/README.md` has to carry the query that finds the offending rows. Nothing is
|
||||
deleted automatically: which copy is canonical is not a decision Relata should make for you.
|
||||
|
||||
**Rejected alternative:** a transaction around check-and-insert. It would work, but it makes
|
||||
every ingest pay for a case that is rare, requires a replica set, and still leaves the
|
||||
invariant expressed in application code where a future caller can bypass it.
|
||||
@@ -0,0 +1,62 @@
|
||||
# 7. Unusable model output degrades the answer; it does not fail the request
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
Model output is validated strictly: strict JSON decoding, enum checks, citations resolved
|
||||
against the references actually supplied, quotations grounded in the source. Anything that
|
||||
fails is rejected rather than repaired.
|
||||
|
||||
Adding degeneracy validation — rejecting free text that is a schema field name, a placeholder,
|
||||
or too short to carry meaning — made a pre-existing weakness visible. `Answer` returned
|
||||
`nil, err` for *any* generation failure. A model that satisfied the schema without answering
|
||||
therefore produced no sources, no citations and no conflicts: the retrieval was discarded
|
||||
along with the response.
|
||||
|
||||
`llama3.2:3b` returns `{"answer": "sufficient", "sufficient": true, "citations": []}` often
|
||||
enough that this was the normal experience on a small model, not an edge case.
|
||||
|
||||
## Decision
|
||||
|
||||
Output rejected as invalid produces an answer with no prose, `sufficient = false`, the
|
||||
retrieved sources intact, the recorded contradictions between them surfaced, and the rejection
|
||||
reason stated in the uncertainty section.
|
||||
|
||||
Transport failures — a timeout, an unreachable provider — still return an error.
|
||||
|
||||
## Rationale
|
||||
|
||||
Retrieval and contradiction detection are deterministic and do not involve the model at all.
|
||||
Discarding them because the model misbehaved throws away the part of the system that worked,
|
||||
in precisely the situation where the user most needs to read the sources themselves.
|
||||
|
||||
It also matches what the surrounding code already does. "Nothing was retrieved" and "the
|
||||
retrieved items had no readable text" both return an answer explaining the situation rather
|
||||
than an error. A useless model response is the third case of the same kind, and it was the
|
||||
only one that behaved differently.
|
||||
|
||||
The distinction between an invalid response and a transport failure is the distinction between
|
||||
a bad answer and a broken dependency. The first is a normal outcome of asking a small model a
|
||||
question, and the user can work around it by reading the sources. The second is an operational
|
||||
condition: it needs to reach the operator as a failure, with the 504 that names the remedy,
|
||||
because no amount of reading sources will fix Ollama being down.
|
||||
|
||||
Stating the rejection reason matters. Silently omitting the prose would look like the model
|
||||
had nothing to say about material that plainly addresses the question.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** a weak model degrades the experience instead of removing it. The guarantees that do
|
||||
not depend on the model — sources shown, contradictions surfaced independently of what the
|
||||
model said — hold in exactly the case where the model is least trustworthy.
|
||||
|
||||
**Bad:** a caller can no longer treat "no error" as "an answer was generated". `Sufficient` and
|
||||
the uncertainty list have to be read, which the API documents and the interface renders, but
|
||||
it is more to get right than a bare error would be.
|
||||
|
||||
**Rejected alternative:** retrying on rejection. A permanently invalid response does not become
|
||||
valid by repetition, and retrying spends the user's time — 100 seconds per attempt on CPU — to
|
||||
reach the same place. Rejections are counted on
|
||||
`relata_model_invalid_output_total{task,reason}` so a model that does this constantly is
|
||||
visible, which is the actionable signal.
|
||||
@@ -0,0 +1,85 @@
|
||||
# 8. Federated identity, keyed on the subject, with a private workspace each
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
Relata needed to sit behind an organisation's identity provider, with a public description of
|
||||
the product in front of it. Two questions had to be answered before any code: what identifies
|
||||
an account, and what does a new person get.
|
||||
|
||||
## Decision
|
||||
|
||||
OAuth 2.0 Authorization Code with PKCE, upgraded to OpenID Connect when the provider offers an
|
||||
issuer. Accounts are keyed on `(provider, subject)`. Each person who signs in gets a **private
|
||||
workspace of their own**, in which they are admin.
|
||||
|
||||
Local password login remains, disabled whenever an identity provider is configured, as a way
|
||||
back in when that provider is misconfigured or unreachable.
|
||||
|
||||
## Rationale
|
||||
|
||||
### The subject, never the email
|
||||
|
||||
`sub` is the only identifier a provider undertakes not to reassign. Email addresses are
|
||||
recycled constantly — a person leaves, their address is given to someone new — and an account
|
||||
keyed on one hands the new holder the previous holder's documents. That is not a subtle
|
||||
failure: it is a silent, total disclosure of somebody's corpus to a stranger, and it looks like
|
||||
a successful login.
|
||||
|
||||
The email is still stored, for display and for the optional domain allowlist. It is never
|
||||
looked up.
|
||||
|
||||
### A private workspace each
|
||||
|
||||
Relata already enforces a workspace boundary in every storage query, from the first commit.
|
||||
Giving each person their own reuses that boundary rather than inventing a second notion of
|
||||
separation on top of it — and a mechanism exercised by every query is worth far more than one
|
||||
added for this feature alone.
|
||||
|
||||
The cost is real and worth stating rather than discovering: **relationship discovery never
|
||||
spans users.** Two people who upload documents about the same contract get no link between
|
||||
them, because as far as Relata is concerned they are separate corpora. The product's central
|
||||
value — finding connections you did not know about — stops at the workspace edge. Anyone
|
||||
choosing this model should choose it knowing that.
|
||||
|
||||
Making them admin of their own workspace follows from it. There is nobody else in there to
|
||||
protect anything from, and a person who cannot review the relationships in their own corpus
|
||||
cannot use the system at all.
|
||||
|
||||
### Provisioning must be idempotent
|
||||
|
||||
First sign-in creates a workspace and an account. Two clicks, two tabs, or a retried callback
|
||||
mean two simultaneous first sign-ins, and creating two workspaces for one person would split
|
||||
their corpus in a way nothing would later reconcile. A unique index on
|
||||
`(oauth_provider, oauth_subject)` decides it: the loser re-reads the winner's account.
|
||||
|
||||
A workspace created by a losing racer is abandoned rather than deleted. Deleting it would race
|
||||
with the winner, and an empty unreferenced workspace costs nothing.
|
||||
|
||||
### Local login stays, and defaults to off
|
||||
|
||||
An identity provider is a single point of failure for getting in at all. A wrong client secret,
|
||||
an expired registration, or an outage would otherwise mean editing the database to recover.
|
||||
|
||||
It defaults to on while no provider is configured and off once one is, so a fresh installation
|
||||
works out of the box and a federated one does not keep a second front door open by accident.
|
||||
Setting the variable explicitly always wins. A federated account has no password hash and is
|
||||
refused by the local path explicitly, rather than relying on hash verification to fail on an
|
||||
empty string — the guarantee should not depend on a parser's behaviour.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** identity is stable across email changes; isolation reuses a boundary already
|
||||
enforced everywhere; a broken provider is recoverable without database surgery.
|
||||
|
||||
**Bad:** no shared corpora, so cross-user discovery is impossible by construction. No account
|
||||
linking either — a local account and an OAuth account with the same email are two people with
|
||||
two workspaces, because merging them is a decision to make deliberately, not to infer from a
|
||||
matching address.
|
||||
|
||||
**Rejected alternative:** one shared workspace with roles mapped from provider groups. It would
|
||||
make discovery work across everyone's documents, which is where Relata is most useful — but it
|
||||
means every person who signs in can read everything anyone has uploaded, and group-to-role
|
||||
mapping is a poor place to discover you got the boundary wrong. If shared workspaces are added
|
||||
later, claim-based role mapping belongs with them.
|
||||
@@ -0,0 +1,80 @@
|
||||
# 9. API tokens are hashed, scoped, and bounded by their owner's role
|
||||
|
||||
**Status:** accepted
|
||||
|
||||
## Context
|
||||
|
||||
Programs need to ingest and query without a browser session. Sessions are the wrong shape for
|
||||
that: they expire on idleness, rotate, and are bound to a cookie and a CSRF token.
|
||||
|
||||
## Decision
|
||||
|
||||
Bearer tokens a person creates for themselves. Only a SHA-256 hash is stored. Each token
|
||||
carries explicit scopes (`ingest`, `query`, `review`), and its effective permission is the
|
||||
intersection of those scopes and its owner's **current** role, re-read on every request.
|
||||
|
||||
Tokens authenticate `/api/v1` only, and cannot manage tokens.
|
||||
|
||||
## Rationale
|
||||
|
||||
### Hash, not encrypt
|
||||
|
||||
An encrypted token is recoverable by whoever holds the key, which is the same process that
|
||||
holds the database connection. Hashing means a database read, a backup, a dump or a stolen
|
||||
snapshot yields nothing that authenticates. The cost — the secret cannot ever be shown again —
|
||||
is the point rather than a limitation, and the interface says so at the moment of creation.
|
||||
|
||||
Authentication is a lookup on a unique index over the whole hash. There is no comparison in
|
||||
application code for an attacker to time, and no way to probe for a prefix. The 12-character
|
||||
display prefix is kept in clear so a person can tell their tokens apart; it is far too short to
|
||||
be a usable fraction of a 32-byte secret.
|
||||
|
||||
### Intersection, not assignment
|
||||
|
||||
Scopes could have been a fixed grant checked on its own. Making permission the *intersection*
|
||||
with the owner's live role means one thing that a fixed grant cannot give: **demoting or
|
||||
disabling a person immediately narrows or kills every token they hold**, without anyone having
|
||||
to find those tokens. The user record is re-read per request precisely to make that true; a
|
||||
role cached on the token would mean the change took effect at expiry, which is exactly when it
|
||||
does not matter.
|
||||
|
||||
### Tokens cannot manage tokens
|
||||
|
||||
The single most valuable thing a leaked credential can do is make itself permanent. Requiring
|
||||
an interactive session for creation, listing and revocation means a leaked token has a fixed
|
||||
expiry it cannot extend, cannot mint replacements, and cannot revoke the tokens that would
|
||||
replace it.
|
||||
|
||||
### The CSRF exemption is about *how*, not *whether*
|
||||
|
||||
Bearer requests carry no CSRF token, and should not have to: CSRF defends against a
|
||||
third-party page spending *ambient* credentials, and an `Authorization` header is not ambient —
|
||||
a browser never attaches one by itself.
|
||||
|
||||
The dangerous way to implement that is to skip CSRF when an `Authorization` header is present,
|
||||
because then any cookie-authenticated request can disable the check by adding one. So the
|
||||
exemption keys off how the request actually authenticated, and authentication resolves the
|
||||
session cookie **first**. A request holding a cookie is never token-authenticated, so the
|
||||
header cannot switch anything off. This was verified against a running instance, not only
|
||||
reasoned about.
|
||||
|
||||
### Rejected: PATs that never expire
|
||||
|
||||
Convenient, and the reason so many leaked credentials stay live for years. Every token has an
|
||||
expiry, capped by configuration, and stops working with no further action.
|
||||
|
||||
### Rejected: tokens in query parameters
|
||||
|
||||
A token in a URL ends up in access logs, browser history, referrer headers and screenshots.
|
||||
Only the `Authorization` header is read.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Good:** a stolen database yields no usable credential; a role change takes effect instantly
|
||||
across every token; a leaked token is bounded in scope, in lifetime and in what it can escalate
|
||||
to.
|
||||
|
||||
**Bad:** a lost token cannot be recovered, only revoked and replaced. Scopes are coarse — three
|
||||
of them, checked per route, with no per-item restriction. Creating a token through the
|
||||
interface puts its value in the browser's history for one page load, which the page warns
|
||||
about; the API route returns it in a response body and avoids that entirely.
|
||||
@@ -0,0 +1,19 @@
|
||||
# Architectural decision records
|
||||
|
||||
One file per decision that would be expensive to reverse or surprising to encounter without
|
||||
explanation. Each records the context, what was decided, why, and what it costs — including
|
||||
the alternative that was rejected, since that is usually the part a later reader wants.
|
||||
|
||||
| # | Decision | Turns on |
|
||||
|---|---|---|
|
||||
| [1](0001-modular-monolith.md) | Modular monolith with in-process workers | One binary and a real dependency rule beat a distributed system for a corpus this size |
|
||||
| [2](0002-scoring-not-model-percentage.md) | Relationship strength is computed from signals, not asked of a model | A model's percentage cannot be explained, reproduced or debugged |
|
||||
| [3](0003-conservative-entity-resolution.md) | Entities merge only on an exact normalised key; similar names become candidate aliases | "Example Ltd" and "Example Holdings Ltd" are different companies more often than they are typos |
|
||||
| [4](0004-leased-job-queue-in-mongodb.md) | A leased job queue in MongoDB rather than a broker | A lease expires by itself; a lock held by a dead worker does not |
|
||||
| [5](0005-untrusted-model-output.md) | Model output is parsed as untrusted input and never repaired | A regex that fixes malformed JSON is inventing content the model did not produce |
|
||||
| [6](0006-duplicate-detection-enforced-by-index.md) | Duplicate detection is enforced by a partial unique index, not by a check | A uniqueness rule that a race can defeat is not a rule |
|
||||
| [7](0007-unusable-model-output-degrades.md) | Unusable model output degrades the answer; it does not fail the request | Retrieval and contradiction detection do not involve the model, so a bad response should not discard them |
|
||||
| [8](0008-federated-identity-private-workspaces.md) | Federated identity keyed on the provider subject, with a private workspace each | Email addresses get reassigned; keying on one hands a recycled address the previous holder's documents |
|
||||
| [9](0009-api-tokens-hashed-and-scoped.md) | API tokens are hashed, scoped, and bounded by their owner's live role | A stolen database should yield no usable credential, and demoting someone should narrow every token they hold |
|
||||
|
||||
Superseding a decision means adding a record that says so, not editing the old one.
|
||||
Reference in New Issue
Block a user