215 lines
12 KiB
Markdown
215 lines
12 KiB
Markdown
# Relata — Implementation Plan
|
||
|
||
Status: living document. Written before implementation, updated as decisions were made.
|
||
|
||
## Goal
|
||
|
||
A self-organising semantic document database: ingest arbitrary information, discover
|
||
**evidence-backed** relationships between stored items, and retrieve related knowledge with
|
||
an explanation of *why* each result was returned.
|
||
|
||
## Non-negotiable invariants
|
||
|
||
These shape every design choice below.
|
||
|
||
1. **Originals are never rewritten.** The raw bytes/text a user supplied are stored verbatim.
|
||
Everything else (extracted text, chunks, entities, claims) is derived and reproducible.
|
||
2. **Three different numbers, never conflated:**
|
||
- *semantic similarity* — cosine distance between embeddings,
|
||
- *relationship strength* — a versioned weighted sum of explicit signals,
|
||
- *query relevance* — a fused retrieval score for one specific query.
|
||
Each is stored in its own field and rendered with its own label.
|
||
3. **Provenance for every inference.** Each relationship records how it came to exist
|
||
(`explicit` / `deterministic` / `model_inferred` / `user_confirmed` / `user_rejected`)
|
||
and carries the signal list that produced its score.
|
||
4. **Model output is untrusted input.** It is parsed with `json.Decoder` against a strict
|
||
schema, enum-validated, and referenced IDs are checked against what was actually sent.
|
||
Invalid output is recorded and discarded, never repaired by regex.
|
||
5. **Stored document content is data, never instruction.** Retrieved text is delimited and
|
||
the LLM is told to treat it as untrusted quoted material.
|
||
|
||
## Architecture
|
||
|
||
Modular monolith + in-process background workers. One binary, several subcommands.
|
||
|
||
```
|
||
cmd/relata serve | createadmin | seed | version
|
||
internal/domain pure types + port interfaces. No I/O imports.
|
||
internal/config env + file config, validated at startup
|
||
internal/parsing text, markdown, JSON, PDF, DOCX -> ExtractedDoc
|
||
internal/chunking structure-aware chunker
|
||
internal/embeddings Embedder port + Ollama adapter + batching/retry
|
||
internal/analysis LLM port + Ollama adapter + strict JSON schema validation
|
||
internal/entities normalisation, deterministic + model-assisted extraction
|
||
internal/relationships candidate discovery, versioned scoring, review
|
||
internal/search hybrid retrieval (lexical + vector + entity + relationship expansion)
|
||
internal/ranking score fusion and explanation building
|
||
internal/ingest pipeline orchestration
|
||
internal/jobs Mongo-backed leased job queue + bounded worker pool
|
||
internal/storage/mongo repositories implementing domain ports
|
||
internal/storage/qdrant vector store implementing domain ports
|
||
internal/auth argon2id, sessions, CSRF, login rate limiting,
|
||
OAuth 2.0 + OIDC sign-in, per-user workspace provisioning,
|
||
hashed and scoped API tokens
|
||
internal/api /api/v1 REST
|
||
internal/web server-rendered HTML + HTMX
|
||
internal/audit audit event recording
|
||
internal/observability structured logs, request IDs, Prometheus metrics
|
||
```
|
||
|
||
**Dependency rule:** `internal/domain` imports only the standard library. Storage adapters
|
||
import domain, never the reverse. `api` and `web` depend on services, not on storage.
|
||
|
||
### Why ports live in `domain`
|
||
|
||
Repository and provider interfaces are declared in `internal/domain` next to the types they
|
||
carry, so the service packages compile against domain alone. Adapters
|
||
(`storage/mongo`, `storage/qdrant`, `embeddings`, `analysis`) implement them.
|
||
|
||
## Data model (MongoDB)
|
||
|
||
`users, sessions, workspaces, items, chunks, entities, entity_aliases, claims,
|
||
relationships, jobs, processing_runs, model_versions, audit_events, api_tokens`
|
||
|
||
- String UUID `_id` everywhere. Chunk IDs double as Qdrant point IDs (Qdrant accepts UUIDs),
|
||
which removes an entire class of ID-mapping bugs.
|
||
- **Every** user-content collection carries `workspace_id`, and every repository method takes
|
||
a workspace and includes it in the filter. There is no repository method that can read
|
||
user content without a workspace.
|
||
- Qdrant payloads hold only `workspace_id, item_id, chunk_id, source_type, created_at,
|
||
embedding_model` — retrieval metadata, not a second document store.
|
||
|
||
## Pipeline
|
||
|
||
`ingest -> parse -> chunk -> embed -> extract entities -> extract claims -> discover relationships`
|
||
|
||
Each stage is a persistent job with a lease. Stages are idempotent: they key derived records
|
||
off deterministic hashes so a retry replaces rather than duplicates.
|
||
|
||
Candidate relationship discovery never does an all-pairs comparison. It gathers a bounded
|
||
candidate set from (a) vector neighbours of the new item's chunks, (b) items sharing a
|
||
normalised entity, (c) items in a temporal window, then scores that set.
|
||
|
||
## Scoring
|
||
|
||
`relationship-score-v1`: weighted sum of independent signals (semantic similarity, exact
|
||
entity overlap, alias overlap, shared concepts, temporal proximity, shared metadata, claim
|
||
support/contradiction, model classification, user confirmation/rejection), each stored with
|
||
its own value and weight so the UI can show the arithmetic. The LLM contributes **one
|
||
signal** — never the score.
|
||
|
||
## Build order
|
||
|
||
1. domain, config, ids, observability
|
||
2. storage/mongo, storage/qdrant
|
||
3. parsing, chunking
|
||
4. embeddings, analysis (ports + Ollama adapters)
|
||
5. jobs queue + workers
|
||
6. entities, claims, relationships, ingest
|
||
7. search, ranking, answer generation
|
||
8. auth, api, web
|
||
9. seed, docs, tests, Docker/Makefile
|
||
|
||
## Deliberate scope limits (MVP)
|
||
|
||
- No OCR. Scanned PDFs are reported as low-extractable-text, original preserved.
|
||
- No archive formats (zip/tar/gz) — rejected at upload.
|
||
- Single workspace in practice, but the boundary is enforced in code and storage from day one.
|
||
- Entity resolution proposes aliases; it never auto-merges on name similarity alone.
|
||
- Relationship scores are heuristic strengths, not calibrated probabilities. Stated in the UI.
|
||
|
||
---
|
||
|
||
## Outcome
|
||
|
||
Built as planned. The notes below record what changed during implementation and what running
|
||
the system revealed, because a plan that claims to have gone exactly to plan is not a useful
|
||
document.
|
||
|
||
### Bugs found by running it, not by reading it
|
||
|
||
Six defects survived code review and were caught by operating the system, testing it under
|
||
concurrency, or tightening validation elsewhere. Each now has a regression test.
|
||
|
||
1. **Worker deadlock.** `execute` waited for the heartbeat goroutine to exit, while that
|
||
goroutine's only exit was the job context being cancelled — which happened via `defer`
|
||
*after* `execute` returned. Every worker wedged after one job; three workers meant three
|
||
stuck jobs and a queue that never drained. Found by watching the queue page stop moving.
|
||
`TestWorkerCompletesJobsAndDoesNotDeadlock` queues more jobs than there are workers.
|
||
|
||
2. **Dates leaking into figures.** The number pattern could begin at the whitespace *before* a
|
||
date and take `202` out of `2025-02-10`. The overlap check tested only the match's start
|
||
offset, so the fragment survived and was then compared against other documents' quantities —
|
||
manufacturing contradictions between documents that agreed. Found by reading the evidence
|
||
behind a suspicious `contradicts`. Fixed by testing the whole span.
|
||
|
||
3. **Mirror-image relationships.** The dedupe key included endpoint order, so a link found
|
||
while processing A and rediscovered while processing B became two records. Every pair in
|
||
the demonstration set was duplicated. `PairKey` now normalises the order.
|
||
|
||
4. **Incomparable ranking curves.** Scaled contributions (entity match, relationship
|
||
expansion) used `weight × strength / 15` while ranked contributions used
|
||
`weight / (60 + rank)`. A full-strength expansion therefore outranked a direct keyword hit —
|
||
"related to a match" beating "is a match". Both now sit on the same reciprocal-rank curve.
|
||
|
||
5. **Duplicate detection that a race could defeat.** Exact-duplicate detection read the
|
||
content hash and then inserted, with only a non-unique index underneath. Eight concurrent
|
||
uploads of identical content stored four items each claiming to be the only one — so
|
||
"an identical item already exists" was simply untrue under load. Found by writing the
|
||
concurrency test rather than by observing it. The fix puts the guarantee where it belongs:
|
||
a partial unique index on `(workspace_id, content_hash)` covering live, non-deliberate
|
||
items, with the insert's duplicate-key error reported as exactly the same `DuplicateError`
|
||
the check produces, so callers see one answer regardless of which path caught it.
|
||
Deliberate duplicates carry `duplicate_of` and sit outside the constraint.
|
||
|
||
6. **A useless model answer failed the whole request.** Adding degeneracy validation to model
|
||
output surfaced this: `Answer` returned `nil, err` for any generation failure, discarding
|
||
the retrieved sources and the recorded contradictions between them. Both are produced
|
||
deterministically and neither depends on the model, so the one case where a user most needs
|
||
to read the sources themselves was the case that showed them nothing. Invalid output now
|
||
degrades — no prose, sources and conflicts retained, rejection reason stated — matching the
|
||
no-material and no-readable-text paths beside it. A timeout or unreachable provider still
|
||
errors, because that is an operational condition rather than a bad answer.
|
||
|
||
A seventh, found by fuzzing rather than running: the Markdown parser set a heading block's `Text`
|
||
to the trimmed line while its offsets spanned the untrimmed one, breaking the invariant that
|
||
`block.Text == doc.Text[block.Start:block.End]` — on which every claim and entity source
|
||
reference depends. The failing input is retained as a corpus seed.
|
||
|
||
### Deliberate changes from the plan
|
||
|
||
- **`MaxUnreviewedScore = 0.95`.** Not in the original design. Without it a candidate with
|
||
enough accumulated signals reached 1.00 — the same score as a human confirmation — and the
|
||
interface presented them identically. That conflates "much evidence" with "confirmed", which
|
||
is the distinction the whole design exists to preserve.
|
||
- **Alias signals require a *confirmed* alias.** The plan listed `alias_entity` as a signal
|
||
without saying which aliases counted. Using unconfirmed candidates would have made a guess
|
||
into evidence, defeating the point of proposing aliases for review.
|
||
- **Claim subjects moved to a two-sided rule.** For a value the subject precedes the figure
|
||
("the cost is £42,000"); for a quantity it follows ("ship 240 **units**"). The one-sided
|
||
rule produced subjects like "ship" and "ltd", which paired unrelated claims.
|
||
- **A model that says "unrelated" no longer also contributes its type suggestion.** Both
|
||
signals were being recorded, so a model could simultaneously argue against and for a link.
|
||
- **No `scripts/openapi` generator.** The specification is hand-written and a test asserts it
|
||
matches the routes the router actually registers, plus every enum and error code. A
|
||
generator would have produced a specification that could only agree with itself.
|
||
|
||
### Verified, with the caveats that apply
|
||
|
||
Run against MongoDB 7, Qdrant 1.12, and Ollama with `nomic-embed-text` and `llama3.1:8b` on a
|
||
CPU-only machine.
|
||
|
||
- The full pipeline completes for all eleven demonstration documents.
|
||
- The two superficially similar depot reports acquire **no** relationship, and the note sharing
|
||
only the entity "Example Ltd" acquires none either — sharing wording, or an entity, is not a
|
||
relationship.
|
||
- The cost contradiction is detected deterministically, with both figures named, and is
|
||
surfaced in generated answers **independently of what the model says**.
|
||
- Deleting an item leaves no orphaned vectors, chunks, claims or entities.
|
||
- Reprocessing twice changes no derived-record count, and a human review decision survives it.
|
||
- Two Relata instances against one MongoDB cooperated correctly through the leased queue —
|
||
unplanned, but a genuine test of the lease design.
|
||
|
||
Known weak points are listed in the README rather than here, so that a reader meets them
|
||
before they meet a surprise.
|