andy/relata

Fork 0

Files

T

Andy Dixon 00ef697b03 first commit

2026-08-08 19:42:49 +01:00

24 KiB

Raw Blame History

Relata

A self-organising semantic document database.

Relata accepts unstructured and semi-structured information, analyses it, discovers evidence-backed relationships between stored items, and lets you search associatively — returning an explanation of why each result came back.

The central idea: store arbitrary information, discover evidence-backed relationships between items, and retrieve related knowledge with an explanation of why each result was returned.

What Relata does

Ingests plain text, Markdown, JSON, PDF and DOCX — pasted, uploaded, or posted to a JSON API. The original is always preserved verbatim.
Extracts text while keeping headings, paragraphs, page numbers and document sections.
Chunks documents along structural boundaries rather than at arbitrary character counts.
Embeds chunks with a local model and stores the vectors in Qdrant.
Extracts entities and claims in two stages: deterministic rules first, then model-assisted extraction for broader concepts. Every record says which produced it.
Discovers candidate relationships between items from a bounded candidate set, and scores them from named, inspectable signals.
Searches with hybrid retrieval — exact matching, MongoDB text search, vector similarity, entity matching and relationship expansion — and explains every result.
Answers questions from stored material only, with citations, an explicit "Sources used" section, and an "Uncertainty and conflicts" section when sources disagree.
Keeps humans in charge: every inferred relationship is a proposal until someone confirms it, and rejecting one never deletes the evidence behind it.

What Relata does not do

These are deliberate limits, not oversights.

It does not decide what is true. A claim records that a source said something. When two sources disagree, Relata shows both and says they conflict; it never picks a winner.
It does not treat similarity as a relationship. Two documents can be near-identical in wording and describe entirely different events. Semantic similarity alone tops out well below the threshold at which a link looks well-evidenced.
It does not produce calibrated probabilities. A relationship's strength is a weighted sum of explicit signals under a versioned scoring model. It is a heuristic strength, and the interface says so wherever it appears.
It does not merge entities on name similarity. "Example Ltd" and "Example Holdings Ltd" stay separate. Similar names become candidate aliases for a person to confirm — and even a confirmed alias does not merge the records; it becomes one weighted signal among many.
It does not ask a language model for a score. A model may classify a relationship type or describe a connection; that contributes exactly one signal among many.
It does not OCR. A scanned PDF with no text layer is stored and preserved, and Relata tells you its contents are not searchable.
It does not accept archives (zip, tar, gz, rar, 7z) or executables.
It does not fetch URLs found inside documents. Nothing in a stored document can cause Relata to make an outbound request.
It is not multi-tenant yet. Every record carries a workspace ID and every query filters on it, so the boundary is enforced from day one — but there is one workspace in practice.

Architecture overview

A modular monolith with in-process background workers. One binary, three data stores.

                    ┌──────────────────────────────────────────┐
   browser ────────▶│  HTTP: server-rendered HTML + /api/v1    │
   API client ─────▶│  auth · CSRF · CSP · rate limits         │
                    └───────────────┬──────────────────────────┘
                                    │
                    ┌───────────────▼──────────────────────────┐
                    │  Services: ingest · search · relationships│
                    │            entities · analysis            │
                    └───────────────┬──────────────────────────┘
                                    │
              ┌─────────────────────┼─────────────────────┐
              ▼                     ▼                     ▼
    ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
    │ MongoDB          │  │ Qdrant           │  │ Ollama           │
    │ authoritative:   │  │ vectors + minimal│  │ embeddings +     │
    │ items, chunks,   │  │ retrieval payload│  │ analysis         │
    │ entities, claims,│  │                  │  │ (replaceable)    │
    │ relationships,   │  │                  │  │                  │
    │ jobs, audit      │  │                  │  │                  │
    └──────────────────┘  └──────────────────┘  └──────────────────┘
              ▲
              │
    ┌─────────┴──────────────────────────────────────────────────┐
    │ Background workers: leased job queue in MongoDB            │
    │ parse → chunk → embed → entities → claims → relationships  │
    └────────────────────────────────────────────────────────────┘

internal/domain holds the types and port interfaces and imports only the standard library. Storage and AI adapters implement those ports; the service layer never sees a MongoDB, Qdrant or HTTP type. See docs/ARCHITECTURE.md.

Prerequisites

Go 1.26.5 or newer — earlier 1.26 patches carry an html/template XSS fix Relata needs
Docker and the Compose plugin, for MongoDB, Qdrant and Ollama
~8 GB free disk for the Ollama models
Linux or macOS. Windows via WSL2.

Ollama can run either in Compose or natively on the host — Relata only needs an HTTP endpoint.

Exact local startup

# 1. Fetch dependencies and create .env from the example
make setup

# 2. Start MongoDB, Qdrant and Ollama
make compose-up

# 3. Pull the models (a few GB, once)
make models

# 4. Create the administrator account (prompts for a password, no echo)
make createadmin

# 5. Run Relata
make dev

Then open http://127.0.0.1:8080 and sign in.

To confirm everything is healthy before signing in:

curl -s http://127.0.0.1:8080/health/ready | jq

mongodb and qdrant must report ok. The AI providers are reported but not required — ingestion, keyword search and browsing all work without them.

Initial administrator creation

relata createadmin --username admin

You are prompted for a password twice, without echo. The minimum length is 12 characters; length is what matters, so a passphrase is fine. The password is hashed with Argon2id (64 MiB, 3 passes) and the plain text is never stored, logged or echoed.

For unattended setup, set RELATA_ADMIN_PASSWORD in the environment instead. The password is deliberately not accepted as a command-line flag, because arguments are visible in the process list and shell history.

Signing in through an identity provider

Relata can sit behind OAuth 2.0 / OpenID Connect. The redirect URI to register is derived from RELATA_BASE_URL and a fixed path:

<RELATA_BASE_URL>/auth/oauth/callback

The path is not configurable, so what you register and what Relata serves cannot drift apart.

RELATA_OAUTH_ENABLED=true
RELATA_OAUTH_PROVIDER_NAME=Blackwall
RELATA_OAUTH_ISSUER=https://www.blackwall.cx      # or the three endpoints directly
RELATA_OAUTH_CLIENT_ID=...
RELATA_OAUTH_CLIENT_SECRET=...

Each person who signs in gets a private workspace of their own and is admin of it. Nobody sees anyone else's documents — which also means relationship discovery never spans users. That trade-off is the point of ADR 8.

Local password login switches itself off once OAuth is configured, and back on if it is not. Set RELATA_LOCAL_LOGIN_ENABLED=true to keep it as a way back in when the provider is broken; relata createadmin works regardless. A federated account has no password and can never sign in locally.

Startup refuses a configuration with no way in at all, and refuses non-HTTPS provider endpoints outside development.

Full detail: docs/AUTHENTICATION.md.

API tokens

Programs authenticate with bearer tokens created in Settings → API tokens, carrying any of ingest, query and review:

curl -H "Authorization: Bearer $RELATA_TOKEN" \
  'http://127.0.0.1:8080/api/v1/search?q=delivery+schedule'

A token is shown once — only its hash is stored — never carries more than its owner's current role, cannot manage tokens, and cannot reach the web interface.

Ollama model setup

Relata uses two models, both replaceable:

Purpose	Default	Why
Embeddings	`nomic-embed-text` (768 dims)	Small, fast, good retrieval quality
Analysis	`llama3.1:8b`	Entity/claim extraction, relationship classification, answers

ollama pull nomic-embed-text
ollama pull llama3.1:8b

On a CPU-only machine, a smaller analysis model is much more comfortable:

ollama pull llama3.2:3b
# then in .env:
RELATA_ANALYSIS_MODEL=llama3.2:3b

To run without a language model entirely, set RELATA_ANALYSIS_ENABLED=false. Ingestion, chunking, embedding, deterministic entity and claim extraction, relationship scoring and search all continue to work; model-assisted extraction and generated answers are disabled.

Changing the embedding model changes the vector geometry. Relata detects a dimension mismatch and refuses to write mixed-width vectors. After such a change, set RELATA_EMBEDDING_DIMENSIONS to match and re-embed the workspace from Settings → Re-embed workspace.

Demonstration dataset

make seed          # load and return
make seed-wait     # load and wait for processing to finish

Eleven documents chosen so that the difference between similarity and evidence is visible:

Document	What it demonstrates
Atlas delivery schedule (original)	Related to, and superseded by, the revision
Atlas revised delivery schedule	Updates the original; shares entities and subject
Atlas invoice reconciliation	Contradicts the original about the project cost
Returns policy v3 / v2	An update that supersedes older information
Northfield fire drill / safety inspection	Superficially similar, different events
Example Ltd sponsorship announcement	Shares an entity but not a subject
Atlas revised delivery — forwarded copy	A likely duplicate, flagged not merged
Office plant care rota	Unrelated to everything
Atlas shipment record (JSON)	Structured data sharing entities and figures

Demonstration searches:

What information relates to the revised delivery date?
Which sources disagree about the project cost?
Show information associated with Example Ltd.
What replaced the original policy?
Find items similar in meaning but not actually about the same event.

Example API requests

The API uses the same session cookie as the browser, and enforces CSRF on state-changing requests. Sign in first and keep the cookie jar.

BASE=http://127.0.0.1:8080

# Sign in (the login form's own CSRF token comes from the page)
TOKEN=$(curl -s -c cj.txt $BASE/login \
  | grep -o 'name="csrf_token" value="[^"]*"' | head -1 | sed 's/.*value="//;s/"//')

curl -s -b cj.txt -c cj.txt -X POST $BASE/login \
  --data-urlencode "username=admin" \
  --data-urlencode "password=YOUR-PASSWORD" \
  --data-urlencode "csrf_token=$TOKEN"

# The session's CSRF token, for subsequent writes
CSRF=$(curl -s -b cj.txt $BASE/ \
  | grep -o 'name="csrf_token" value="[^"]*"' | head -1 | sed 's/.*value="//;s/"//')

Store a note

curl -s -b cj.txt -X POST $BASE/api/v1/items \
  -H 'Content-Type: application/json' -H "X-CSRF-Token: $CSRF" \
  -d '{
        "title": "Delivery update",
        "content": "The revised delivery date is 14 March 2025. Example Ltd will ship 240 units.",
        "source_type": "note",
        "metadata": {"project": "atlas"},
        "created_at": "2025-02-28T09:00:00Z"
      }'

Returns 202 Accepted with the item and the ID of the job that will process it. Identical content returns 409 with existing_item_id; add "allow_duplicate": true to store it anyway.

Upload a document

curl -s -b cj.txt -X POST $BASE/api/v1/items/upload \
  -H "X-CSRF-Token: $CSRF" \
  -F 'file=@contract.pdf' \
  -F 'title=Supply contract' \
  -F 'metadata[project]=atlas'

Search

curl -s -b cj.txt \
  "$BASE/api/v1/search?q=revised+delivery+date&limit=5" | jq '.search.results[] | {
     title: .item.title,
     query_relevance,
     semantic_similarity,
     why: .why_matched
   }'

Ask a question, answered only from stored material

curl -s -b cj.txt -X POST $BASE/api/v1/search/answer \
  -H 'Content-Type: application/json' -H "X-CSRF-Token: $CSRF" \
  -d '{"question": "Which sources disagree about the project cost?"}' | jq

Confirm that two names refer to the same thing

curl -s -b cj.txt -X POST "$BASE/api/v1/aliases/$ALIAS_ID/confirm" -H "X-CSRF-Token: $CSRF"

Candidate aliases appear on an entity's page. Confirming one records a human judgement that becomes evidence during relationship discovery; the two entities remain separate records.

Inspect and review a relationship

curl -s -b cj.txt "$BASE/api/v1/items/$ITEM_ID/relationships" | jq '.outgoing[0].explanation'

curl -s -b cj.txt -X POST "$BASE/api/v1/relationships/$REL_ID/confirm" \
  -H 'Content-Type: application/json' -H "X-CSRF-Token: $CSRF" \
  -d '{"note": "Checked against the signed contract."}'

Preview a deletion before committing

curl -s -b cj.txt -X DELETE -H "X-CSRF-Token: $CSRF" \
  "$BASE/api/v1/items/$ITEM_ID?dry_run=true" | jq .plan

The full specification is in docs/openapi.yaml.

Running the tests

make test               # unit tests with the race detector
make test-integration   # against disposable MongoDB and Qdrant containers
make test-security      # security regression tests
make fuzz               # short run over all 15 fuzz targets: parsers, model output,
                        #   API decoding, filenames, entity and claim extraction, scoring
make lint               # gofmt check, go vet, staticcheck
make security           # govulncheck + security tests + secret check
make check              # everything CI runs

Some of those tests exercise concurrency rather than behaviour, because two of the defects found in this codebase were only reachable under it: duplicate detection is a read followed by a write, and asserting a relationship used to page through a node's links. Both are now enforced by unique indexes and covered by tests/race_test.go.

Integration tests need Docker. They start their own containers, use a database named after the run, and clean up afterwards. To point them at instances you already have:

RELATA_TEST_MONGO_URI=mongodb://127.0.0.1:27017 \
RELATA_TEST_QDRANT_URL=http://127.0.0.1:6333 \
go test -tags=integration ./tests/...

Resetting the local environment

make backup     # capture MongoDB and the uploaded originals first
make reset      # asks for confirmation, then removes all volumes and uploads

Or piecemeal:

docker compose down -v          # drop MongoDB and Qdrant data
rm -rf data/uploads/*           # drop stored original files
make createadmin                # create an account again

To clear only the vectors and rebuild them, use Settings → Re-embed workspace.

Configuration

Every setting is an environment variable, documented in .env.example, which make setup copies to .env. Real environment variables take precedence over the file.

The settings you are most likely to change:

Variable	Default	Notes
`RELATA_HTTP_ADDR`	`127.0.0.1:8080`	Non-loopback requires `RELATA_ALLOW_EXTERNAL_BIND=true`
`RELATA_EMBEDDING_MODEL`	`nomic-embed-text`	Changing it requires re-embedding
`RELATA_ANALYSIS_MODEL`	`llama3.1:8b`	Use `llama3.2:3b` on CPU-only machines
`RELATA_ANALYSIS_ENABLED`	`true`	`false` runs the deterministic pipeline only
`RELATA_WORKER_COUNT`	`3`	Background workers
`RELATA_AI_CONCURRENCY`	`2`	Concurrent model calls, independent of workers
`RELATA_REL_STORE_SCORE`	`0.25`	Strength below which a candidate is discarded
`RELATA_MAX_UPLOAD_BYTES`	`26214400`	25 MB

Configuration is validated at startup: Relata refuses to start with a clear list of every problem rather than failing later inside a request.

.env must never be committed. It is in .gitignore, and make security fails if it is tracked.

Known limitations

Stated plainly, because a system that discovers relationships should be honest about its own.

Relationship scoring

Scores are heuristic strengths, not calibrated probabilities. The weights in relationship-score-v1 were chosen by judgement, not fitted to labelled data. A 0.7 does not mean "70% likely to be a real relationship".
Changing weights changes every future score. Existing relationships keep the version that produced them; they are not recomputed automatically.
Human confirmations and rejections influence ranking and expansion. They do not retrain any model, and Relata never sends your decisions anywhere.

Extraction quality

Deterministic entity extraction is intentionally narrow: organisations with a legal suffix, people with an honorific, email addresses and dates. It will miss a person named without a title. Broader coverage comes from the model stage and is labelled as such.
Deterministic claim extraction only picks up sentences carrying a figure or a date, because those are the statements that can actually agree or conflict. Prose claims need the model.
Contradiction detection compares figures and dates about the same coarse subject. It will not catch a contradiction expressed entirely in prose.
Model output quality varies considerably with model size. A 3B model produces noticeably weaker entity and relationship classification than an 8B one, and its generated answers are often unusable — though the safeguards still hold: an uncited answer is downgraded to "not sufficient", and recorded contradictions are surfaced regardless of what the model says. A model that answers with the schema itself (llama3.2:3b returned {"answer": "sufficient"}) is rejected as degenerate_content rather than displayed; the request still returns the retrieved sources and any recorded contradictions, since neither depends on the model. That check catches obvious emptiness only — it cannot distinguish a weak answer from a good one, so a small model still produces answers not worth reading, it just no longer produces answers that are not answers.
Answer generation on CPU is slow. With llama3.1:8b, three sources take about 33 seconds. The defaults (six passages, a 300-second model timeout) are set to complete on a CPU-only machine; exceeding them returns a 504 that names the remedy. A GPU changes this by an order of magnitude. See docs/OPERATIONS.md.

Search

Query relevance is a fused rank score normalised against the best result in the same response. It is comparable within one result set, not across queries.
MongoDB text search is English-stemmed by default. Other languages match less well.
Relationship expansion only follows links from the strongest matches, bounded by RELATA_SEARCH_MAX_EXPANDED_ITEMS.

Scale

Designed for a personal or small-team corpus — thousands to low tens of thousands of items. Alias proposal compares new entities against up to 200 existing ones of the same type per run, which is bounded but not free at large entity counts.
Relationship discovery indexes at most 500 confirmed aliases per workspace. Beyond that, further confirmed aliases stop contributing their signal; discovery weakens rather than breaking, and a warning is logged rather than the loss happening silently.
One binary, deliberately — no sharding, no coordination layer, no service mesh. Running more than one instance against the same MongoDB does work: the job queue leases rather than locks, so instances share the workload and an instance that dies has its jobs requeued when the lease expires. That was verified with two instances, but it is a property of the queue design rather than a supported deployment topology, and nothing else is coordinated.

Formats

No OCR. Scanned PDFs are preserved and flagged as having little extractable text.
PDF extraction handles ordinary text layouts; complex multi-column layouts and tables may come out in an awkward order.
DOCX extraction reads paragraph text and heading styles. Tables, footnotes, comments and tracked changes are not extracted.

Authentication

No refresh tokens. The provider's access token establishes identity once and is then discarded rather than stored. A revoked upstream account keeps its Relata session until that session expires (idle 12h, absolute 7 days by default); revoke the Relata session too if that matters.
No group or role mapping. Every federated user is admin of their own workspace. There is nothing to map a provider group onto, because there is no shared workspace to have roles in.
No account linking. A local account and a federated account with the same email address are two separate people with two separate workspaces. Merging them is deliberate manual work, not something to infer from a matching address.
Token scopes are coarse — three of them, checked per route, with no per-item restriction.
Creating a token through the interface puts its value in the browser's history for one page load; the page says so. POST /api/v1/tokens returns it in a response body instead.

Security

Prompt-injection defences (delimited untrusted blocks, an unforgeable fence, strict output schemas, no tools available to the model) make injection much harder and bound its effect. They are not a proof. Treat generated answers as summaries of sources, not as authority.
Relata trusts its operator. There is no protection against a malicious administrator.
Sessions are stored server-side and revocable, but there is no device management, no two-factor authentication, and no password reset flow — recreate the account instead.

The threat model, including what is explicitly out of scope, is in docs/THREAT_MODEL.md.

Documentation

Document	Contents
docs/ARCHITECTURE.md	Package layout, dependency rules, data model
docs/THREAT_MODEL.md	Trust boundaries, defences, known limitations
docs/SEARCH_AND_SCORING.md	The three scores and how each is computed
docs/INGESTION_PIPELINE.md	Stage by stage, with failure behaviour
docs/AI_PROVIDER_INTERFACE.md	Adding a provider without touching the pipeline
docs/BACKUP_AND_RESTORE.md	What to back up and how to restore it
docs/OPERATIONS.md	Running it: metrics, logs, common problems
docs/IMPLEMENTATION_PLAN.md	The plan this was built to
docs/AUTHENTICATION.md	Sign-in, workspaces and API tokens
docs/decisions/	Architectural decision records

Licence

Provided as-is for evaluation.

24 KiB Raw Blame History