relata/docs/INGESTION_PIPELINE.md

Ingestion pipeline

From "a user pastes a note" to "the note has evidence-backed relationships".

The governing rule: the original is preserved. Whatever a user supplies is stored verbatim before any parsing is attempted. If parsing later fails, the failure is recorded as a warning on the item and the original remains retrievable and reprocessable.

---

Overview

   HTTP  ──▶  ingest.Service.Ingest  ──▶  items collection + parse_item job
                                                      │
   ┌──────────────────────────────────────────────────┘
   ▼
 parse_item ─▶ chunk_item ─▶ generate_embeddings ─▶ extract_entities
                                                          │
                          discover_relationships ◀── extract_claims

Each stage is a separate job type, so progress is visible on the queue page and a failure is attributable to a specific stage. Stages hand off by enqueuing the next.

---

Stage 0 — Acceptance

internal/ingest/service.go

Before anything is written:

1. Admission control. A semaphore bounds concurrent ingestion, because each in-flight request holds its content in memory.
2. Size check against RELATA_MAX_UPLOAD_BYTES.
3. Content-based type detection. parsing.Sniff decides from magic bytes. The browser's Content-Type and the filename are hints, never authorities.
   • PDF: %PDF-
   • DOCX: a zip containing word/document.xml — checked by reading the central directory
   • Archives (gzip, bzip2, xz, rar, 7z, plain zip): rejected
   • Executables (ELF, PE, Mach-O, #!): rejected
   • Text: must be valid UTF-8, no NUL bytes, under 2% control characters
   • JSON: parses as a single well-formed value
   • Markdown vs plain text: decided by filename or declared type, then by content signals
4. Validation of title length, metadata keys and values, and the user-asserted creation date (a date far in the future is refused, since it would distort every temporal signal).
5. Exact-duplicate detection. SHA-256 over the supplied bytes, looked up against (workspace, content_hash). An identical item returns 409 with existing_item_id unless allow_duplicate is set, in which case duplicate_of records the link.
6. Original storage. Binary formats are written to {upload_dir}/{workspace_id}/{item_id}.{ext} with mode 0600. The filename is the item's own UUID and the extension comes from a fixed allow-list — the user's filename never influences the path. Text formats are kept inline on the item.
7. Item created, then parse_item enqueued.

If the declared type and the detected type disagree, a mime_mismatch warning is stored. It may be innocent, or it may be an attempt to smuggle one format past a check for another; either way it is visible.

---

Stage 1 — Parse

internal/parsing, handler in pipeline.go

Opens a processing run, then extracts text from the preserved original.

Format	Approach	Retains
Plain text	Paragraph splitting on blank lines; RFC-822 headers recognised as a block	Paragraph offsets
Markdown	Line-based structural parse: headings, lists, fences, quotes, tables	Heading hierarchy as a section path
JSON	Depth-bounded streaming scan, then flattening to path: value lines, keys sorted	Object boundaries as sections
PDF	Page by page; a page that fails is skipped with a warning	Page numbers
DOCX	Streaming XML over word/document.xml; Word heading styles → structural headings	Heading hierarchy

Every block records byte offsets into the extracted text, and the invariant block.Text == doc.Text[block.Start:block.End] holds. Every derived record's source reference depends on that identity; it is asserted in unit tests and fuzz-tested.

Failure behaviour. A parse failure is not a job failure. The warning is recorded, the item is marked failed, the original is untouched, and the job returns successfully — the job did its work correctly, and retrying would only repeat the same failure. The user sees the warning on the item and can reprocess after fixing the cause.

Quality warnings raised here:

• low_extractable_text — a large file yielding almost no letters, usually a scan. Relata does not OCR, and says so rather than silently indexing nothing.
• truncated_text — output exceeded the configured limit; blocks past the cut are dropped and the straddling one clamped, so offsets stay valid.
• json_depth_exceeded — nesting or width beyond the limit.
• parse_failed — recorded per page for PDFs, or for the whole document.

---

Stage 2 — Chunk

internal/chunking

Structure first, character counts only as a ceiling.

Rules, in order:

1. A document below RELATA_CHUNK_MIN_SPLIT_CHARS becomes one chunk. Splitting a two-sentence note produces fragments that embed worse than the whole.
2. A heading starts a new chunk, once the current one is large enough to stand alone.
3. A page change (PDF) is a real boundary.
4. A JSON object boundary is respected once the chunk has reached its target size.
5. Adding a block that would exceed RELATA_CHUNK_MAX_CHARS starts a new chunk.
6. A single block larger than the ceiling is split internally, preferring blank lines, then sentence terminators, then line breaks, then word breaks.
7. Chunks below RELATA_CHUNK_MIN_CHARS are merged into a neighbour.

Overlap. Each chunk after the first extends backwards by up to RELATA_CHUNK_OVERLAP_CHARS, aligned to a word boundary, so a sentence spanning a boundary is retrievable from both sides. The chunk's Start moves back with it, so chunk.Text == text[Start:End] still holds; OverlapChars records how much was borrowed.

Each chunk stores: parent item, sequence, source offsets, page, section, heading, text, content hash and (after embedding) the embedding model.

Chunk IDs are deterministic — derived from (workspace, item, sequence, content_hash) — so re-chunking unchanged content reuses the same Qdrant point IDs and overwrites vectors rather than orphaning them. Before writing new chunks, the item's existing vectors are deleted.

---

Stage 3 — Embed

internal/embeddings

• Chunks are embedded in batches of RELATA_EMBED_BATCH_SIZE.
• An AI semaphore is held for the duration of the provider call, so concurrent model usage stays within RELATA_AI_CONCURRENCY regardless of worker count.
• Dimension validation. The first response fixes the width; a later mismatch is a permanent error, because writing differently-sized vectors into one collection would corrupt it. The error names the fix.
• Bounded retry. Only transport failures and explicitly transient HTTP statuses retry, with exponential backoff and full jitter, within RELATA_AI_MAX_RETRIES. A 400 or an unknown model never retries. Jitter matters because the worker pool dispatches in parallel: without it, a provider restart would be met by every worker retrying in lockstep.
• Vectors go to Qdrant with the minimal payload; chunks are marked embedded with the model name.
• Document text is never logged.

---

Stage 4 — Entities

internal/entities

Two stages, separately labelled so a user can tell a matched pattern from a model's guess.

Deterministic (method = deterministic) — narrow by design, because a pattern that fires on ordinary prose would flood the index and make shared-entity evidence meaningless:

• organisations ending in a legal suffix (Ltd, PLC, GmbH, Inc, …)
• people preceded by an honorific (Dr, Prof, Ms, …)
• email addresses
• dates in ISO, 14 March 2025, March 14, 2025 and 14/03/2025 forms, normalised to ISO-8601 (day/month/year for the ambiguous numeric form, matching the application's British English conventions)

Model-assisted (method = model) — broader concepts, over a bounded sample of chunks. Output is enum-validated per row; one bad row is skipped rather than invalidating the response. A claimed evidence quotation that does not appear in the source is dropped, because a fabricated quotation would break the promise that source references point at real text.

Resolution. Entities are keyed on (workspace, type, normalised_name) and merged only on an exact key match. Normalisation is shallow — case folding, whitespace, punctuation, apostrophes, diacritics — and deliberately does not strip legal suffixes: "Example Ltd" and "Example Holdings Ltd" are different companies far more often than they are typos.

Names that look alike produce a candidate alias for a human to confirm, using a looser key plus token overlap, edit distance and acronym detection. Nothing is merged automatically.

Every mention carries item-relative byte offsets and the verbatim quote.

---

Stage 5 — Claims

A claim records that a source said something. It is not a fact, and it never loses its link to the original text.

Deterministic — sentences carrying a figure or a date, because those are the statements that can actually agree or conflict with another source. Sentence splitting knows common abbreviations, so "Ltd. reported" stays one sentence.

For each, Relata derives comparison keys:

• subject — the content word governing the figure ("cost" in "the total project cost is £42,000")
• numbers — currency- and separator-aware
• dates — normalised to ISO-8601
• polarity — +1 affirmative, −1 negated

Model-assisted — broader claims, with attribution and a quoted/paraphrased/inferred nature. A claim whose quotation cannot be located in the source is discarded; whitespace reflow is tolerated, because that locates the same span rather than inventing one.

---

Stage 6 — Discover relationships

internal/relationships/discover.go

Candidates never come from an all-pairs comparison. Three bounded, indexed sources:

1. Vector neighbours — a sample of the item's chunks is embedded and searched, filtered to the workspace and excluding the item itself.
2. Shared entities — an indexed aggregation over entities.item_ids, ordered by how many entities are shared. Concepts are excluded as a source (too broad) while still contributing a signal.
3. Shared claim subjects — an indexed lookup on claims.subject.

The union is capped at RELATA_REL_MAX_CANDIDATES before any scoring or model call.

Evidence gathering per candidate: shared entities (discounted by corpus frequency), concept overlap, temporal proximity, shared metadata, claim comparison and content-hash match.

Claim comparison is what makes "which sources disagree about the cost?" answerable without a language model: two claims about the same subject with materially different figures (beyond a 0.1% tolerance) or different dates are a contradiction, recorded with both quotations attached. Matching figures are support.

Scoring, then a bounded number of model classification calls — only for candidates that already have real evidence, so model time is not spent on noise. The model contributes one signal; a maximally confident opinion cannot on its own reach the storage threshold.

Storage. Candidates below RELATA_REL_STORE_SCORE are discarded. The rest are stored as pending review with full evidence. Non-directional types are stored in a canonical endpoint order so AB and BA are one relationship. A near-duplicate (similarity above RELATA_NEAR_DUPLICATE_SIMILARITY) is flagged with an explicit note — never merged.

---

Failure and recovery

Failure	Behaviour
Parse fails	Warning recorded, item marked failed, original preserved, job succeeds
Embedding provider down	Transient — retried with backoff, then the job fails and can be retried from the queue page
Model returns invalid JSON	Recorded with a reason, not retried; deterministic results are kept
Model unavailable	Entity and claim extraction degrade to deterministic only; the pipeline continues
Worker crashes	The lease expires and the job returns to the queue
Job exceeds its attempts	Marked failed; retryable manually
Item deleted mid-pipeline	Permanent failure — the stage stops rather than resurrecting derived data

---

Reprocessing

RequestReprocess opens a new processing run and restarts the chain at parsing, with the run ID in the job's idempotency key so it is never suppressed as a duplicate of the original ingest.

Nothing multiplies:

Records	Why not
Chunks	Replaced wholesale
Vectors	Deterministic IDs; the item's vectors are cleared before re-chunking
Claims	Replaced wholesale; unique dedupe key
Entities	Upserted on the normalised key; observations accumulate
Relationships	Upserted on the dedupe key — and a human review decision is never overwritten

Previous processing runs are retained, recording the parser, embedding model, analysis model and scoring version that produced each generation. That is the audit trail for "what did the old parser make of this?"

Reprocess when a parser changes, the embedding model changes, the analysis model changes, or the relationship scoring algorithm changes. For an embedding-model change across the whole corpus, use Settings → Re-embed workspace, which processes in bounded batches and re-enqueues itself rather than holding one lease for hours.