# Operations Running Relata: what to watch, what the logs mean, and what to do when something is wrong. --- ## Health | Endpoint | Purpose | Fails when | |---|---|---| | `GET /health/live` | Liveness | The process is gone | | `GET /health/ready` | Readiness | MongoDB or Qdrant is unreachable | `/health/live` deliberately checks nothing else: a liveness probe that fails during a brief database blip causes an orchestrator to restart a process that was working perfectly well. `/health/ready` is unauthenticated, so it names each dependency and whether it is reachable, but never why. **The reason is in the server log**, at `WARN` with `dependency` and the request ID: ```bash grep "readiness check failed" relata.log | tail ``` Returning the driver's error to an unauthenticated caller would publish the database host, port and — when the connection string carries them — its credentials. `/health/ready` reports the AI providers but **does not require them**. Ingestion, keyword search and browsing all work without a model, so taking the service out of rotation because Ollama restarted would be wrong. ```json { "status": "ready", "checks": { "mongodb": {"status": "ok", "required": true}, "qdrant": {"status": "ok", "required": true}, "embedding_provider": {"status": "ok", "required": false}, "analysis_provider": {"status": "unavailable", "required": false, "detail": "model \"llama3.1:8b\" is not installed — run: ollama pull llama3.1:8b"} } } ``` --- ## Metrics Prometheus exposition at `GET /metrics`. These describe **the service**, not the stored knowledge: no item ID, entity name or query string ever becomes a label value. | Metric | Type | Labels | Watch for | |---|---|---|---| | `relata_http_requests_total` | counter | route, method, status | A rise in `5xx` | | `relata_http_request_duration_seconds` | histogram | route, method | Search latency drift | | `relata_jobs_processed_total` | counter | type, outcome | `outcome="permanent"` climbing | | `relata_job_duration_seconds` | histogram | type | Embedding or analysis slowing | | `relata_jobs_queued` | gauge | status | A backlog that never drains | | `relata_ai_requests_total` | counter | provider, operation, outcome | Provider errors | | `relata_ai_request_duration_seconds` | histogram | provider, operation | Model host under strain | | `relata_storage_operations_total` | counter | store, operation, outcome | Database errors | | `relata_model_invalid_output_total` | counter | task, reason | A model producing unusable output | | `relata_auth_attempts_total` | counter | outcome | `bad_password` or `rate_limited` spikes | | `relata_ingest_items_total` | counter | source_type, outcome | Rejections | | `relata_search_duration_seconds` | histogram | — | Retrieval latency | ### Alerts worth having ```yaml - alert: RelataJobsFailing expr: rate(relata_jobs_processed_total{outcome=~"permanent|invalid_model_output"}[15m]) > 0.1 annotations: summary: Jobs are failing permanently — check /jobs for the errors - alert: RelataQueueBacklog expr: relata_jobs_queued{status="pending"} > 100 for: 30m annotations: summary: The queue is not draining — is the AI provider reachable? - alert: RelataModelOutputRejected expr: rate(relata_model_invalid_output_total[15m]) > 0.05 annotations: summary: The model is producing output that fails validation — consider a larger model - alert: RelataLoginAttacks expr: rate(relata_auth_attempts_total{outcome="rate_limited"}[5m]) > 1 annotations: summary: Sustained failed sign-ins ``` --- ## Logs Structured, via `log/slog`. Set `RELATA_LOG_JSON=true` in production. Every line carries `request_id` or `job_id`, so a user's report ("I got reference `3f2504e0-…`") leads straight to the failure. ```json {"time":"...","level":"INFO","msg":"job completed","job_id":"...","job_type":"extract_entities", "item_id":"...","worker":"host-a1b2-w0","duration_ms":4210} ``` **Never logged:** passwords, session tokens, API credentials, complete uploaded documents, embedding vectors. Content appears as lengths and hashes; user agents as a truncated hash. Levels: `debug` (per-chunk detail, noisy), `info` (requests, job outcomes — the default), `warn` (degraded: model unavailable, lease lost, parse warnings), `error` (failures needing attention). --- ## Common problems ### Items stay "Waiting" and the queue does not move The workers are not running or cannot reach a dependency. ```bash grep "background workers started" relata.log # should appear at startup curl -s localhost:8080/health/ready | jq .checks curl -s localhost:11434/api/tags | jq '.models[].name' ``` If `RELATA_WORKERS_ENABLED=false`, nothing will ever process. If the embedding model is missing, `ollama pull` it — jobs retry and will drain once it is back. ### "model ... is not installed" ```bash ollama pull nomic-embed-text ollama pull llama3.1:8b ``` On a CPU-only machine, use `llama3.2:3b` for analysis and set `RELATA_ANALYSIS_MODEL` accordingly. Expect entity and relationship quality to be noticeably lower with a small model. ### Jobs fail with "invalid model output" The model is returning JSON that fails schema validation. These are **not retried**, by design. ```bash curl -s -b cj.txt 'localhost:8080/api/v1/jobs?status=failed' | jq '.jobs[].last_error' ``` Usually a model too small to follow a schema. Use a larger one, or set `RELATA_ANALYSIS_ENABLED=false` to run the deterministic pipeline only — ingestion, search and relationships still work. ### Search returns nothing for content that is clearly stored 1. Is the item `ready`? Check the queue page. 2. Is meaning-based search available? Search responses carry a note when it is not. 3. Do the vector and chunk counts match on the settings page? If not, re-embed. 4. Did the embedding model change? Vectors written by a different model are excluded from search by design, because comparing across models is meaningless. ### "qdrant collection stores N-dimensional vectors but the model produces M" The embedding model changed. Either restore the previous model, or: ```bash # set RELATA_EMBEDDING_DIMENSIONS to M, then curl -X DELETE http://127.0.0.1:6333/collections/relata_chunks # restart Relata, then Settings → Re-embed workspace ``` ### Answer generation times out A large model on CPU can exceed the default 120-second analysis timeout when asked to read several passages. The API returns **504** with code `dependency_timeout` and names the fix. Measured on a CPU-only machine with `llama3.1:8b`: | Sources in the prompt | Time | |---|---| | 3 | ~33 s | | 6 | exceeds 120 s | The shipped defaults already account for this — six passages, a 300-second analysis timeout and a 360-second write timeout. If you still hit it: ```bash RELATA_ANSWER_MAX_CHUNKS=3 # fewer passages per answer (cheapest fix) RELATA_ANALYSIS_TIMEOUT=600s # allow the model longer RELATA_HTTP_WRITE_TIMEOUT=660s # must exceed the analysis timeout RELATA_ANALYSIS_MODEL=llama3.2:3b # a smaller model ``` `RELATA_HTTP_WRITE_TIMEOUT` must exceed `RELATA_ANALYSIS_TIMEOUT`, or the connection closes while the model is still working and the user sees nothing at all. Configuration validation refuses to start on that combination rather than letting it fail silently at request time. ### Everything is slow Almost always the model host. Check `relata_ai_request_duration_seconds`. A 8B model on CPU takes seconds per call, and entity plus claim extraction makes several calls per chunk. Options: a smaller analysis model; lower `RELATA_REL_MAX_MODEL_CALLS`; raise `RELATA_AI_CONCURRENCY` **only** if the model host has capacity; or disable analysis entirely. ### A worker died mid-job Nothing to do. The lease expires, `ReleaseExpired` returns the job to the queue, and another worker picks it up. You will see: ``` WARN returned jobs from expired leases to the queue count=2 ``` Jobs out of attempts are marked failed and can be retried from the queue page. --- ## Routine maintenance **Daily** — check `/health/ready`; check failed jobs on the queue page. **Weekly** — review relationships awaiting review (the dashboard shows the count); confirming and rejecting improves search, because confirmed links carry more weight in expansion and rejected ones stop pulling results in. **Monthly** — `make security` (includes `govulncheck`); check the settings page for a vector/chunk mismatch; verify a backup restores. **On any model change** — reprocess affected items, or re-embed the workspace. Both are idempotent. --- ## Capacity Rough figures on a modern laptop, CPU-only, with `llama3.2:3b`: | Operation | Time | |---|---| | Ingest and parse a small note | < 100 ms | | Chunk and embed one chunk | 0.3–1 s | | Model entity extraction per chunk | 2–10 s | | Model claim extraction per chunk | 2–10 s | | Relationship discovery per item | 5–30 s, depending on model calls | | Search (no answer generation) | 20–200 ms | | Answer generation | 5–30 s | A GPU changes the model figures by an order of magnitude. Deterministic-only mode (`RELATA_ANALYSIS_ENABLED=false`) processes an item in well under a second. Storage: roughly 3 KB per chunk in Qdrant at 768 dimensions, plus the original and extracted text in MongoDB. --- ## Deployment notes **Keep the loopback binding** unless you have a specific reason not to. If you must expose it: 1. Put it behind a reverse proxy terminating TLS. 2. Set `RELATA_BASE_URL` to the https URL — this enables `Secure` cookies and sets the trusted CSRF origin. 3. Set `RELATA_HTTP_ADDR=0.0.0.0:8080` **and** `RELATA_ALLOW_EXTERNAL_BIND=true`; Relata refuses to start otherwise, so exposure is always deliberate. 4. Set `RELATA_TRUST_PROXY_HEADERS=true` **only** if the proxy strips client-supplied forwarding headers. Otherwise anyone can defeat login rate limiting with a forged header. **Least-privilege MongoDB:** ```javascript use relata db.createUser({ user: "relata_app", pwd: passwordPrompt(), roles: [{ role: "readWrite", db: "relata" }] }) ``` Relata creates its own indexes at startup, which `readWrite` permits. It never needs `dbAdmin` or `clusterAdmin`. **Container:** the image runs as UID 65532 on a read-only root filesystem with all capabilities dropped. The upload volume is the only writable path and must be writable by that UID. --- ## Upgrading 1. Back up (see [BACKUP_AND_RESTORE.md](BACKUP_AND_RESTORE.md)). 2. Read the release notes for changes to the parser, scoring or embedding defaults. 3. Deploy. Index creation is idempotent and runs at startup. 4. If the parser version changed, reprocess items to benefit — old items keep working with their existing extraction. 5. If the scoring version changed, existing relationships keep the version that produced them. Reprocess to recompute; human review decisions survive.