296 lines
11 KiB
Markdown
296 lines
11 KiB
Markdown
# Operations
|
||
|
||
Running Relata: what to watch, what the logs mean, and what to do when something is wrong.
|
||
|
||
---
|
||
|
||
## Health
|
||
|
||
| Endpoint | Purpose | Fails when |
|
||
|---|---|---|
|
||
| `GET /health/live` | Liveness | The process is gone |
|
||
| `GET /health/ready` | Readiness | MongoDB or Qdrant is unreachable |
|
||
|
||
`/health/live` deliberately checks nothing else: a liveness probe that fails during a brief
|
||
database blip causes an orchestrator to restart a process that was working perfectly well.
|
||
|
||
`/health/ready` is unauthenticated, so it names each dependency and whether it is reachable,
|
||
but never why. **The reason is in the server log**, at `WARN` with `dependency` and the
|
||
request ID:
|
||
|
||
```bash
|
||
grep "readiness check failed" relata.log | tail
|
||
```
|
||
|
||
Returning the driver's error to an unauthenticated caller would publish the database host,
|
||
port and — when the connection string carries them — its credentials.
|
||
|
||
`/health/ready` reports the AI providers but **does not require them**. Ingestion, keyword
|
||
search and browsing all work without a model, so taking the service out of rotation because
|
||
Ollama restarted would be wrong.
|
||
|
||
```json
|
||
{
|
||
"status": "ready",
|
||
"checks": {
|
||
"mongodb": {"status": "ok", "required": true},
|
||
"qdrant": {"status": "ok", "required": true},
|
||
"embedding_provider": {"status": "ok", "required": false},
|
||
"analysis_provider": {"status": "unavailable", "required": false,
|
||
"detail": "model \"llama3.1:8b\" is not installed — run: ollama pull llama3.1:8b"}
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Metrics
|
||
|
||
Prometheus exposition at `GET /metrics`. These describe **the service**, not the stored
|
||
knowledge: no item ID, entity name or query string ever becomes a label value.
|
||
|
||
| Metric | Type | Labels | Watch for |
|
||
|---|---|---|---|
|
||
| `relata_http_requests_total` | counter | route, method, status | A rise in `5xx` |
|
||
| `relata_http_request_duration_seconds` | histogram | route, method | Search latency drift |
|
||
| `relata_jobs_processed_total` | counter | type, outcome | `outcome="permanent"` climbing |
|
||
| `relata_job_duration_seconds` | histogram | type | Embedding or analysis slowing |
|
||
| `relata_jobs_queued` | gauge | status | A backlog that never drains |
|
||
| `relata_ai_requests_total` | counter | provider, operation, outcome | Provider errors |
|
||
| `relata_ai_request_duration_seconds` | histogram | provider, operation | Model host under strain |
|
||
| `relata_storage_operations_total` | counter | store, operation, outcome | Database errors |
|
||
| `relata_model_invalid_output_total` | counter | task, reason | A model producing unusable output |
|
||
| `relata_auth_attempts_total` | counter | outcome | `bad_password` or `rate_limited` spikes |
|
||
| `relata_ingest_items_total` | counter | source_type, outcome | Rejections |
|
||
| `relata_search_duration_seconds` | histogram | — | Retrieval latency |
|
||
|
||
### Alerts worth having
|
||
|
||
```yaml
|
||
- alert: RelataJobsFailing
|
||
expr: rate(relata_jobs_processed_total{outcome=~"permanent|invalid_model_output"}[15m]) > 0.1
|
||
annotations:
|
||
summary: Jobs are failing permanently — check /jobs for the errors
|
||
|
||
- alert: RelataQueueBacklog
|
||
expr: relata_jobs_queued{status="pending"} > 100
|
||
for: 30m
|
||
annotations:
|
||
summary: The queue is not draining — is the AI provider reachable?
|
||
|
||
- alert: RelataModelOutputRejected
|
||
expr: rate(relata_model_invalid_output_total[15m]) > 0.05
|
||
annotations:
|
||
summary: The model is producing output that fails validation — consider a larger model
|
||
|
||
- alert: RelataLoginAttacks
|
||
expr: rate(relata_auth_attempts_total{outcome="rate_limited"}[5m]) > 1
|
||
annotations:
|
||
summary: Sustained failed sign-ins
|
||
```
|
||
|
||
---
|
||
|
||
## Logs
|
||
|
||
Structured, via `log/slog`. Set `RELATA_LOG_JSON=true` in production.
|
||
|
||
Every line carries `request_id` or `job_id`, so a user's report ("I got reference
|
||
`3f2504e0-…`") leads straight to the failure.
|
||
|
||
```json
|
||
{"time":"...","level":"INFO","msg":"job completed","job_id":"...","job_type":"extract_entities",
|
||
"item_id":"...","worker":"host-a1b2-w0","duration_ms":4210}
|
||
```
|
||
|
||
**Never logged:** passwords, session tokens, API credentials, complete uploaded documents,
|
||
embedding vectors. Content appears as lengths and hashes; user agents as a truncated hash.
|
||
|
||
Levels: `debug` (per-chunk detail, noisy), `info` (requests, job outcomes — the default),
|
||
`warn` (degraded: model unavailable, lease lost, parse warnings), `error` (failures needing
|
||
attention).
|
||
|
||
---
|
||
|
||
## Common problems
|
||
|
||
### Items stay "Waiting" and the queue does not move
|
||
|
||
The workers are not running or cannot reach a dependency.
|
||
|
||
```bash
|
||
grep "background workers started" relata.log # should appear at startup
|
||
curl -s localhost:8080/health/ready | jq .checks
|
||
curl -s localhost:11434/api/tags | jq '.models[].name'
|
||
```
|
||
|
||
If `RELATA_WORKERS_ENABLED=false`, nothing will ever process. If the embedding model is
|
||
missing, `ollama pull` it — jobs retry and will drain once it is back.
|
||
|
||
### "model ... is not installed"
|
||
|
||
```bash
|
||
ollama pull nomic-embed-text
|
||
ollama pull llama3.1:8b
|
||
```
|
||
|
||
On a CPU-only machine, use `llama3.2:3b` for analysis and set `RELATA_ANALYSIS_MODEL`
|
||
accordingly. Expect entity and relationship quality to be noticeably lower with a small model.
|
||
|
||
### Jobs fail with "invalid model output"
|
||
|
||
The model is returning JSON that fails schema validation. These are **not retried**, by design.
|
||
|
||
```bash
|
||
curl -s -b cj.txt 'localhost:8080/api/v1/jobs?status=failed' | jq '.jobs[].last_error'
|
||
```
|
||
|
||
Usually a model too small to follow a schema. Use a larger one, or set
|
||
`RELATA_ANALYSIS_ENABLED=false` to run the deterministic pipeline only — ingestion, search and
|
||
relationships still work.
|
||
|
||
### Search returns nothing for content that is clearly stored
|
||
|
||
1. Is the item `ready`? Check the queue page.
|
||
2. Is meaning-based search available? Search responses carry a note when it is not.
|
||
3. Do the vector and chunk counts match on the settings page? If not, re-embed.
|
||
4. Did the embedding model change? Vectors written by a different model are excluded from
|
||
search by design, because comparing across models is meaningless.
|
||
|
||
### "qdrant collection stores N-dimensional vectors but the model produces M"
|
||
|
||
The embedding model changed. Either restore the previous model, or:
|
||
|
||
```bash
|
||
# set RELATA_EMBEDDING_DIMENSIONS to M, then
|
||
curl -X DELETE http://127.0.0.1:6333/collections/relata_chunks
|
||
# restart Relata, then Settings → Re-embed workspace
|
||
```
|
||
|
||
### Answer generation times out
|
||
|
||
A large model on CPU can exceed the default 120-second analysis timeout when asked to read
|
||
several passages. The API returns **504** with code `dependency_timeout` and names the fix.
|
||
|
||
Measured on a CPU-only machine with `llama3.1:8b`:
|
||
|
||
| Sources in the prompt | Time |
|
||
|---|---|
|
||
| 3 | ~33 s |
|
||
| 6 | exceeds 120 s |
|
||
|
||
The shipped defaults already account for this — six passages, a 300-second analysis timeout
|
||
and a 360-second write timeout. If you still hit it:
|
||
|
||
```bash
|
||
RELATA_ANSWER_MAX_CHUNKS=3 # fewer passages per answer (cheapest fix)
|
||
RELATA_ANALYSIS_TIMEOUT=600s # allow the model longer
|
||
RELATA_HTTP_WRITE_TIMEOUT=660s # must exceed the analysis timeout
|
||
RELATA_ANALYSIS_MODEL=llama3.2:3b # a smaller model
|
||
```
|
||
|
||
`RELATA_HTTP_WRITE_TIMEOUT` must exceed `RELATA_ANALYSIS_TIMEOUT`, or the connection closes
|
||
while the model is still working and the user sees nothing at all. Configuration validation
|
||
refuses to start on that combination rather than letting it fail silently at request time.
|
||
|
||
### Everything is slow
|
||
|
||
Almost always the model host. Check `relata_ai_request_duration_seconds`. A 8B model on CPU
|
||
takes seconds per call, and entity plus claim extraction makes several calls per chunk.
|
||
|
||
Options: a smaller analysis model; lower `RELATA_REL_MAX_MODEL_CALLS`; raise
|
||
`RELATA_AI_CONCURRENCY` **only** if the model host has capacity; or disable analysis entirely.
|
||
|
||
### A worker died mid-job
|
||
|
||
Nothing to do. The lease expires, `ReleaseExpired` returns the job to the queue, and another
|
||
worker picks it up. You will see:
|
||
|
||
```
|
||
WARN returned jobs from expired leases to the queue count=2
|
||
```
|
||
|
||
Jobs out of attempts are marked failed and can be retried from the queue page.
|
||
|
||
---
|
||
|
||
## Routine maintenance
|
||
|
||
**Daily** — check `/health/ready`; check failed jobs on the queue page.
|
||
|
||
**Weekly** — review relationships awaiting review (the dashboard shows the count); confirming
|
||
and rejecting improves search, because confirmed links carry more weight in expansion and
|
||
rejected ones stop pulling results in.
|
||
|
||
**Monthly** — `make security` (includes `govulncheck`); check the settings page for a
|
||
vector/chunk mismatch; verify a backup restores.
|
||
|
||
**On any model change** — reprocess affected items, or re-embed the workspace. Both are
|
||
idempotent.
|
||
|
||
---
|
||
|
||
## Capacity
|
||
|
||
Rough figures on a modern laptop, CPU-only, with `llama3.2:3b`:
|
||
|
||
| Operation | Time |
|
||
|---|---|
|
||
| Ingest and parse a small note | < 100 ms |
|
||
| Chunk and embed one chunk | 0.3–1 s |
|
||
| Model entity extraction per chunk | 2–10 s |
|
||
| Model claim extraction per chunk | 2–10 s |
|
||
| Relationship discovery per item | 5–30 s, depending on model calls |
|
||
| Search (no answer generation) | 20–200 ms |
|
||
| Answer generation | 5–30 s |
|
||
|
||
A GPU changes the model figures by an order of magnitude. Deterministic-only mode
|
||
(`RELATA_ANALYSIS_ENABLED=false`) processes an item in well under a second.
|
||
|
||
Storage: roughly 3 KB per chunk in Qdrant at 768 dimensions, plus the original and extracted
|
||
text in MongoDB.
|
||
|
||
---
|
||
|
||
## Deployment notes
|
||
|
||
**Keep the loopback binding** unless you have a specific reason not to. If you must expose it:
|
||
|
||
1. Put it behind a reverse proxy terminating TLS.
|
||
2. Set `RELATA_BASE_URL` to the https URL — this enables `Secure` cookies and sets the trusted
|
||
CSRF origin.
|
||
3. Set `RELATA_HTTP_ADDR=0.0.0.0:8080` **and** `RELATA_ALLOW_EXTERNAL_BIND=true`; Relata
|
||
refuses to start otherwise, so exposure is always deliberate.
|
||
4. Set `RELATA_TRUST_PROXY_HEADERS=true` **only** if the proxy strips client-supplied
|
||
forwarding headers. Otherwise anyone can defeat login rate limiting with a forged header.
|
||
|
||
**Least-privilege MongoDB:**
|
||
|
||
```javascript
|
||
use relata
|
||
db.createUser({
|
||
user: "relata_app",
|
||
pwd: passwordPrompt(),
|
||
roles: [{ role: "readWrite", db: "relata" }]
|
||
})
|
||
```
|
||
|
||
Relata creates its own indexes at startup, which `readWrite` permits. It never needs
|
||
`dbAdmin` or `clusterAdmin`.
|
||
|
||
**Container:** the image runs as UID 65532 on a read-only root filesystem with all
|
||
capabilities dropped. The upload volume is the only writable path and must be writable by
|
||
that UID.
|
||
|
||
---
|
||
|
||
## Upgrading
|
||
|
||
1. Back up (see [BACKUP_AND_RESTORE.md](BACKUP_AND_RESTORE.md)).
|
||
2. Read the release notes for changes to the parser, scoring or embedding defaults.
|
||
3. Deploy. Index creation is idempotent and runs at startup.
|
||
4. If the parser version changed, reprocess items to benefit — old items keep working with
|
||
their existing extraction.
|
||
5. If the scoring version changed, existing relationships keep the version that produced them.
|
||
Reprocess to recompute; human review decisions survive.
|