Deployment — relata.dixon.cx

Relata runs on server.dixon.cx as a Docker Compose stack managed by systemd, behind the existing nginx.

relata.dixon.cx is grey-clouded in Cloudflare — DNS only, no proxy — so requests reach this server directly. That was a deliberate change: the zone's managed challenge blocks non-browser clients, which made API tokens unusable, and Cloudflare's 100-second proxy timeout would have cut off answer generation. The cost is that the origin address is public.

client ──► nginx :443 ──► 127.0.0.1:8477 ──► app container
                                                    ├── mongodb   (compose network only)
                                                    ├── qdrant    (compose network only)
                                                    └── ollama    (host, via bridge proxy)

Files this deployment owns

Path	What
`deploy/docker-compose.prod.yml`	The stack. Committed; contains no secrets.
`/etc/relata/relata.env`	Site configuration and secrets. Mode 600, root. Not in the repository.
`/etc/systemd/system/relata.service`	Brings the stack up at boot.
`/etc/systemd/system/relata-ollama-proxy.{socket,service}`	Bridges the host's Ollama onto the Docker network.
`/etc/systemd/system/ollama.service.d/keep-alive.conf`	Keeps models resident for 30 minutes.
`/etc/nginx/sites-available/relata.dixon.cx.conf`	TLS termination and proxying.
`/etc/letsencrypt/live/relata.dixon.cx/`	Certificate, renewed by the existing certbot timer.

Everyday operations

sudo systemctl status relata            # is it up
sudo systemctl restart relata           # restart the stack
sudo systemctl stop relata              # stop it
docker compose -f deploy/docker-compose.prod.yml logs -f app    # follow the app log
curl -s http://127.0.0.1:8477/health/ready | jq .               # dependency status

Note: /etc/relata/relata.env is root-readable only, so docker compose must be run as root (or through systemd). Running it as your own user fails with a permission error on the env file — that is the file permissions working, not a fault.

Deploying a new version

cd /home/andy/src/relata
make check                                    # do not deploy something that does not pass

cd deploy
RELATA_VERSION=1.0.2 sudo -E docker compose -f docker-compose.prod.yml build app
sudo docker tag relata:1.0.2 relata:latest    # systemd deploys the `latest` tag
sudo systemctl restart relata

Tagging both a version and latest means latest is always a tag you can name, so a rollback is docker tag relata:1.0.1 relata:latest && sudo systemctl restart relata.

Why the pieces are as they are

MongoDB and Qdrant publish no ports. They are reachable only over the compose network. The development compose file publishes them for tooling; production has no reason to, and this host has no firewall, so anything published is a decision to publish.

Ollama is the host's, not a container. The host already runs one with the models pulled; a second copy would duplicate several gigabytes of weights and, on a CPU-only host, would not be any faster — the bottleneck is the CPU, not the process boundary. The host binds it to 127.0.0.1, which containers cannot reach, so relata-ollama-proxy.socket listens on the Docker bridge address (172.17.0.1:11434) and forwards to it.

The obvious alternative — OLLAMA_HOST=0.0.0.0 — would have been one line, and on a host with no firewall it publishes an unauthenticated model API to the internet. The socket proxy exposes it to containers on the bridge and nothing else. That is still slightly wider than Relata alone: any container on this host can reach Ollama through it.

Ollama runs as a systemd service, and did not used to. It was previously an ollama serve process started by hand inside a tmux session, owned by andy and orphaned to init. It worked, but it would not have survived a reboot — and Relata is enabled at boot and depends on it, so a restart would have brought the site up with no embedding and no analysis.

It is now ollama.service, enabled, with the 6.8GB model store moved from /home/andy/.ollama to /usr/share/ollama/.ollama where the service's own user can read it. relata.service declares After=ollama.service.

Models stay resident for 30 minutes (/etc/systemd/system/ollama.service.d/keep-alive.conf). Measured on this host: an answer takes 47s cold and 17s warm, and almost all of the difference is loading 5.2GB of llama3.1:8b. The default of 5 minutes expired between one question and the next, so nearly every answer paid the load. The two models total ~5.8GB against 31GB of RAM.

The Cloudflare real-IP snippet is deliberately NOT included. With the hostname grey-clouded, $remote_addr is already the true client. That snippet trusts CF-Connecting-IP from any Cloudflare address — with no Cloudflare proxy actually in front, anyone willing to route a request through a Cloudflare Worker could then set their own address and give themselves a fresh login-rate-limit bucket.

If this hostname is ever put back behind the orange cloud, re-add the include. Without it every request would appear to come from a Cloudflare edge node and rate limiting would bucket the whole internet together.

nginx adds no security headers. Relata sets its own CSP, X-Frame-Options, X-Content-Type-Options and Referrer-Policy on every response. add_header in nginx appends rather than replaces, and two conflicting X-Frame-Options headers are worse than one.

Two locations have long timeouts. GET /search and POST /api/v1/search/answer run a language model inside the request. Everything else gets 120s.

Known constraints

Answer generation is slow, because this host has no GPU

8 cores, 31GB RAM, CPU-only inference. Measured with llama3.1:8b:

	Time
Answer, model warm	~17s
Answer, model cold	~47s
Full ingestion pipeline, one short note	~120s

Ingestion is asynchronous — jobs run in the background and nobody waits — so 120s is a non-issue. Answer generation is synchronous and is what a person actually feels.

A second Ollama container would not change any of this: same CPU, same weights, plus several gigabytes of duplicated model store. What would:

A GPU. That is the whole of the gap.
RELATA_ANALYSIS_MODEL=llama3.2:3b — several times faster, noticeably worse, and the safeguards still hold (an uncited answer is downgraded, a schema-echo answer is rejected).
RELATA_SEARCH_ANSWER_MAX_CHUNKS=4 to send the model less material.

Cloudflare (resolved, but worth knowing)

Both of the following applied while relata.dixon.cx was proxied, and are the reason it is now grey-clouded. They return the moment the orange cloud goes back on:

Non-browser clients are challenged. Requests without a browser get 403 with cf-mitigated: challenge at the edge. This is zone-wide on dixon.cx — grafana.dixon.cx and intel.dixon.cx behave the same way; blackwall.cx has an exception. API tokens are unusable through it.
A 100-second proxy timeout on non-enterprise plans, which cannot be raised from the origin, returns 524 for a slow answer even though nginx and Relata would both have waited.

If you want the proxy back for its other benefits, both are fixable from the Cloudflare dashboard: a WAF → Custom rules → Skip matching (http.host eq "relata.dixon.cx" and starts_with(http.request.uri.path, "/api/v1/")) skipping Managed Challenge and Bot Fight Mode, and either keeping answers under 100s (see above) or accepting that they will be cut off. Re-add snippets/cloudflare-realip.conf to the site at the same time.

Everything else

First sign-in provisions a private workspace. Each Blackwall identity gets its own, and relationship discovery never crosses between them. See ADR 8.
Local password login is off. Set RELATA_LOCAL_LOGIN_ENABLED=true in /etc/relata/relata.env and restart to open the break-glass form if Blackwall is ever unreachable. relata createadmin works regardless.
Backups. scripts/backup.sh covers MongoDB and Qdrant. The compose volumes are relata-prod_mongo_data, relata-prod_qdrant_data and relata-prod_uploads.

8.3 KiB Raw Permalink Blame History