Code sync, security fixes
This commit is contained in:
Andy Dixon
@@ -0,0 +1,159 @@
|
||||
# Deployment — relata.dixon.cx
|
||||
|
||||
Relata runs on `server.dixon.cx` as a Docker Compose stack managed by systemd, behind the
|
||||
existing nginx.
|
||||
|
||||
`relata.dixon.cx` is **grey-clouded** in Cloudflare — DNS only, no proxy — so requests reach
|
||||
this server directly. That was a deliberate change: the zone's managed challenge blocks
|
||||
non-browser clients, which made API tokens unusable, and Cloudflare's 100-second proxy timeout
|
||||
would have cut off answer generation. The cost is that the origin address is public.
|
||||
|
||||
```
|
||||
client ──► nginx :443 ──► 127.0.0.1:8477 ──► app container
|
||||
├── mongodb (compose network only)
|
||||
├── qdrant (compose network only)
|
||||
└── ollama (host, via bridge proxy)
|
||||
```
|
||||
|
||||
## Files this deployment owns
|
||||
|
||||
| Path | What |
|
||||
|---|---|
|
||||
| `deploy/docker-compose.prod.yml` | The stack. Committed; contains no secrets. |
|
||||
| `/etc/relata/relata.env` | Site configuration **and secrets**. Mode 600, root. Not in the repository. |
|
||||
| `/etc/systemd/system/relata.service` | Brings the stack up at boot. |
|
||||
| `/etc/systemd/system/relata-ollama-proxy.{socket,service}` | Bridges the host's Ollama onto the Docker network. |
|
||||
| `/etc/systemd/system/ollama.service.d/keep-alive.conf` | Keeps models resident for 30 minutes. |
|
||||
| `/etc/nginx/sites-available/relata.dixon.cx.conf` | TLS termination and proxying. |
|
||||
| `/etc/letsencrypt/live/relata.dixon.cx/` | Certificate, renewed by the existing certbot timer. |
|
||||
|
||||
## Everyday operations
|
||||
|
||||
```bash
|
||||
sudo systemctl status relata # is it up
|
||||
sudo systemctl restart relata # restart the stack
|
||||
sudo systemctl stop relata # stop it
|
||||
docker compose -f deploy/docker-compose.prod.yml logs -f app # follow the app log
|
||||
curl -s http://127.0.0.1:8477/health/ready | jq . # dependency status
|
||||
```
|
||||
|
||||
**Note:** `/etc/relata/relata.env` is root-readable only, so `docker compose` must be run as
|
||||
root (or through systemd). Running it as your own user fails with a permission error on the
|
||||
env file — that is the file permissions working, not a fault.
|
||||
|
||||
## Deploying a new version
|
||||
|
||||
```bash
|
||||
cd /home/andy/src/relata
|
||||
make check # do not deploy something that does not pass
|
||||
|
||||
cd deploy
|
||||
RELATA_VERSION=1.0.2 sudo -E docker compose -f docker-compose.prod.yml build app
|
||||
sudo docker tag relata:1.0.2 relata:latest # systemd deploys the `latest` tag
|
||||
sudo systemctl restart relata
|
||||
```
|
||||
|
||||
Tagging both a version and `latest` means `latest` is always a tag you can name, so a rollback
|
||||
is `docker tag relata:1.0.1 relata:latest && sudo systemctl restart relata`.
|
||||
|
||||
## Why the pieces are as they are
|
||||
|
||||
**MongoDB and Qdrant publish no ports.** They are reachable only over the compose network.
|
||||
The development compose file publishes them for tooling; production has no reason to, and this
|
||||
host has no firewall, so anything published is a decision to publish.
|
||||
|
||||
**Ollama is the host's, not a container.** The host already runs one with the models pulled; a
|
||||
second copy would duplicate several gigabytes of weights and, on a CPU-only host, would not be
|
||||
any faster — the bottleneck is the CPU, not the process boundary. The host binds it to
|
||||
`127.0.0.1`, which containers cannot reach, so `relata-ollama-proxy.socket` listens on the
|
||||
Docker bridge address (`172.17.0.1:11434`) and forwards to it.
|
||||
|
||||
The obvious alternative — `OLLAMA_HOST=0.0.0.0` — would have been one line, and on a host with
|
||||
no firewall it publishes an unauthenticated model API to the internet. The socket proxy exposes
|
||||
it to containers on the bridge and nothing else. That is still slightly wider than Relata alone:
|
||||
any container on this host can reach Ollama through it.
|
||||
|
||||
**Ollama runs as a systemd service, and did not used to.** It was previously an `ollama serve`
|
||||
process started by hand inside a tmux session, owned by `andy` and orphaned to init. It worked,
|
||||
but it would not have survived a reboot — and Relata is enabled at boot and depends on it, so a
|
||||
restart would have brought the site up with no embedding and no analysis.
|
||||
|
||||
It is now `ollama.service`, enabled, with the 6.8GB model store moved from `/home/andy/.ollama`
|
||||
to `/usr/share/ollama/.ollama` where the service's own user can read it. `relata.service`
|
||||
declares `After=ollama.service`.
|
||||
|
||||
**Models stay resident for 30 minutes** (`/etc/systemd/system/ollama.service.d/keep-alive.conf`).
|
||||
Measured on this host: an answer takes **47s cold and 17s warm**, and almost all of the
|
||||
difference is loading 5.2GB of `llama3.1:8b`. The default of 5 minutes expired between one
|
||||
question and the next, so nearly every answer paid the load. The two models total ~5.8GB
|
||||
against 31GB of RAM.
|
||||
|
||||
**The Cloudflare real-IP snippet is deliberately NOT included.** With the hostname grey-clouded,
|
||||
`$remote_addr` is already the true client. That snippet trusts `CF-Connecting-IP` from any
|
||||
Cloudflare address — with no Cloudflare proxy actually in front, anyone willing to route a
|
||||
request through a Cloudflare Worker could then set their own address and give themselves a
|
||||
fresh login-rate-limit bucket.
|
||||
|
||||
**If this hostname is ever put back behind the orange cloud, re-add the include.** Without it
|
||||
every request would appear to come from a Cloudflare edge node and rate limiting would bucket
|
||||
the whole internet together.
|
||||
|
||||
**nginx adds no security headers.** Relata sets its own CSP, X-Frame-Options,
|
||||
X-Content-Type-Options and Referrer-Policy on every response. `add_header` in nginx appends
|
||||
rather than replaces, and two conflicting X-Frame-Options headers are worse than one.
|
||||
|
||||
**Two locations have long timeouts.** `GET /search` and `POST /api/v1/search/answer` run a
|
||||
language model inside the request. Everything else gets 120s.
|
||||
|
||||
## Known constraints
|
||||
|
||||
### Answer generation is slow, because this host has no GPU
|
||||
|
||||
8 cores, 31GB RAM, CPU-only inference. Measured with `llama3.1:8b`:
|
||||
|
||||
| | Time |
|
||||
|---|---|
|
||||
| Answer, model warm | ~17s |
|
||||
| Answer, model cold | ~47s |
|
||||
| Full ingestion pipeline, one short note | ~120s |
|
||||
|
||||
Ingestion is asynchronous — jobs run in the background and nobody waits — so 120s is a
|
||||
non-issue. Answer generation is synchronous and is what a person actually feels.
|
||||
|
||||
A second Ollama container would not change any of this: same CPU, same weights, plus several
|
||||
gigabytes of duplicated model store. What would:
|
||||
|
||||
1. A GPU. That is the whole of the gap.
|
||||
2. `RELATA_ANALYSIS_MODEL=llama3.2:3b` — several times faster, noticeably worse, and the
|
||||
safeguards still hold (an uncited answer is downgraded, a schema-echo answer is rejected).
|
||||
3. `RELATA_SEARCH_ANSWER_MAX_CHUNKS=4` to send the model less material.
|
||||
|
||||
### Cloudflare (resolved, but worth knowing)
|
||||
|
||||
Both of the following applied while `relata.dixon.cx` was proxied, and are the reason it is now
|
||||
grey-clouded. They return the moment the orange cloud goes back on:
|
||||
|
||||
- **Non-browser clients are challenged.** Requests without a browser get `403` with
|
||||
`cf-mitigated: challenge` at the edge. This is zone-wide on `dixon.cx` — `grafana.dixon.cx`
|
||||
and `intel.dixon.cx` behave the same way; `blackwall.cx` has an exception. API tokens are
|
||||
unusable through it.
|
||||
- **A 100-second proxy timeout** on non-enterprise plans, which cannot be raised from the
|
||||
origin, returns 524 for a slow answer even though nginx and Relata would both have waited.
|
||||
|
||||
If you want the proxy back for its other benefits, both are fixable from the Cloudflare
|
||||
dashboard: a **WAF → Custom rules → Skip** matching
|
||||
`(http.host eq "relata.dixon.cx" and starts_with(http.request.uri.path, "/api/v1/"))` skipping
|
||||
*Managed Challenge* and *Bot Fight Mode*, and either keeping answers under 100s (see above) or
|
||||
accepting that they will be cut off. Re-add `snippets/cloudflare-realip.conf` to the site at the
|
||||
same time.
|
||||
|
||||
### Everything else
|
||||
|
||||
- **First sign-in provisions a private workspace.** Each Blackwall identity gets its own, and
|
||||
relationship discovery never crosses between them. See
|
||||
[ADR 8](../docs/decisions/0008-federated-identity-private-workspaces.md).
|
||||
- **Local password login is off.** Set `RELATA_LOCAL_LOGIN_ENABLED=true` in
|
||||
`/etc/relata/relata.env` and restart to open the break-glass form if Blackwall is ever
|
||||
unreachable. `relata createadmin` works regardless.
|
||||
- **Backups.** `scripts/backup.sh` covers MongoDB and Qdrant. The compose volumes are
|
||||
`relata-prod_mongo_data`, `relata-prod_qdrant_data` and `relata-prod_uploads`.
|
||||
Reference in New Issue
Block a user