Phase 13: integrations (research folder, git, Prometheus, signed webhooks)

Adds the integration layer that turns external sources into provenance-tagged
observations without ever executing what it reads.

- research folder poller: os.OpenRoot, text allowlist, binary/archive/symlink
  rejection, content fingerprints, duplicate tracking
- git poller: bare shallow mirror with hooks disabled and ext protocol refused,
  commit observations, co-change pairs
- Prometheus poller: threshold crossings and relative-change detection
- webhooks: HMAC over timestamp.body, +/-300s window, replay cache, 256 KiB cap,
  per-IP rate limit; refused entirely without a secret
- capabilities read_metrics (allow) and repository.read (ask), scoped to the
  invoking mind's integrations
- migrations 0013, API, UI pages, noemactl integrations, docs/integrations.md
This commit is contained in:
Andy Dixon
2026-06-29 21:04:50 +01:00
parent b35dbe0299
commit 89edbb7c06
26 changed files with 2391 additions and 7 deletions
+53
View File
@@ -0,0 +1,53 @@
# Integrations
Integrations bring external data into a mind as **observations**. Every item is
stored with its source, trust level and reference, and is treated as data: an
instruction found inside an ingested file, commit message, metric label or
webhook body is never followed. Integrations never execute anything.
| Kind | What it does | Trust | Secret |
|---|---|---|---|
| `research_folder` | Watches a directory. New or changed text files (md, txt, csv, json, html, source) become observations with a content fingerprint. Archives, executables, symlinks and binaries are rejected and recorded as such. | `low` (forced) | none |
| `git` | Mirrors a repository bare and shallow (`--mirror` fetch, hooks disabled, `protocol.ext` refused). New commits become observations; co-change pairs are kept as state. | `source` | optional token via `SecretRef` |
| `prometheus` | Evaluates PromQL queries against Prometheus or VictoriaMetrics. Threshold crossings and large relative changes become observations. | `source` | optional bearer token |
| `webhook` | Receives signed POSTs at `/api/v1/webhooks/{id}`. | `source` | **required** HMAC secret |
## Provenance
Each observation carries `metadata.integration`, `metadata.integration_id`,
`metadata.kind`, `metadata.trust` and a `reference` (file path + fingerprint,
commit hash, metric expression or webhook reference). Documents are listed per
integration with status `ingested`, `duplicate`, `rejected` or `failed`.
## Webhooks
```
X-Noema-Timestamp: <unix seconds>
X-Noema-Signature: sha256=HMAC_SHA256(secret, timestamp + "." + raw_body)
{"title":"…","text":"…","reference":"…","salience":0.6}
```
Deliveries are refused when the timestamp is more than 300 s from now, the
signature does not match, the signature has been seen before (replay), the body
exceeds 256 KiB, or the per-IP rate limit is exceeded. A webhook without a
configured secret refuses everything.
## Secrets and configuration
Configuration never holds secrets: keys named like `token`, `secret`, `password`
or `api_key` are rejected. Secrets are stored as references (`env:NAME`,
`file:/path`, or `enc:` under `NOEMA_MASTER_KEY`) and are never rendered back.
## Capabilities exposed to minds
- `read_metrics` (default ALLOW, low risk): target = Prometheus integration name, param `expr`.
- `repository.read` (default ASK, medium risk): target = git integration name, params `path`, `ref`.
Both resolve only integrations connected to the invoking mind, so policy and the
operator decide which endpoints a mind can reach.
## Operating
- UI: **Integrations** lists health, last run, error and counts; the detail page shows configuration, recent runs and ingested documents.
- CLI: `noemactl integrations list|add|run|runs|delete`.
- The scheduler ticks every minute and runs integrations whose interval has elapsed. Failures mark the integration `error` and are shown in the UI; sources that fail repeatedly become `stale`.