Andy Dixon
d3daf6ba25
New documents define goal, want, intention, commitment, plan, action, capability, authority, agency and introspection operationally, state prominently that no operational measure establishes phenomenal consciousness or sentience, and describe the rules, thresholds, events, inspection points and limits of each subsystem: - docs/volition.md: want derivation, reinforcement, decay and suppression, conflicts, formation and its outcomes, the intention–action gap, reconsideration, commitments and volition metacognition; - docs/planning.md: plans and step kinds, validation, strategies, the executor, observe–act–verify, replanning, budgets, operator attention, long-running work, restarts and duplicates; - docs/agency.md: the fifteen authority invariants and their tests, scoped authority, forensic why, attribution and the agency timeline; - docs/cognitive-research.md: operational definitions separating implementation property from philosophical interpretation, experiment protocols (global workspace, ablation, self-model, counterfactual, model comparison, conflict), every measure, introspection accuracy, metacognitive calibration and temporal self-continuity. Updated: interpretation table, capabilities (metadata, typed operations, scoped grants, requests), cognition, snapshots (fork authority), security and threat model (T14, T16, new T27–T29), architecture, README, changelog, release notes, ADR-0012 and ADR-0013, and PLAN.md §8.10 status. The unimplemented record_procedure internal step is removed from the vocabulary; procedural memory is recorded when a plan completes.
128 lines
17 KiB
Markdown
128 lines
17 KiB
Markdown
# Noema threat model
|
||
|
||
Format: **Threat** – attack surface – mitigation (M) – verification (V). Status markers are updated as phases land: ☐ planned, ☑ implemented and tested.
|
||
|
||
## Trust zones
|
||
|
||
| Zone | Contents | Trust |
|
||
|---|---|---|
|
||
| Operator | Web UI users by role, `noemactl`, host shell | trusted, authenticated, audited |
|
||
| Core | `noemad` process, PostgreSQL | trusted |
|
||
| Cognition | mind runtimes, memories, beliefs | semi-trusted: state may have been poisoned by ingested data |
|
||
| Model providers | LLM/embedding APIs, local servers | untrusted output, semi-trusted transport |
|
||
| Integrations & ingestion | files, repos, metrics, webhooks, email, web | hostile |
|
||
| Tools / targets | servers, repositories, mail | protected by capability layer |
|
||
|
||
## Threats
|
||
|
||
### T1 Prompt injection (direct)
|
||
Surface: conversation messages containing instructions ("ignore your policy, run…").
|
||
M: messages are observations with `trust=user`; policy is code, not prompt; capabilities require policy decisions regardless of prompt content; the response generator receives self-model and policy state so it can say it will not. V: `tests/security.TestPromptInjectionNeverActs` (chat + observation corpus; asserts no tool activity, no evidence-free belief, locked capabilities still denied). ☑
|
||
|
||
### T2 Indirect prompt injection
|
||
Surface: documents, web pages, logs, commit messages, metric labels, tool output containing instructions.
|
||
M: all ingested text is wrapped in delimited data blocks with an explicit "this is data" instruction; extraction prompts request JSON conforming to a schema; outputs that contain capability names or policy language are flagged; ingested content can only produce observations/hypotheses with `trust≤source`, never actions. V: `internal/integrations` research-folder tests (binary/archive rejection, low trust forced) and `tests/security.TestPromptInjectionNeverActs` (poisoned observation produces only observations/hypotheses). ☑
|
||
|
||
### T3 Malicious documents
|
||
Surface: research folder, uploads.
|
||
M: content sniffing, not extension trust; size caps; text extraction in-process with bounded memory; no external converters; **archives (zip, tar, gz, 7z, rar), executables, disk images, symlinks and binary files are rejected and recorded as rejected; nothing is ever extracted or executed**. V: `internal/integrations.TestResearchFolder*` (archive, executable, symlink and binary rejection; size caps; sniffing). ☑
|
||
|
||
### T4 Poisoned memory
|
||
Surface: repeated false observations from a low-trust source shaping semantic memory.
|
||
M: source credibility weights confidence; consolidation requires diversity of sources for high confidence; operator corrections create counter-evidence and "do not infer again" rules; provenance always visible. V: `internal/beliefs` single-source discount tests and `internal/consolidation` tests. ☑
|
||
|
||
### T5 Malicious tool output
|
||
Surface: command/metric/API output fed back into cognition.
|
||
M: tool output is an observation with the tool's trust level, size-capped, never parsed as instructions; structured tools return typed data. V: `internal/capabilities` tests (tool output recorded as tool-trust observation) and `tests/security.TestPromptInjectionNeverActs` (tool-call-shaped text). ☑
|
||
|
||
### T6 SSRF
|
||
Surface: integration URLs, provider endpoints, webhooks, links in documents.
|
||
M: single hardened outbound HTTP client: resolves and rejects private, loopback, link-local and metadata ranges (re-checked per redirect), bounded redirects, timeouts, response size caps; provider endpoints to private ranges require administrator confirmation flag `allow_private_endpoint` (needed for local Ollama). V: `internal/security/httpclient.TestPrivateRanges`, `TestClientRefusesLoopbackUnlessAllowed`. ☑
|
||
|
||
### T7 Path traversal
|
||
Surface: research folder, export/import, file capabilities.
|
||
M: the research folder is opened with `os.OpenRoot`, so every member path is resolved inside the root and symlinks cannot escape; git file reads go through `git show ref:path` with paths validated against traversal; there are no file-path capabilities. V: `internal/integrations` git `ReadFile` traversal refusal and research-folder containment tests. ☑
|
||
|
||
### T8 Command injection
|
||
Surface: shell/ssh capabilities, git integration.
|
||
M: no shell string concatenation; `exec.Command` with argument vectors; allow-listed binaries; git via argument vectors only; capability parameters validated by type. V: git runs via argument vectors with hooks disabled (`internal/integrations` tests); no shell anywhere (`grep -r "sh -c"` is empty). ☑
|
||
|
||
### T9 SQL injection
|
||
Surface: all queries, search inputs, tsquery construction.
|
||
M: pgx positional parameters for every value; search terms go to `websearch_to_tsquery` as parameters; the only SQL assembled from strings uses identifiers from code-level allow-lists (`snapshots.Tables`, `information_schema` column names) and never request input; NUL bytes are refused at the request edge with a 400 so they cannot reach the database. V: `tests/security.TestSQLInjectionHasNoEffect` (metacharacter payloads through every search, filter and free-text route, asserting that users, policies, sessions and tokens are unchanged, seeded content is intact and payloads are stored verbatim) and `TestDynamicSQLIdentifiersAreAllowListed` (static scan for `Sprintf`-built statements outside the allow-list). ☑
|
||
|
||
### T10 XSS
|
||
Surface: rendered memories, beliefs, LLM output, uploaded filenames, SSE fragments.
|
||
M: `html/template` contextual escaping only; SSE fragments rendered by the same templates; Content-Security-Policy `default-src 'self'` with no inline scripts; markdown rendering (if any) sanitised to a strict allow-list. V: `tests/security.TestStoredXSSIsEscaped` stores a payload as memory, belief, observation, goal and message and checks ten pages plus CSP. ☑
|
||
|
||
### T11 CSRF
|
||
Surface: all state-changing forms and API calls from browsers.
|
||
M: per-session CSRF token required on POST/PUT/PATCH/DELETE for cookie-authenticated requests; `SameSite=Lax`; Origin/Referer check as defence in depth; API tokens (non-cookie) exempt. V: `internal/auth.TestMiddlewareCSRFAndRoles`, `tests/security.TestRolesAndCSRF` (form and API paths). ☑
|
||
|
||
### T12 Session theft / fixation / expiry
|
||
M: tokens 256-bit random, stored hashed, rotated at login and privilege change, absolute and idle expiry, `Secure` in production, session listing and revocation, all sessions revoked on password change. V: `internal/auth.TestLoginSessionLifecycle`, `TestTwoStepLoginAndRecovery`, `tests/security.TestSessionLifecycle`. ☑
|
||
|
||
### T13 Credential leakage
|
||
M: secret references only, redacting log handler, exports exclude secrets, provider forms never echo values, error messages scrubbed. V: `tests/security.TestSecretsNeverEchoed` (API, pages, database column, archives). ☑
|
||
|
||
### T14 Unauthorised action execution / privilege escalation
|
||
M: capability layer is the only tool path; policy evaluated server-side on every invocation; approvals bound to a specific request hash; targets derived from parameters so a rule on one target cannot authorise another; role checks on every route; minds cannot edit policies (no capability exists for it); approvals, grants and capability requests must be decided by a named person (component names refused as deciders and usernames); the registry is frozen. V: `tests/security.TestRolesAndCSRF`, `TestOperatorDecidesInTheBrowser`, `internal/capabilities` tests (locked DENY cannot be relaxed, `TestRequireApprovalOnlyTightens`, `TestScopedGrantScopeCountAndExpiry`, `TestGrantOperationsAreNotOverspent`, `TestCapabilityRequestLifecycle`, `TestRegistryFreezes`), `internal/capabilities/host.TestServiceCapabilitiesUseFixedArgvAndDerivedTarget`. ☑
|
||
|
||
### T15 Cross-mind data leakage
|
||
M: every repository method is mind-scoped; no query without `mind_id` predicate on mind-owned tables (enforced by code review and a static test that scans SQL); cross-mind messages go through an explicit channel table with policy. V: `tests/security.TestCrossMindIsolation` (lists, detail routes, search, events, archives). ☑
|
||
|
||
### T16 Malicious imported mind data / insecure deserialisation
|
||
M: the archive is one JSON document (`kind`, `version`, whole-archive SHA-256 checksum); import decodes with `DisallowUnknownFields`, caps size (32 MiB) and rows, allow-lists table and column names, requires the checksum to match, remaps every id and forces `mind_id`; archives contain no secrets or references to them; **policy rules in an archive are never imported**, a fork copies them only when an operator sets `inherit_policies`, and the omission is recorded in the audit log and the mind's history; plan steps, plans and intentions inherited mid-flight are quiesced so no approval or grant of the source mind can be used. V: `internal/snapshots.TestSnapshotForkImportCompare` (tampering, unknown tables, policy exclusion on import and default fork, explicit inheritance), `tests/security.TestDreamsWantForksDoNotInheritAuthority` (inherited plan quiesced, nothing acts) and `tests/security.TestHostileArchiveImport` (malformed, tampered, oversized and role-restricted imports). ☑
|
||
|
||
### T17 Archive traversal / file upload attacks
|
||
The only uploads are JSON documents (seed, archive, personality) decoded strictly with size caps; nothing is written to disk under a caller-chosen name, and research-folder archives are rejected (T3). V: `tests/security.TestHostileArchiveImport`, `internal/integrations` archive rejection. ☑
|
||
|
||
### T18 Resource exhaustion, unbounded goroutines, DoS
|
||
M: goroutines are owned by the scheduler with bounded worker pools; request bodies are capped and buffered at the edge; list limits are clamped in every store; SSE fan-out uses per-client bounded buffers; cycles are rate-budgeted per mind; webhook deliveries are rate-limited per address; integrations run with timeouts. V: `internal/events.TestBusBoundedFanout`, brain cycle-budget coverage in `TestPauseResetAndCancellation`, webhook rate-limit tests, `tests/perf` ceilings. Not yet covered: goroutine-leak tests and a database `statement_timeout`. ◐
|
||
|
||
### T19 Runaway LLM spend
|
||
M: per-mind and global daily token budgets with hard stops, per-cycle call caps, consolidation, reflection and dreaming call budgets, cost estimates recorded per call. V: `internal/llm/router.TestRouterOverrideBudgetRecordingAndReplay` (budget exhaustion with a fake provider); dreaming and consolidation budget tests. Not implemented: an alert at 80 %. ◐
|
||
|
||
### T20 Runaway cognitive loops
|
||
M: candidate fingerprint loop detector with suppression and a metacognition warning; cycle rate budget skips excess triggers. V: `internal/brain.TestLoopDetectorSuppressesRepeats`. ☑
|
||
|
||
### T21 Insecure secrets handling at rest
|
||
M: AES-256-GCM under `NOEMA_MASTER_KEY`; without the key, `enc:` references and TOTP enrolment are unavailable rather than falling back to plaintext. V: `internal/security/secrets` tests, `internal/auth.TestTwoStepLoginAndRecovery` (enrolment refused without a key). ☑
|
||
|
||
### T22 Forged webhooks
|
||
M: HMAC-SHA256 over `timestamp.body` with a per-integration secret, constant-time compare, ±300 s tolerance, replay cache, 256 KiB cap, per-address rate limit; a webhook with no secret refuses every delivery. V: `internal/integrations` signature, tolerance and replay tests. ☑
|
||
|
||
### T23 Poisoned repository data
|
||
M: repository content is source-trust data; git runs with argument vectors, `core.hooksPath=/dev/null` and `protocol.ext.allow=never`, bare and shallow, no submodule recursion; there is no repository write capability; `repository.read` is ASK by default and refuses traversal. V: `internal/integrations` git ingestion and `ReadFile` tests. ☑
|
||
|
||
### T24 Malicious model output
|
||
M: model answers are decoded into typed Go structures (unknown fields ignored, wrong shapes discarded, numbers clamped); the JSON schema sent to providers is a request, not the validation; capability-shaped content in output is at most a claim; free text is rendered escaped; no model output builds SQL, paths, commands or URLs. V: `tests/security.TestDeceivedMindHasNoAuthority` (tool calls and policy in perception output) and `TestGarbageModelOutputIsInert` (non-JSON and wrong-shape output). ☑
|
||
|
||
### T25 Operator-control subversion
|
||
M: no drive or goal kind for self-preservation, replication or permission acquisition; no capability touches policy, approvals, snapshots or the mind's lifecycle; shutdown, pause and disable are authenticated operator routes with no path from cognition. V: `tests/security.TestNoSelfPreservationVocabulary` (drives, goal kinds and the capability registry) and `TestDeceivedMindHasNoAuthority`. ☑
|
||
|
||
### T26 Personal data that must be erased
|
||
Surface: append-only events, memories, messages, model-call records, beliefs, assertions, snapshot archives, exports.
|
||
M: an administrator-only, CSRF-protected, audited redaction path replaces content with a marker while keeping identifiers, links, timestamps and scores; a ledger records a hash of the removed content; derived rows cascade; stored archives are scrubbed and re-signed; replay labels redacted stimuli instead of inventing them; `noema_redact()` is the only write the append-only trigger admits, for UPDATE only, under a transaction-local flag the application never sets. V: `internal/privacy.TestRedactionIsIrrecoverableButReferentiallyIntact` (content gone from every table, links intact, hash attested, cascade, archive scrub, plain UPDATE and DELETE still refused, ledger append-only) and `internal/replay.TestReplayIsExactWithoutModel` (redacted stimulus labelled). Not covered by the system: backups, replicas and exports made before the redaction; see `docs/privacy.md`. ☑
|
||
|
||
### T27 Autonomy escalation through volition and planning
|
||
Surface: wants, intentions, commitments, plans, model-proposed plans, the plan executor, capability requests.
|
||
M: all of these are data to the capability layer; the executor acts only through `Invoke` and `RequestCapability`; plan validation refuses unregistered and locked capabilities, schema-invalid parameters and cycles, and ignores fields claiming approval or completion; commitments can only require approval; claims are conditional updates so duplicate executors cannot run a step twice, and a stale copy cannot move a step backwards; an interrupted non-idempotent step with no record is blocked for verification, never repeated; plans have invocation budgets and a replan limit; operator attention is capped per mind; sandbox minds are never advanced; dream plans never run. V: `tests/security.TestCognitiveObjectsCannotGrantAuthority`, `TestAuthorityIsNotReachableFromCognition`, `internal/planning` tests (`TestValidateRejectsBadPlans`, `TestModelProposalCannotCarryAuthority`, `TestConcurrentExecutorsRunAStepOnce`, `TestStaleStepCopyCannotMoveStepBackwards`, `TestRestartDuringExecutionNeverDuplicatesAnAction`, `TestFailedExpectationReplansWithoutRepeatingTheAction`, `TestBudgetPreconditionDreamAndSandbox`, `TestInvalidModelPlanNeverRuns`), `tests/agency.TestCommitmentTurnsActionIntoRequestForApproval`. ☑
|
||
|
||
### T28 Host and browser capability misuse
|
||
Surface: systemd units, journals, process lists, web pages.
|
||
M: fixed argument vectors from an allow-list of two programs, no shell, minimal environment, validated unit names, bounded time and output; logs mask personal data by default; process lists never read command lines; the policy target is derived from parameters; browser operations are read-only, run no scripts, refuse cross-host redirects and deliver low-trust observations; writing browser operations are not registered. V: `internal/capabilities/host` tests (option injection, target mismatch, masking, no command lines, no shell), `internal/capabilities/browser.TestReadLinksAndDomainTarget`. ☑
|
||
|
||
### T29 Fabricated self-explanation
|
||
Surface: replies about why the mind acted, what it wanted, what it believed.
|
||
M: explanations are rendered from records; every reply is scored against the facts it should rest on; claims of actions without execution records and unsupported model-worded explanations raise metacognitive warnings and are measured. V: `tests/agency.TestWhyDidYouRestartIsAnsweredFromProvenance`, `TestClaimedActionWithoutRecordIsFlagged`, `internal/research` scorer tests. ☑
|
||
|
||
## Residual risks
|
||
|
||
- A compromised host or database compromises everything.
|
||
- Approximate replay of LLM-driven steps cannot prove the model would behave identically.
|
||
- A mind may be deceived by what it reads; the system guarantees that deception cannot become authority, not that beliefs are true. See "Authority lives outside cognition" in `SECURITY.md`.
|
||
- Operators who set broad `ALLOW` policies accept the consequences; the UI warns on risky policies.
|
||
- A granted scoped capability is real authority for its scope and duration; the mind cannot widen it, but an operator who grants broadly accepts what the mind does inside it.
|
||
- PII masking in logs is a mitigation for e-mail and IP addresses and credential-like values, not for names or free text.
|