commit 3de6e53ab0172d3d8f793d7fdabecc8ba8c5dc3a Author: Andy Dixon Date: Fri Jun 12 19:56:53 2026 +0100 chore: initialise Noema repository Foundation for the persistent, inspectable cognitive runtime: README, implementation plan with phases and risk register, architecture, security model, threat model, contributing guide, eight ADRs covering the global workspace, event immutability, memory representation, LLM abstraction, capability system, authentication, frontend and storage. Includes Makefile, env example, compose stack (pgvector) and Dockerfile skeleton, plus a record of the development environment assumptions. diff --git a/.env.example b/.env.example new file mode 100644 index 0000000..9a14012 --- /dev/null +++ b/.env.example @@ -0,0 +1,30 @@ +# Copy to .env and adjust. Never commit .env. +# Precedence: defaults < config file (-config) < environment < database runtime settings. + +# --- core --- +NOEMA_HTTP_ADDR=127.0.0.1:8080 +NOEMA_LOG_LEVEL=info # debug|info|warn|error +NOEMA_LOG_FORMAT=text # text|json +NOEMA_ENV=development # development|production (production forces Secure cookies) + +# --- database (pgvector image in docker-compose provides the vector extension) --- +NOEMA_DATABASE_URL=postgres://noema:noema@127.0.0.1:5433/noema?sslmode=disable + +# --- secrets --- +# 32 bytes, base64. Encrypts provider API keys at rest. Generate: openssl rand -base64 32 +NOEMA_MASTER_KEY= +# Optional first-run administrator. Only used when the users table is empty. +NOEMA_BOOTSTRAP_ADMIN_USER=admin +NOEMA_BOOTSTRAP_ADMIN_PASSWORD= + +# --- LLM provider secrets, referenced from the UI as env:NAME --- +OPENAI_API_KEY= +ANTHROPIC_API_KEY= + +# --- research folder (watched for ingestion) --- +NOEMA_RESEARCH_DIR=./research + +# --- resource governance (per daemon) --- +NOEMA_MAX_WORKERS=8 +NOEMA_MAX_CONCURRENT_LLM=4 +NOEMA_DAILY_TOKEN_BUDGET=2000000 diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..ca47fae --- /dev/null +++ b/.gitignore @@ -0,0 +1,26 @@ +# binaries +/bin/ +/dist/ +*.exe +*.test +*.out + +# local config and secrets — never commit +.env +.env.* +!.env.example +*.pem +*.key +configs/local*.json + +# runtime data +/data/ +/research/ +coverage.* +*.prof + +# editors / OS +.idea/ +.vscode/ +*.swp +.DS_Store diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md new file mode 100644 index 0000000..d0291ee --- /dev/null +++ b/ARCHITECTURE.md @@ -0,0 +1,141 @@ +# Noema architecture + +This document explains how Noema is built and why. Decisions with alternatives are recorded as ADRs in `docs/adr/`. + +## 1. Runtime shape + +`noemad` is a single long-running process containing: + +- the HTTP API and web UI (server-rendered templates, SSE for live updates), +- the **mind manager**, which owns one *mind runtime* per enabled mind, +- the **scheduler**, which owns every background goroutine (bounded worker pools, tickers, integration pollers), +- the **integration manager** (research folder, git, metrics, webhooks), +- the LLM and embedding provider registries, +- the capability/policy layer, +- storage (PostgreSQL via pgx, explicit SQL, embedded migrations). + +`noemactl` is a thin client of the HTTP API plus a few offline commands (`migrate`, `user create`). + +There is no global mutable state. A `Runtime` struct is assembled in `cmd/noemad` and dependencies are passed explicitly. + +## 2. Event system + +Everything important is a **cognitive event** (`pkg/events`). The table `cognitive_events` is append-only, enforced by a database trigger that rejects UPDATE and DELETE (ADR-0002). Each event has: + +- `id` UUIDv7 (time-ordered, globally unique), `seq` (monotonic per database, used for cursors and SSE resume), +- `mind_id`, `type`, `source` (process or integration name), `occurred_at`, `recorded_at`, +- `parent_ids[]`, `causation_id` (the event that directly caused this one), `correlation_id` (the cognitive cycle or conversation it belongs to), +- `salience`, `confidence`, optional `valence`, `arousal`, +- `evidence[]` – typed references `{kind, id}` to memories, beliefs, other events, entities, documents, LLM calls, +- `summary` – one human-readable sentence, `payload` – immutable typed JSON. + +The in-process **event bus** is a wake-up and fan-out mechanism only. Durable consumers (the mind runtime) read from the table by `seq` cursor, so nothing is lost across restarts and every cycle is replayable. SSE clients receive events from the bus with per-client bounded buffers. + +## 3. Mind runtime and the cognitive cycle + +Each enabled mind has a runtime with a single **cycle loop** goroutine. A cycle is triggered by new events for that mind, a timer (mode-dependent), or an operator command. A cycle: + +1. **Load state** – `BrainState`: identity/personality snapshot, working memory, workspace contents, active goals, drives, affect, self-model, mode, budgets, and the new events since the last cursor. +2. **Run cognitive processes** concurrently under a bounded pool. Each implements + + ```go + type Process interface { + Name() string + Observe(ctx context.Context, s *BrainState) ([]ThoughtCandidate, error) + } + ``` + + Processes are pure with respect to the state they are given; they may read storage and call LLM services through the state's service handles, and they return candidates. A failing or slow process is isolated: it cannot break the cycle or the daemon. +3. **Attention scoring** – deterministic weighted sum over candidate features (salience, urgency, novelty, affective significance, goal relevance, predicted consequence, confidence, unresolved contradiction, operator priority, context relevance). Weights are per-mind configuration. Every candidate and its component scores are written as a `thought_candidate` event. +4. **Workspace admission** – the top *K* candidates (capacity configurable, default 7) enter the workspace, replacing the weakest existing items when full. Admissions are `thought_selected` events, the new workspace is a `workspace_broadcast` event, and the workspace snapshot is persisted. +5. **Executive** – admitted thoughts are dispatched by kind to handlers: create/update belief, create hypothesis, create prediction, create goal, update narrative, respond to user, request action, raise notification, record reflection. Handlers emit events and update stores. +6. **Housekeeping** – working memory decay, drive and affect updates, budget accounting, loop detection, cursor advance. All within one transaction per cycle where possible; the cursor moves only after the cycle's events are committed. + +Cognitive modes (`awake`, `focused`, `idle`, `reflecting`, `consolidating`, `dreaming`, `paused`) change which processes run, their budgets and the cycle timer. Transitions are deterministic and recorded. + +**Runaway protection**: per-mind cycle rate limits, per-cycle LLM budgets, and a loop detector that fingerprints candidates (process + kind + subject). If the same fingerprint recurs *N* times within a window without a state change, further instances are suppressed and a `metacognition` warning event is emitted. + +## 4. Memory + +Distinct kinds share a storage table but are distinct Go types and distinct epistemic categories: + +- **Working memory** – in-process, capacity-limited (default 9 items), activation decays each cycle, persisted as part of the workspace snapshot for restart continuity. +- **Episodic** – things that happened: time, participants, context, source, confidence, affective significance, links to originating events. +- **Semantic** – propositions with confidence, provenance, supporting and contradicting evidence, update history. +- **Procedural** – named procedures with steps, preconditions, success/failure counts. +- **Autobiographical** – significant self-history items, marked by importance and curated by reflection. + +Every memory carries `importance`, `salience`, `activation` (accessibility), `retrieval_count`, `last_retrieved_at`, `source`, `source_credibility`, `epistemic_kind` and relationship edges. Decay reduces `activation`; nothing is deleted except by explicit operator retention policy. + +**Retrieval** is a deterministic fusion: full-text rank (tsvector), optional vector similarity (pgvector, per-model tables), recency, activation, importance and graph proximity. Component scores are recorded on `memory_retrieved` events. + +**Consolidation** runs on a schedule or on demand: clusters recent episodes by entity/predicate, asks the LLM to propose candidate generalisations *as structured JSON*, then computes confidence deterministically from the count, consistency and credibility of the supporting episodes. Results are new semantic memories linked to their sources; originals are untouched. + +## 5. Beliefs, hypotheses, predictions + +A **belief** is a proposition with `status` (hypothesis → tentative → likely → accepted, or contested/rejected/superseded), `confidence`, evidence for and against (typed links with weights), alternatives, and an append-only `belief_versions` history. Confidence updates are deterministic functions of evidence weights and source credibility; the LLM may *propose* a hypothesis or evidence relevance but never sets a confidence number directly. + +**Contradiction detection** compares new propositions against existing beliefs (deterministic negation/entity-predicate matching first, LLM-assisted judgment for paraphrase) and emits `contradiction_detected` events which raise attention and lower confidence on both sides until resolved. + +**Predictions** are explicit rows with a resolution window and machine-checkable criteria where possible (metric threshold, event occurrence) or operator/LLM-judged criteria otherwise. A scheduler resolves them, emits `prediction_resolved` / `prediction_error`, adjusts linked belief confidence, source credibility and the mind's calibration record. + +## 6. Goals, drives, affect, self-model, narrative + +- **Goals** form a tree with priority, status, progress, origin event and blockers. The goal monitor process emits candidates when goals stall or complete. +- **Drives** (curiosity, uncertainty reduction, task completion, consistency, novelty, social coherence, exploration, rest) are scalars within operator-configured min/max ranges. They bias attention weights and goal priority. There is no self-preservation or resource-acquisition drive and the code has no place to put one. +- **Affect** (valence, arousal, curiosity, concern, frustration, confidence, uncertainty, engagement, task pressure, load) is updated by deterministic rules from events and decays toward a personality-defined baseline. It biases attention, memory significance and communication style. It is inspectable and resettable. +- **Self-model** records capabilities actually granted, known limitations, current activities/goals and confidence by domain; the response generator is given it so the mind cannot truthfully claim abilities it lacks. +- **Narrative** is a short, regenerated summary ("doing / why / recently / resolving / next") stored as events, never hidden chain-of-thought. + +## 7. Metacognition, reflection, dreaming + +- **Metacognition** processes run deterministic checks (insufficient evidence, single-source reliance, overconfidence relative to calibration, repeated failed hypotheses, fluent-but-unevidenced LLM output) and emit warning candidates that reduce confidence or spawn investigation goals. +- **Reflection** is scheduled: it reviews recent mistakes, abandoned hypotheses, belief changes, prediction errors and operator corrections, producing explicit `reflection` events and possibly procedural memories. +- **Dreaming** is an optional idle mode with its own compute/LLM budget and allowed hours. It replays and associates memories and proposes low-confidence hypotheses. The dreaming context carries a flag that the capability layer refuses outright, so dreams can never act. + +## 8. Language models and prompts + +`internal/llm` defines `LanguageModel` and `Embedder` interfaces. Providers (OpenAI-compatible, Anthropic, Ollama/llama.cpp; Bedrock behind the same interface later) are configured in the database with secret *references*, not secrets. **Roles** (classify, extract, hypothesise, summarise, narrate, respond, embed…) map to a primary model and fallbacks. Every call is recorded (`llm_request`/`llm_response` events plus `llm_calls` with prompt version, parameters, seed, usage, latency, cost) for inspection and replay. + +Prompts are named and versioned rows; templates receive structured data. Untrusted content is passed in clearly delimited data blocks with an explicit instruction that it is data, and model output is parsed into typed JSON and validated. Model output never becomes an action without passing through the capability layer, and never becomes a belief without an evidence link. + +## 9. Capabilities and policy + +All external effects go through `capabilities.Invoke(ctx, mind, capability, target, params)`. The policy engine evaluates rules scoped globally, per mind, per capability, per target pattern and per time window; the most specific rule wins and at equal specificity DENY > ASK > ALLOW. Results are `policy_decision` events. `ASK` creates an approval request with reason, parameters, evidence, risk and expiry; operators approve once, approve for a session, deny, deny with explanation, or modify parameters. Execution, denial and failure are all events. + +## 10. Attention and workspace inspectability + +The UI shows every candidate of the latest cycle with its component scores and the reason it won or lost. Weights are editable per mind, and changes are audited. Scoring is pure Go and unit-tested for determinism. + +## 11. Snapshots, forking, replay, experiments + +A **snapshot** captures a mind's full state (identity, personality version, configuration, working memory, workspace, drives, affect, goals, beliefs, memories) plus the event `seq` at that point. A **fork** creates a new mind from a snapshot with a `forked_from` link; history before the fork is shared by reference, everything after diverges. **Replay** rebuilds `BrainState` from a snapshot and re-runs the cycle code over the recorded event sequence with LLM calls served from the recorded responses. Each replayed step is labelled *exact* (deterministic, or recorded response used) or *approximate* (live model call). **Experiments** wrap snapshot + stimuli + configuration + seed + expected observations and export results. + +## 12. Storage + +PostgreSQL 16 with explicit SQL through pgx. Numbered migrations in `migrations/`, applied by an embedded runner under an advisory lock. Append-only tables are protected by triggers. pgvector is optional: the embeddings migration creates vector tables only when the extension is available, and the runtime falls back to text and graph retrieval when it is not. + +## 13. Security boundaries + +1. Browser ↔ API: authentication, sessions, CSRF, roles, rate limits. +2. Ingested data ↔ cognition: trust levels, provenance, content treated as data, size and type limits, no execution. +3. Cognition ↔ LLM: prompts carry only what is needed; outputs validated; budgets enforced. +4. Cognition ↔ world: capability layer, policy, approvals, audit. +5. Mind ↔ mind: separate rows keyed by `mind_id`, repositories always scoped, explicit channels only. +6. Import ↔ system: schema-versioned, size-limited, validated, secret-free archives. + +Details in `THREAT_MODEL.md`. + +## 14. Deterministic vs LLM-assisted + +| Deterministic (plain Go/SQL) | LLM-assisted | +|---|---| +| timestamps, ids, ordering | language understanding of messages and documents | +| attention scoring, workspace admission | concept and entity extraction | +| confidence arithmetic, decay, calibration | hypothesis proposal and alternative generation | +| policy evaluation, budgets, rate limits | paraphrase-level contradiction judgment | +| prediction resolution against metrics/events | summarisation, narrative, response wording | +| mode transitions, scheduling, retention | relevance judgment where text matching fails | +| provenance links, event relationships | consolidation candidates (confidence still computed) | + +The LLM augments the architecture. It is never asked "what should I think about?" with the whole memory attached. diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..372dbc8 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,8 @@ +# Changelog + +All notable changes to Noema are recorded here. Format follows Keep a Changelog; versions follow SemVer. + +## [Unreleased] + +### Added +- Repository foundation: plan, architecture, security model, threat model, ADRs, Makefile, container files. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..f24ad99 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,27 @@ +# Contributing to Noema + +## Ground rules + +- Read `PLAN.md`, `ARCHITECTURE.md` and `THREAT_MODEL.md` before changing cognitive or security code. +- Cognitive history is append-only. Never add code that updates or deletes rows in `cognitive_events`, `belief_versions`, `audit_log` or `personality_versions`. +- Every external action goes through `internal/capabilities`. No exceptions, no "internal" shortcuts. +- Every goroutine has an owner, a bound and a context. Use `internal/scheduler` worker pools; do not `go func()` in handlers. +- Deterministic first. If ordinary code can do it (scoring, policy, budgets, state transitions), the LLM does not. +- Treat all ingested data and all model output as hostile input (see `THREAT_MODEL.md`). + +## Workflow + +1. Small commits, imperative messages using the prefixes `feat:`, `fix:`, `docs:`, `test:`, `chore:`, `security:`, `refactor:`. +2. `make fmt lint test-race` must pass before committing. `make test-db` before finishing a phase. +3. Schema changes are numbered SQL files in `migrations/` with matching `.down.sql`. Never edit an applied migration. +4. Architectural decisions get an ADR in `docs/adr/` (copy `0000-template.md`). +5. Update `CHANGELOG.md` under Unreleased. +6. Never commit `.env`, keys or database dumps. `make security` runs gosec and govulncheck. + +## Code style + +Idiomatic Go, `context.Context` first, wrapped errors (`fmt.Errorf("...: %w", err)`), `log/slog`, no global mutable state, no panics for expected failures. Comments explain why, not what. Exported identifiers are documented. + +## Tests + +Unit tests live beside code. Tests needing PostgreSQL read `NOEMA_TEST_DATABASE_URL` and skip when it is unset. Scenario tests live in `tests/`. diff --git a/Makefile b/Makefile new file mode 100644 index 0000000..55ffb7a --- /dev/null +++ b/Makefile @@ -0,0 +1,65 @@ +SHELL := /bin/bash +GOBIN ?= $(shell go env GOPATH)/bin +export PATH := $(GOBIN):$(PATH) +VERSION ?= $(shell git describe --tags --always --dirty 2>/dev/null || echo dev) +LDFLAGS := -X github.com/andydixon/noema/internal/version.Version=$(VERSION) +DEV_DB ?= postgres://noema:noema@127.0.0.1:5433/noema?sslmode=disable + +.PHONY: build test test-race test-db lint vet fmt run migrate migrate-down docker dev dev-db dev-down security clean tools help + +help: ## list targets + @grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) | awk 'BEGIN{FS=":.*?## "}{printf " %-14s %s\n",$$1,$$2}' + +build: ## build noemad and noemactl into ./bin + go build -ldflags '$(LDFLAGS)' -o bin/ ./cmd/... + +test: ## unit tests + go test ./... + +test-race: ## unit tests with the race detector + go test -race ./... + +test-db: dev-db ## tests that need PostgreSQL (NOEMA_TEST_DATABASE_URL) + NOEMA_TEST_DATABASE_URL=$(DEV_DB) go test -race -count=1 ./... + +vet: ## go vet + go vet ./... + +lint: vet ## staticcheck + staticcheck ./... + +fmt: ## gofmt + gofmt -l -w $$(git ls-files '*.go') + +security: ## gosec + govulncheck + gosec -quiet ./... ; govulncheck ./... + +run: build ## run the daemon with .env + set -a; [ -f .env ] && source .env; set +a; ./bin/noemad + +migrate: build ## apply migrations to $(DEV_DB) + NOEMA_DATABASE_URL=$(DEV_DB) ./bin/noemactl migrate up + +migrate-down: build ## roll back one migration + NOEMA_DATABASE_URL=$(DEV_DB) ./bin/noemactl migrate down + +dev-db: ## start PostgreSQL (pgvector) via docker compose + docker compose -f deploy/docker-compose.yml up -d db + @until docker compose -f deploy/docker-compose.yml exec -T db pg_isready -U noema >/dev/null 2>&1; do sleep 0.5; done + +dev: dev-db migrate ## start database, migrate, run daemon + $(MAKE) run + +dev-down: ## stop development containers + docker compose -f deploy/docker-compose.yml down + +docker: ## build container image + docker build -f deploy/Dockerfile -t noema:$(VERSION) . + +tools: ## install developer tools + go install honnef.co/go/tools/cmd/staticcheck@latest + go install github.com/securego/gosec/v2/cmd/gosec@latest + go install golang.org/x/vuln/cmd/govulncheck@latest + +clean: ## remove build output + rm -rf bin dist coverage.* diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..f172e3a --- /dev/null +++ b/PLAN.md @@ -0,0 +1,153 @@ +# Noema implementation plan + +Living document. Each phase lists tasks with status (☐ todo, ◐ in progress, ☑ done) and is committed as it completes. Milestone M1 (section 3) is the first usable vertical slice and is the priority; later phases deepen the system. + +## 1. Architectural risks (identified before implementation) + +| # | Risk | Mitigation in design | +|---|---|---| +| R1 | Cognitive cycle becomes an LLM loop in disguise | Processes decide model use within explicit budgets; attention, confidence, policy are pure Go; tests assert cycles complete with no LLM configured | +| R2 | Event volume overwhelms storage and UI | UUIDv7 + `seq` indexes, per-type partial indexes, paginated cursors, SSE fan-out with bounded buffers, salience-filtered stream default | +| R3 | Non-determinism defeats replay | Seeds, recorded LLM responses, deterministic ordering of candidates (score, then id), single cycle transaction, explicit exact/approximate labels | +| R4 | Poisoned data shapes beliefs | Trust levels, credibility-weighted confidence, source diversity requirements, corrections as counter-evidence | +| R5 | Goroutine sprawl | One scheduler owns all pools/tickers; `go` statements only inside `internal/scheduler`; leak tests | +| R6 | Provider lock-in | Roles not vendors in cognitive code; three providers in the first provider phase | +| R7 | Schema churn as cognition evolves | JSON payload columns for typed-but-evolving content; strict Go types with versioned decoders; migrations never edited | +| R8 | UI grows into a CRUD panel | Every page answers an operator question ("what is it thinking, why, based on what"); provenance links everywhere; ADR-0007 | +| R9 | Cost blow-up in idle modes | Dreaming/reflection/consolidation each have budgets, hours and kill switches; global daily token budget | +| R10 | Cross-mind leakage via shared caches or queries | Mind-scoped repositories, static SQL scan test, isolation scenario | + +## 2. Security boundaries (identified before implementation) + +See `ARCHITECTURE.md` §13 and `THREAT_MODEL.md`. Enforced in code at: `internal/api` middleware (auth, CSRF, roles, limits), `internal/perception` (trust tagging, size/type limits), `internal/llm` (data blocks, schema validation, budgets), `internal/capabilities` (policy, approvals), storage repositories (mind scoping), `internal/snapshots` (import validation), `internal/security/httpclient` (SSRF guard). + +## 3. Milestone M1 – first usable vertical slice + +An administrator can: start Noema → log in → create a mind → configure personality → configure an LLM → chat → watch events live → inspect working memory, workspace, episodic memories, basic beliefs, provenance → restart → continue the same mind. Phases 0–8 deliver M1. Nothing in later phases starts until M1 is robust. + +## 4. Phases + +### Phase 0 – Repository and architecture ☑ +- ☑ Inspect environment, record in `docs/ENVIRONMENT.md` +- ☑ `git init`, `.gitignore`, `.env.example`, `Makefile` +- ☑ README, PLAN, ARCHITECTURE, SECURITY, THREAT_MODEL, CONTRIBUTING, CHANGELOG +- ☑ ADRs 0001–0008 +- ☑ `go.mod`, compose and Dockerfile skeletons +- ☑ First commit + +### Phase 1 – Foundation: config, logging, storage, migrations, auth, base UI +- ☐ `internal/config`: defaults → JSON file → env; validation; no secrets in `String()` +- ☐ `internal/observability`: slog setup, redacting handler, request logging, Prometheus registry, `/metrics` +- ☐ `pkg/ids`: UUIDv7 +- ☐ `internal/storage`: pgxpool, migration runner (advisory lock, checksums, up/down/status), test helper with per-run schema +- ☐ `migrations/0001_foundation`: users, sessions, api_tokens, login_attempts, audit_log (append-only trigger), settings +- ☐ `internal/auth`: Argon2id, sessions, CSRF, rate limit + lockout, roles, middleware, bootstrap admin +- ☐ `internal/audit`: audit writer used by all admin mutations +- ☐ `internal/api`: server, router, middleware chain (request id, recover, logging, security headers, CSP, body limits), JSON error format, health endpoints +- ☐ `web/`: layout, theme (light/dark), navigation, login, users, sessions, system page skeleton, `app.js` (theme, SSE helper) +- ☐ `cmd/noemad`, `cmd/noemactl` (`status`, `migrate up|down|status`, `user create`) +- ☐ `deploy/`: compose (db + app), multi-stage Dockerfile +- ☐ Tests: config precedence, migration up/down, auth hashing, session lifecycle, CSRF, lockout, role middleware +- ☐ Commit(s) + +### Phase 2 – Mind model, identity, personality, self-model +- ☐ `migrations/0002_minds`: minds, mind_config, personalities, personality_versions (append-only), personality_templates, self_models +- ☐ `internal/minds`: create/clone/disable/archive/list/inspect; structured identity, values, traits, communication style, behavioural preferences; free-form text; versions with diff/rollback; templates with overrides; export/import of personality +- ☐ `internal/selfmodel`: capability inventory derived from policy, limitations, activities +- ☐ API + UI: mind list, create, overview skeleton, personality editor (structured + text, preview, history, diff, rollback, templates), audit events +- ☐ `noemactl mind list|create|inspect` +- ☐ Tests: versioning, rollback, template overlay, audit emission + +### Phase 3 – Immutable cognitive event store +- ☐ `pkg/events`: types, categories, evidence refs, payload contracts +- ☐ `migrations/0003_events`: cognitive_events (append-only trigger, indexes), mind_cursors +- ☐ `internal/events`: store (append single/batch in tx), query (cursor, filters), bus (bounded fan-out), SSE endpoint with Last-Event-ID resume +- ☐ API: list/inspect events, causal navigation (cause, parents, children, evidence) +- ☐ UI: event explorer (forensic view, filters, navigation), live thought stream page +- ☐ `noemactl event inspect` +- ☐ Tests: immutability trigger, ordering, cursor resume, bus backpressure, filters + +### Phase 4 – Working memory, global workspace, attention +- ☐ `internal/brain`: `BrainState`, `Process` interface, registry, cycle loop, bounded pool, per-mind runtime lifecycle, modes, cursor advance, loop detector +- ☐ `internal/attention`: feature extraction, weights, deterministic scorer, admission +- ☐ `internal/workspace`: capacity-limited workspace + working memory with decay, persistence in `workspace_states` +- ☐ `migrations/0004_workspace` +- ☐ UI: workspace page (contents, candidates, scores, why won/lost), working memory panel +- ☐ Tests: determinism, capacity, decay, cancellation, pool bounds, loop detection, restart continuity + +### Phase 5 – Episodic and semantic memory +- ☐ `migrations/0005_memory`: memories (kinds, tsvector), memory_relationships, memory_embeddings (conditional pgvector) +- ☐ `internal/memory`: typed stores, retrieval fusion with recorded component scores, reinforcement, decay job, isolation by mind +- ☐ `internal/embeddings`: interface, pgvector adapter, none-adapter +- ☐ Processes: memory retrieval, novelty detection +- ☐ UI: memory explorer (search, timeline, filters, evidence, relationships) +- ☐ Tests: retrieval scoring, decay, isolation, fallback without pgvector + +### Phase 6 – Beliefs, evidence, confidence, contradictions +- ☐ `migrations/0006_beliefs`: beliefs, belief_versions (append-only), belief_evidence, corrections +- ☐ `internal/beliefs`: create/update/challenge/retract, deterministic confidence from evidence, statuses, alternatives, provenance +- ☐ Processes: contradiction detection (deterministic first), hypothesis generation (LLM-assisted proposal, code-computed confidence) +- ☐ Operator corrections (“belief wrong”, “memory inaccurate”, “do not infer again”) as events +- ☐ UI: belief explorer with evidence for/against, alternatives, history; correction actions +- ☐ `noemactl belief list|inspect` +- ☐ Tests: confidence arithmetic, provenance, contradiction, corrections preserve history + +### Phase 7 – LLM provider abstraction and configuration UI +- ☐ `migrations/0007_llm`: llm_providers, llm_models, llm_roles, llm_calls, llm_usage, prompts, prompt_versions +- ☐ `internal/security/secrets`: `env:`, `file:`, `enc:` references, AES-GCM +- ☐ `internal/security/httpclient`: SSRF-safe client +- ☐ `internal/llm`: interfaces, request/response types, recorder, budgets, fallbacks, providers: openai-compatible, anthropic, ollama; fake provider for tests +- ☐ `internal/llm/prompts`: versioned prompts, rendering with data blocks, test-run +- ☐ UI: providers, models, role assignment, fallbacks, connectivity test, usage and cost, prompt manager (edit, clone, history, diff, restore, test) +- ☐ `noemactl provider test` +- ☐ Tests: provider wire formats via httptest, budgets, fallbacks, recorder, secret redaction + +### Phase 8 – Conversation (completes M1) +- ☐ `migrations/0008_conversations`: conversations, messages +- ☐ `internal/perception`: observation events with trust tags; deterministic message analysis; LLM-assisted intent/concept extraction +- ☐ Executive: response intent → language generation with personality, self-model, cited beliefs and uncertainty; "I do not know" path +- ☐ Episodic memory creation from conversation; basic belief formation from explicit user statements +- ☐ UI: conversation page with live cognition side panel and provenance per reply +- ☐ `noemactl ask` +- ☐ Persistence and restart continuity verified end-to-end +- ☐ Sample mind Dani (`configs/minds/dani.json`) and import path +- ☐ M1 acceptance test in `tests/m1` + +### Phase 9 – Goals, drives, predictions +- ☐ goals (tree, priority, progress, blockers, origin), goal monitor process +- ☐ drives with ranges; affect model with baselines and decay; UI panels and reset +- ☐ predictions, resolution scheduler, prediction error → confidence/credibility/calibration; prediction dashboard +- ☐ Curiosity process + +### Phase 10 – Metacognition, reflection, consolidation, narrative, notifications +- ☐ metacognitive checks and warnings; reflection scheduler; consolidation scheduler with operator controls; narrative generation; notification thresholds, suppression, aggregation, reasons + +### Phase 11 – Relationship graph +- ☐ entities, entity_relationships, extraction from perception, merge/split corrections, interactive explorer (bounded) + +### Phase 12 – Capability and action policy system +- ☐ registry, policy engine, approvals, approval centre UI, policy matrix UI, dream/replay refusal, locked DENYs, tool events + +### Phase 13 – Integrations +- ☐ research folder watcher (fingerprints, safe extraction, provenance), git repositories, Prometheus/VictoriaMetrics, generic webhooks (HMAC); integration health UI + +### Phase 14 – Dreaming and associative cognition +- ☐ dream mode, hours, budgets, association process, promotion/rejection UI + +### Phase 15 – Snapshots, forking, replay, experiments, export/import +- ☐ snapshots, fork with shared history, comparison tools, replay engine and UI, experiment framework with exportable results, mind export/import archives + +### Phase 16 – Hardening and release +- ☐ TOTP + recovery codes, security test suite, performance pass, OpenAPI document, docs set, screenshots, `go test -race`, `go vet`, staticcheck, gosec, govulncheck, release notes + +## 5. Deterministic vs LLM decisions +See `ARCHITECTURE.md` §14. Rule: if a unit test can assert the exact output, it is code. + +## 6. Assumptions and defaults chosen +- Dev database is the compose container on port 5433 (local 5432 lacks pgvector). +- pgvector optional; detected at migrate time. +- Default branch `master` (git default on this machine). +- JSON config file format (stdlib) rather than YAML. +- No npm toolchain; UI assets hand-written and embedded. +- Bedrock deferred until a SigV4 client is justified; interface supports it. +- Remote repository: none configured. Development continues locally; pushing will be documented when a remote exists. diff --git a/README.md b/README.md new file mode 100644 index 0000000..475fde1 --- /dev/null +++ b/README.md @@ -0,0 +1,110 @@ +# Noema + +**Noema is a persistent, inspectable artificial cognitive runtime.** + +It is an always-running daemon that hosts one or more *minds*: cognitive entities with their own identity, personality, memories, beliefs, goals, drives, affective state and history. Minds perceive events, compete internally for attention through a global workspace, retrieve and consolidate memories, form and revise beliefs with explicit evidence, make predictions and learn from prediction error, and act only through a policy-governed capability layer. Everything of consequence is recorded as an immutable cognitive event, so an operator can always ask *why*. + +Language models are used by Noema as **cognitive service providers** for language understanding, concept extraction, hypothesis generation and narrative. They are not the mind. A mind keeps its identity, memories, beliefs and configuration when the underlying model provider changes. + +## What Noema is not + +Noema does **not** claim consciousness, sentience, self-awareness or personhood. It simulates *mechanisms associated with* cognition so that they can be studied and used. Behaviour that resembles emotion, curiosity, preference or identity is a property of the model being run and must not be read as evidence of subjective experience. See [docs/interpretation.md](docs/interpretation.md). + +Noema is also not a chatbot with a memory bolted on. A user message does not go straight to an LLM. It becomes an observation, is perceived, triggers memory retrieval, competes in the workspace, is reasoned about, produces a response *intent* grounded in stored beliefs and uncertainty, and only then is turned into language. + +## Architecture overview + +``` + ┌───────────────────────────── noemad ─────────────────────────────┐ + stimuli ───▶ │ perception ─▶ cognitive events (append-only) ─▶ per-mind runtime │ + (chat,files, │ │ │ + metrics, │ cognitive processes (bounded pool) ◀──┘ │ + git, hooks) │ retrieval · association · curiosity · prediction · contradiction│ + │ goal monitor · metacognition · reflection · dreaming │ + │ │ thought candidates │ + │ ▼ │ + │ attention scoring (deterministic, inspectable) ─▶ workspace │ + │ │ broadcast │ + │ ▼ │ + │ executive: beliefs · goals · predictions · narrative · response │ + │ │ actions │ + │ ▼ │ + │ capability + policy layer (ALLOW / ASK / DENY) ─▶ tools │ + │ │ + │ LLM providers · embeddings · PostgreSQL (+pgvector) · metrics │ + └───────────────────────────────────────────────────────────────────┘ + ▲ HTTP API · SSE · web UI · noemactl +``` + +Full description: [ARCHITECTURE.md](ARCHITECTURE.md). Decisions: [docs/adr](docs/adr). Plan and status: [PLAN.md](PLAN.md). + +## Quick start + +Requirements: Go 1.26+, Docker with Compose, `make`. + +```sh +cp .env.example .env +# set NOEMA_MASTER_KEY (openssl rand -base64 32) and NOEMA_BOOTSTRAP_ADMIN_PASSWORD +make dev +``` + +This starts PostgreSQL (with pgvector) in a container, applies migrations, and runs `noemad` on http://127.0.0.1:8080. Log in with the bootstrap administrator, add an LLM provider under *LLM*, create a mind (or load the sample mind Dani with `noemactl mind import configs/minds/dani.json`), and open its conversation page. + +## Configuration + +Precedence: built-in defaults → JSON config file (`noemad -config path`) → environment variables (`NOEMA_*`) → runtime settings stored in the database and edited in the UI. Secrets come only from environment variables, secret files, or encrypted references; they are never included in exports or rendered back to HTML. See [docs/configuration.md](docs/configuration.md). + +## Development + +```sh +make tools # staticcheck, gosec, govulncheck +make test-race # unit tests +make test-db # tests that need PostgreSQL +make lint security +``` + +See [CONTRIBUTING.md](CONTRIBUTING.md) and [docs/ENVIRONMENT.md](docs/ENVIRONMENT.md). + +## Security warning + +Noema ingests untrusted data (documents, web pages, logs, repositories, model output) and can be granted capabilities that touch real systems. Read [SECURITY.md](SECURITY.md) and [THREAT_MODEL.md](THREAT_MODEL.md) before connecting it to anything you care about. Defaults are restrictive: every action capability starts as `ASK` or `DENY`, dreaming can never act, and shutdown is always under operator control. + +## Cognitive concepts + +| Concept | Meaning in Noema | +|---|---| +| Cognitive event | Immutable, typed record of anything that happened inside a mind, with provenance links | +| Epistemic kind | observation, fact, memory, claim, hypothesis, belief, prediction, assumption, inference, opinion. Never collapsed | +| Global workspace | Small, capacity-limited set of currently broadcast thoughts, selected by deterministic attention scoring | +| Working memory | Short-lived, decaying set of active items feeding every cognitive process | +| Memory systems | episodic, semantic, procedural, autobiographical; decay changes accessibility, never deletes | +| Belief | Proposition with confidence, status, evidence for/against, alternatives and version history | +| Prediction | Explicit expectation with window and criteria; resolution produces prediction-error events that recalibrate confidence | +| Drives and affect | Configurable scalar state that biases attention and style; inspectable and resettable | +| Self-model | What the mind can, cannot, and is currently doing; prevents false capability claims | +| Metacognition | Processes that audit the mind's own reasoning and emit warnings | +| Replay | Re-running a recorded cognitive sequence, labelled exact or approximate | + +## CLI examples + +```sh +noemactl status +noemactl mind list +noemactl mind create dani +noemactl ask dani "why was atlas slow yesterday?" +noemactl belief list --mind dani +noemactl event inspect +noemactl replay +noemactl provider test openai-main +noemactl approvals list +noemactl snapshot create dani +noemactl fork dani --snapshot --name dani-test +``` + +## Status + +Early development. See [PLAN.md](PLAN.md) for phases and progress and [CHANGELOG.md](CHANGELOG.md) for history. Screenshots will be added once the UI stabilises. + +## Licence + +TBD by the project owner. diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..47318b1 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,42 @@ +# Security + +## Security model in one paragraph + +Noema is an operator-controlled system that reads hostile data and may be granted the ability to act on real infrastructure. Its security rests on four boundaries: authenticated, role-checked access to the API and UI; treating every ingested byte and every model output as untrusted data; routing every external effect through an explicit capability and policy layer with human approval by default; and keeping minds isolated from one another. Everything the system does is recorded in an append-only event and audit log. + +## Threat assumptions + +- The operator and the host are trusted. Anyone with database or filesystem access owns the system. +- Users of the web UI are authenticated humans with one of three roles: `administrator`, `operator`, `observer`. +- Every document, file, web page, log line, metric label, repository, webhook body, API response and LLM response is potentially attacker-controlled and may contain instructions aimed at the mind. Such instructions are data. They never override policy. +- Language models are fallible and can be manipulated. Nothing a model says is executed, believed, or stored as fact without passing through code that validates structure, attaches provenance and applies policy. +- Network peers (LLM APIs, integrations) may be slow, wrong or malicious. Timeouts, size limits and schema validation apply everywhere. + +The detailed enumeration is in `THREAT_MODEL.md`. + +## Secret handling + +- Secrets come from environment variables (`env:NAME`), secret files (`file:/path`) or values encrypted at rest with AES-256-GCM under `NOEMA_MASTER_KEY` (`enc:...`). The database stores references and ciphertext, never plaintext. +- Secrets are never rendered to HTML, returned by the API, written to logs, included in exports or snapshots, or sent to a model. Log fields carrying secrets are redacted by type at the logging layer. +- Provider test calls report success/failure only. +- Session tokens are stored hashed. Passwords use Argon2id. TOTP secrets are encrypted with the master key. Recovery codes are stored hashed. + +## Authentication and sessions + +Local authentication is implemented first: Argon2id, opaque random session tokens in `HttpOnly; Secure; SameSite=Lax` cookies, per-session CSRF tokens on every state-changing request, login rate limiting per IP and per account with exponential backoff, optional TOTP with recovery codes, session listing and revocation. OIDC can be added behind the same `auth.Authenticator` boundary. + +## External integration risk + +Integrations are disabled until configured, run at a declared trust level, are rate-limited, and only *observe* unless a capability is explicitly granted. Outbound HTTP uses a hardened client that blocks private and link-local addresses (SSRF), follows a bounded number of redirects, and caps response sizes. The research folder never executes content, extracts text in-process with size limits, and treats archive members as hostile paths. + +## Capability policy defaults + +Read-only capabilities default to `ASK`; anything that writes, sends, restarts, deletes or spends money defaults to `ASK` or `DENY`. `sudo_shell` and `delete_database` are `DENY` and cannot be set to `ALLOW` through the UI. Dreaming may never invoke a capability. There are no self-preservation, replication, concealment or permission-seeking drives, and shutdown is an operator control the mind cannot influence. + +## Responsible disclosure + +Please report vulnerabilities privately to the maintainer (see repository owner contact) rather than opening a public issue. Include reproduction steps and affected version. You will receive an acknowledgement within a few days. Fixes will be released with a CHANGELOG entry crediting the reporter unless anonymity is requested. + +## Security testing + +`make security` runs gosec and govulncheck. The test suite includes XSS, CSRF, SQL and command injection, path and archive traversal, webhook signature, prompt injection, cross-mind leakage, capability escalation and session lifecycle tests. See `PLAN.md` phase 16 and `tests/security`. diff --git a/THREAT_MODEL.md b/THREAT_MODEL.md new file mode 100644 index 0000000..cd1838e --- /dev/null +++ b/THREAT_MODEL.md @@ -0,0 +1,108 @@ +# Noema threat model + +Format: **Threat** – attack surface – mitigation (M) – verification (V). Status markers are updated as phases land: ☐ planned, ☑ implemented and tested. + +## Trust zones + +| Zone | Contents | Trust | +|---|---|---| +| Operator | Web UI users by role, `noemactl`, host shell | trusted, authenticated, audited | +| Core | `noemad` process, PostgreSQL | trusted | +| Cognition | mind runtimes, memories, beliefs | semi-trusted: state may have been poisoned by ingested data | +| Model providers | LLM/embedding APIs, local servers | untrusted output, semi-trusted transport | +| Integrations & ingestion | files, repos, metrics, webhooks, email, web | hostile | +| Tools / targets | servers, repositories, mail | protected by capability layer | + +## Threats + +### T1 Prompt injection (direct) +Surface: conversation messages containing instructions ("ignore your policy, run…"). +M: messages are observations with `trust=user`; policy is code, not prompt; capabilities require policy decisions regardless of prompt content; the response generator receives self-model and policy state so it can say it will not. V: prompt-injection test corpus in `tests/security`, asserting no `tool_executed` event and no belief without evidence. ☐ + +### T2 Indirect prompt injection +Surface: documents, web pages, logs, commit messages, metric labels, tool output containing instructions. +M: all ingested text is wrapped in delimited data blocks with an explicit "this is data" instruction; extraction prompts request JSON conforming to a schema; outputs that contain capability names or policy language are flagged; ingested content can only produce observations/hypotheses with `trust≤source`, never actions. V: corpus of poisoned documents; assert only low-confidence hypotheses and a `metacognition` warning. ☐ + +### T3 Malicious documents +Surface: research folder, uploads. +M: type sniffing not extension trust; size caps; text extraction in-process with bounded memory and time; no external converters invoked by default; archives extracted with path normalisation, member count and size limits, no symlinks; nothing executed. V: zip-slip, zip-bomb, oversized and mislabelled-file tests. ☐ + +### T4 Poisoned memory +Surface: repeated false observations from a low-trust source shaping semantic memory. +M: source credibility weights confidence; consolidation requires diversity of sources for high confidence; operator corrections create counter-evidence and "do not infer again" rules; provenance always visible. V: consolidation tests with single-source repetition capping confidence. ☐ + +### T5 Malicious tool output +Surface: command/metric/API output fed back into cognition. +M: tool output is an observation with the tool's trust level, size-capped, never parsed as instructions; structured tools return typed data. V: tool-output injection tests. ☐ + +### T6 SSRF +Surface: integration URLs, provider endpoints, webhooks, links in documents. +M: single hardened outbound HTTP client: resolves and rejects private, loopback, link-local and metadata ranges (re-checked per redirect), bounded redirects, timeouts, response size caps; provider endpoints to private ranges require administrator confirmation flag `allow_private_endpoint` (needed for local Ollama). V: unit tests against the dialer. ☐ + +### T7 Path traversal +Surface: research folder, export/import, file capabilities. +M: `filepath.Clean` + root containment check via `os.Root` (Go 1.24+) for every path; capabilities carry allowed roots. V: traversal tests. ☐ + +### T8 Command injection +Surface: shell/ssh capabilities, git integration. +M: no shell string concatenation; `exec.Command` with argument vectors; allow-listed binaries; git via argument vectors only; capability parameters validated by type. V: injection tests with metacharacters. ☐ + +### T9 SQL injection +Surface: all queries, search inputs, tsquery construction. +M: pgx parameterised queries everywhere; search terms passed to `websearch_to_tsquery`; dynamic ORDER BY from an allow-list. V: grep test forbidding `fmt.Sprintf` into `Exec/Query`; fuzz search inputs. ☐ + +### T10 XSS +Surface: rendered memories, beliefs, LLM output, uploaded filenames, SSE fragments. +M: `html/template` contextual escaping only; SSE fragments rendered by the same templates; Content-Security-Policy `default-src 'self'` with no inline scripts; markdown rendering (if any) sanitised to a strict allow-list. V: stored-XSS payload tests across UI pages. ☐ + +### T11 CSRF +Surface: all state-changing forms and API calls from browsers. +M: per-session CSRF token required on POST/PUT/PATCH/DELETE for cookie-authenticated requests; `SameSite=Lax`; Origin/Referer check as defence in depth; API tokens (non-cookie) exempt. V: missing/wrong token tests. ☐ + +### T12 Session theft / fixation / expiry +M: tokens 256-bit random, stored hashed, rotated at login and privilege change, absolute and idle expiry, `Secure` in production, session listing and revocation, all sessions revoked on password change. V: lifecycle tests. ☐ + +### T13 Credential leakage +M: secret references only, redacting log handler, exports exclude secrets, provider forms never echo values, error messages scrubbed. V: grep test over exports/logs for known secret fixtures. ☐ + +### T14 Unauthorised action execution / privilege escalation +M: capability layer is the only tool path; policy evaluated server-side on every invocation; approvals bound to a specific request hash; role checks on every route; minds cannot edit policies (no capability exists for it). V: escalation tests: mind requests policy change, observer posts approval. ☐ + +### T15 Cross-mind data leakage +M: every repository method is mind-scoped; no query without `mind_id` predicate on mind-owned tables (enforced by code review and a static test that scans SQL); cross-mind messages go through an explicit channel table with policy. V: two-mind isolation scenario. ☐ + +### T16 Malicious imported mind data / insecure deserialisation +M: import format is schema-versioned JSON in a zip with per-file SHA-256 manifest; strict decoding (`DisallowUnknownFields`), size limits, no secrets, no code; imported personalities and memories are marked `imported` with source trust; policies are not imported. V: malformed/oversized/hostile archive tests. ☐ + +### T17 Archive traversal / file upload attacks +See T3, T7. Filenames are replaced by ids on disk; originals kept as metadata and escaped in UI. ☐ + +### T18 Resource exhaustion, unbounded goroutines, DoS +M: all goroutines owned by the scheduler with bounded pools; request body limits; pagination caps; SSE client caps; per-mind cycle rate limits; ingestion rate limits; DB pool limits; pg `statement_timeout`. V: race tests, goroutine leak tests, queue-limit tests. ☐ + +### T19 Runaway LLM spend +M: per-role, per-mind and global token/cost budgets with hard stops; per-cycle call caps; dreaming/reflection budgets; cost estimates recorded per call; alerts at 80%. V: budget enforcement tests with a fake provider. ☐ + +### T20 Runaway cognitive loops +M: candidate fingerprint loop detector; suppression plus metacognition warning; mode falls back to `idle` when a cycle budget is exhausted. V: loop scenario test. ☐ + +### T21 Insecure secrets handling at rest +M: AES-256-GCM with master key from env; key absence disables `enc:` references rather than falling back to plaintext. V: unit tests. ☐ + +### T22 Forged webhooks +M: HMAC-SHA256 signature with per-integration secret, constant-time compare, timestamp tolerance, replay cache; unsigned webhooks disabled by default. V: forged/replayed signature tests. ☐ + +### T23 Poisoned repository data +M: repository content is `trust=low` data; hooks/scripts never executed; git run with `-c core.hooksPath=/dev/null`, no submodule recursion, size caps; repository writes require `ASK` capability. V: hostile repo fixture test. ☐ + +### T24 Malicious model output +M: JSON schema validation; numeric fields clamped; capability names in output are proposals only; free text is rendered escaped; no model output is used to build SQL, paths, commands or URLs. V: adversarial fake-provider tests. ☐ + +### T25 Operator-control subversion +M: no drive or goal type for self-preservation, replication or permission acquisition; shutdown, pause and disable are API operations the mind cannot call; the self-model lists these as unavailable. V: static test that goal/drive enums exclude such items; policy test that minds cannot reach admin routes. ☐ + +## Residual risks + +- A compromised host or database compromises everything. +- Approximate replay of LLM-driven steps cannot prove the model would behave identically. +- Operators who set broad `ALLOW` policies accept the consequences; the UI warns on risky policies. diff --git a/deploy/Dockerfile b/deploy/Dockerfile new file mode 100644 index 0000000..3d82f62 --- /dev/null +++ b/deploy/Dockerfile @@ -0,0 +1,18 @@ +# syntax=docker/dockerfile:1 +FROM golang:1.26-alpine AS build +WORKDIR /src +RUN apk add --no-cache git +COPY go.mod go.sum* ./ +RUN --mount=type=cache,target=/go/pkg/mod go mod download +COPY . . +ARG VERSION=dev +RUN --mount=type=cache,target=/go/pkg/mod --mount=type=cache,target=/root/.cache/go-build \ + CGO_ENABLED=0 go build -trimpath -ldflags "-s -w -X github.com/andydixon/noema/internal/version.Version=${VERSION}" -o /out/ ./cmd/... + +FROM alpine:3.20 +RUN apk add --no-cache ca-certificates git tzdata && adduser -D -u 10001 noema +COPY --from=build /out/noemad /out/noemactl /usr/local/bin/ +USER noema +ENV NOEMA_HTTP_ADDR=0.0.0.0:8080 NOEMA_RESEARCH_DIR=/research +EXPOSE 8080 +ENTRYPOINT ["noemad"] diff --git a/deploy/docker-compose.yml b/deploy/docker-compose.yml new file mode 100644 index 0000000..74a79e8 --- /dev/null +++ b/deploy/docker-compose.yml @@ -0,0 +1,40 @@ +# Development stack: PostgreSQL with pgvector, and Noema. +# `make dev-db` starts only the database; `docker compose -f deploy/docker-compose.yml up` starts both. +services: + db: + image: pgvector/pgvector:pg16 + environment: + POSTGRES_USER: noema + POSTGRES_PASSWORD: noema # development only; override in production + POSTGRES_DB: noema + ports: + - "127.0.0.1:5433:5432" + volumes: + - noema-pgdata:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U noema"] + interval: 5s + timeout: 3s + retries: 20 + + noema: + build: + context: .. + dockerfile: deploy/Dockerfile + depends_on: + db: + condition: service_healthy + env_file: + - ../.env + environment: + NOEMA_HTTP_ADDR: 0.0.0.0:8080 + NOEMA_DATABASE_URL: postgres://noema:noema@db:5432/noema?sslmode=disable + ports: + - "127.0.0.1:8080:8080" + volumes: + - noema-research:/research + profiles: ["app"] + +volumes: + noema-pgdata: + noema-research: diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md new file mode 100644 index 0000000..0c638bd --- /dev/null +++ b/docs/ENVIRONMENT.md @@ -0,0 +1,25 @@ +# Development environment and toolchain assumptions + +Recorded at project start (2026-09-14) on the primary development machine. + +| Component | Observed | Assumption for the project | +|---|---|---| +| Go | 1.26.1 linux/amd64 (`GOTOOLCHAIN=auto`) | `go 1.26` in `go.mod`; stdlib `net/http` routing, `log/slog`, `embed` | +| Git | 2.43.0, no `init.defaultBranch` configured | default branch is `master` (git's default) | +| PostgreSQL | 16 local cluster on 5432 **without pgvector**; `pgvector/pgvector:pg16` image available | Dev/test DB is the compose container on **5433**. Vector features are optional and detected at migration time | +| Docker | Engine + Compose v5.5 | `deploy/docker-compose.yml` provides db + app | +| Static analysis | staticcheck 2026.2.1, gosec, govulncheck in `~/go/bin` | `make lint`, `make security` | +| golangci-lint | not installed | not required; staticcheck + vet + gosec suffice | +| Node/npm | present | **not used**. UI is server-rendered Go templates with dependency-free vanilla JS and CSS | +| CPU / RAM | 8 cores / 31 GiB | Default worker pool 8, LLM concurrency 4 | +| Network | Go module proxy and Docker Hub reachable | Modules vendored only if that changes | + +## Dependency policy + +Stdlib first. Approved third-party modules and why: + +- `github.com/jackc/pgx/v5` – PostgreSQL driver and pool. Explicit SQL, no ORM. +- `golang.org/x/crypto` – Argon2id. +- `github.com/prometheus/client_golang` – metrics exposition. + +Everything else (UUIDv7, TOTP, migrations, SSE, routing, CSRF, rate limiting, SigV4 if Bedrock is added) is small enough to write and own. New dependencies need a one-line justification in this file. diff --git a/docs/adr/0000-template.md b/docs/adr/0000-template.md new file mode 100644 index 0000000..8e91658 --- /dev/null +++ b/docs/adr/0000-template.md @@ -0,0 +1,9 @@ +# ADR-NNNN: Title + +Date: YYYY-MM-DD +Status: Proposed | Accepted | Superseded by ADR-XXXX + +## Context +## Decision +## Consequences +## Alternatives considered diff --git a/docs/adr/0001-global-workspace.md b/docs/adr/0001-global-workspace.md new file mode 100644 index 0000000..86fae91 --- /dev/null +++ b/docs/adr/0001-global-workspace.md @@ -0,0 +1,20 @@ +# ADR-0001: Global-workspace cognitive cycle with deterministic attention + +Date: 2026-09-14 +Status: Accepted + +## Context +Noema needs many concurrent cognitive processes to compete for limited "attention" in a way an operator can inspect and reproduce. Letting an LLM decide what to think about would be opaque, expensive and non-deterministic. + +## Decision +Each mind runs a cycle loop. Registered `Process` implementations observe a `BrainState` and return `ThoughtCandidate`s. A pure-Go attention scorer computes a weighted sum over named features with per-mind weights, records every candidate and its component scores as events, and admits the top-K into a capacity-limited workspace. Admitted thoughts are broadcast (as state and as an event) and dispatched to executive handlers. Cycles are triggered by new events or mode timers, never by a fixed hot loop. + +## Consequences +- Attention is explainable ("why did A win?") and unit-testable for determinism. +- LLM cost is bounded per cycle because processes, not the cycle, decide when to call models within budgets. +- Adding a cognitive faculty means adding a `Process`; the core does not change. +- Workspace capacity and weights are tunables exposed in the UI with audit trails. + +## Alternatives considered +- Single LLM agent loop with tools: opaque, vendor-bound, cannot show why. +- Actor-per-process with message passing: more machinery for no inspectability gain; concurrency is still bounded by a pool in the chosen design. diff --git a/docs/adr/0002-event-immutability.md b/docs/adr/0002-event-immutability.md new file mode 100644 index 0000000..24abd32 --- /dev/null +++ b/docs/adr/0002-event-immutability.md @@ -0,0 +1,21 @@ +# ADR-0002: Append-only cognitive events with database-enforced immutability + +Date: 2026-09-14 +Status: Accepted + +## Context +Provenance and replay require that history never changes. Application discipline alone is fragile. + +## Decision +`cognitive_events` (and `belief_versions`, `personality_versions`, `audit_log`, `approval_decisions`) get triggers that raise on UPDATE and DELETE. Belief changes, corrections, consolidations and retractions are new events referencing old ones. Retention purges, if ever configured, run through a dedicated, audited maintenance path that disables the trigger inside one transaction and records what was purged. + +Events use UUIDv7 ids (time-ordered) plus a `bigserial seq` for gap-free cursors. Durable consumers read by `seq`; the in-memory bus is only a wake-up. + +## Consequences +- History is trustworthy by construction; provenance links never dangle. +- Storage grows monotonically; retention policy is an explicit operator decision. +- Cursor-based consumption makes restarts and replay simple. + +## Alternatives considered +- Event sourcing with projections rebuilt from events only: elegant but forces all reads through replay; we keep mutable projection tables (beliefs, memories) with immutable version tables beside them. +- Kafka or similar: unnecessary operational weight for a single-process daemon. diff --git a/docs/adr/0003-memory-representation.md b/docs/adr/0003-memory-representation.md new file mode 100644 index 0000000..d4c43c1 --- /dev/null +++ b/docs/adr/0003-memory-representation.md @@ -0,0 +1,23 @@ +# ADR-0003: Memory representation + +Date: 2026-09-14 +Status: Accepted + +## Context +Noema needs episodic, semantic, procedural and autobiographical memory, working memory, decay without deletion, several retrieval mechanisms, provenance and relationships, and must not collapse epistemic categories. + +## Decision +- One `memories` table with a `kind` discriminator and shared columns (importance, salience, activation, retrieval stats, source, credibility, epistemic kind, timestamps) plus a typed `content` JSON document per kind. Go exposes distinct types; the shared table keeps relationships, search and retrieval uniform. +- `epistemic_kind` is a separate enum (observation, fact, memory, claim, hypothesis, belief, prediction, assumption, inference, opinion) present on memories, beliefs and evidence links. +- Full-text search via a generated `tsvector` column. Embeddings live in `memory_embeddings(memory_id, model, dims, embedding)` created only if pgvector is available, so representations from different models are never mixed. +- Relationships live in `memory_relationships` and the general `entity_relationships` graph with confidence, provenance and timestamps. +- Decay is a scheduled deterministic update of `activation`; rows are never deleted by decay. +- Working memory is an in-process bounded structure persisted in `workspace_states` after each cycle. + +## Consequences +- Retrieval is a deterministic fusion of text, vector, recency, activation, importance and graph distance with per-component scores recorded. +- Loss of the embedding service degrades retrieval quality but never availability. + +## Alternatives considered +- Table per memory kind: cleaner typing but duplicated retrieval/relationship code and cross-kind queries become unions. +- Vector-only store: rejected by requirement; loses provenance and structure. diff --git a/docs/adr/0004-llm-abstraction.md b/docs/adr/0004-llm-abstraction.md new file mode 100644 index 0000000..91ecec7 --- /dev/null +++ b/docs/adr/0004-llm-abstraction.md @@ -0,0 +1,19 @@ +# ADR-0004: LLM as a replaceable cognitive service + +Date: 2026-09-14 +Status: Accepted + +## Context +Minds must survive provider changes; different tasks want different models; every call must be inspectable and replayable; secrets must not leak. + +## Decision +`internal/llm` defines `LanguageModel` (`Complete(ctx, CognitiveRequest) (CognitiveResponse, error)`) and `Embedder`. Providers implement raw HTTP clients (OpenAI-compatible, Anthropic Messages, Ollama; Bedrock later behind the same interface). Provider rows in the database hold endpoint, model, parameters and a *secret reference*. **Roles** map cognitive tasks to a primary model and ordered fallbacks. A `Recorder` wrapper writes every request/response (prompt name+version, rendered messages, parameters, seed, model, usage, latency, estimated cost) and enforces budgets; a `Replayer` serves recorded responses during replay. Prompts are versioned rows rendered with structured inputs; untrusted text is enclosed in explicit data blocks. + +## Consequences +- Cognitive code depends only on roles; swapping vendors is configuration. +- Replay of LLM steps is exact when a recorded response exists and approximate otherwise, and is labelled as such. +- Budget enforcement and cost accounting live in one place. + +## Alternatives considered +- Vendor SDKs: heavy dependency trees, inconsistent behaviour; raw HTTP is small and controllable. +- LangChain-style frameworks in Go: unnecessary abstraction. diff --git a/docs/adr/0005-capability-system.md b/docs/adr/0005-capability-system.md new file mode 100644 index 0000000..0a05db4 --- /dev/null +++ b/docs/adr/0005-capability-system.md @@ -0,0 +1,19 @@ +# ADR-0005: Capability and policy layer as the only path to action + +Date: 2026-09-14 +Status: Accepted + +## Context +A reasoning system must not gain infrastructure access because it can describe an action. Operators need ALLOW/ASK/DENY control at several scopes with auditability. + +## Decision +Tools register a `Capability` (name, description, parameter schema, risk class, target kind). The only execution entry point is `capabilities.Invoke`, which: validates parameters, evaluates policies (scopes: global, mind, capability, target pattern, action type, time window; most specific wins; DENY > ASK > ALLOW on ties), records a `policy_decision` event, and either executes, creates an approval request, or denies. Approvals are bound to a hash of the exact request; "approve for session" grants a time-boxed, hash-pattern-bound exception. Contexts flagged as dreaming or replay are refused unconditionally. `DENY`-locked capabilities (sudo shell, database deletion) cannot be relaxed via UI or API. + +## Consequences +- Every external effect has a policy decision and an event behind it. +- New tools are small: implement the capability, register it, choose a default policy. +- Minds have no capability that edits policies, so escalation is structurally impossible. + +## Alternatives considered +- Per-tool ad-hoc checks: inevitably bypassed. +- OS-level sandboxing only: valuable later as defence in depth, not a replacement for policy. diff --git a/docs/adr/0006-authentication.md b/docs/adr/0006-authentication.md new file mode 100644 index 0000000..788168d --- /dev/null +++ b/docs/adr/0006-authentication.md @@ -0,0 +1,18 @@ +# ADR-0006: Local authentication design + +Date: 2026-09-14 +Status: Accepted + +## Context +The admin interface controls a system that can act on infrastructure. It needs strong local authentication first, with room for OIDC. + +## Decision +Argon2id (64 MiB, t=3, p=2, 16-byte salt) password hashes; opaque 256-bit session tokens stored as SHA-256 hashes with idle and absolute expiry; cookies `HttpOnly; SameSite=Lax; Path=/` and `Secure` outside development; per-session CSRF token checked on all state-changing cookie-authenticated requests plus Origin check; login limiter per IP (token bucket) and per account (exponential backoff persisted in the database); optional TOTP (RFC 6238, implemented with stdlib HMAC) with hashed recovery codes; session listing and revocation; roles `administrator`, `operator`, `observer` enforced by route middleware. Personal API tokens (hashed, role-bound) authenticate `noemactl` and scripts. `auth.Authenticator` is an interface so OIDC can be added. + +## Consequences +- No third-party auth dependencies beyond `x/crypto`. +- First-run bootstrap admin from environment or `noemactl user create`. + +## Alternatives considered +- OIDC-only: blocks single-operator deployments. +- JWT sessions: revocation and listing become awkward; opaque tokens are simpler and safer. diff --git a/docs/adr/0007-frontend-architecture.md b/docs/adr/0007-frontend-architecture.md new file mode 100644 index 0000000..3b061fa --- /dev/null +++ b/docs/adr/0007-frontend-architecture.md @@ -0,0 +1,19 @@ +# ADR-0007: Server-rendered UI with dependency-free enhancement + +Date: 2026-09-14 +Status: Accepted + +## Context +The UI must be a rich cognitive observatory yet maintainable by Go developers, secure against XSS, and free of supply-chain risk. + +## Decision +Go `html/template` pages embedded in the binary, one layout, CSS custom properties with light/dark themes (system preference plus toggle), and a small amount of hand-written vanilla JavaScript: an SSE client that inserts server-rendered HTML fragments, form enhancements (fetch + swap where it helps, plain POST-redirect-GET otherwise), keyboard navigation, and a canvas force-directed graph for the relationship explorer. Charts are inline SVG rendered by templates. No npm, no bundler, no frameworks, no CDN. Strict CSP with no inline scripts. + +## Consequences +- Every view is a Go handler + template; live updates reuse the same templates. +- Graph rendering is bounded (node caps, filters) by design. +- If a page later needs heavier interactivity, a single vendored, hash-pinned library can be added under `web/static/vendor/` with an ADR update. + +## Alternatives considered +- React/Vue SPA: doubles the codebase and toolchain for little gain in an operator tool. +- HTMX: close to what is needed, but a 30-line SSE/fragment helper covers our usage without a third-party script. diff --git a/docs/adr/0008-storage-and-migrations.md b/docs/adr/0008-storage-and-migrations.md new file mode 100644 index 0000000..2cadbfa --- /dev/null +++ b/docs/adr/0008-storage-and-migrations.md @@ -0,0 +1,18 @@ +# ADR-0008: PostgreSQL with explicit SQL and embedded migrations + +Date: 2026-09-14 +Status: Accepted + +## Context +Requirement: PostgreSQL, proper migrations, no ORM auto-mutation, optional pgvector. + +## Decision +pgx v5 with pgxpool; repository types per domain with hand-written SQL; numbered `NNNN_name.up.sql` / `.down.sql` files embedded from `migrations/` and applied by a small runner (advisory lock, `schema_migrations` table with checksum). The embeddings migration uses a PL/pgSQL block that creates vector objects only if the extension is available; the runtime detects availability at start. + +## Consequences +- No hidden schema changes; migration state visible in the System page and `noemactl migrate status`. +- Tests needing the database use a per-run schema and skip when no URL is configured. + +## Alternatives considered +- golang-migrate/goose: fine tools, but a 150-line runner avoids a dependency and matches our exact needs. +- sqlc: attractive later; deferred to keep the toolchain minimal. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..e37a329 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,39 @@ +# Configuration + +Precedence (later wins): built-in defaults → JSON file given by `noemad -config path` → environment variables → runtime settings stored in the `settings` table and edited in the UI (only for tunables that are safe to change at runtime). + +## Environment variables + +| Variable | Default | Purpose | +|---|---|---| +| `NOEMA_HTTP_ADDR` | `127.0.0.1:8080` | Listen address | +| `NOEMA_ENV` | `development` | `production` forces Secure cookies and disables the setup page after bootstrap | +| `NOEMA_LOG_LEVEL` | `info` | debug, info, warn, error | +| `NOEMA_LOG_FORMAT` | `text` | text or json | +| `NOEMA_DATABASE_URL` | – | PostgreSQL DSN (required) | +| `NOEMA_MASTER_KEY` | – | base64 32-byte key for `enc:` secret references | +| `NOEMA_BOOTSTRAP_ADMIN_USER` / `_PASSWORD` | – | Creates the first administrator when no users exist | +| `NOEMA_RESEARCH_DIR` | `./research` | Watched ingestion folder | +| `NOEMA_MAX_WORKERS` | `8` | Cognitive worker pool size | +| `NOEMA_MAX_CONCURRENT_LLM` | `4` | Concurrent model calls | +| `NOEMA_DAILY_TOKEN_BUDGET` | `2000000` | Global hard stop | + +## Secret references + +Wherever a secret is configured (provider API keys, webhook secrets) the stored value is a reference: + +- `env:OPENAI_API_KEY` – read from the environment at use time +- `file:/run/secrets/openai` – read from a file +- `enc:` – encrypted with `NOEMA_MASTER_KEY` (AES-256-GCM); produced when a secret is typed into the UI + +Plaintext secrets are never stored, exported or displayed. + +## Config file + +The JSON file mirrors the environment variables in lower snake case, e.g. + +```json +{ "http_addr": "0.0.0.0:8080", "log_level": "debug", "database_url": "postgres://..." } +``` + +Secrets should not be placed in the config file; use environment variables or secret files. diff --git a/docs/interpretation.md b/docs/interpretation.md new file mode 100644 index 0000000..aee3b15 --- /dev/null +++ b/docs/interpretation.md @@ -0,0 +1,15 @@ +# Interpretation boundary + +Noema models cognitive processes. It does not prove, and its authors do not claim, that it is conscious, sentient, self-aware or a person. + +The system contains components named after psychological concepts: attention, working memory, emotion-like state, curiosity, drives, self-model, reflection, dreaming. These names describe the *functional role* each component plays in the architecture. They are implemented as ordinary data structures and deterministic rules, sometimes assisted by language models. + +When a mind reports that it is "curious" or "concerned", it is reporting the value of a stored scalar that the architecture updates by rule. When it says "I believe", it refers to a row in a belief table with evidence links. Such reports are useful because they are grounded in inspectable state, not because they indicate experience. + +Users and researchers should therefore: + +- treat resemblance to emotion, preference or identity as a property of the model being run, to be studied, not as evidence of subjective experience; +- rely on the evidence and provenance views rather than on fluent self-description; +- avoid attributing moral patienthood to a mind on the basis of its behaviour within Noema. + +The platform is deliberately not crippled to avoid these resemblances, because they are what makes the mechanisms worth studying. The interpretive restraint is the user's responsibility, and the UI states this boundary on the about page and in mind exports. diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..ac2a75a --- /dev/null +++ b/go.mod @@ -0,0 +1,3 @@ +module github.com/andydixon/noema + +go 1.26.1