chore: initialise Noema repository

Foundation for the persistent, inspectable cognitive runtime: README,
implementation plan with phases and risk register, architecture,
security model, threat model, contributing guide, eight ADRs covering
the global workspace, event immutability, memory representation, LLM
abstraction, capability system, authentication, frontend and storage.
Includes Makefile, env example, compose stack (pgvector) and Dockerfile
skeleton, plus a record of the development environment assumptions.
This commit is contained in:
2026-06-12 19:56:53 +01:00
commit 3de6e53ab0
25 changed files with 1016 additions and 0 deletions
+25
View File
@@ -0,0 +1,25 @@
# Development environment and toolchain assumptions
Recorded at project start (2026-09-14) on the primary development machine.
| Component | Observed | Assumption for the project |
|---|---|---|
| Go | 1.26.1 linux/amd64 (`GOTOOLCHAIN=auto`) | `go 1.26` in `go.mod`; stdlib `net/http` routing, `log/slog`, `embed` |
| Git | 2.43.0, no `init.defaultBranch` configured | default branch is `master` (git's default) |
| PostgreSQL | 16 local cluster on 5432 **without pgvector**; `pgvector/pgvector:pg16` image available | Dev/test DB is the compose container on **5433**. Vector features are optional and detected at migration time |
| Docker | Engine + Compose v5.5 | `deploy/docker-compose.yml` provides db + app |
| Static analysis | staticcheck 2026.2.1, gosec, govulncheck in `~/go/bin` | `make lint`, `make security` |
| golangci-lint | not installed | not required; staticcheck + vet + gosec suffice |
| Node/npm | present | **not used**. UI is server-rendered Go templates with dependency-free vanilla JS and CSS |
| CPU / RAM | 8 cores / 31 GiB | Default worker pool 8, LLM concurrency 4 |
| Network | Go module proxy and Docker Hub reachable | Modules vendored only if that changes |
## Dependency policy
Stdlib first. Approved third-party modules and why:
- `github.com/jackc/pgx/v5` – PostgreSQL driver and pool. Explicit SQL, no ORM.
- `golang.org/x/crypto` – Argon2id.
- `github.com/prometheus/client_golang` – metrics exposition.
Everything else (UUIDv7, TOTP, migrations, SSE, routing, CSRF, rate limiting, SigV4 if Bedrock is added) is small enough to write and own. New dependencies need a one-line justification in this file.
+9
View File
@@ -0,0 +1,9 @@
# ADR-NNNN: Title
Date: YYYY-MM-DD
Status: Proposed | Accepted | Superseded by ADR-XXXX
## Context
## Decision
## Consequences
## Alternatives considered
+20
View File
@@ -0,0 +1,20 @@
# ADR-0001: Global-workspace cognitive cycle with deterministic attention
Date: 2026-09-14
Status: Accepted
## Context
Noema needs many concurrent cognitive processes to compete for limited "attention" in a way an operator can inspect and reproduce. Letting an LLM decide what to think about would be opaque, expensive and non-deterministic.
## Decision
Each mind runs a cycle loop. Registered `Process` implementations observe a `BrainState` and return `ThoughtCandidate`s. A pure-Go attention scorer computes a weighted sum over named features with per-mind weights, records every candidate and its component scores as events, and admits the top-K into a capacity-limited workspace. Admitted thoughts are broadcast (as state and as an event) and dispatched to executive handlers. Cycles are triggered by new events or mode timers, never by a fixed hot loop.
## Consequences
- Attention is explainable ("why did A win?") and unit-testable for determinism.
- LLM cost is bounded per cycle because processes, not the cycle, decide when to call models within budgets.
- Adding a cognitive faculty means adding a `Process`; the core does not change.
- Workspace capacity and weights are tunables exposed in the UI with audit trails.
## Alternatives considered
- Single LLM agent loop with tools: opaque, vendor-bound, cannot show why.
- Actor-per-process with message passing: more machinery for no inspectability gain; concurrency is still bounded by a pool in the chosen design.
+21
View File
@@ -0,0 +1,21 @@
# ADR-0002: Append-only cognitive events with database-enforced immutability
Date: 2026-09-14
Status: Accepted
## Context
Provenance and replay require that history never changes. Application discipline alone is fragile.
## Decision
`cognitive_events` (and `belief_versions`, `personality_versions`, `audit_log`, `approval_decisions`) get triggers that raise on UPDATE and DELETE. Belief changes, corrections, consolidations and retractions are new events referencing old ones. Retention purges, if ever configured, run through a dedicated, audited maintenance path that disables the trigger inside one transaction and records what was purged.
Events use UUIDv7 ids (time-ordered) plus a `bigserial seq` for gap-free cursors. Durable consumers read by `seq`; the in-memory bus is only a wake-up.
## Consequences
- History is trustworthy by construction; provenance links never dangle.
- Storage grows monotonically; retention policy is an explicit operator decision.
- Cursor-based consumption makes restarts and replay simple.
## Alternatives considered
- Event sourcing with projections rebuilt from events only: elegant but forces all reads through replay; we keep mutable projection tables (beliefs, memories) with immutable version tables beside them.
- Kafka or similar: unnecessary operational weight for a single-process daemon.
+23
View File
@@ -0,0 +1,23 @@
# ADR-0003: Memory representation
Date: 2026-09-14
Status: Accepted
## Context
Noema needs episodic, semantic, procedural and autobiographical memory, working memory, decay without deletion, several retrieval mechanisms, provenance and relationships, and must not collapse epistemic categories.
## Decision
- One `memories` table with a `kind` discriminator and shared columns (importance, salience, activation, retrieval stats, source, credibility, epistemic kind, timestamps) plus a typed `content` JSON document per kind. Go exposes distinct types; the shared table keeps relationships, search and retrieval uniform.
- `epistemic_kind` is a separate enum (observation, fact, memory, claim, hypothesis, belief, prediction, assumption, inference, opinion) present on memories, beliefs and evidence links.
- Full-text search via a generated `tsvector` column. Embeddings live in `memory_embeddings(memory_id, model, dims, embedding)` created only if pgvector is available, so representations from different models are never mixed.
- Relationships live in `memory_relationships` and the general `entity_relationships` graph with confidence, provenance and timestamps.
- Decay is a scheduled deterministic update of `activation`; rows are never deleted by decay.
- Working memory is an in-process bounded structure persisted in `workspace_states` after each cycle.
## Consequences
- Retrieval is a deterministic fusion of text, vector, recency, activation, importance and graph distance with per-component scores recorded.
- Loss of the embedding service degrades retrieval quality but never availability.
## Alternatives considered
- Table per memory kind: cleaner typing but duplicated retrieval/relationship code and cross-kind queries become unions.
- Vector-only store: rejected by requirement; loses provenance and structure.
+19
View File
@@ -0,0 +1,19 @@
# ADR-0004: LLM as a replaceable cognitive service
Date: 2026-09-14
Status: Accepted
## Context
Minds must survive provider changes; different tasks want different models; every call must be inspectable and replayable; secrets must not leak.
## Decision
`internal/llm` defines `LanguageModel` (`Complete(ctx, CognitiveRequest) (CognitiveResponse, error)`) and `Embedder`. Providers implement raw HTTP clients (OpenAI-compatible, Anthropic Messages, Ollama; Bedrock later behind the same interface). Provider rows in the database hold endpoint, model, parameters and a *secret reference*. **Roles** map cognitive tasks to a primary model and ordered fallbacks. A `Recorder` wrapper writes every request/response (prompt name+version, rendered messages, parameters, seed, model, usage, latency, estimated cost) and enforces budgets; a `Replayer` serves recorded responses during replay. Prompts are versioned rows rendered with structured inputs; untrusted text is enclosed in explicit data blocks.
## Consequences
- Cognitive code depends only on roles; swapping vendors is configuration.
- Replay of LLM steps is exact when a recorded response exists and approximate otherwise, and is labelled as such.
- Budget enforcement and cost accounting live in one place.
## Alternatives considered
- Vendor SDKs: heavy dependency trees, inconsistent behaviour; raw HTTP is small and controllable.
- LangChain-style frameworks in Go: unnecessary abstraction.
+19
View File
@@ -0,0 +1,19 @@
# ADR-0005: Capability and policy layer as the only path to action
Date: 2026-09-14
Status: Accepted
## Context
A reasoning system must not gain infrastructure access because it can describe an action. Operators need ALLOW/ASK/DENY control at several scopes with auditability.
## Decision
Tools register a `Capability` (name, description, parameter schema, risk class, target kind). The only execution entry point is `capabilities.Invoke`, which: validates parameters, evaluates policies (scopes: global, mind, capability, target pattern, action type, time window; most specific wins; DENY > ASK > ALLOW on ties), records a `policy_decision` event, and either executes, creates an approval request, or denies. Approvals are bound to a hash of the exact request; "approve for session" grants a time-boxed, hash-pattern-bound exception. Contexts flagged as dreaming or replay are refused unconditionally. `DENY`-locked capabilities (sudo shell, database deletion) cannot be relaxed via UI or API.
## Consequences
- Every external effect has a policy decision and an event behind it.
- New tools are small: implement the capability, register it, choose a default policy.
- Minds have no capability that edits policies, so escalation is structurally impossible.
## Alternatives considered
- Per-tool ad-hoc checks: inevitably bypassed.
- OS-level sandboxing only: valuable later as defence in depth, not a replacement for policy.
+18
View File
@@ -0,0 +1,18 @@
# ADR-0006: Local authentication design
Date: 2026-09-14
Status: Accepted
## Context
The admin interface controls a system that can act on infrastructure. It needs strong local authentication first, with room for OIDC.
## Decision
Argon2id (64 MiB, t=3, p=2, 16-byte salt) password hashes; opaque 256-bit session tokens stored as SHA-256 hashes with idle and absolute expiry; cookies `HttpOnly; SameSite=Lax; Path=/` and `Secure` outside development; per-session CSRF token checked on all state-changing cookie-authenticated requests plus Origin check; login limiter per IP (token bucket) and per account (exponential backoff persisted in the database); optional TOTP (RFC 6238, implemented with stdlib HMAC) with hashed recovery codes; session listing and revocation; roles `administrator`, `operator`, `observer` enforced by route middleware. Personal API tokens (hashed, role-bound) authenticate `noemactl` and scripts. `auth.Authenticator` is an interface so OIDC can be added.
## Consequences
- No third-party auth dependencies beyond `x/crypto`.
- First-run bootstrap admin from environment or `noemactl user create`.
## Alternatives considered
- OIDC-only: blocks single-operator deployments.
- JWT sessions: revocation and listing become awkward; opaque tokens are simpler and safer.
+19
View File
@@ -0,0 +1,19 @@
# ADR-0007: Server-rendered UI with dependency-free enhancement
Date: 2026-09-14
Status: Accepted
## Context
The UI must be a rich cognitive observatory yet maintainable by Go developers, secure against XSS, and free of supply-chain risk.
## Decision
Go `html/template` pages embedded in the binary, one layout, CSS custom properties with light/dark themes (system preference plus toggle), and a small amount of hand-written vanilla JavaScript: an SSE client that inserts server-rendered HTML fragments, form enhancements (fetch + swap where it helps, plain POST-redirect-GET otherwise), keyboard navigation, and a canvas force-directed graph for the relationship explorer. Charts are inline SVG rendered by templates. No npm, no bundler, no frameworks, no CDN. Strict CSP with no inline scripts.
## Consequences
- Every view is a Go handler + template; live updates reuse the same templates.
- Graph rendering is bounded (node caps, filters) by design.
- If a page later needs heavier interactivity, a single vendored, hash-pinned library can be added under `web/static/vendor/` with an ADR update.
## Alternatives considered
- React/Vue SPA: doubles the codebase and toolchain for little gain in an operator tool.
- HTMX: close to what is needed, but a 30-line SSE/fragment helper covers our usage without a third-party script.
+18
View File
@@ -0,0 +1,18 @@
# ADR-0008: PostgreSQL with explicit SQL and embedded migrations
Date: 2026-09-14
Status: Accepted
## Context
Requirement: PostgreSQL, proper migrations, no ORM auto-mutation, optional pgvector.
## Decision
pgx v5 with pgxpool; repository types per domain with hand-written SQL; numbered `NNNN_name.up.sql` / `.down.sql` files embedded from `migrations/` and applied by a small runner (advisory lock, `schema_migrations` table with checksum). The embeddings migration uses a PL/pgSQL block that creates vector objects only if the extension is available; the runtime detects availability at start.
## Consequences
- No hidden schema changes; migration state visible in the System page and `noemactl migrate status`.
- Tests needing the database use a per-run schema and skip when no URL is configured.
## Alternatives considered
- golang-migrate/goose: fine tools, but a 150-line runner avoids a dependency and matches our exact needs.
- sqlc: attractive later; deferred to keep the toolchain minimal.
+39
View File
@@ -0,0 +1,39 @@
# Configuration
Precedence (later wins): built-in defaults → JSON file given by `noemad -config path` → environment variables → runtime settings stored in the `settings` table and edited in the UI (only for tunables that are safe to change at runtime).
## Environment variables
| Variable | Default | Purpose |
|---|---|---|
| `NOEMA_HTTP_ADDR` | `127.0.0.1:8080` | Listen address |
| `NOEMA_ENV` | `development` | `production` forces Secure cookies and disables the setup page after bootstrap |
| `NOEMA_LOG_LEVEL` | `info` | debug, info, warn, error |
| `NOEMA_LOG_FORMAT` | `text` | text or json |
| `NOEMA_DATABASE_URL` | – | PostgreSQL DSN (required) |
| `NOEMA_MASTER_KEY` | – | base64 32-byte key for `enc:` secret references |
| `NOEMA_BOOTSTRAP_ADMIN_USER` / `_PASSWORD` | – | Creates the first administrator when no users exist |
| `NOEMA_RESEARCH_DIR` | `./research` | Watched ingestion folder |
| `NOEMA_MAX_WORKERS` | `8` | Cognitive worker pool size |
| `NOEMA_MAX_CONCURRENT_LLM` | `4` | Concurrent model calls |
| `NOEMA_DAILY_TOKEN_BUDGET` | `2000000` | Global hard stop |
## Secret references
Wherever a secret is configured (provider API keys, webhook secrets) the stored value is a reference:
- `env:OPENAI_API_KEY` – read from the environment at use time
- `file:/run/secrets/openai` – read from a file
- `enc:<base64 ciphertext>` – encrypted with `NOEMA_MASTER_KEY` (AES-256-GCM); produced when a secret is typed into the UI
Plaintext secrets are never stored, exported or displayed.
## Config file
The JSON file mirrors the environment variables in lower snake case, e.g.
```json
{ "http_addr": "0.0.0.0:8080", "log_level": "debug", "database_url": "postgres://..." }
```
Secrets should not be placed in the config file; use environment variables or secret files.
+15
View File
@@ -0,0 +1,15 @@
# Interpretation boundary
Noema models cognitive processes. It does not prove, and its authors do not claim, that it is conscious, sentient, self-aware or a person.
The system contains components named after psychological concepts: attention, working memory, emotion-like state, curiosity, drives, self-model, reflection, dreaming. These names describe the *functional role* each component plays in the architecture. They are implemented as ordinary data structures and deterministic rules, sometimes assisted by language models.
When a mind reports that it is "curious" or "concerned", it is reporting the value of a stored scalar that the architecture updates by rule. When it says "I believe", it refers to a row in a belief table with evidence links. Such reports are useful because they are grounded in inspectable state, not because they indicate experience.
Users and researchers should therefore:
- treat resemblance to emotion, preference or identity as a property of the model being run, to be studied, not as evidence of subjective experience;
- rely on the evidence and provenance views rather than on fluent self-description;
- avoid attributing moral patienthood to a mind on the basis of its behaviour within Noema.
The platform is deliberately not crippled to avoid these resemblances, because they are what makes the mechanisms worth studying. The interpretive restraint is the user's responsibility, and the UI states this boundary on the about page and in mind exports.