The API's belief emitter published every event as soon as it was appended.
Belief retraction runs inside its own transaction, so subscribers on the
event bus saw belief_retracted while the belief row still held its old
status, breaking the rule that subscribers never see uncommitted events;
a plan reacting to a retracted assumption read the belief before the
change and did nothing. The emitter now only appends, and every API and UI
caller publishes the operation's events after it succeeds.
Observers can read wants (with effective strength), want conflicts,
commitments, intentions with their plans, subordinate intentions and
decisions, the decision ledger, plans with steps, "why" explanations for
executions, approvals, capability requests, plan steps, intentions and
events, attribution of an outcome event, the agency timeline, the
measure definitions with their disclaimer, agency metrics, metacognitive
calibration and raw research measurements.
Operators can create and close commitments, abandon an intention (its
plan is cancelled and pending requests withdrawn), cancel a plan and
release a blocked step. Every write is audited; none grants capability
authority. OpenAPI documents all 21 routes, and an end-to-end test
covers reads, writes and observer refusals.
Part H invariants, enforced and tested:
- Snapshots now carry wants, want conflicts, commitments, intentions,
plans, plan steps and the volition decision ledger.
- A fork no longer copies the source mind's policies unless the operator
asks for it explicitly (inherit_policies in the API, a checkbox in the
UI; audited), and the fork's history records the omission. Imports
never restore policies, as before. Replays and experiments never
inherit.
- Work inherited mid-flight by a fork or import is quiesced in the
restore transaction: open steps are blocked with their approval and
request ids cleared, open plans blocked and live intentions suspended,
each with the reason. Approvals, grants and executions stay with the
source mind.
- Dreams may leave a want to test a hypothesis they proposed; they never
request authority, and dream-origin plans never run.
- Snapshot column discovery now reads the current schema instead of
"public", which only worked in tests because the development database
happened to hold the same tables.
- Security tests: a static scan that only the capability service runs
executors and that no cognitive package can set or delete policy,
approve, deny, grant or revoke; forged wants, commitments, intentions,
model plans and model output end in refusal or a pending approval with
policy and grants unchanged; component names cannot decide requests or
approvals; no capability names an operator control; the registry cannot
grow at runtime; audit, event and decision history cannot be altered;
forks and dreams as above.
- Capabilities declare access (read/write, unknown fails closed as
write), side effects, idempotency, timeout, cost estimate and the trust
of their output (web content is low trust). A capability may derive its
policy target from its parameters; a disagreeing target is refused, and
approvals whose modified parameters change the target are refused.
- The registry is frozen at start-up; registration afterwards panics.
- Invocation.RequireApproval turns ALLOW into ASK for commitments; it can
only tighten, and scoped grants never satisfy it.
- Scoped grants (migration 0025): capability, target and parameter globs,
access, expiry of at most seven days and an optional operation count
consumed atomically; applied only where policy says ASK. A sweep marks
lapsed grants and refreshes the self-model, which now lists live grants.
- Capability requests carry scope, duration, operation cap, reason,
evidence, risk, mitigation, fallback, alternatives and explicit decision
history; they batch per intention, are capped per mind (operator
attention budget) and can be decided only by a named person. Component
names are refused as deciders, for approvals too, and as usernames.
- API routes, OpenAPI entries, scheduler sweeps and approval-centre
sections for requests and temporary grants.
Rows in append-only history are never deleted; their content can now be
made irrecoverable through one audited path. A redaction replaces the
content columns of an event, memory, message, model call, belief or
assertion with a fixed marker, keeps identifiers, links, timestamps,
scores and breakdowns, writes an append-only ledger row with a SHA-256 of
the removed content, appends a redaction event, optionally cascades to
derived rows and to the events that quote them, and scrubs stored snapshot
archives before re-signing them. noema_redact() is the only write the
append-only trigger admits, for UPDATE only, under a transaction-local flag
the application never sets; plain updates and all deletes stay refused.
Replay labels redacted stimuli instead of inventing them.
Cryptographic erasure was considered and deferred with reasons (ADR-0009);
docs/privacy.md defines retained, redacted, irrecoverable and deleted and
states what backups and earlier exports are not covered. Migration 0023.
A bounded walk (six hops, two hundred events) over the links events
actually record: causation ids, parent ids and cited-event evidence,
backwards towards causes and forwards towards consequences, with the edge
kind on every arrow. Nodes carry their cycle for grouping, labelled as
correlation rather than causation, and the artefacts (beliefs, memories,
model calls) they rest on. Cyclic references terminate. Exposed at
GET /api/v1/events/{id}/chain and /events/{id}/timeline, linked from the
event page, and documented.
A source's track record is kept per (evidence family, belief subject) and
updated only by outcomes: predictions derived from a belief resolving, and
operators marking a belief wrong. With three or more outcomes in a domain
the record adjusts the credibility of that source's new evidence in that
domain, and the adjustment is written on the evidence row. Standing in one
domain never moves standing in another, and the trust-level credibility
constants are never rewritten. Migration 0021 is additive; belief pages
show each cited family's record in the belief's domain.
Every breakdown now carries its meaning: an uncalibrated epistemic score
derived from recorded evidence, not a probability of truth. Belief detail
adds a calibrated estimate read from the mind's resolved predictions when
at least twenty were decided in the score's range, and otherwise says why
none is shown; and names the primary uncertainty deterministically (no live
evidence, conflicting evidence, second-hand only, unclear reading, a single
source family, little evidence). The belief list heading becomes Score.
A statement made to a mind is now three things kept apart: the observation
that it arrived, an append-only assertion (attributed actor, stance,
certainty, modality, negation, reported or quoted speech, hypothetical
status, interpretation confidence and notes), and, only for factual
assertions, one piece of evidence for a proposition belief attributed to
whoever actually stands behind it. Reported speech is attributed to the
person reported and weighed as second-hand; hypotheticals, questions and
imperatives create no belief; expressed doubt counts against; a possibly
sarcastic remark is recorded with low interpretation confidence rather
than guessed at. Interpretation is deterministic code; a model may enrich
the claim list but never sets stance or confidence.
Association no longer treats conversational turns as world events:
hypotheticals, questions, requests and consecutive turns by one speaker
cannot seed causal hypotheses.
Adds internal/epistemics, migration 0018, the assertions API, assertions
on belief pages and in belief detail, a shared end-to-end test harness
(tests/harness), end-to-end tests for eight speech patterns, and updates
the milestone test to the new provenance.
Every evidence row now records its derivation (direct, assertion, operator,
external, prediction, consolidation, inference, reflection, dream,
retrieval), its independent-source family (an actor, an integration, an
operator, a prediction) and its root event or memory ids. Confidence
groups evidence by family: the strongest primary item counts in full, each
further primary item adds a quarter, capped at 1.5x, and derived items
count only when the family has no primary member. Three derived
re-expressions of one utterance therefore weigh exactly what the utterance
weighs, while three independent families still add up.
Producers (conversation, association, consolidation, predictions,
metacognition, dreaming, operator and seed input, contradictions) supply
their ancestry. Existing rows are marked legacy_unknown with family =
source; nothing is invented. The belief page shows per-family
contributions and the replay test checks that provenance is reconstructed
identically in the sandbox. Migration 0017 is additive.
The SQL-injection effect test found that a NUL byte in a query string,
path or JSON body reached PostgreSQL and surfaced as a 500. Requests are
now buffered up to the existing limit and NUL in any form (raw, JSON
escaped, form encoded) is refused with a 400 before a query runs.
TestSQLInjectionHasNoEffect sends metacharacter payloads through every
search, filter and free-text route and asserts effect: users, policies,
sessions and tokens unchanged, seeded content intact, payloads stored
verbatim. TestDynamicSQLIdentifiersAreAllowListed scans for statements
assembled with Sprintf outside the reviewed allow-list.
An archive is data about a mind, not a grant of permissions on this
instance. Import now skips the policies table (forks on the same instance
keep theirs), reports the omitted rows in the audit log and the mind's
history, and the snapshot test proves both paths.
- TOTP second factor (RFC 6238, secret encrypted under the master key),
ten hashed single-use recovery codes, two-step login with a pending token
bound to the client address, account enrolment/disable, admin reset
- tests/security: prompt-injection corpus never acts or creates evidence-free
beliefs; roles and CSRF on form and API paths; cross-mind isolation
(demo scenario 5); stored-XSS escaping and CSP across pages; secrets never
echoed (API, pages, database, archives); hostile archive import; session
lifecycle and security headers
- OpenAPI 3.1 document generated from the route table, served at
/api/v1/openapi.json and kept complete by a test
- gosec clean (justified #nosec annotations, cookie/redirect hardening,
bounded multipart parsing, UUID packing without narrowing conversions);
govulncheck clean with toolchain pinned to go1.26.8
- THREAT_MODEL verification pointers, SECURITY.md testing section, README,
CHANGELOG 0.1.0, docs/RELEASE_NOTES.md, PLAN status
- snapshots: checksummed JSON archive of every mind-scoped state table plus
identity/personality/config and the event position; stored, downloadable
- fork: restore into a new mind with all ids remapped; event references are
kept so history before the fork is shared by reference
- export/import: same archive; import is hostile-input validated (size,
allow-listed tables, row caps, column names, checksum, forced mind_id)
- compare: deterministic counts, belief diffs by proposition, personality
and config diffs, event mix
- replay: Runtime.Step for synchronous drivers; sandbox fork driven one
stimulus at a time under a forbidden capability context; recorded model
responses via a replaying router; steps labelled exact / approximate /
diverged with recorded-vs-replayed digests
- experiments: definition (overrides, stimuli, expectations, seed, duration),
background run in a disabled subject mind, stored and exportable results
- migration 0015, API, UI (Snapshots and Replay tabs, Experiments section,
compare page), noemactl snapshots/compare/replay/experiments, docs
- dreaming.Service: deterministic memory replay sample keyed by dream id,
vocabulary-based association of unlinked memory pairs (similar_to edges +
dream_association events), speculative hypotheses as hypothesis-kind beliefs
with two weak memory citations (starts ~0.30, asserted <= 0.35)
- Run marks the context forbidden for capabilities before anything else;
test proves Invoke refuses inside a dream
- per-mind settings: enabled (default off), hours window (wraps midnight),
interval, model-call / memory / hypothesis caps; global pause switch
- optional model wording via new `dream` prompt (role hypothesise)
- promotion adds operator evidence, rejection retracts the belief; both
audited and recorded as operator actions
- migration 0014, API, Dreams tab and page, noemactl dreams, docs/dreaming.md
Phase 12 (ADR-0005). Capabilities are registered in code with parameter
schemas, risk classes and default effects; sudo_shell and delete_database
are locked to DENY and cannot be relaxed. Policies are scoped globally or
per mind with capability, target glob, action kind and optional hour
window; evaluation is deterministic, most specific wins, and ties go to
the stricter effect. Invoke is the single entry point: it refuses
dreaming and replay contexts, validates parameters (unknown keys, non
http URLs and path traversal rejected), evaluates policy and session
grants, hashes the exact request and records a policy_decision event.
ALLOW executes with timeout and panic isolation and turns textual output
into a tool-trust observation; ASK creates an approval request bound to
the hash with an alert notification; DENY is counted and recorded.
Operators approve once, approve for a time-boxed session grant, deny
with a note, or modify parameters (re-validated, re-hashed, original
kept in the append-only decision log). The self-model's capability list
is recomputed from effective policy on start and on every policy change.
The act process maps fetch requests to http.fetch and the action handler
invokes through the layer; replies report capabilities and outcomes.
Policy matrix and approval centre pages, dashboard count, API, CLI
(approvals, policy, invoke) and docs/capabilities.md.
Phase 11. Entities are mind-scoped nodes with canonical matching,
aliases and deterministic kind guessing (hostnames and hyphenated ids
are systems, capitalised names people, file suffixes documents).
Relationships carry a relation from the shared vocabulary, confidence,
source, reinforcement count and the event that produced them; links tie
entities to memories, beliefs, events, goals and predictions.
The perceive and attend handlers upsert the entities a message or
observation mentions and record co-mention edges cited to the event, so
the graph grows only from what was actually seen. Operators can merge
("these are the same": edges and links move, the old name becomes an
alias, the old id still resolves), record that two entities are
different, correct kinds and add typed edges; every correction is an
operator action and a corrections row.
The explorer renders a bounded subgraph on a canvas with a small
force-directed layout in plain JavaScript, with kind filters, focus on
a neighbourhood and a side panel per entity. API, entity page, CLI and
docs/graph.md.
Phase 10. Metacognition is a cognitive process with deterministic
self-checks (insufficient evidence, single-source reliance, model output
without evidence, repeated failed hypotheses, stale contradictions,
calibration overconfidence); its warnings compete for attention and,
when admitted, add counter-evidence or open an investigation goal.
Reflection reviews the period since the last run (prediction errors,
rejected beliefs, corrections, status changes, completed goals,
contradictions, calibration), turns repeated failures into procedural
lessons written once, and records a reflection event; a model may only
word the prose within the mind's budget.
Consolidation clusters unconsolidated episodes by shared stemmed terms,
creates a belief whose confidence is computed from the episodes as
evidence and a semantic memory linked to each episode by derived_from,
and records jobs with inputs and outputs. Wording may come from the
consolidate prompt within a per-run model budget; the job says which.
Operators can run, pause and reject; originals are never touched.
The narrative hook keeps a five-part first-person narrative built from
state and stored with narrative_updated events. The notification hook
applies explicit thresholded rules, aggregates repeats within a cooldown
and records suppressions with reasons. A scheduled job runs
consolidation and reflection per mind, switching the runtime mode while
it works. API, Background tab, notifications page with unread badge,
overview narrative and warnings, CLI commands, docs/cognition.md.
Phase 9. Goals form a mind-scoped tree with kind, priority, progress and
blockers; their deterministic relevance to a candidate feeds the
goal_relevance attention feature and a goal monitor raises stalled goals
and observations that touch one. Drives are a closed set of eight
scalars with operator-configured ranges and personality-derived
baselines; there is no self-preservation, replication or acquisition
drive and no way to add one at runtime. Affect is ten scalars moved by an
explicit rule table over events and decaying towards baseline; both
modulate attention weights each cycle and the effective weights are
recorded with every cycle and shown beside the configured ones.
Predictions carry a window and machine-checkable criteria (event terms
occurring or staying absent, a belief reaching a confidence, or operator
judgement). A resolver job judges them deterministically, keeps an
append-only outcome history for calibration (accuracy, Brier score,
confidence buckets, overconfidence flag), emits prediction_error events
scaled by how confident the miss was, and feeds the outcome back into the
linked belief as evidence. Causal hypotheses derive predictions
automatically. Curiosity turns misses, contradictions and unexplained
observations into questions that compete for attention and become
investigation goals when admitted.
Brain gains after-cycle hooks and goal/motivation state. API, UI (goals,
prediction dashboard, cognitive state panel, effective weights) and CLI
(goal, prediction, state). Demo scenario 4 verified end to end.
Phase 8, completing milestone M1. A user message is stored and appended
as an observation event carrying trust, actor, channel and reference. The
perceive process analyses it deterministically (intent, introspection
kind, entities, claims, urgency) and, within the per-cycle model
allowance, enriches it through the perceive prompt. Its handler writes a
perception event, an episodic memory with participants, turns claims into
beliefs whose evidence is the message, and builds a response intent from
state: beliefs and memories for the topic, evidence rows for why-questions,
the uncertain band, contested beliefs and their links, recent learning,
status changes, and honest declines for subsystems that do not exist yet
or for requests with no capabilities. The respond prompt phrases the
intent when a model is assigned; otherwise the reply is a direct
rendering of the intent. response_generated cites the message, the
perception, every belief and memory used, the episode and the model call.
API: conversations, messages with optional wait, one-shot ask; chat UI
with live refresh and a per-reply intent view. Sample mind Dani as a
strictly decoded seed document with an import endpoint and noemactl mind
import. tests/m1 drives all fourteen milestone steps including restart.
Phase 7 (ADR-0004). Cognitive code asks the router for a role; the
router maps roles to configured models with ordered fallbacks, applies
global and per-mind daily token budgets and a concurrency limit, records
every call (request, response, usage, latency, cost, seed, request hash)
and can serve recordings back for exact replay. Providers are raw HTTP
clients for OpenAI-compatible, Anthropic and Ollama endpoints, with
embedding support for the embed role wired into memory retrieval.
Secrets are references only: env:, file:, or enc: values encrypted with
AES-256-GCM under NOEMA_MASTER_KEY; keys are resolved at call time and
never rendered, logged or exported. Outbound HTTP refuses private and
loopback addresses at dial time unless a provider explicitly allows it,
bounds redirects and caps bodies. Provider errors are scrubbed.
Prompts are named, versioned rows with system and user templates and an
optional JSON schema; untrusted text enters only through delimited data
blocks. Six defaults are seeded and never overwritten. The UI adds the
LLM page (providers, models, role matrix, usage, errors, budgets), a
provider page with connectivity test, and a prompt manager with editor,
test runner, history, restore, clone and diff. CLI: provider, prompt.
Phase 6. Beliefs are propositions with a prior set by epistemic kind and a
confidence computed from typed evidence: sigmoid(logit(prior) + slope ×
(support × diversity − opposition)), where support from a single source
is discounted so repetition cannot manufacture certainty, and bounds keep
the mind away from absolute certainty. Every evaluation stores its
breakdown and appends a version row protected by the append-only
trigger; the belief page shows the arithmetic, evidence for and against,
alternatives, contradictions and history.
Contradiction detection is deterministic first (negation plus token
overlap): a hit emits contradiction_detected, links both beliefs and
makes each counter-evidence for the other so both trend to contested.
Operators can assert beliefs (their statement is the evidence), add
evidence, mark a belief wrong (pinned rejected, history kept) and forbid
an inference; do-not-infer corrections block matching hypotheses.
Retracting a memory retracts the evidence that cited it and re-evaluates
the beliefs concerned.
The associate process proposes low-confidence causal hypotheses from
co-occurring notable observations, and the hypothesis handler turns
admitted ones into beliefs with automatic alternatives, closing demo
scenarios 1 and 2 without any model call.
Phase 5 (ADR-0003). pkg/concepts fixes the epistemic vocabulary
(observation, fact, memory, claim, hypothesis, belief, prediction,
assumption, inference, opinion), memory kinds and relation names.
internal/memory stores episodic, semantic, procedural and
autobiographical memories in one mind-scoped table with trust-derived
credibility, importance, salience and decaying activation. Retrieval
fuses full-text rank (any-term by default), optional pgvector similarity,
recency, activation, importance and credibility, and returns the
component scores so every recall can be explained. Decay lowers
activation and never deletes; retraction and supersession keep rows for
provenance. Relationships are constrained to one mind at insert time.
Embeddings live in a conditional pgvector table keyed by model so vector
spaces are never mixed; without an embedder the system degrades to text
and graph retrieval.
Attention gates encoding: the attend handler turns admitted observations
into episodic memories (operator changes become autobiographical ones)
with memory_created events; the retrieve process proposes recalled
memories as candidates and admitted recalls are reinforced with
memory_retrieved events carrying their scoring components.
Memory explorer UI (search with component breakdown, timeline, filters,
operator knowledge entry), memory detail with provenance, relationships,
citing events and an inaccurate-memory correction; API; noemactl memory
commands; decay and embed-missing background jobs.
Phase 4 (ADR-0001). internal/attention scores thought candidates as a
normalised weighted sum over ten named features with a per-term
breakdown, ranked by (score, id) so ordering is total and replayable.
internal/workspace holds the capacity-limited workspace (refresh by
subject, decayed competition, explained admit/reject outcomes) and the
decaying working memory.
internal/brain runs one runtime per active mind: debounced wake-ups from
the event bus (own output never re-triggers), a settle window so
concurrently committing events are not skipped, bounded concurrent
Observe calls with per-process isolation and timeouts, deterministic
scoring, a loop detector that suppresses recurring thoughts and raises a
metacognition warning, executive dispatch by thought kind, and one
transaction per cycle whose events are published only after commit.
Every candidate, selection, attention shift and broadcast is an event
with provenance back to the triggering observation. Operators can run a
cycle, pause, resume, reset transient state and change attention
weights live; the manager reconciles runtimes with mind status.
cognition/attend is the first process: deterministic candidates from
external events using an urgency lexicon, novelty against context and
trust-based operator priority.
Phase 3. Events are the append-only record of everything a mind does
(ADR-0002). pkg/events defines the closed type vocabulary, categories,
evidence references and the observation payload with explicit trust
levels. The store validates and appends inside the caller's transaction,
publishes to a bounded fan-out bus only after commit, queries by seq
cursor with type/category/source/salience/confidence/time/search
filters, and walks provenance: cause, parents, children, cited evidence,
derived events and correlation groups. Durable per-consumer cursors never
move backwards.
Server-Sent Events streams (per mind and global) subscribe before
replaying history after Last-Event-ID so nothing is missed, apply the
same filter vocabulary live, and heartbeat. Operators can inject
observations tagged with operator trust for demos and tests.
UI: live thought stream with filters and paging, forensic event page
with causal chain and evidence navigation, dashboard important events.
Personality and status changes now leave operator_action events in the
mind's own history. CLI: event inspect/list/tail, observe.
Phase 2. Minds are rows with a slug, status (active/disabled/archived),
cognitive configuration and a pointer to the current personality version.
Personalities are structured (identity, values, traits, communication,
behaviour, free text) with a deterministic Brief() used by prompts and the
editor preview. Every change appends a personality_versions row protected
by the append-only trigger; restore copies an old version forward.
Templates carry reusable personalities; minds record their overrides so a
template can be re-applied without losing mind-specific edits. The
self-model starts with structural limitations the mind cannot drop
(no policy changes, no influence over shutdown, no cross-mind access).
API, UI (list, overview, editor with sliders/diff/history/import/export,
template editor) and noemactl mind/template commands. All mutations are
audited with field-level diffs.
Phase 1 foundation. Local auth per ADR-0006: Argon2id passwords, opaque
hashed session tokens with idle/absolute expiry, CSRF token plus Origin
check on cookie sessions, per-IP token bucket and per-account exponential
lockout, three roles enforced by middleware, personal API tokens for
noemactl. Audit log writer used by every admin mutation.
API: request ids, panic recovery, strict CSP and security headers, body
limits, single JSON error envelope, health/ready/status, users, sessions,
tokens, audit, token-protected /metrics.
UI: embedded html/template pages (login, first-run setup, dashboard,
system, users, account, audit, interpretation boundary), light/dark
theme, dependency-free app.js with CSRF-aware fetch and SSE helper.
Binaries: noemad with graceful shutdown, migrations at start, bootstrap
admin from environment, supervised background tasks; noemactl with
status, migrate, user create and meaningful exit codes.
Andy Dixon