andy/noema
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(experiments): cognitive forks per model with per-mind routing and comparison

Browse Source 
A mind's configuration may override model routing per role (llm_roles);
the router honours it for that mind only and fails closed when the named
model is missing or disabled, so a comparison between providers can never
silently fall back to the default. Experiments may declare variants: each
becomes its own fork of the same snapshot with its own configuration,
personality or routing, receives identical stimuli, and is measured. The
results carry one entry per variant and a deterministic comparison: first
divergent stimulus, belief and attention overlap, mean score difference on
shared propositions, and the ranges of memory counts, goal counts, cost and
capability requests. Single-subject definitions are unchanged.
This commit is contained in:
Andy Dixon
2026-08-20 22:26:42 +01:00
parent 522de3b026
commit 844de93c8b
10 changed files with 481 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/llm-providers.md
+4
View File
@@ -2,6 +2,10 @@
Language models are **cognitive service providers**. Cognitive code never names a vendor; it asks the router for a *role* and receives a response with a call id it can cite as evidence (ADR-0004).
## Per-mind routing
A mind's cognitive configuration may carry `llm_roles`, a map from role to model id, which overrides the daemon-wide routing for that mind alone. It exists so that forks of one snapshot can run on different providers in a controlled experiment. An override names exactly one model; if that model is missing or disabled the call fails with "no model" rather than falling back, and the runtime identity records the override next to the daemon routing.
## Roles
| Role | Used for | Suggested tier |
docs/snapshots-replay.md
+6
View File
@@ -72,6 +72,12 @@ The replaying router serves recorded responses by request hash
(`llm_calls.request_hash`) and records the replayed call with `replayed=true`.
Replays run under a forbidden capability context; no action can execute.
## Cognitive forks per model
An experiment may declare **variants**: named sets of overrides, each of which becomes its own fork of the same snapshot and receives the same stimuli. A variant can override configuration, personality and, through `llm_roles` (role → model id), the model used for each role **for that fork only**; the router honours a mind's own routing and fails closed if the named model is missing or disabled, so a comparison between providers can never silently fall back to the default. Every fork starts from identical cognitive state, which is exactly what makes the model a replaceable service rather than the identity.
The results carry one entry per variant (subject, routing in force, produced-event digests per stimulus, beliefs, memory and goal counts, expectations, model calls, tokens, cost, capability requests, cycles) and a deterministic **comparison**: the first stimulus at which any two variants produced different event digests, belief-set overlap and mean score difference on shared propositions, attention overlap (selected-thought digests), and the ranges of memory counts, goal counts, cost and capability requests. Every subject mind stays for the two-mind comparison page. With live models the differences include provider nondeterminism, which is measured rather than hidden.
## Experiments (spec §57)
An experiment is a snapshot, optional configuration and personality