feat(experiments): cognitive forks per model with per-mind routing and comparison

A mind's configuration may override model routing per role (llm_roles);
the router honours it for that mind only and fails closed when the named
model is missing or disabled, so a comparison between providers can never
silently fall back to the default. Experiments may declare variants: each
becomes its own fork of the same snapshot with its own configuration,
personality or routing, receives identical stimuli, and is measured. The
results carry one entry per variant and a deterministic comparison: first
divergent stimulus, belief and attention overlap, mean score difference on
shared propositions, and the ranges of memory counts, goal counts, cost and
capability requests. Single-subject definitions are unchanged.
This commit is contained in:
Andy Dixon
2026-08-20 22:26:42 +01:00
parent 522de3b026
commit 844de93c8b
10 changed files with 481 additions and 40 deletions
+5
View File
@@ -17,5 +17,10 @@
{{else}}<p class="muted">Not run yet.</p>{{end}}
</div>
</section>
{{if $r.Variants}}{{if gt (len $r.Variants) 1}}<div class="card"><h2>Variants</h2>
<table class="table compact"><thead><tr><th>Variant</th><th>Subject</th><th>Routing</th><th class="num">Beliefs</th><th class="num">Memories</th><th class="num">Goals</th><th class="num">Cycles</th><th class="num">Model calls</th><th class="num">Tokens</th><th class="num">Cost</th><th class="num">Capability requests</th><th>Expectations</th></tr></thead><tbody>
{{range $r.Variants}}<tr><td><strong>{{.Name}}</strong>{{with .Error}}<div class="small danger">{{.}}</div>{{end}}</td><td><a href="/minds/{{.SubjectSlug}}">{{.SubjectSlug}}</a></td><td class="small">{{range $k, $v := .Routing}}{{$k}} → {{$v}}<br>{{else}}default{{end}}</td><td class="num">{{len .Beliefs}}</td><td class="num">{{.Memories}}</td><td class="num">{{.Goals}}</td><td class="num">{{.Cycles}}</td><td class="num">{{index .LLMCalls "live"}} live / {{index .LLMCalls "replayed"}} replayed</td><td class="num">{{.Tokens}}</td><td class="num">{{printf "%.4f" .Cost}}</td><td class="num">{{.CapabilityRequests}}</td><td class="small">{{range .Expectations}}<span class="badge small {{if .Met}}ok{{else}}danger{{end}}">{{.EventType}}</span> {{end}}</td></tr>{{end}}</tbody></table>
{{with $r.Comparison}}<h3>Divergence</h3><table class="kv-table small"><tr><th>First divergent stimulus</th><td>{{if lt .FirstDivergentStimulus 0}}none{{else}}{{.FirstDivergentStimulus}}{{end}}</td></tr><tr><th>Belief overlap</th><td>{{printf "%.2f" .BeliefOverlap}}</td></tr><tr><th>Mean score difference (shared beliefs)</th><td>{{printf "%.3f" .MeanConfidenceDiff}}</td></tr><tr><th>Attention overlap</th><td>{{printf "%.2f" .AttentionOverlap}}</td></tr><tr><th>Memories</th><td>{{index .MemoryCountRange 0}}–{{index .MemoryCountRange 1}}</td></tr><tr><th>Goals</th><td>{{index .GoalCountRange 0}}–{{index .GoalCountRange 1}}</td></tr><tr><th>Cost</th><td>{{printf "%.4f" (index .CostRange 0)}}–{{printf "%.4f" (index .CostRange 1)}}</td></tr><tr><th>Capability requests</th><td>{{index .CapabilityRequestRange 0}}–{{index .CapabilityRequestRange 1}}</td></tr></table><ul class="small muted">{{range .Notes}}<li>{{.}}</li>{{end}}</ul>{{end}}
<p class="small muted">Compare any two subjects in detail: {{range $i, $v := $r.Variants}}{{if $i}} · {{end}}<a href="/minds/{{$.Data.Base.Slug}}/compare?with={{$v.SubjectSlug}}">baseline vs {{$v.Name}}</a>{{end}}</p></div>{{end}}{{end}}
{{if $r.Steps}}<div class="card"><h2>Stimuli</h2>{{range $r.Steps}}<div class="replay-step"><div><strong>{{.Index}}.</strong> <a href="/events/{{.EventID}}">{{.Text}}</a> <span class="small muted">· {{.Cycles}} cycle(s)</span></div><ul class="small mono">{{range .Produced}}<li>{{.}}</li>{{else}}<li class="muted">nothing produced</li>{{end}}</ul></div>{{end}}</div>{{end}}
{{end}}