Close the plan: screenshots and a measured performance pass

- docs/screenshots: nineteen pages captured with headless Chromium from a
  running instance; README gains a screenshot section
- tests/perf: opt-in load test (NOEMA_PERF=1) seeding 60k events, 6k
  memories and 1.5k beliefs across two minds, timing sixteen hot reads
  against a 400 ms ceiling and rejecting sequential-scan plans
- migration 0016: indexes the test found missing (memories by mind and
  recency, beliefs by mind and confidence, events by mind and category,
  belief versions by mind and time)
- docs/performance.md records the numbers and how to rerun
This commit is contained in:
Andy Dixon
2026-07-06 23:18:31 +01:00
parent 89580c0a79
commit d9ae91e406
25 changed files with 264 additions and 2 deletions
+2 -2
View File
@@ -171,8 +171,8 @@ An administrator can: start Noema → log in → create a mind → configure per
### Phase 16 – Hardening and release
- ☑ TOTP + recovery codes, security test suite (`tests/security`), OpenAPI document (`docs/openapi.json`, served at `/api/v1/openapi.json`, coverage-tested), docs set, `go test -race`, `go vet`, staticcheck, gosec, govulncheck clean, release notes
- ☐ screenshots (no browser available in the build environment; capture from a running instance and add to `docs/screenshots/`)
- ☐ performance pass beyond indexes already in migrations (profile under a realistic load before tuning)
- ☑ screenshots captured with headless Chromium into `docs/screenshots/` (19 pages)
- ☑ performance pass: `tests/perf` seeds 60k events / 6k memories / 1.5k beliefs, times hot reads against a ceiling and rejects sequential-scan plans; migration 0016 adds the indexes it found missing (see `docs/performance.md`)
## 5. Deterministic vs LLM decisions
See `ARCHITECTURE.md` §14. Rule: if a unit test can assert the exact output, it is code.
+10
View File
@@ -38,6 +38,16 @@ Noema is also not a chatbot with a memory bolted on. A user message does not go
Full description: [ARCHITECTURE.md](ARCHITECTURE.md). Decisions: [docs/adr](docs/adr). Plan and status: [PLAN.md](PLAN.md).
## Screenshots
Captured from a running instance with the sample mind Dani (full set in [docs/screenshots](docs/screenshots)).
| | |
|---|---|
| ![Mind overview](docs/screenshots/mind.png) | ![Thought stream](docs/screenshots/thoughts.png) |
| ![Beliefs](docs/screenshots/beliefs.png) | ![Relationship graph](docs/screenshots/graph.png) |
| ![Cognitive replay](docs/screenshots/replay.png) | ![Policies](docs/screenshots/policies.png) |
## Quick start
Requirements: Go 1.26+, Docker with Compose, `make`.
+50
View File
@@ -0,0 +1,50 @@
# Performance pass
`tests/perf` is an opt-in load test. It seeds a fresh schema with a realistic
volume for one mind plus a second "noise" mind (so scoping matters), times
the hot read paths in steady state, and rejects any plan that sequentially
scans the three large tables.
```
NOEMA_PERF=1 NOEMA_TEST_DATABASE_URL=postgres://... go test -count=1 -v -run TestHotPaths ./tests/perf/
```
Seed: 60 000 events, 6 000 memories, 1 500 beliefs with evidence. Ceiling per
read: 400 ms (a regression guard, not a target).
## Results (PostgreSQL 16 in Docker, laptop, 2026-09-14)
| Read | Before 0016 | After 0016 |
|---|---|---|
| events recent 50 | 3.2 ms | 1.2 ms |
| events by type | 1.8 ms | 1.1 ms |
| events after seq (runtime cursor) | 2.2 ms | 1.2 ms |
| events summary search | 3.3 ms | 1.4 ms |
| events stats (group by category) | 13.8 ms | 15.8 ms |
| memory list 50 | 4.1 ms (seq scan) | 0.5 ms |
| memory text search | 1.7 ms | 0.5 ms |
| memory fused retrieve | 31 ms | 27 ms |
| beliefs list | 3.6 ms (seq scan) | 3.6 ms |
| beliefs search / contested / uncertain / stats / recent changes | ≤ 3 ms | ≤ 3 ms |
| snapshot capture (all state tables) | 288 ms | 296 ms |
Migration `0016_hot_path_indexes` adds the indexes the planner needed for the
two list orders that had fallen back to sequential scans, plus
`(mind_id, category, seq)` on events for replay's stimulus scan and
`(mind_id, created_at)` on belief versions for the recent-changes view.
## What was deliberately left alone
- **Fused retrieval (~27 ms)** is the most expensive per-cycle read; it runs a
ranked full-text query capped at 50 candidates plus optional vector search.
It is well inside the cycle budget; tune only if cycle latency becomes visible.
- **Snapshot capture (~300 ms)** serialises every state table; it is an
operator action, not a cycle path.
- **Event stats (~16 ms)** is a group-by over all of a mind's events, used on
the overview page. A materialised counter would be premature at this volume.
## Rerun before touching queries
The plan checks fail loudly when a future query drifts off its index, which is
the cheapest way to catch a regression that would otherwise only show up with
months of history.
Binary file not shown.

After

Width:  |  Height:  |  Size: 110 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 72 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 163 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 97 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 70 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 167 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 164 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 139 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 107 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 152 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 153 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 127 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 165 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 112 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 145 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 129 KiB

@@ -0,0 +1,4 @@
DROP INDEX IF EXISTS belief_versions_mind_created_idx;
DROP INDEX IF EXISTS cognitive_events_mind_category_seq_idx;
DROP INDEX IF EXISTS beliefs_mind_confidence_idx;
DROP INDEX IF EXISTS memories_mind_recent_idx;
+7
View File
@@ -0,0 +1,7 @@
-- Performance pass (Phase 16): indexes matching the hot list orders found by
-- tests/perf. The planner chose sequential scans because the existing
-- (mind_id, kind|status) indexes could not serve the ORDER BY.
CREATE INDEX IF NOT EXISTS memories_mind_recent_idx ON memories (mind_id, occurred_at DESC, id DESC) WHERE status = 'active';
CREATE INDEX IF NOT EXISTS beliefs_mind_confidence_idx ON beliefs (mind_id, confidence DESC) WHERE status <> 'superseded';
CREATE INDEX IF NOT EXISTS cognitive_events_mind_category_seq_idx ON cognitive_events (mind_id, category, seq);
CREATE INDEX IF NOT EXISTS belief_versions_mind_created_idx ON belief_versions (mind_id, created_at DESC);
+191
View File
@@ -0,0 +1,191 @@
// Package perf seeds a mind with a realistic volume of state and times the
// hot read paths. It runs only with NOEMA_PERF=1 because seeding takes a
// while; CI runs it nightly, developers run it before touching queries.
//
// NOEMA_PERF=1 NOEMA_TEST_DATABASE_URL=... go test -run TestHotPaths -v ./tests/perf/
package perf
import (
"context"
"fmt"
"os"
"strings"
"testing"
"time"
"github.com/andydixon/noema/internal/beliefs"
evstore "github.com/andydixon/noema/internal/events"
"github.com/andydixon/noema/internal/memory"
"github.com/andydixon/noema/internal/minds"
"github.com/andydixon/noema/internal/snapshots"
"github.com/andydixon/noema/internal/storage"
"github.com/andydixon/noema/migrations"
"github.com/andydixon/noema/pkg/concepts"
"github.com/andydixon/noema/pkg/events"
"github.com/andydixon/noema/pkg/ids"
)
const (
nEvents = 60_000
nMemories = 6_000
nBeliefs = 1_500
ceiling = 400 * time.Millisecond // any hot read slower than this is a regression
)
var vocab = strings.Fields("payments database migration latency deploy atlas server disk cpu alert go rust python andy project service host backup replica queue cache timeout retry release rollback metric dashboard incident postmortem")
func phrase(i int) string {
var w []string
for k := 0; k < 8; k++ {
w = append(w, vocab[(i*7+k*3)%len(vocab)])
}
return fmt.Sprintf("%s (item %d)", strings.Join(w, " "), i)
}
func TestHotPaths(t *testing.T) {
if os.Getenv("NOEMA_PERF") == "" {
t.Skip("set NOEMA_PERF=1 to run the performance pass")
}
pool := storage.TestPool(t, migrations.FS)
ctx := context.Background()
ms := minds.NewStore(pool)
es := evstore.NewStore(pool, evstore.NewBus(8), nil)
mem := memory.NewStore(ctx, pool, nil)
bs := beliefs.NewStore(pool)
m, err := ms.Create(ctx, minds.CreateParams{Slug: "perf", DisplayName: "Perf"})
if err != nil {
t.Fatal(err)
}
other, _ := ms.Create(ctx, minds.CreateParams{Slug: "noise", DisplayName: "Noise"})
seed := time.Now()
batch := make([]events.New, 0, 500)
flush := func(mind ids.ID) {
if len(batch) == 0 {
return
}
if _, err := es.Append(ctx, batch...); err != nil {
t.Fatal(err)
}
batch = batch[:0]
}
types := []events.Type{events.Observation, events.ThoughtSelected, events.MemoryRetrieved, events.BeliefUpdated, events.WorkspaceBroadcast, events.CycleCompleted}
for i := 0; i < nEvents; i++ {
mind := m.ID
if i%5 == 0 {
mind = other.ID // a second mind so scoping matters
}
batch = append(batch, events.New{MindID: mind, Type: types[i%len(types)], Source: "perf", OccurredAt: seed.Add(-time.Duration(nEvents-i) * time.Second), Summary: phrase(i), Salience: float64(i%10) / 10, Confidence: 0.8})
if len(batch) == 500 {
flush(mind)
}
}
flush(m.ID)
for i := 0; i < nMemories; i++ {
kind := concepts.Episodic
if i%4 == 0 {
kind = concepts.Semantic
}
if _, err := mem.Create(ctx, pool, memory.New{MindID: m.ID, Kind: kind, Epistemic: concepts.Observation, Content: phrase(i), Source: "perf", Trust: events.TrustUser, Confidence: 0.7, Importance: float64(i%10) / 10, OccurredAt: seed.Add(-time.Duration(i) * time.Minute)}); err != nil {
t.Fatal(err)
}
}
emit := func(ctx context.Context, q storage.Querier, n events.New) (events.Event, error) {
out, err := es.AppendIn(ctx, q, n)
if err != nil {
return events.Event{}, err
}
return out[0], nil
}
for i := 0; i < nBeliefs; i++ {
if _, err := bs.Create(ctx, pool, emit, beliefs.New{MindID: m.ID, Proposition: phrase(i * 3), Subject: vocab[i%len(vocab)], Epistemic: concepts.Claim, Source: "perf",
Evidence: []beliefs.EvidenceInput{{Direction: "for", Kind: "memory", RefID: ids.New().String(), Source: "user", Weight: 0.4, Credibility: 0.8}}}); err != nil {
t.Fatal(err)
}
}
t.Logf("seeded %d events, %d memories, %d beliefs in %s", nEvents, nMemories, nBeliefs, time.Since(seed).Round(time.Millisecond))
type probe struct {
name string
fn func() error
}
probes := []probe{
{"events recent 50", func() error { _, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Limit: 50}); return err }},
{"events by type", func() error {
_, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Types: []events.Type{events.BeliefUpdated}, Limit: 50})
return err
}},
{"events after seq (cursor)", func() error {
_, err := es.Query(ctx, evstore.Filter{MindID: m.ID, AfterSeq: 1000, Ascending: true, Limit: 200})
return err
}},
{"events search", func() error {
_, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Search: "rollback", Limit: 50})
return err
}},
{"events stats", func() error { _, err := es.Stats(ctx, m.ID, time.Time{}); return err }},
{"memory list 50", func() error { _, err := mem.List(ctx, m.ID, memory.Filter{Limit: 50}); return err }},
{"memory text search", func() error {
_, err := mem.List(ctx, m.ID, memory.Filter{Search: "payments latency", Limit: 50})
return err
}},
{"memory fused retrieve", func() error {
_, err := mem.Retrieve(ctx, m.ID, memory.Query{Text: "database migration latency alert", Limit: 10})
return err
}},
{"memory stats", func() error { _, err := mem.Stats(ctx, m.ID); return err }},
{"beliefs list", func() error { _, err := bs.List(ctx, m.ID, beliefs.Filter{Limit: 50}); return err }},
{"beliefs search", func() error { _, err := bs.Search(ctx, m.ID, "deploy rollback", 20); return err }},
{"beliefs contested", func() error { _, err := bs.Contested(ctx, m.ID, 20); return err }},
{"beliefs uncertain", func() error { _, err := bs.Uncertain(ctx, m.ID, 20); return err }},
{"beliefs stats", func() error { _, err := bs.Stats(ctx, m.ID); return err }},
{"beliefs recent changes", func() error { _, err := bs.RecentChanges(ctx, m.ID, seed.Add(-time.Hour), 50); return err }},
{"snapshot capture", func() error { _, err := snapshots.Capture(ctx, pool, es, m); return err }},
}
failed := false
for _, p := range probes {
p.fn() //nolint:errcheck // warm the cache once; we time the steady state
start := time.Now()
var err error
for i := 0; i < 3; i++ {
err = p.fn()
}
d := time.Since(start) / 3
mark := ""
if d > ceiling {
mark, failed = " <-- over ceiling", true
}
t.Logf("%-28s %8s%s", p.name, d.Round(100*time.Microsecond), mark)
if err != nil {
t.Errorf("%s: %v", p.name, err)
}
}
// Plans: the hot event and memory queries must be index-driven.
plans := map[string]string{
"events recent": `EXPLAIN SELECT id FROM cognitive_events WHERE mind_id=$1 ORDER BY seq DESC LIMIT 50`,
"events cursor": `EXPLAIN SELECT id FROM cognitive_events WHERE mind_id=$1 AND seq > 1000 ORDER BY seq ASC LIMIT 200`,
"memory list": `EXPLAIN SELECT id FROM memories WHERE mind_id=$1 AND status='active' ORDER BY occurred_at DESC, id DESC LIMIT 50`,
"beliefs list": `EXPLAIN SELECT id FROM beliefs WHERE mind_id=$1 AND status NOT IN ('superseded') ORDER BY confidence DESC LIMIT 50`,
}
for name, q := range plans {
rows, err := pool.Query(ctx, q, m.ID)
if err != nil {
t.Fatal(err)
}
var plan []string
for rows.Next() {
var line string
rows.Scan(&line) //nolint:errcheck
plan = append(plan, strings.TrimSpace(line))
}
rows.Close()
joined := strings.Join(plan, " | ")
if strings.Contains(joined, "Seq Scan on cognitive_events") || strings.Contains(joined, "Seq Scan on memories") || strings.Contains(joined, "Seq Scan on beliefs") {
t.Errorf("%s uses a sequential scan: %s", name, joined)
}
t.Logf("plan %-14s %s", name, plan[0])
}
if failed {
t.Fatalf("a hot read exceeded %s", ceiling)
}
}