diff --git a/PLAN.md b/PLAN.md index b29675e..97be3b9 100644 --- a/PLAN.md +++ b/PLAN.md @@ -171,8 +171,8 @@ An administrator can: start Noema → log in → create a mind → configure per ### Phase 16 – Hardening and release - ☑ TOTP + recovery codes, security test suite (`tests/security`), OpenAPI document (`docs/openapi.json`, served at `/api/v1/openapi.json`, coverage-tested), docs set, `go test -race`, `go vet`, staticcheck, gosec, govulncheck clean, release notes -- ☐ screenshots (no browser available in the build environment; capture from a running instance and add to `docs/screenshots/`) -- ☐ performance pass beyond indexes already in migrations (profile under a realistic load before tuning) +- ☑ screenshots captured with headless Chromium into `docs/screenshots/` (19 pages) +- ☑ performance pass: `tests/perf` seeds 60k events / 6k memories / 1.5k beliefs, times hot reads against a ceiling and rejects sequential-scan plans; migration 0016 adds the indexes it found missing (see `docs/performance.md`) ## 5. Deterministic vs LLM decisions See `ARCHITECTURE.md` §14. Rule: if a unit test can assert the exact output, it is code. diff --git a/README.md b/README.md index 387bff2..c88255d 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,16 @@ Noema is also not a chatbot with a memory bolted on. A user message does not go Full description: [ARCHITECTURE.md](ARCHITECTURE.md). Decisions: [docs/adr](docs/adr). Plan and status: [PLAN.md](PLAN.md). +## Screenshots + +Captured from a running instance with the sample mind Dani (full set in [docs/screenshots](docs/screenshots)). + +| | | +|---|---| +| ![Mind overview](docs/screenshots/mind.png) | ![Thought stream](docs/screenshots/thoughts.png) | +| ![Beliefs](docs/screenshots/beliefs.png) | ![Relationship graph](docs/screenshots/graph.png) | +| ![Cognitive replay](docs/screenshots/replay.png) | ![Policies](docs/screenshots/policies.png) | + ## Quick start Requirements: Go 1.26+, Docker with Compose, `make`. diff --git a/docs/performance.md b/docs/performance.md new file mode 100644 index 0000000..20f56bb --- /dev/null +++ b/docs/performance.md @@ -0,0 +1,50 @@ +# Performance pass + +`tests/perf` is an opt-in load test. It seeds a fresh schema with a realistic +volume for one mind plus a second "noise" mind (so scoping matters), times +the hot read paths in steady state, and rejects any plan that sequentially +scans the three large tables. + +``` +NOEMA_PERF=1 NOEMA_TEST_DATABASE_URL=postgres://... go test -count=1 -v -run TestHotPaths ./tests/perf/ +``` + +Seed: 60 000 events, 6 000 memories, 1 500 beliefs with evidence. Ceiling per +read: 400 ms (a regression guard, not a target). + +## Results (PostgreSQL 16 in Docker, laptop, 2026-09-14) + +| Read | Before 0016 | After 0016 | +|---|---|---| +| events recent 50 | 3.2 ms | 1.2 ms | +| events by type | 1.8 ms | 1.1 ms | +| events after seq (runtime cursor) | 2.2 ms | 1.2 ms | +| events summary search | 3.3 ms | 1.4 ms | +| events stats (group by category) | 13.8 ms | 15.8 ms | +| memory list 50 | 4.1 ms (seq scan) | 0.5 ms | +| memory text search | 1.7 ms | 0.5 ms | +| memory fused retrieve | 31 ms | 27 ms | +| beliefs list | 3.6 ms (seq scan) | 3.6 ms | +| beliefs search / contested / uncertain / stats / recent changes | ≤ 3 ms | ≤ 3 ms | +| snapshot capture (all state tables) | 288 ms | 296 ms | + +Migration `0016_hot_path_indexes` adds the indexes the planner needed for the +two list orders that had fallen back to sequential scans, plus +`(mind_id, category, seq)` on events for replay's stimulus scan and +`(mind_id, created_at)` on belief versions for the recent-changes view. + +## What was deliberately left alone + +- **Fused retrieval (~27 ms)** is the most expensive per-cycle read; it runs a + ranked full-text query capped at 50 candidates plus optional vector search. + It is well inside the cycle budget; tune only if cycle latency becomes visible. +- **Snapshot capture (~300 ms)** serialises every state table; it is an + operator action, not a cycle path. +- **Event stats (~16 ms)** is a group-by over all of a mind's events, used on + the overview page. A materialised counter would be premature at this volume. + +## Rerun before touching queries + +The plan checks fail loudly when a future query drifts off its index, which is +the cheapest way to catch a regression that would otherwise only show up with +months of history. diff --git a/docs/screenshots/account.png b/docs/screenshots/account.png new file mode 100644 index 0000000..e9b493e Binary files /dev/null and b/docs/screenshots/account.png differ diff --git a/docs/screenshots/approvals.png b/docs/screenshots/approvals.png new file mode 100644 index 0000000..b7f36b3 Binary files /dev/null and b/docs/screenshots/approvals.png differ diff --git a/docs/screenshots/beliefs.png b/docs/screenshots/beliefs.png new file mode 100644 index 0000000..37e64d1 Binary files /dev/null and b/docs/screenshots/beliefs.png differ diff --git a/docs/screenshots/compare.png b/docs/screenshots/compare.png new file mode 100644 index 0000000..0c8d2ea Binary files /dev/null and b/docs/screenshots/compare.png differ diff --git a/docs/screenshots/conversation.png b/docs/screenshots/conversation.png new file mode 100644 index 0000000..5194652 Binary files /dev/null and b/docs/screenshots/conversation.png differ diff --git a/docs/screenshots/dashboard.png b/docs/screenshots/dashboard.png new file mode 100644 index 0000000..ea006ce Binary files /dev/null and b/docs/screenshots/dashboard.png differ diff --git a/docs/screenshots/dreams.png b/docs/screenshots/dreams.png new file mode 100644 index 0000000..2639905 Binary files /dev/null and b/docs/screenshots/dreams.png differ diff --git a/docs/screenshots/experiment.png b/docs/screenshots/experiment.png new file mode 100644 index 0000000..9605d9d Binary files /dev/null and b/docs/screenshots/experiment.png differ diff --git a/docs/screenshots/graph.png b/docs/screenshots/graph.png new file mode 100644 index 0000000..70f2cff Binary files /dev/null and b/docs/screenshots/graph.png differ diff --git a/docs/screenshots/integrations.png b/docs/screenshots/integrations.png new file mode 100644 index 0000000..c1dd6f2 Binary files /dev/null and b/docs/screenshots/integrations.png differ diff --git a/docs/screenshots/llm.png b/docs/screenshots/llm.png new file mode 100644 index 0000000..aedc311 Binary files /dev/null and b/docs/screenshots/llm.png differ diff --git a/docs/screenshots/memory.png b/docs/screenshots/memory.png new file mode 100644 index 0000000..cd30b19 Binary files /dev/null and b/docs/screenshots/memory.png differ diff --git a/docs/screenshots/mind.png b/docs/screenshots/mind.png new file mode 100644 index 0000000..8cc5d3b Binary files /dev/null and b/docs/screenshots/mind.png differ diff --git a/docs/screenshots/minds.png b/docs/screenshots/minds.png new file mode 100644 index 0000000..d48b2c5 Binary files /dev/null and b/docs/screenshots/minds.png differ diff --git a/docs/screenshots/policies.png b/docs/screenshots/policies.png new file mode 100644 index 0000000..7c9c64f Binary files /dev/null and b/docs/screenshots/policies.png differ diff --git a/docs/screenshots/replay.png b/docs/screenshots/replay.png new file mode 100644 index 0000000..866319c Binary files /dev/null and b/docs/screenshots/replay.png differ diff --git a/docs/screenshots/snapshots.png b/docs/screenshots/snapshots.png new file mode 100644 index 0000000..d22a5b4 Binary files /dev/null and b/docs/screenshots/snapshots.png differ diff --git a/docs/screenshots/thoughts.png b/docs/screenshots/thoughts.png new file mode 100644 index 0000000..b0aa27f Binary files /dev/null and b/docs/screenshots/thoughts.png differ diff --git a/docs/screenshots/workspace.png b/docs/screenshots/workspace.png new file mode 100644 index 0000000..e12e598 Binary files /dev/null and b/docs/screenshots/workspace.png differ diff --git a/migrations/0016_hot_path_indexes.down.sql b/migrations/0016_hot_path_indexes.down.sql new file mode 100644 index 0000000..a2972ed --- /dev/null +++ b/migrations/0016_hot_path_indexes.down.sql @@ -0,0 +1,4 @@ +DROP INDEX IF EXISTS belief_versions_mind_created_idx; +DROP INDEX IF EXISTS cognitive_events_mind_category_seq_idx; +DROP INDEX IF EXISTS beliefs_mind_confidence_idx; +DROP INDEX IF EXISTS memories_mind_recent_idx; diff --git a/migrations/0016_hot_path_indexes.up.sql b/migrations/0016_hot_path_indexes.up.sql new file mode 100644 index 0000000..d885456 --- /dev/null +++ b/migrations/0016_hot_path_indexes.up.sql @@ -0,0 +1,7 @@ +-- Performance pass (Phase 16): indexes matching the hot list orders found by +-- tests/perf. The planner chose sequential scans because the existing +-- (mind_id, kind|status) indexes could not serve the ORDER BY. +CREATE INDEX IF NOT EXISTS memories_mind_recent_idx ON memories (mind_id, occurred_at DESC, id DESC) WHERE status = 'active'; +CREATE INDEX IF NOT EXISTS beliefs_mind_confidence_idx ON beliefs (mind_id, confidence DESC) WHERE status <> 'superseded'; +CREATE INDEX IF NOT EXISTS cognitive_events_mind_category_seq_idx ON cognitive_events (mind_id, category, seq); +CREATE INDEX IF NOT EXISTS belief_versions_mind_created_idx ON belief_versions (mind_id, created_at DESC); diff --git a/tests/perf/perf_test.go b/tests/perf/perf_test.go new file mode 100644 index 0000000..52d3579 --- /dev/null +++ b/tests/perf/perf_test.go @@ -0,0 +1,191 @@ +// Package perf seeds a mind with a realistic volume of state and times the +// hot read paths. It runs only with NOEMA_PERF=1 because seeding takes a +// while; CI runs it nightly, developers run it before touching queries. +// +// NOEMA_PERF=1 NOEMA_TEST_DATABASE_URL=... go test -run TestHotPaths -v ./tests/perf/ +package perf + +import ( + "context" + "fmt" + "os" + "strings" + "testing" + "time" + + "github.com/andydixon/noema/internal/beliefs" + evstore "github.com/andydixon/noema/internal/events" + "github.com/andydixon/noema/internal/memory" + "github.com/andydixon/noema/internal/minds" + "github.com/andydixon/noema/internal/snapshots" + "github.com/andydixon/noema/internal/storage" + "github.com/andydixon/noema/migrations" + "github.com/andydixon/noema/pkg/concepts" + "github.com/andydixon/noema/pkg/events" + "github.com/andydixon/noema/pkg/ids" +) + +const ( + nEvents = 60_000 + nMemories = 6_000 + nBeliefs = 1_500 + ceiling = 400 * time.Millisecond // any hot read slower than this is a regression +) + +var vocab = strings.Fields("payments database migration latency deploy atlas server disk cpu alert go rust python andy project service host backup replica queue cache timeout retry release rollback metric dashboard incident postmortem") + +func phrase(i int) string { + var w []string + for k := 0; k < 8; k++ { + w = append(w, vocab[(i*7+k*3)%len(vocab)]) + } + return fmt.Sprintf("%s (item %d)", strings.Join(w, " "), i) +} + +func TestHotPaths(t *testing.T) { + if os.Getenv("NOEMA_PERF") == "" { + t.Skip("set NOEMA_PERF=1 to run the performance pass") + } + pool := storage.TestPool(t, migrations.FS) + ctx := context.Background() + ms := minds.NewStore(pool) + es := evstore.NewStore(pool, evstore.NewBus(8), nil) + mem := memory.NewStore(ctx, pool, nil) + bs := beliefs.NewStore(pool) + m, err := ms.Create(ctx, minds.CreateParams{Slug: "perf", DisplayName: "Perf"}) + if err != nil { + t.Fatal(err) + } + other, _ := ms.Create(ctx, minds.CreateParams{Slug: "noise", DisplayName: "Noise"}) + + seed := time.Now() + batch := make([]events.New, 0, 500) + flush := func(mind ids.ID) { + if len(batch) == 0 { + return + } + if _, err := es.Append(ctx, batch...); err != nil { + t.Fatal(err) + } + batch = batch[:0] + } + types := []events.Type{events.Observation, events.ThoughtSelected, events.MemoryRetrieved, events.BeliefUpdated, events.WorkspaceBroadcast, events.CycleCompleted} + for i := 0; i < nEvents; i++ { + mind := m.ID + if i%5 == 0 { + mind = other.ID // a second mind so scoping matters + } + batch = append(batch, events.New{MindID: mind, Type: types[i%len(types)], Source: "perf", OccurredAt: seed.Add(-time.Duration(nEvents-i) * time.Second), Summary: phrase(i), Salience: float64(i%10) / 10, Confidence: 0.8}) + if len(batch) == 500 { + flush(mind) + } + } + flush(m.ID) + for i := 0; i < nMemories; i++ { + kind := concepts.Episodic + if i%4 == 0 { + kind = concepts.Semantic + } + if _, err := mem.Create(ctx, pool, memory.New{MindID: m.ID, Kind: kind, Epistemic: concepts.Observation, Content: phrase(i), Source: "perf", Trust: events.TrustUser, Confidence: 0.7, Importance: float64(i%10) / 10, OccurredAt: seed.Add(-time.Duration(i) * time.Minute)}); err != nil { + t.Fatal(err) + } + } + emit := func(ctx context.Context, q storage.Querier, n events.New) (events.Event, error) { + out, err := es.AppendIn(ctx, q, n) + if err != nil { + return events.Event{}, err + } + return out[0], nil + } + for i := 0; i < nBeliefs; i++ { + if _, err := bs.Create(ctx, pool, emit, beliefs.New{MindID: m.ID, Proposition: phrase(i * 3), Subject: vocab[i%len(vocab)], Epistemic: concepts.Claim, Source: "perf", + Evidence: []beliefs.EvidenceInput{{Direction: "for", Kind: "memory", RefID: ids.New().String(), Source: "user", Weight: 0.4, Credibility: 0.8}}}); err != nil { + t.Fatal(err) + } + } + t.Logf("seeded %d events, %d memories, %d beliefs in %s", nEvents, nMemories, nBeliefs, time.Since(seed).Round(time.Millisecond)) + + type probe struct { + name string + fn func() error + } + probes := []probe{ + {"events recent 50", func() error { _, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Limit: 50}); return err }}, + {"events by type", func() error { + _, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Types: []events.Type{events.BeliefUpdated}, Limit: 50}) + return err + }}, + {"events after seq (cursor)", func() error { + _, err := es.Query(ctx, evstore.Filter{MindID: m.ID, AfterSeq: 1000, Ascending: true, Limit: 200}) + return err + }}, + {"events search", func() error { + _, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Search: "rollback", Limit: 50}) + return err + }}, + {"events stats", func() error { _, err := es.Stats(ctx, m.ID, time.Time{}); return err }}, + {"memory list 50", func() error { _, err := mem.List(ctx, m.ID, memory.Filter{Limit: 50}); return err }}, + {"memory text search", func() error { + _, err := mem.List(ctx, m.ID, memory.Filter{Search: "payments latency", Limit: 50}) + return err + }}, + {"memory fused retrieve", func() error { + _, err := mem.Retrieve(ctx, m.ID, memory.Query{Text: "database migration latency alert", Limit: 10}) + return err + }}, + {"memory stats", func() error { _, err := mem.Stats(ctx, m.ID); return err }}, + {"beliefs list", func() error { _, err := bs.List(ctx, m.ID, beliefs.Filter{Limit: 50}); return err }}, + {"beliefs search", func() error { _, err := bs.Search(ctx, m.ID, "deploy rollback", 20); return err }}, + {"beliefs contested", func() error { _, err := bs.Contested(ctx, m.ID, 20); return err }}, + {"beliefs uncertain", func() error { _, err := bs.Uncertain(ctx, m.ID, 20); return err }}, + {"beliefs stats", func() error { _, err := bs.Stats(ctx, m.ID); return err }}, + {"beliefs recent changes", func() error { _, err := bs.RecentChanges(ctx, m.ID, seed.Add(-time.Hour), 50); return err }}, + {"snapshot capture", func() error { _, err := snapshots.Capture(ctx, pool, es, m); return err }}, + } + failed := false + for _, p := range probes { + p.fn() //nolint:errcheck // warm the cache once; we time the steady state + start := time.Now() + var err error + for i := 0; i < 3; i++ { + err = p.fn() + } + d := time.Since(start) / 3 + mark := "" + if d > ceiling { + mark, failed = " <-- over ceiling", true + } + t.Logf("%-28s %8s%s", p.name, d.Round(100*time.Microsecond), mark) + if err != nil { + t.Errorf("%s: %v", p.name, err) + } + } + // Plans: the hot event and memory queries must be index-driven. + plans := map[string]string{ + "events recent": `EXPLAIN SELECT id FROM cognitive_events WHERE mind_id=$1 ORDER BY seq DESC LIMIT 50`, + "events cursor": `EXPLAIN SELECT id FROM cognitive_events WHERE mind_id=$1 AND seq > 1000 ORDER BY seq ASC LIMIT 200`, + "memory list": `EXPLAIN SELECT id FROM memories WHERE mind_id=$1 AND status='active' ORDER BY occurred_at DESC, id DESC LIMIT 50`, + "beliefs list": `EXPLAIN SELECT id FROM beliefs WHERE mind_id=$1 AND status NOT IN ('superseded') ORDER BY confidence DESC LIMIT 50`, + } + for name, q := range plans { + rows, err := pool.Query(ctx, q, m.ID) + if err != nil { + t.Fatal(err) + } + var plan []string + for rows.Next() { + var line string + rows.Scan(&line) //nolint:errcheck + plan = append(plan, strings.TrimSpace(line)) + } + rows.Close() + joined := strings.Join(plan, " | ") + if strings.Contains(joined, "Seq Scan on cognitive_events") || strings.Contains(joined, "Seq Scan on memories") || strings.Contains(joined, "Seq Scan on beliefs") { + t.Errorf("%s uses a sequential scan: %s", name, joined) + } + t.Logf("plan %-14s %s", name, plan[0]) + } + if failed { + t.Fatalf("a hot read exceeded %s", ceiling) + } +}