Close the plan: screenshots and a measured performance pass
- docs/screenshots: nineteen pages captured with headless Chromium from a running instance; README gains a screenshot section - tests/perf: opt-in load test (NOEMA_PERF=1) seeding 60k events, 6k memories and 1.5k beliefs across two minds, timing sixteen hot reads against a 400 ms ceiling and rejecting sequential-scan plans - migration 0016: indexes the test found missing (memories by mind and recency, beliefs by mind and confidence, events by mind and category, belief versions by mind and time) - docs/performance.md records the numbers and how to rerun
@@ -171,8 +171,8 @@ An administrator can: start Noema → log in → create a mind → configure per
|
||||
|
||||
### Phase 16 – Hardening and release
|
||||
- ☑ TOTP + recovery codes, security test suite (`tests/security`), OpenAPI document (`docs/openapi.json`, served at `/api/v1/openapi.json`, coverage-tested), docs set, `go test -race`, `go vet`, staticcheck, gosec, govulncheck clean, release notes
|
||||
- ☐ screenshots (no browser available in the build environment; capture from a running instance and add to `docs/screenshots/`)
|
||||
- ☐ performance pass beyond indexes already in migrations (profile under a realistic load before tuning)
|
||||
- ☑ screenshots captured with headless Chromium into `docs/screenshots/` (19 pages)
|
||||
- ☑ performance pass: `tests/perf` seeds 60k events / 6k memories / 1.5k beliefs, times hot reads against a ceiling and rejects sequential-scan plans; migration 0016 adds the indexes it found missing (see `docs/performance.md`)
|
||||
|
||||
## 5. Deterministic vs LLM decisions
|
||||
See `ARCHITECTURE.md` §14. Rule: if a unit test can assert the exact output, it is code.
|
||||
|
||||
@@ -38,6 +38,16 @@ Noema is also not a chatbot with a memory bolted on. A user message does not go
|
||||
|
||||
Full description: [ARCHITECTURE.md](ARCHITECTURE.md). Decisions: [docs/adr](docs/adr). Plan and status: [PLAN.md](PLAN.md).
|
||||
|
||||
## Screenshots
|
||||
|
||||
Captured from a running instance with the sample mind Dani (full set in [docs/screenshots](docs/screenshots)).
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
|  |  |
|
||||
|  |  |
|
||||
|  |  |
|
||||
|
||||
## Quick start
|
||||
|
||||
Requirements: Go 1.26+, Docker with Compose, `make`.
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
# Performance pass
|
||||
|
||||
`tests/perf` is an opt-in load test. It seeds a fresh schema with a realistic
|
||||
volume for one mind plus a second "noise" mind (so scoping matters), times
|
||||
the hot read paths in steady state, and rejects any plan that sequentially
|
||||
scans the three large tables.
|
||||
|
||||
```
|
||||
NOEMA_PERF=1 NOEMA_TEST_DATABASE_URL=postgres://... go test -count=1 -v -run TestHotPaths ./tests/perf/
|
||||
```
|
||||
|
||||
Seed: 60 000 events, 6 000 memories, 1 500 beliefs with evidence. Ceiling per
|
||||
read: 400 ms (a regression guard, not a target).
|
||||
|
||||
## Results (PostgreSQL 16 in Docker, laptop, 2026-09-14)
|
||||
|
||||
| Read | Before 0016 | After 0016 |
|
||||
|---|---|---|
|
||||
| events recent 50 | 3.2 ms | 1.2 ms |
|
||||
| events by type | 1.8 ms | 1.1 ms |
|
||||
| events after seq (runtime cursor) | 2.2 ms | 1.2 ms |
|
||||
| events summary search | 3.3 ms | 1.4 ms |
|
||||
| events stats (group by category) | 13.8 ms | 15.8 ms |
|
||||
| memory list 50 | 4.1 ms (seq scan) | 0.5 ms |
|
||||
| memory text search | 1.7 ms | 0.5 ms |
|
||||
| memory fused retrieve | 31 ms | 27 ms |
|
||||
| beliefs list | 3.6 ms (seq scan) | 3.6 ms |
|
||||
| beliefs search / contested / uncertain / stats / recent changes | ≤ 3 ms | ≤ 3 ms |
|
||||
| snapshot capture (all state tables) | 288 ms | 296 ms |
|
||||
|
||||
Migration `0016_hot_path_indexes` adds the indexes the planner needed for the
|
||||
two list orders that had fallen back to sequential scans, plus
|
||||
`(mind_id, category, seq)` on events for replay's stimulus scan and
|
||||
`(mind_id, created_at)` on belief versions for the recent-changes view.
|
||||
|
||||
## What was deliberately left alone
|
||||
|
||||
- **Fused retrieval (~27 ms)** is the most expensive per-cycle read; it runs a
|
||||
ranked full-text query capped at 50 candidates plus optional vector search.
|
||||
It is well inside the cycle budget; tune only if cycle latency becomes visible.
|
||||
- **Snapshot capture (~300 ms)** serialises every state table; it is an
|
||||
operator action, not a cycle path.
|
||||
- **Event stats (~16 ms)** is a group-by over all of a mind's events, used on
|
||||
the overview page. A materialised counter would be premature at this volume.
|
||||
|
||||
## Rerun before touching queries
|
||||
|
||||
The plan checks fail loudly when a future query drifts off its index, which is
|
||||
the cheapest way to catch a regression that would otherwise only show up with
|
||||
months of history.
|
||||
|
After Width: | Height: | Size: 110 KiB |
|
After Width: | Height: | Size: 72 KiB |
|
After Width: | Height: | Size: 163 KiB |
|
After Width: | Height: | Size: 97 KiB |
|
After Width: | Height: | Size: 70 KiB |
|
After Width: | Height: | Size: 167 KiB |
|
After Width: | Height: | Size: 164 KiB |
|
After Width: | Height: | Size: 139 KiB |
|
After Width: | Height: | Size: 112 KiB |
|
After Width: | Height: | Size: 88 KiB |
|
After Width: | Height: | Size: 107 KiB |
|
After Width: | Height: | Size: 152 KiB |
|
After Width: | Height: | Size: 153 KiB |
|
After Width: | Height: | Size: 127 KiB |
|
After Width: | Height: | Size: 118 KiB |
|
After Width: | Height: | Size: 165 KiB |
|
After Width: | Height: | Size: 112 KiB |
|
After Width: | Height: | Size: 145 KiB |
|
After Width: | Height: | Size: 129 KiB |
@@ -0,0 +1,4 @@
|
||||
DROP INDEX IF EXISTS belief_versions_mind_created_idx;
|
||||
DROP INDEX IF EXISTS cognitive_events_mind_category_seq_idx;
|
||||
DROP INDEX IF EXISTS beliefs_mind_confidence_idx;
|
||||
DROP INDEX IF EXISTS memories_mind_recent_idx;
|
||||
@@ -0,0 +1,7 @@
|
||||
-- Performance pass (Phase 16): indexes matching the hot list orders found by
|
||||
-- tests/perf. The planner chose sequential scans because the existing
|
||||
-- (mind_id, kind|status) indexes could not serve the ORDER BY.
|
||||
CREATE INDEX IF NOT EXISTS memories_mind_recent_idx ON memories (mind_id, occurred_at DESC, id DESC) WHERE status = 'active';
|
||||
CREATE INDEX IF NOT EXISTS beliefs_mind_confidence_idx ON beliefs (mind_id, confidence DESC) WHERE status <> 'superseded';
|
||||
CREATE INDEX IF NOT EXISTS cognitive_events_mind_category_seq_idx ON cognitive_events (mind_id, category, seq);
|
||||
CREATE INDEX IF NOT EXISTS belief_versions_mind_created_idx ON belief_versions (mind_id, created_at DESC);
|
||||
@@ -0,0 +1,191 @@
|
||||
// Package perf seeds a mind with a realistic volume of state and times the
|
||||
// hot read paths. It runs only with NOEMA_PERF=1 because seeding takes a
|
||||
// while; CI runs it nightly, developers run it before touching queries.
|
||||
//
|
||||
// NOEMA_PERF=1 NOEMA_TEST_DATABASE_URL=... go test -run TestHotPaths -v ./tests/perf/
|
||||
package perf
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/andydixon/noema/internal/beliefs"
|
||||
evstore "github.com/andydixon/noema/internal/events"
|
||||
"github.com/andydixon/noema/internal/memory"
|
||||
"github.com/andydixon/noema/internal/minds"
|
||||
"github.com/andydixon/noema/internal/snapshots"
|
||||
"github.com/andydixon/noema/internal/storage"
|
||||
"github.com/andydixon/noema/migrations"
|
||||
"github.com/andydixon/noema/pkg/concepts"
|
||||
"github.com/andydixon/noema/pkg/events"
|
||||
"github.com/andydixon/noema/pkg/ids"
|
||||
)
|
||||
|
||||
const (
|
||||
nEvents = 60_000
|
||||
nMemories = 6_000
|
||||
nBeliefs = 1_500
|
||||
ceiling = 400 * time.Millisecond // any hot read slower than this is a regression
|
||||
)
|
||||
|
||||
var vocab = strings.Fields("payments database migration latency deploy atlas server disk cpu alert go rust python andy project service host backup replica queue cache timeout retry release rollback metric dashboard incident postmortem")
|
||||
|
||||
func phrase(i int) string {
|
||||
var w []string
|
||||
for k := 0; k < 8; k++ {
|
||||
w = append(w, vocab[(i*7+k*3)%len(vocab)])
|
||||
}
|
||||
return fmt.Sprintf("%s (item %d)", strings.Join(w, " "), i)
|
||||
}
|
||||
|
||||
func TestHotPaths(t *testing.T) {
|
||||
if os.Getenv("NOEMA_PERF") == "" {
|
||||
t.Skip("set NOEMA_PERF=1 to run the performance pass")
|
||||
}
|
||||
pool := storage.TestPool(t, migrations.FS)
|
||||
ctx := context.Background()
|
||||
ms := minds.NewStore(pool)
|
||||
es := evstore.NewStore(pool, evstore.NewBus(8), nil)
|
||||
mem := memory.NewStore(ctx, pool, nil)
|
||||
bs := beliefs.NewStore(pool)
|
||||
m, err := ms.Create(ctx, minds.CreateParams{Slug: "perf", DisplayName: "Perf"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
other, _ := ms.Create(ctx, minds.CreateParams{Slug: "noise", DisplayName: "Noise"})
|
||||
|
||||
seed := time.Now()
|
||||
batch := make([]events.New, 0, 500)
|
||||
flush := func(mind ids.ID) {
|
||||
if len(batch) == 0 {
|
||||
return
|
||||
}
|
||||
if _, err := es.Append(ctx, batch...); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
batch = batch[:0]
|
||||
}
|
||||
types := []events.Type{events.Observation, events.ThoughtSelected, events.MemoryRetrieved, events.BeliefUpdated, events.WorkspaceBroadcast, events.CycleCompleted}
|
||||
for i := 0; i < nEvents; i++ {
|
||||
mind := m.ID
|
||||
if i%5 == 0 {
|
||||
mind = other.ID // a second mind so scoping matters
|
||||
}
|
||||
batch = append(batch, events.New{MindID: mind, Type: types[i%len(types)], Source: "perf", OccurredAt: seed.Add(-time.Duration(nEvents-i) * time.Second), Summary: phrase(i), Salience: float64(i%10) / 10, Confidence: 0.8})
|
||||
if len(batch) == 500 {
|
||||
flush(mind)
|
||||
}
|
||||
}
|
||||
flush(m.ID)
|
||||
for i := 0; i < nMemories; i++ {
|
||||
kind := concepts.Episodic
|
||||
if i%4 == 0 {
|
||||
kind = concepts.Semantic
|
||||
}
|
||||
if _, err := mem.Create(ctx, pool, memory.New{MindID: m.ID, Kind: kind, Epistemic: concepts.Observation, Content: phrase(i), Source: "perf", Trust: events.TrustUser, Confidence: 0.7, Importance: float64(i%10) / 10, OccurredAt: seed.Add(-time.Duration(i) * time.Minute)}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
emit := func(ctx context.Context, q storage.Querier, n events.New) (events.Event, error) {
|
||||
out, err := es.AppendIn(ctx, q, n)
|
||||
if err != nil {
|
||||
return events.Event{}, err
|
||||
}
|
||||
return out[0], nil
|
||||
}
|
||||
for i := 0; i < nBeliefs; i++ {
|
||||
if _, err := bs.Create(ctx, pool, emit, beliefs.New{MindID: m.ID, Proposition: phrase(i * 3), Subject: vocab[i%len(vocab)], Epistemic: concepts.Claim, Source: "perf",
|
||||
Evidence: []beliefs.EvidenceInput{{Direction: "for", Kind: "memory", RefID: ids.New().String(), Source: "user", Weight: 0.4, Credibility: 0.8}}}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
t.Logf("seeded %d events, %d memories, %d beliefs in %s", nEvents, nMemories, nBeliefs, time.Since(seed).Round(time.Millisecond))
|
||||
|
||||
type probe struct {
|
||||
name string
|
||||
fn func() error
|
||||
}
|
||||
probes := []probe{
|
||||
{"events recent 50", func() error { _, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Limit: 50}); return err }},
|
||||
{"events by type", func() error {
|
||||
_, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Types: []events.Type{events.BeliefUpdated}, Limit: 50})
|
||||
return err
|
||||
}},
|
||||
{"events after seq (cursor)", func() error {
|
||||
_, err := es.Query(ctx, evstore.Filter{MindID: m.ID, AfterSeq: 1000, Ascending: true, Limit: 200})
|
||||
return err
|
||||
}},
|
||||
{"events search", func() error {
|
||||
_, err := es.Query(ctx, evstore.Filter{MindID: m.ID, Search: "rollback", Limit: 50})
|
||||
return err
|
||||
}},
|
||||
{"events stats", func() error { _, err := es.Stats(ctx, m.ID, time.Time{}); return err }},
|
||||
{"memory list 50", func() error { _, err := mem.List(ctx, m.ID, memory.Filter{Limit: 50}); return err }},
|
||||
{"memory text search", func() error {
|
||||
_, err := mem.List(ctx, m.ID, memory.Filter{Search: "payments latency", Limit: 50})
|
||||
return err
|
||||
}},
|
||||
{"memory fused retrieve", func() error {
|
||||
_, err := mem.Retrieve(ctx, m.ID, memory.Query{Text: "database migration latency alert", Limit: 10})
|
||||
return err
|
||||
}},
|
||||
{"memory stats", func() error { _, err := mem.Stats(ctx, m.ID); return err }},
|
||||
{"beliefs list", func() error { _, err := bs.List(ctx, m.ID, beliefs.Filter{Limit: 50}); return err }},
|
||||
{"beliefs search", func() error { _, err := bs.Search(ctx, m.ID, "deploy rollback", 20); return err }},
|
||||
{"beliefs contested", func() error { _, err := bs.Contested(ctx, m.ID, 20); return err }},
|
||||
{"beliefs uncertain", func() error { _, err := bs.Uncertain(ctx, m.ID, 20); return err }},
|
||||
{"beliefs stats", func() error { _, err := bs.Stats(ctx, m.ID); return err }},
|
||||
{"beliefs recent changes", func() error { _, err := bs.RecentChanges(ctx, m.ID, seed.Add(-time.Hour), 50); return err }},
|
||||
{"snapshot capture", func() error { _, err := snapshots.Capture(ctx, pool, es, m); return err }},
|
||||
}
|
||||
failed := false
|
||||
for _, p := range probes {
|
||||
p.fn() //nolint:errcheck // warm the cache once; we time the steady state
|
||||
start := time.Now()
|
||||
var err error
|
||||
for i := 0; i < 3; i++ {
|
||||
err = p.fn()
|
||||
}
|
||||
d := time.Since(start) / 3
|
||||
mark := ""
|
||||
if d > ceiling {
|
||||
mark, failed = " <-- over ceiling", true
|
||||
}
|
||||
t.Logf("%-28s %8s%s", p.name, d.Round(100*time.Microsecond), mark)
|
||||
if err != nil {
|
||||
t.Errorf("%s: %v", p.name, err)
|
||||
}
|
||||
}
|
||||
// Plans: the hot event and memory queries must be index-driven.
|
||||
plans := map[string]string{
|
||||
"events recent": `EXPLAIN SELECT id FROM cognitive_events WHERE mind_id=$1 ORDER BY seq DESC LIMIT 50`,
|
||||
"events cursor": `EXPLAIN SELECT id FROM cognitive_events WHERE mind_id=$1 AND seq > 1000 ORDER BY seq ASC LIMIT 200`,
|
||||
"memory list": `EXPLAIN SELECT id FROM memories WHERE mind_id=$1 AND status='active' ORDER BY occurred_at DESC, id DESC LIMIT 50`,
|
||||
"beliefs list": `EXPLAIN SELECT id FROM beliefs WHERE mind_id=$1 AND status NOT IN ('superseded') ORDER BY confidence DESC LIMIT 50`,
|
||||
}
|
||||
for name, q := range plans {
|
||||
rows, err := pool.Query(ctx, q, m.ID)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var plan []string
|
||||
for rows.Next() {
|
||||
var line string
|
||||
rows.Scan(&line) //nolint:errcheck
|
||||
plan = append(plan, strings.TrimSpace(line))
|
||||
}
|
||||
rows.Close()
|
||||
joined := strings.Join(plan, " | ")
|
||||
if strings.Contains(joined, "Seq Scan on cognitive_events") || strings.Contains(joined, "Seq Scan on memories") || strings.Contains(joined, "Seq Scan on beliefs") {
|
||||
t.Errorf("%s uses a sequential scan: %s", name, joined)
|
||||
}
|
||||
t.Logf("plan %-14s %s", name, plan[0])
|
||||
}
|
||||
if failed {
|
||||
t.Fatalf("a hot read exceeded %s", ceiling)
|
||||
}
|
||||
}
|
||||