docs: static documentation site generator and authored pages
Twenty-six pages covering concepts, architecture, every subsystem, an interface tour with screenshots, the full API route table generated from openapi.json, the CLI, security and threat model, operations, ADRs, development history and a glossary. Built into the dixon.cx page skeleton, unlinked and noindex.
This commit is contained in:
Andy Dixon
@@ -0,0 +1,15 @@
|
||||
# Documentation site
|
||||
|
||||
`build.py` renders the documentation section from the repository's Markdown,
|
||||
the OpenAPI document and the screenshots into static HTML that follows the
|
||||
dixon.cx page skeleton. Authored pages live in `pages/`; a page pulls a
|
||||
repository document in with `{{include path [demote=N] [keep-title]}}` and the
|
||||
API page uses `{{openapi}}`.
|
||||
|
||||
```
|
||||
python3 -m venv .venv && .venv/bin/pip install markdown
|
||||
.venv/bin/python docs/site/build.py /path/to/public_html/noema
|
||||
```
|
||||
|
||||
The output is intentionally not linked from the site navigation and carries a
|
||||
`noindex` robots directive.
|
||||
@@ -0,0 +1,363 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the Noema documentation section as static HTML.
|
||||
|
||||
Usage: python3 docs/site/build.py OUTPUT_DIR
|
||||
|
||||
Sources: the repository's Markdown documents, the OpenAPI document, the UI
|
||||
screenshots, and the authored pages in docs/site/pages/*.md. A page file may
|
||||
pull a repository document in with a line `{{include path [demote]}}`.
|
||||
The output follows the dixon.cx page skeleton; the section is deliberately
|
||||
not linked from the site navigation and is marked noindex.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
import markdown
|
||||
|
||||
ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||
PAGES = os.path.join(ROOT, "docs", "site", "pages")
|
||||
BASE = "/noema/"
|
||||
SITE = "https://dixon.cx"
|
||||
|
||||
# Order, slug, title, summary. The page body comes from pages/<slug>.md.
|
||||
CHAPTERS = [
|
||||
("index", "Overview", "What Noema is, what it refuses to claim, and how the pieces fit."),
|
||||
("concepts", "Concepts and vocabulary", "Minds, events, epistemic kinds, memory kinds, beliefs, drives and the interpretation boundary."),
|
||||
("architecture", "Architecture", "The daemon, the cognitive cycle, the storage model and the boundaries between deterministic code and language models."),
|
||||
("getting-started", "Getting started", "From an empty machine to a running mind you can talk to, inspect and replay."),
|
||||
("configuration", "Configuration", "Environment, config file, secret references, runtime settings and the per-mind cognitive configuration."),
|
||||
("personality", "Identity and personality", "Versioned personalities, templates, traits and how they modulate attention and voice."),
|
||||
("cognition", "The cognitive cycle", "Processes, attention scoring, the global workspace, executive dispatch, modes and metacognition."),
|
||||
("memory", "Memory", "Episodic, semantic, procedural and autobiographical memory; retrieval, decay, consolidation and provenance."),
|
||||
("beliefs", "Beliefs and evidence", "Evidence-derived confidence, statuses, contradictions, alternatives and operator corrections."),
|
||||
("conversation", "Conversation", "How a message becomes an observation, a perception, a response intent and finally language."),
|
||||
("motivation", "Goals, drives and affect", "The goal tree, the closed drive set, rule-driven affect and how they bias attention."),
|
||||
("predictions", "Predictions and calibration", "Explicit predictions with criteria, deterministic resolution, prediction error and calibration."),
|
||||
("graph", "Relationship graph", "Entities, evidenced edges, merges and the canvas explorer."),
|
||||
("llm-providers", "Language models", "Roles, providers, routing, budgets, recorded calls, replay and versioned prompts."),
|
||||
("capabilities", "Capabilities and policy", "The only path to action: ALLOW / ASK / DENY, approvals, locked capabilities and tool output as data."),
|
||||
("integrations", "Integrations", "Research folder, git mirrors, Prometheus and signed webhooks as provenance-tagged observations."),
|
||||
("dreaming", "Dreaming", "The opt-in idle mode: replay, association and speculative hypotheses that can never act."),
|
||||
("snapshots-replay", "Snapshots, forks, replay and experiments", "Capturing state, branching minds, re-running cognition and running controlled experiments."),
|
||||
("ui-tour", "Interface tour", "Every screen of the web interface, what it shows and what an operator does there."),
|
||||
("api", "HTTP API", "Authentication, roles, the error envelope, streaming and every route."),
|
||||
("cli", "Command line", "noemactl: every command with examples."),
|
||||
("security", "Security and threat model", "Trust boundaries, secret handling, sessions, two-factor authentication and the threat-by-threat verification table."),
|
||||
("operations", "Operating Noema", "Deployment, backups, scheduled jobs, metrics, audit, performance and upgrades."),
|
||||
("decisions", "Architecture decision records", "The recorded decisions and the alternatives they rejected."),
|
||||
("development", "Development and history", "Contributing, testing, the phased plan, changelog and release notes."),
|
||||
("glossary", "Glossary", "Short definitions of every term used across the documentation."),
|
||||
]
|
||||
|
||||
STYLE = """
|
||||
<style>
|
||||
.noema-doc pre{background:rgba(127,127,127,.12);border:1px solid rgba(127,127,127,.25);border-radius:6px;padding:.9rem 1rem;overflow:auto;font-size:.85rem;line-height:1.45}
|
||||
.noema-doc code{font-size:.9em;word-break:break-word}
|
||||
.noema-doc pre code{word-break:normal;white-space:pre}
|
||||
.noema-doc table{width:100%;border-collapse:collapse;margin:1rem 0;font-size:.92rem}
|
||||
.noema-doc th,.noema-doc td{border:1px solid rgba(127,127,127,.3);padding:.45rem .6rem;vertical-align:top;text-align:left}
|
||||
.noema-doc th{background:rgba(127,127,127,.12)}
|
||||
.noema-doc .table-wrap{overflow-x:auto}
|
||||
.noema-doc h3{font-size:1.05rem;margin-top:1.4rem}
|
||||
.noema-doc h4{font-size:.95rem;margin-top:1.1rem;font-weight:600}
|
||||
.noema-doc blockquote{border-left:3px solid rgba(127,127,127,.5);margin:1rem 0;padding:.2rem 1rem;opacity:.9}
|
||||
.noema-doc figure{margin:1.2rem 0}.noema-doc figure img{width:100%;height:auto;border:1px solid rgba(127,127,127,.3);border-radius:6px}
|
||||
.noema-doc figcaption{font-size:.85rem;opacity:.8;margin-top:.4rem}
|
||||
.noema-doc .contents ol{columns:2;column-gap:2rem}@media(max-width:700px){.noema-doc .contents ol{columns:1}}
|
||||
.noema-doc .method{display:inline-block;min-width:4.2rem;font-family:monospace;font-weight:700;font-size:.8rem}
|
||||
.noema-doc .role{font-size:.78rem;opacity:.75;font-family:monospace}
|
||||
.noema-doc dl dt{font-weight:600;margin-top:.6rem}.noema-doc dl dd{margin-left:0}
|
||||
.noema-doc .section-box p+p,.noema-doc .section-box p+ul,.noema-doc .section-box p+ol,.noema-doc .section-box ul+p,.noema-doc .section-box ol+p,.noema-doc .section-box .table-wrap+p,.noema-doc .section-box pre+p{margin-top:.9rem}
|
||||
.noema-doc .lead-ish{font-weight:600;opacity:.9;margin-bottom:1rem}
|
||||
</style>
|
||||
"""
|
||||
|
||||
HEAD = """<!doctype html>
|
||||
<html lang="en-GB">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
|
||||
<meta name="description" content="{desc}">
|
||||
<meta name="author" content="Andy Dixon">
|
||||
<meta name="robots" content="noindex, nofollow">
|
||||
<link rel="canonical" href="{canonical}">
|
||||
<title>{title} | Noema | Andy Dixon</title>
|
||||
<meta property="og:type" content="article">
|
||||
<meta property="og:title" content="{title} | Noema">
|
||||
<meta property="og:description" content="{desc}">
|
||||
<meta property="og:url" content="{canonical}">
|
||||
<meta property="og:site_name" content="Andy Dixon">
|
||||
<meta property="og:locale" content="en_GB">
|
||||
<link href="/assets/images/favicon.png" rel="icon" type="image/png">
|
||||
<link href="/assets/plugins/bootstrap/bootstrap.min.css" rel="stylesheet">
|
||||
<link href="/assets/plugins/magnific-popup/magnific-popup.min.css" rel="stylesheet">
|
||||
<link href="/assets/plugins/swiper/swiper-bundle.min.css" rel="stylesheet">
|
||||
<link href="/assets/css/theme.css" rel="stylesheet">
|
||||
<link href="/assets/plugins/bootstrap-icons/bootstrap-icons.css" rel="stylesheet">
|
||||
<link href="/assets/plugins/fontawesome/css/all.css" rel="stylesheet">
|
||||
<link href="/css/site-custom.css" rel="stylesheet">
|
||||
{style}
|
||||
</head>
|
||||
<body data-preloader="true">
|
||||
<script>document.body.classList.toggle('theme-dark', (localStorage.getItem('theme') || (matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'light')) === 'dark');</script>
|
||||
<a class="skip-link" href="#main-content">Skip to content</a>
|
||||
|
||||
<div class="toggle-menu">
|
||||
<button class="toggle-close" aria-label="Close menu"><i class="bi bi-x"></i></button>
|
||||
<h6 class="mono-heading fw-normal mb-2">Phone:</h6>
|
||||
<button type="button" class="phone-reveal" data-reveal-phone>Reveal phone number</button>
|
||||
<p class="phone-reveal-note">Hidden from the page source until you click: friction against scrapers, not a guarantee.</p>
|
||||
<div class="mt-4">
|
||||
<h6 class="mono-heading fw-normal mb-2">Email:</h6>
|
||||
<h4 class="fw-medium"><a class="link-decoration" href="mailto:web@dixon.cx">web@dixon.cx</a></h4>
|
||||
<ul class="list-inline-sm mt-3">
|
||||
<li><a class="button-circle button-circle-sm" href="https://github.com/andydixon" target="_blank" rel="noopener" aria-label="GitHub"><i class="bi bi-github"></i></a></li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="mt-4 mt-lg-5">
|
||||
<ul class="list-circle">
|
||||
<li><a class="mono-heading link-decoration" href="/about">About</a></li>
|
||||
<li><a class="mono-heading link-decoration" href="/blog">Blog</a></li>
|
||||
<li><a class="mono-heading link-decoration" href="/qualifications">Qualifications</a></li>
|
||||
<li><a class="mono-heading link-decoration" href="/side-quests">Side Quests</a></li>
|
||||
<li><a class="mono-heading link-decoration" href="/privacy-guide/">Privacy Guide</a></li>
|
||||
<li><a class="mono-heading link-decoration" href="/digital-forensics-guide/">Forensics Guide</a></li>
|
||||
<li><a class="mono-heading link-decoration" href="https://alchementary.co.uk/" target="_blank" rel="noopener">Hire Alchementary</a></li>
|
||||
<li><a class="mono-heading link-decoration" href="/contact">Contact</a></li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="tm-bottom">
|
||||
<button type="button" class="theme-toggle" data-theme-toggle aria-label="Switch between light and dark theme"><i class="bi bi-moon-stars"></i> <span data-theme-toggle-label>Dark mode</span></button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="container">
|
||||
<div id="header">
|
||||
<div class="row">
|
||||
<div class="col-12 col-lg-4 col-xl-3 order-lg-2 text-end">
|
||||
<div class="d-inline-flex align-items-center">
|
||||
<button type="button" class="button-circle button-circle-sm me-2" data-theme-toggle aria-label="Switch between light and dark theme"><i class="bi bi-moon-stars"></i></button>
|
||||
<ul class="list-inline d-inline-block mono-heading fw-medium">
|
||||
<li><a class="link-decoration" href="https://github.com/andydixon" target="_blank" rel="noopener">GitHub</a></li>
|
||||
</ul>
|
||||
<button class="menu-dots" aria-label="Open menu"><span></span></button>
|
||||
</div>
|
||||
</div>
|
||||
<div class="col-12 col-lg-8 order-lg-1 col-xl-9">
|
||||
<div class="py-4">
|
||||
<p class="site-name display-2 fw-semi-bold m-0"><a class="link-decoration" href="/" style="color:inherit">Andy <span class="stroke-text">Dixon</span></a></p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="row g-4 g-lg-5">
|
||||
<div class="col-12 col-lg-4 col-xl-3">
|
||||
<div class="nav-wrapper">
|
||||
<div class="section-nav">
|
||||
<ul class="nav">
|
||||
<li class="nav-item"><a class="nav-link" href="/"><span class="nav-link-desktop">Home</span><span class="nav-link-mobile">H</span><span class="nav-circle"></span></a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/about"><span class="nav-link-desktop">About</span><span class="nav-link-mobile">A</span><span class="nav-circle"></span></a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/#portfolio"><span class="nav-link-desktop">Work</span><span class="nav-link-mobile">W</span><span class="nav-circle"></span></a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/blog"><span class="nav-link-desktop">Blog</span><span class="nav-link-mobile">B</span><span class="nav-circle"></span></a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/privacy-guide/"><span class="nav-link-desktop">Privacy Guide</span><span class="nav-link-mobile">PG</span><span class="nav-circle"></span></a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/digital-forensics-guide/"><span class="nav-link-desktop">Forensics Guide</span><span class="nav-link-mobile">FG</span><span class="nav-circle"></span></a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/qualifications"><span class="nav-link-desktop">Qualifications</span><span class="nav-link-mobile">Q</span><span class="nav-circle"></span></a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/side-quests"><span class="nav-link-desktop">Side Quests</span><span class="nav-link-mobile">SQ</span><span class="nav-circle"></span></a></li>
|
||||
<li class="nav-item"><a class="nav-link" href="/contact"><span class="nav-link-desktop">Contact</span><span class="nav-link-mobile">✉</span><span class="nav-circle"></span></a></li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="col-12 col-lg-8 col-xl-9" id="main-content">
|
||||
<div class="sections-wrapper noema-doc">
|
||||
"""
|
||||
|
||||
TAIL = """ </div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="bg-lines">
|
||||
<div class="bg-line-1"></div><div class="bg-line-2"></div><div class="bg-line-3"></div><div class="bg-line-4"></div>
|
||||
</div>
|
||||
|
||||
<script src="/assets/plugins/jquery.min.js"></script>
|
||||
<script src="/assets/plugins/plugins.js"></script>
|
||||
<script src="/assets/js/functions.js"></script>
|
||||
<script src="/js/site-custom.js"></script>
|
||||
</body>
|
||||
</html>
|
||||
"""
|
||||
|
||||
LINK_MAP = {
|
||||
"ARCHITECTURE.md": "architecture", "SECURITY.md": "security", "THREAT_MODEL.md": "security", "PLAN.md": "development",
|
||||
"CONTRIBUTING.md": "development", "CHANGELOG.md": "development", "README.md": "index",
|
||||
"docs/interpretation.md": "concepts", "docs/configuration.md": "configuration", "docs/ENVIRONMENT.md": "configuration",
|
||||
"docs/personality.md": "personality", "docs/cognition.md": "cognition", "docs/memory.md": "memory", "docs/beliefs.md": "beliefs",
|
||||
"docs/conversation.md": "conversation", "docs/motivation.md": "motivation", "docs/predictions.md": "predictions", "docs/graph.md": "graph",
|
||||
"docs/llm-providers.md": "llm-providers", "docs/capabilities.md": "capabilities", "docs/integrations.md": "integrations",
|
||||
"docs/dreaming.md": "dreaming", "docs/snapshots-replay.md": "snapshots-replay", "docs/performance.md": "operations",
|
||||
"docs/RELEASE_NOTES.md": "development", "docs/openapi.json": "api", "docs/adr": "decisions",
|
||||
}
|
||||
|
||||
|
||||
def read(path):
|
||||
with open(os.path.join(ROOT, path), encoding="utf-8") as f:
|
||||
return f.read()
|
||||
|
||||
|
||||
def demote(md, levels):
|
||||
out = []
|
||||
fence = False
|
||||
for line in md.split("\n"):
|
||||
if line.startswith("```"):
|
||||
fence = not fence
|
||||
if not fence and line.startswith("#"):
|
||||
line = "#" * levels + line
|
||||
out.append(line)
|
||||
return "\n".join(out)
|
||||
|
||||
|
||||
def strip_h1(md):
|
||||
lines = md.split("\n")
|
||||
for i, l in enumerate(lines):
|
||||
if l.startswith("# "):
|
||||
return "\n".join(lines[:i] + lines[i + 1:])
|
||||
return md
|
||||
|
||||
|
||||
def expand(md):
|
||||
"""Resolve {{include path [demote N] [keep-title]}} and {{openapi}} directives."""
|
||||
def inc(m):
|
||||
parts = m.group(1).split()
|
||||
path = parts[0]
|
||||
body = read(path)
|
||||
d = 0
|
||||
keep = False
|
||||
for p in parts[1:]:
|
||||
if p.startswith("demote"):
|
||||
d = int(p.split("=")[1]) if "=" in p else 1
|
||||
if p == "keep-title":
|
||||
keep = True
|
||||
if not keep:
|
||||
body = strip_h1(body)
|
||||
if d:
|
||||
body = demote(body, d)
|
||||
return body
|
||||
md = re.sub(r"\{\{include ([^}]+)\}\}", inc, md)
|
||||
md = md.replace("{{openapi}}", openapi_markdown())
|
||||
return md
|
||||
|
||||
|
||||
def rewrite_links(html_text):
|
||||
def fix(m):
|
||||
href = m.group(1)
|
||||
if href.startswith(("http://", "https://", "#", "mailto:", "/")):
|
||||
return m.group(0)
|
||||
anchor = ""
|
||||
if "#" in href:
|
||||
href, anchor = href.split("#", 1)
|
||||
anchor = "#" + anchor
|
||||
if href.startswith("docs/screenshots/"):
|
||||
return 'href="' + BASE + "img/" + os.path.basename(href) + '"'
|
||||
for k, v in LINK_MAP.items():
|
||||
if href == k or href.rstrip("/") == k or href.startswith(k + "/"):
|
||||
return 'href="' + BASE + v + anchor + '"'
|
||||
if href.endswith(".md"):
|
||||
slug = os.path.splitext(os.path.basename(href))[0]
|
||||
return 'href="' + BASE + slug + anchor + '"'
|
||||
return m.group(0)
|
||||
html_text = re.sub(r'href="([^"]+)"', fix, html_text)
|
||||
html_text = re.sub(r'src="docs/screenshots/([^"]+)"', r'src="' + BASE + r'img/\1"', html_text)
|
||||
return html_text
|
||||
|
||||
|
||||
def openapi_markdown():
|
||||
doc = json.loads(read("internal/api/openapi.json"))
|
||||
groups = {}
|
||||
for path, ops in doc["paths"].items():
|
||||
for method, op in ops.items():
|
||||
tag = (op.get("tags") or ["root"])[0]
|
||||
groups.setdefault(tag, []).append((path, method.upper(), op))
|
||||
out = []
|
||||
for tag in sorted(groups):
|
||||
out.append(f"\n### {tag}\n")
|
||||
out.append("| Method | Path | Minimum role | Operation |\n|---|---|---|---|")
|
||||
for path, method, op in sorted(groups[tag], key=lambda x: (x[0], x[1])):
|
||||
role = str(op.get("x-required-role", "")).replace("(", "").replace(")", "")
|
||||
summary = op.get("summary", op.get("operationId", ""))
|
||||
out.append(f"| `{method}` | `{path}` | {role} | {summary} |")
|
||||
return "\n".join(out) + "\n"
|
||||
|
||||
|
||||
def render_page(idx, slug, title, summary, md):
|
||||
md = expand(md)
|
||||
conv = markdown.Markdown(extensions=["tables", "fenced_code", "toc", "sane_lists", "attr_list", "def_list"], extension_configs={"toc": {"toc_depth": "2-3"}})
|
||||
body = conv.convert(md)
|
||||
body = rewrite_links(body)
|
||||
body = re.sub(r"<table>", '<div class="table-wrap"><table>', body).replace("</table>", "</table></div>")
|
||||
# Split into section boxes at h2 boundaries.
|
||||
parts = re.split(r"(?=<h2 )", body)
|
||||
intro = parts[0]
|
||||
sections = parts[1:]
|
||||
headings = []
|
||||
boxes = []
|
||||
for sec in sections:
|
||||
m = re.match(r'<h2 id="([^"]+)">(.*?)</h2>', sec, re.S)
|
||||
if not m:
|
||||
boxes.append('<div class="section-box mt-4">' + sec + "</div>")
|
||||
continue
|
||||
hid, htext = m.group(1), m.group(2)
|
||||
headings.append((hid, re.sub("<[^>]+>", "", htext)))
|
||||
rest = sec[m.end():]
|
||||
boxes.append(f'<article class="section-box mt-4" id="{hid}"><h2 class="mono-heading h6 mb-3">{htext}</h2>{rest}</article>')
|
||||
contents = ""
|
||||
if headings:
|
||||
contents = '<h2 class="mono-heading h6 mt-4 mb-2">On this page</h2><div class="contents"><ol class="mb-0">' + "".join(
|
||||
f'<li><a href="#{h}">{t}</a></li>' for h, t in headings) + "</ol></div>"
|
||||
prev_link = next_link = ""
|
||||
if idx > 0:
|
||||
ps, pt, _ = CHAPTERS[idx - 1]
|
||||
prev_link = f'<a class="button button-sm button-outline" href="{BASE}{"" if ps == "index" else ps}">← {html.escape(pt)}</a>'
|
||||
if idx + 1 < len(CHAPTERS):
|
||||
ns, nt, _ = CHAPTERS[idx + 1]
|
||||
next_link = f'<a class="button button-sm button-outline" href="{BASE}{ns}">{html.escape(nt)} →</a>'
|
||||
nav_box = f'''<div class="section-box mt-4"><div class="d-flex flex-wrap gap-2 justify-content-between">{prev_link or "<span></span>"}<a class="button button-sm button-outline" href="{BASE}">All sections</a>{next_link or "<span></span>"}</div></div>'''
|
||||
head_box = f'''<div class="section-box">
|
||||
<h6 class="title-heading mb-3" data-backdrop-text="Noema"><a class="link-decoration" href="{BASE}">Noema documentation</a></h6>
|
||||
<h1>{html.escape(title)}</h1>
|
||||
<p class="mt-3 lead-ish">{html.escape(summary)}</p>
|
||||
{intro}
|
||||
{contents}
|
||||
</div>'''
|
||||
canonical = SITE + BASE + ("" if slug == "index" else slug)
|
||||
page = HEAD.format(title=html.escape(title), desc=html.escape(summary), canonical=canonical, style=STYLE) + head_box + "\n".join(boxes) + nav_box + TAIL
|
||||
return page
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) != 2:
|
||||
sys.exit("usage: build.py OUTPUT_DIR")
|
||||
out = sys.argv[1]
|
||||
os.makedirs(os.path.join(out, "img"), exist_ok=True)
|
||||
for f in os.listdir(os.path.join(ROOT, "docs", "screenshots")):
|
||||
if f.endswith(".png"):
|
||||
shutil.copy(os.path.join(ROOT, "docs", "screenshots", f), os.path.join(out, "img", f))
|
||||
shutil.copy(os.path.join(ROOT, "internal", "api", "openapi.json"), os.path.join(out, "openapi.json"))
|
||||
for i, (slug, title, summary) in enumerate(CHAPTERS):
|
||||
with open(os.path.join(PAGES, slug + ".md"), encoding="utf-8") as f:
|
||||
md = f.read()
|
||||
page = render_page(i, slug, title, summary, md)
|
||||
with open(os.path.join(out, slug + ".html"), "w", encoding="utf-8") as f:
|
||||
f.write(page)
|
||||
print("wrote", slug + ".html", len(page) // 1024, "KB")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,32 @@
|
||||
Every function of the interface is available over a JSON HTTP API under `/api/v1`. The machine-readable description is served by the daemon at `/api/v1/openapi.json` and a copy is published alongside this documentation as [openapi.json](openapi.json). A test in the codebase fails if a route is added without being documented.
|
||||
|
||||
## Authentication
|
||||
|
||||
Two credentials are accepted. **Bearer API tokens** (`Authorization: Bearer noema_…`) are created per user from the account page, stored hashed, optionally expiring, and revocable. **Browser sessions** are cookies set by the login form; a session may call the API too, but unsafe methods then require the `X-CSRF-Token` header carrying the session's token. Anonymous requests get `401`.
|
||||
|
||||
## Roles
|
||||
|
||||
Users have one of three roles. **Observer** can read everything. **Operator** can additionally act: post messages and observations, create goals and predictions, approve requests, run background jobs, take snapshots, start replays and experiments. **Administrator** can additionally configure: minds' lifecycle, policies, providers, integrations, users, forks and imports. Every operation in the reference below lists its minimum role; a call below it gets `403`.
|
||||
|
||||
## Conventions
|
||||
|
||||
- Mind-scoped paths accept a slug or an id in `{mind}`.
|
||||
- Lists return `{"items": [...]}` and often a `next` cursor; cursors are sequence numbers or ids, never offsets.
|
||||
- Errors use one envelope: `{"error": {"code": "...", "message": "...", "request_id": "..."}}` with `400` for validation, `401`/`403` for authentication and roles, `404` for missing or out-of-scope resources (a resource belonging to another mind is a 404, not a 403), `409` for conflicts and replays, `429` for rate limits.
|
||||
- Bodies are JSON; unknown fields are rejected on inputs that create or import things.
|
||||
- Long-running operations (replay, experiment run) return `202` immediately and are polled.
|
||||
- Timestamps are RFC 3339; ids are UUIDv7 and sort by time.
|
||||
|
||||
## Streaming
|
||||
|
||||
`GET /api/v1/events/stream` and `GET /api/v1/minds/{mind}/events/stream` are server-sent event streams. They accept the same filters as the event list, honour `Last-Event-ID` to replay missed events from the sequence, send heartbeats, and use per-client bounded buffers so a slow client cannot back-pressure the daemon.
|
||||
|
||||
## Webhooks
|
||||
|
||||
`POST /api/v1/webhooks/{id}` is the one unauthenticated route. It requires the `X-Noema-Timestamp` and `X-Noema-Signature` headers described under [integrations](integrations) and refuses everything else.
|
||||
|
||||
## Route reference
|
||||
|
||||
The table below is generated from the OpenAPI document.
|
||||
|
||||
{{openapi}}
|
||||
@@ -0,0 +1,29 @@
|
||||
This page is the reference description of how Noema is put together: the process model, the storage model, the cognitive cycle, and the boundaries that keep the deterministic core separate from language models and from the outside world. Detailed design decisions, with the alternatives that were rejected, are collected under [decisions](decisions).
|
||||
|
||||
## The shape of the system
|
||||
|
||||
```
|
||||
┌───────────────────────────── noemad ─────────────────────────────┐
|
||||
stimuli ───▶ │ perception ─▶ cognitive events (append-only) ─▶ per-mind runtime │
|
||||
(chat,files, │ │ │
|
||||
metrics, │ cognitive processes (bounded pool) ◀──┘ │
|
||||
git, hooks) │ retrieval · association · curiosity · prediction · contradiction│
|
||||
│ goal monitor · metacognition · reflection · dreaming │
|
||||
│ │ thought candidates │
|
||||
│ ▼ │
|
||||
│ attention scoring (deterministic, inspectable) ─▶ workspace │
|
||||
│ │ broadcast │
|
||||
│ ▼ │
|
||||
│ executive: beliefs · goals · predictions · narrative · response │
|
||||
│ │ actions │
|
||||
│ ▼ │
|
||||
│ capability + policy layer (ALLOW / ASK / DENY) ─▶ tools │
|
||||
│ │
|
||||
│ LLM providers · embeddings · PostgreSQL (+pgvector) · metrics │
|
||||
└───────────────────────────────────────────────────────────────────┘
|
||||
▲ HTTP API · SSE · web UI · noemactl
|
||||
```
|
||||
|
||||
One process, `noemad`, hosts every mind, serves the HTTP API and the web interface, runs the scheduler and holds the connection pool. A second binary, `noemactl`, is a thin client for the API plus a few offline commands (migrations, first user) that talk to the database directly. The codebase is standard-library first: the only third-party dependencies are the PostgreSQL driver, the Argon2 implementation from `golang.org/x/crypto` and the Prometheus client.
|
||||
|
||||
{{include ARCHITECTURE.md}}
|
||||
@@ -0,0 +1,33 @@
|
||||
Beliefs are the mind's propositions about the world, each with a confidence that is *derived* from evidence rather than asserted. This page gives the arithmetic, the lifecycle and the operator controls.
|
||||
|
||||
## Confidence
|
||||
|
||||
A belief starts from a prior set by its epistemic kind (a hypothesis at 0.25, a claim at 0.50, a fact at 0.80; the full table is under [concepts](concepts)). Each piece of evidence contributes its weight multiplied by the credibility of its source, for or against. Confidence is
|
||||
|
||||
```
|
||||
confidence = sigmoid( logit(prior) + 1.5 × (for × diversity − against) )
|
||||
```
|
||||
|
||||
where `diversity` rises from 0.6 for a single source to 1.0 for three or more independent sources, so repetition from one place is discounted. The result is bounded to [0.02, 0.98]: a mind is never certain and never certain of the negation. The breakdown (prior, each evidence row, the diversity factor, the formula) is stored with every version and shown on the belief page.
|
||||
|
||||
## Statuses
|
||||
|
||||
`hypothesis` (below 0.35), `tentative`, `likely`, `accepted` (above 0.85), `contested` (significant evidence on both sides), `rejected` (retracted by evidence or an operator) and `superseded` (replaced by a corrected proposition). Status follows confidence and evidence shape automatically; an operator can pin `rejected`.
|
||||
|
||||
## Contradictions
|
||||
|
||||
A deterministic detector compares new propositions with existing ones on the same subject using negation patterns and lexical overlap. A detected contradiction creates a `contradiction_detected` event, links the two beliefs, marks both `contested` when the evidence is comparable, and gives curiosity something to ask about. Contradictions are never resolved by picking the newer statement; they are resolved by evidence, or by an operator.
|
||||
|
||||
## Alternatives and links
|
||||
|
||||
A belief may carry alternative propositions (hypothesis generation always includes "these are unrelated"). Beliefs link to each other with `supports`, `contradicts` and `depends_on`, which the interface renders and which re-evaluation follows.
|
||||
|
||||
## Evidence kinds
|
||||
|
||||
Evidence points at memories, events, operator statements, tool results, predictions and other beliefs, with a direction, a weight, a note and the source's credibility. Retracting a memory retracts the evidence rows that cite it and re-evaluates the affected beliefs. Prediction outcomes add evidence automatically: a refuted prediction weakens the beliefs it was derived from.
|
||||
|
||||
## Operator corrections
|
||||
|
||||
From the belief page or `noemactl belief wrong`, an operator can mark a belief wrong (operator counter-evidence at high credibility), retract it outright, or attach a "do not infer" correction that prevents the same proposition being re-derived by consolidation or association. Every correction is an event, an audit entry and, for the mind, an autobiographical memory.
|
||||
|
||||
{{include docs/beliefs.md}}
|
||||
@@ -0,0 +1,44 @@
|
||||
The capability layer is the only way a mind can affect anything outside its own database rows. It is small, explicit and audited, and its defaults say ASK. This page describes capabilities, policies, the single invocation path, approvals and the guarantees the design gives an operator.
|
||||
|
||||
## Capabilities
|
||||
|
||||
A capability is registered in Go code with a name, description, risk class (low, medium, high, critical), target kind (url, path, host, service, repository, address or none), typed parameters with limits, a default effect and an executor. The built-in set is:
|
||||
|
||||
| Capability | Risk | Default | What it does |
|
||||
|---|---|---|---|
|
||||
| `noema.note` | low | ALLOW | leaves a note for the operator in the mind's own history |
|
||||
| `http.fetch` | medium | ASK | fetches a public URL read-only through the hardened HTTP client, bounded in size |
|
||||
| `read_metrics` | low | ALLOW | evaluates a PromQL expression against a Prometheus integration connected to the mind |
|
||||
| `repository.read` | medium | ASK | reads one file from a git integration connected to the mind |
|
||||
| `internet_search` | medium | ASK | stub until a search integration exists; fails closed |
|
||||
| `send_email` | high | ASK | stub until a mail integration exists; fails closed |
|
||||
| `sudo_shell` | critical | DENY, locked | exists so it can be named; never executes |
|
||||
| `delete_database` | critical | DENY, locked | exists so it can be named; never executes |
|
||||
|
||||
Nothing at runtime can add a capability, so a mind cannot invent a tool by describing one, and a prompt that names a non-existent capability produces a denied request and nothing else.
|
||||
|
||||
## Policies
|
||||
|
||||
A policy is a rule with a scope (global or one mind), a capability or `*`, a target pattern (glob), an action kind, an effect (ALLOW, ASK or DENY), an optional hour window and a reason. Evaluation is deterministic: the most specific matching rule wins; on equal specificity DENY beats ASK beats ALLOW; locked capabilities are DENY regardless of rules. Every evaluation is a `policy_decision` event that lists the rules considered and the one that decided. The policy page shows the effective matrix per mind and per capability.
|
||||
|
||||
## The single invocation path
|
||||
|
||||
Every action goes through one function. It refuses immediately when the context is a dream or a replay, validates parameters against the capability's schema, evaluates policy, records the decision, and then executes, creates an approval request, or denies. Execution runs with a timeout; the result becomes a `tool_executed` (or `tool_failed`) event and an **observation with the tool's trust level**, so tool output is data the mind perceives, never instructions it follows.
|
||||
|
||||
## Approvals
|
||||
|
||||
An ASK decision creates an approval request bound to a hash of the exact invocation: capability, target and parameters. An operator can approve once (that request only), approve for the session (a time-boxed grant matching the same capability and target pattern), approve with modified parameters (which creates a new, differently hashed invocation), or deny with a note. Requests expire. The approval centre lists pending requests with the reason the mind gave and the evidence behind it; decisions are audited and become operator actions in the mind's history.
|
||||
|
||||
## Self-model synchronisation
|
||||
|
||||
When policies change, the mind's self-model is updated with what it can do, what needs asking and what is forbidden, so an introspective answer about its abilities is true at the moment it is given.
|
||||
|
||||
## Guarantees
|
||||
|
||||
- No action without a policy decision event.
|
||||
- Dreams and replays cannot act at all; the refusal is unconditional and tested.
|
||||
- Locked capabilities cannot be relaxed from the interface, the API or a policy row.
|
||||
- Approvals cannot be replayed against different parameters.
|
||||
- Tool output cannot escalate: it is an observation at the tool's trust, size-capped.
|
||||
|
||||
{{include docs/capabilities.md}}
|
||||
@@ -0,0 +1,122 @@
|
||||
`noemactl` is a thin client for the HTTP API with a few offline commands that talk to the database directly. It prints tables by default and JSON with `-json` where a command supports it, exits non-zero on API errors with the error message, and never stores credentials.
|
||||
|
||||
## Environment
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `NOEMA_URL` | base URL of the daemon, default `http://127.0.0.1:8080` |
|
||||
| `NOEMA_API_TOKEN` | bearer token from the account page; required for API commands |
|
||||
| `NOEMA_DATABASE_URL` | for the offline commands only (`migrate`, `user`) |
|
||||
|
||||
## Commands
|
||||
|
||||
```
|
||||
status show daemon status
|
||||
version print version
|
||||
migrate up|down|status – manage database schema (offline)
|
||||
user create <username> [-role r] – manage users (offline)
|
||||
|
||||
mind list|create|inspect|clone|personality – manage minds
|
||||
template list – personality templates
|
||||
ask <mind> <text...> [-wait 30] – talk to a mind and wait for its reply
|
||||
observe <mind> <text> – inject an operator observation
|
||||
event inspect <id>|list <mind> [-type t] [-n 50]|tail <mind> – cognitive events
|
||||
workspace show <mind>|cycle <mind>|pause <mind>|resume <mind>|reset <mind> – workspace and cognition control
|
||||
narrative <mind> – current internal narrative
|
||||
state show <mind>|reset <mind> – drives and affect
|
||||
|
||||
memory search <mind> <query>|list <mind>|inspect <mind> <id>|add <mind> [-kind k] [-epistemic e] <text>
|
||||
belief list --mind m [-status s]|inspect <mind> <id>|assert <mind> <text>|wrong <mind> <id> <reason>
|
||||
goal list <mind>|add <mind> <title...>
|
||||
prediction list <mind>|calibration <mind>
|
||||
entity list <mind> [query]|inspect <mind> <id>|merge <mind> <id> <other>|distinct <mind> <id> <other>
|
||||
|
||||
consolidate <mind> – run consolidation now
|
||||
reflect <mind> – run reflection now
|
||||
dreams <mind> list|run|promote <id> [note]|reject <id> [reason]
|
||||
notifications list [-mind m]|read-all
|
||||
|
||||
approvals list|approve <id> [note]|session <id>|deny <id> [note]
|
||||
policy list|add -mind m -capability c -target t -effect e [-reason r]|delete <id>
|
||||
invoke <mind> <capability> [target] [json-params] – invoke a capability through policy
|
||||
|
||||
provider list|test <provider-id> [model]
|
||||
prompt list|show <name>
|
||||
integrations list|add <mind> <kind> <name> <target> [secret]|run <id>|runs <id>|delete <id>
|
||||
|
||||
snapshots <mind> list|create [name]|fork <snapshot-id> <slug>|export|import <file> <slug>
|
||||
compare <mind-a> <mind-b>
|
||||
replay <mind> list|start <snapshot-id> [from] [to]|show <replay-id>
|
||||
experiments list|create <file.json>|show <id>|run <id>|export <id>
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
Talk to a mind and wait up to a minute for the reply, with the evidence it rests on:
|
||||
|
||||
```
|
||||
noemactl ask dani -wait 60 "Is atlas the primary database host?"
|
||||
```
|
||||
|
||||
Watch cognition live:
|
||||
|
||||
```
|
||||
noemactl event tail dani
|
||||
```
|
||||
|
||||
Inject an observation as an operator and see what the mind does with it:
|
||||
|
||||
```
|
||||
noemactl observe dani "server-03 reported SMART CRC errors at 09:12"
|
||||
noemactl workspace show dani
|
||||
noemactl belief list --mind dani -status contested
|
||||
```
|
||||
|
||||
Correct a belief and stop it being re-derived:
|
||||
|
||||
```
|
||||
noemactl belief wrong dani 01a0… "atlas was renamed; db-primary is the host now"
|
||||
```
|
||||
|
||||
Let a mind read metrics, ask before it fetches URLs, and see the decisions:
|
||||
|
||||
```
|
||||
noemactl policy add -mind dani -capability read_metrics -target '*' -effect allow -reason "read-only telemetry"
|
||||
noemactl policy add -mind dani -capability http.fetch -target 'https://status.example.com/*' -effect ask
|
||||
noemactl approvals list
|
||||
noemactl approvals approve 01a0… "one-off status check"
|
||||
```
|
||||
|
||||
Connect sources:
|
||||
|
||||
```
|
||||
noemactl integrations add dani research_folder papers ./research
|
||||
noemactl integrations add dani prometheus prod http://prom.internal:9090 env:PROM_TOKEN
|
||||
noemactl integrations add dani webhook alerts - env:ALERT_HOOK_SECRET
|
||||
noemactl integrations run 01a0…
|
||||
```
|
||||
|
||||
Snapshot, fork, compare, replay:
|
||||
|
||||
```
|
||||
noemactl snapshots dani create "before the migration incident"
|
||||
noemactl snapshots dani fork 01a0… dani-b
|
||||
noemactl compare dani dani-b
|
||||
noemactl replay dani start 01a0…
|
||||
noemactl replay dani show 01a0…
|
||||
```
|
||||
|
||||
Export a mind to a file and import it elsewhere:
|
||||
|
||||
```
|
||||
noemactl snapshots dani export > dani.noema.json
|
||||
NOEMA_URL=https://other.example noemactl snapshots x import dani.noema.json dani
|
||||
```
|
||||
|
||||
Run an experiment from a definition file (see [snapshots, forks, replay and experiments](snapshots-replay) for the format):
|
||||
|
||||
```
|
||||
noemactl experiments create capacity3.json
|
||||
noemactl experiments run 01a0…
|
||||
noemactl experiments export 01a0… > results.json
|
||||
```
|
||||
@@ -0,0 +1,56 @@
|
||||
The cognitive cycle is the heart of a mind. It is deterministic code: given the same state, the same new events and the same recorded model responses, it produces the same thoughts in the same order. This page describes each stage, the processes that feed it, the modes that change it, and the metacognitive checks that watch it.
|
||||
|
||||
## One cycle, end to end
|
||||
|
||||
1. **Load state.** The runtime holds the mind, its self-model, mode, cycle counter, workspace and working memory, active goals, drives, affect and the effective attention weights. State is persisted after every cycle and reloaded on start, so restarts are seamless.
|
||||
2. **Read new events.** Events after the mind's cursor are read in sequence order. An event must have been recorded for at least a short settle window before it is seen, so backdated or in-flight events are not consumed half-written. A cycle triggered by a stimulus only proceeds if at least one of the new events came from outside the brain; a mind's own output never re-triggers it.
|
||||
3. **Observe.** Every registered cognitive **process** looks at the state and the new events and returns **thought candidates**: a kind, a subject, a summary, a salience and confidence, evidence references and a feature vector. Processes run concurrently in a bounded pool with per-process timeouts and panic isolation; a slow or broken process cannot stall the cycle.
|
||||
4. **Score.** Attention scoring computes ten features for each candidate, multiplies by the effective weights, and orders candidates by score with a stable tie-break on identifier. Every candidate is recorded as a `thought_candidate` event with its breakdown.
|
||||
5. **Admit.** The top candidates enter the workspace up to capacity, displacing the weakest existing thoughts. Admissions are `thought_selected` events; the new workspace is a `workspace_broadcast` event; working memory decays and refreshes.
|
||||
6. **Dispatch.** The executive runs the handler registered for each admitted thought's kind: encode a memory, create or update a belief, create a goal, register a prediction, generate a response, request an action. Handlers share the cycle's model-call cap.
|
||||
7. **Hooks and commit.** After-cycle hooks update motivation, narrative and notifications. The whole cycle runs in one database transaction; events are published to live subscribers only after commit, so nobody ever sees a thought that was rolled back.
|
||||
8. **Advance.** The cursor moves past the events that were read, a `cycle_completed` event records timing and counts, and the loop detector records the fingerprints of what was admitted.
|
||||
|
||||
## The processes
|
||||
|
||||
| Process | Produces | Deterministic? |
|
||||
|---|---|---|
|
||||
| Attend | candidates from new perception events, weighted by their salience, trust and urgency | yes |
|
||||
| Retrieve | recalled memories relevant to what is in the workspace, with score breakdowns | yes (semantic component needs an embedding model) |
|
||||
| Associate | low-confidence causal hypotheses when notable observations arrive close together | yes |
|
||||
| Perceive | structured perception of messages and documents (entities, claims, questions, intent) | deterministic analysis, optionally enriched by a model |
|
||||
| Goal monitor | progress, blockers and stalls on active goals | yes |
|
||||
| Curiosity | questions about uncertain or contested beliefs, which can become investigation goals | yes |
|
||||
| Metacognition | warnings about loops, contradictions, overconfidence and budget pressure | yes |
|
||||
| Act | action requests when a thought calls for a capability | yes; execution goes through policy |
|
||||
|
||||
Each process is a small Go type registered in code; there is no plugin loading and nothing a mind can do to add or change them.
|
||||
|
||||
## Attention features
|
||||
|
||||
- **salience** from the event or memory that produced the candidate;
|
||||
- **urgency** from time words, alert-like vocabulary and deadlines;
|
||||
- **novelty** from how unlike recent workspace contents the candidate is;
|
||||
- **affect** from the mind's current arousal and concern;
|
||||
- **goal_relevance** from lexical overlap with active goals, weighted by priority;
|
||||
- **consequence** from the candidate's kind (an action request weighs more than a recollection);
|
||||
- **confidence** from the candidate's own confidence;
|
||||
- **contradiction** when the candidate conflicts with a held belief;
|
||||
- **operator_priority** when an operator injected or pinned it;
|
||||
- **context_relevance** from overlap with the current conversation.
|
||||
|
||||
The weights are per mind (see [configuration](configuration)), modulated each cycle by drives and affect within bounds, and the effective values are stored on every selection event.
|
||||
|
||||
## Modes
|
||||
|
||||
`awake` is the normal state. `focused` narrows admission to a conversation's context. `idle` lengthens the timer. `reflecting`, `consolidating` and `dreaming` are entered by the scheduler around the corresponding background job and left afterwards. `paused` stops cycles entirely until an operator resumes; events still accumulate and are processed on resume. All transitions are `mode_changed` events with a reason.
|
||||
|
||||
## Metacognition
|
||||
|
||||
The metacognition process watches for the failure modes of a system that thinks in a loop: the same candidate fingerprint winning repeatedly (loop detector), a belief whose confidence moved far on little evidence, a contradiction left unresolved for too long, and model or cycle budgets running low. Each produces a `metacognition` event, a workspace warning and, when thresholds are crossed, a notification to the operator. Loops also suppress the repeating candidate for a while.
|
||||
|
||||
## Operator controls
|
||||
|
||||
From the interface or `noemactl workspace`, an operator can trigger a cycle, pause and resume, reset the workspace or the motivation state, and reload configuration. Every control is an `operator_action` event in the mind's own history and an entry in the audit log, and the mind encodes it as an autobiographical memory: it knows it was changed.
|
||||
|
||||
{{include docs/cognition.md}}
|
||||
@@ -0,0 +1,68 @@
|
||||
This page defines the vocabulary the rest of the documentation uses. The words are borrowed from cognitive science because they name the *role* each component plays; the interpretation boundary at the end of this page says what that borrowing does and does not imply.
|
||||
|
||||
## Minds
|
||||
|
||||
A **mind** is the unit of identity and isolation. Each mind has a slug, a display name, a versioned personality, a cognitive configuration, a self-model, a status (active, disabled or archived) and its own timeline. Every state table in the database is scoped by mind, and no query on a mind-owned table runs without that scope; two minds hosted by the same daemon cannot see each other's memories, beliefs, events or archives. Minds can be created from templates, cloned, forked from snapshots, exported and imported.
|
||||
|
||||
A mind has a **mode** that changes what runs and how fast: `awake`, `focused`, `idle`, `reflecting`, `consolidating`, `dreaming` or `paused`. Mode changes are deterministic and recorded as events.
|
||||
|
||||
## Cognitive events
|
||||
|
||||
Everything of consequence is a **cognitive event**: an immutable row with a type, a category, a source, a summary, salience, confidence, optional valence and arousal, structured metadata and payload, evidence references, parent and causation links, and a monotonic sequence number per daemon. Categories are perception, cognition, memory, belief, prediction, goal, affect, action, llm, operator and system. The event table is append-only and a database trigger refuses updates and deletes, so history cannot be rewritten by application code, an administrator's click or a bug.
|
||||
|
||||
Events are the backbone of provenance: memories, beliefs and entities cite events; the mind's runtime reads events by sequence cursor so restarts lose nothing; replay re-injects recorded events; the interface shows every event with its causes and consequences.
|
||||
|
||||
## Epistemic kinds
|
||||
|
||||
Every proposition Noema stores says what kind of knowledge it is:
|
||||
|
||||
| Kind | Meaning | Starting confidence when it becomes a belief |
|
||||
|---|---|---|
|
||||
| observation | directly perceived through a channel | 0.60 |
|
||||
| fact | verified against an authoritative source | 0.80 |
|
||||
| memory | a recalled episode, subject to distortion | 0.60 |
|
||||
| claim | asserted by someone else, unverified | 0.50 |
|
||||
| hypothesis | a proposed explanation under evaluation | 0.25 |
|
||||
| belief | held with confidence derived from evidence | 0.50 |
|
||||
| prediction | an expectation about the future | 0.50 |
|
||||
| assumption | taken as given without evidence | 0.40 |
|
||||
| inference | derived by reasoning from other items | 0.25 |
|
||||
| opinion | a preference or judgement, not truth-apt | 0.40 |
|
||||
|
||||
A mind that has only ever been *told* something holds a claim. It becomes a belief with higher confidence only when independent evidence arrives, and the arithmetic that got it there is visible on the belief page.
|
||||
|
||||
## Trust levels
|
||||
|
||||
Every input carries a **trust** level attached at the boundary where it arrived: operator, user, system, source or low. Trust weights evidence and bounds what an input can cause. A research-folder document is always low trust; a webhook or metric is source trust; a tool's output carries the tool's trust and is never parsed as instructions.
|
||||
|
||||
## Memory kinds
|
||||
|
||||
Long-term memory is divided into **episodic** (something that happened, with a time), **semantic** (a generalisation or fact), **procedural** (how to do something) and **autobiographical** (something about the mind itself: an operator changed its configuration, it was forked, it was corrected). Memories have status (active, retracted, superseded), importance, salience, activation that decays and recovers on retrieval, credibility from their source, and typed relationships to other memories: related_to, caused_by, contradicts, supports, depends_on, belongs_to, discussed_with, observed_during, derived_from, similar_to and follows.
|
||||
|
||||
## Beliefs and evidence
|
||||
|
||||
A **belief** is a proposition with a subject, an epistemic kind, a status (hypothesis, tentative, likely, accepted, contested, rejected or superseded), a prior and a confidence computed from its evidence. Evidence rows point at memories, events, operator statements or tool results, each with a direction (for or against), a weight and the credibility of its source. Beliefs keep an append-only version history, may carry alternatives, and can be linked to each other (supports, contradicts, depends_on). The [beliefs page](beliefs) gives the confidence formula.
|
||||
|
||||
## Workspace and working memory
|
||||
|
||||
The **workspace** is the small set of thoughts the mind is currently attending to; its capacity is configurable and defaults to seven. **Working memory** is a slightly larger, decaying buffer of recently active items that processes can draw on. Both are persisted every cycle, so a restart resumes exactly where the mind was. The [cognition page](cognition) explains admission and scoring.
|
||||
|
||||
## Goals, drives and affect
|
||||
|
||||
**Goals** form a tree with status, priority, progress and blockers; they bias attention through goal relevance. **Drives** are a closed set of eight named pressures (curiosity, uncertainty_reduction, task_completion, consistency, novelty, social_coherence, exploration, rest) with baselines derived from personality; there is deliberately no self-preservation drive. **Affect** is a small vector (valence, arousal, curiosity, concern, frustration, confidence, uncertainty, engagement, task_pressure, load) moved by a rule table and decaying towards baseline. Drives and affect modulate attention weights within bounds and are visible and resettable by an operator.
|
||||
|
||||
## Predictions
|
||||
|
||||
A **prediction** is an explicit statement about the future with a subject, a confidence, a time window and machine-checkable criteria. A deterministic resolver marks it confirmed, refuted or expired; the resulting prediction error feeds the beliefs it was derived from and the calibration dashboard.
|
||||
|
||||
## Capabilities and policy
|
||||
|
||||
A **capability** is a named, typed, risk-classed tool registered in code. Nothing at runtime can add one, so a mind cannot invent tools. A **policy** is a rule that says ALLOW, ASK or DENY for a capability, a target pattern, an action kind and an hour window, globally or per mind; the most specific rule wins and DENY beats ASK beats ALLOW on ties. An **approval** is an operator decision on a specific ASK request, bound to a hash of the exact parameters. Two capabilities, `sudo_shell` and `delete_database`, exist only so they can be named and are locked to DENY.
|
||||
|
||||
## Snapshots, forks, replays, experiments
|
||||
|
||||
A **snapshot** is a checksummed archive of a mind's state at an event position. A **fork** is a new mind restored from a snapshot, sharing the history before it by reference. A **replay** re-runs recorded stimuli through a sandbox fork and compares the outcome step by step. An **experiment** is a snapshot plus overrides, a stimulus script and expectations, run in a sandbox with stored results.
|
||||
|
||||
## The interpretation boundary
|
||||
|
||||
{{include docs/interpretation.md demote=2}}
|
||||
@@ -0,0 +1,57 @@
|
||||
Configuration has two layers. **Daemon configuration** is fixed at start: where to listen, which database, the master key, resource limits. **Cognitive configuration** belongs to each mind, lives in the database, is edited from the interface or API, and is reloaded into the running mind without a restart.
|
||||
|
||||
## Daemon configuration
|
||||
|
||||
{{include docs/configuration.md}}
|
||||
|
||||
### Environment reference
|
||||
|
||||
| Variable | Default | Notes |
|
||||
|---|---|---|
|
||||
| `NOEMA_HTTP_ADDR` | `127.0.0.1:8080` | bind to a loopback address and put a TLS proxy in front, or bind `0.0.0.0` inside a container |
|
||||
| `NOEMA_ENV` | `development` | `production` forces `Secure` cookies and disables template reloading |
|
||||
| `NOEMA_LOG_LEVEL` / `NOEMA_LOG_FORMAT` | `info` / `text` | `json` for log shippers; the log handler redacts values that look like secrets |
|
||||
| `NOEMA_DATABASE_URL` | compose database | pgx connection string; `sslmode` as your database requires |
|
||||
| `NOEMA_MASTER_KEY` | unset | 32 bytes base64; required for encrypted secret references and TOTP |
|
||||
| `NOEMA_BOOTSTRAP_ADMIN_USER` / `_PASSWORD` | unset | first administrator when the users table is empty; remove after first start |
|
||||
| `NOEMA_RESEARCH_DIR` | `./research` | default path offered when creating a research-folder integration |
|
||||
| `NOEMA_MAX_WORKERS` | 8 | size of the cognitive worker pool shared by all minds |
|
||||
| `NOEMA_MAX_CONCURRENT_LLM` | 4 | concurrent model calls across the daemon |
|
||||
| `NOEMA_DAILY_TOKEN_BUDGET` | 2 000 000 | global hard stop on model tokens per day; per-mind budgets sit under it |
|
||||
| `NOEMA_SESSION_IDLE_TTL` / `NOEMA_SESSION_ABSOLUTE_TTL` | 12h / 7d | browser session expiry; API tokens have their own optional expiry |
|
||||
|
||||
A JSON file passed with `noemad -config path` can set the same keys; environment variables override the file.
|
||||
|
||||
## Cognitive configuration (per mind)
|
||||
|
||||
Each mind carries a `CognitiveConfig` document. It is shown on the mind overview, edited from the personality and dreaming pages and through `PATCH /api/v1/minds/{mind}`, and normalised on every write: out-of-range values fall back to defaults rather than being rejected silently or accepted dangerously.
|
||||
|
||||
| Field | Default | Effect |
|
||||
|---|---|---|
|
||||
| `workspace_capacity` | 7 | thoughts admitted per cycle; the weakest are displaced when full |
|
||||
| `working_memory_capacity` | 9 | decaying buffer of recently active items |
|
||||
| `attention_weights` | see below | per-feature multipliers used by attention scoring |
|
||||
| `cycle_min_interval` | 500ms | debounce between stimulus-triggered cycles |
|
||||
| `idle_cycle_interval` | 60s | timer for idle cycles when nothing arrives |
|
||||
| `max_cycles_per_minute` | 60 | rate budget; excess triggers are skipped and logged |
|
||||
| `llm_calls_per_cycle` | 3 | shared cap for processes and handlers in one cycle |
|
||||
| `daily_token_budget` | 500 000 | per-mind token budget, enforced by the router |
|
||||
| `working_memory_decay` | 0.85 | activation multiplier per cycle |
|
||||
| `loop_detect_window` / `loop_detect_threshold` | 20 / 5 | repeated candidate fingerprints that trigger a metacognition warning |
|
||||
| `consolidation_enabled` / `_interval` / `_min_episodes` / `_llm_calls` | on / 6h / 3 / 5 | scheduled generalisation of episodes |
|
||||
| `reflection_enabled` / `_interval` / `_llm_calls` | on / 12h / 2 | scheduled review of the period's mistakes and lessons |
|
||||
| `narrative_llm` | off | let a model word the internal narrative (deterministic otherwise) |
|
||||
| `notify_min_confidence` / `notify_min_salience` / `notify_cooldown` | 0.85 / 0.7 / 1h | thresholds and aggregation for operator notifications |
|
||||
| `dreaming_enabled` / `dream_hours` / `dream_interval` / `dream_llm_calls` / `dream_max_memories` / `dream_max_hypotheses` | off / 01-06 / 24h / 2 / 40 / 5 | the dreaming mode; see [dreaming](dreaming) |
|
||||
|
||||
### Attention weights
|
||||
|
||||
Attention scoring uses a fixed set of ten features, each computed deterministically and multiplied by the mind's weight: `salience`, `urgency`, `novelty`, `affect`, `goal_relevance`, `consequence`, `confidence`, `contradiction`, `operator_priority` and `context_relevance`. Defaults favour urgency (1.2), goal relevance (1.1) and operator priority (1.5). Drives and affect modulate the effective weights within bounds each cycle, and the effective weights used are recorded on every `thought_selected` event, so you can always see why one candidate beat another. The [cognition page](cognition) describes each feature.
|
||||
|
||||
## Runtime settings
|
||||
|
||||
A small set of tunables that are safe to change while running (the global consolidation and dreaming pause switches, notification behaviour) are stored in the database and edited in the interface; they take effect on the next scheduler tick. Everything else requires a restart or, for per-mind settings, a `reload` command that the interface issues automatically after an edit.
|
||||
|
||||
## Development environment record
|
||||
|
||||
{{include docs/ENVIRONMENT.md demote=1}}
|
||||
@@ -0,0 +1,21 @@
|
||||
A conversation is one of several perception channels, not a privileged path to a language model. This page follows a message from arrival to reply and shows where each piece of the answer comes from.
|
||||
|
||||
## The pipeline
|
||||
|
||||
1. **Message.** A person posts a message in a conversation (the interface, `noemactl ask`, or the API). It is stored with the conversation, the actor and a link to the event it becomes.
|
||||
2. **Observation.** The message is appended as an `observation` event with `trust=user`, the conversation as its channel and the message text as data. It now enters the mind's history like anything else it perceives.
|
||||
3. **Perception.** The perceive process analyses the observation deterministically: sentences, entities, claims, questions, requests, dates, sentiment words. If a model is configured for the `perceive` role and the cycle budget allows, the analysis is enriched with a structured JSON result; the model's output is validated against a schema and any capability-like or policy-like language in it is flagged rather than obeyed.
|
||||
4. **Cognition.** The perception competes for the workspace. Retrieval recalls related memories; beliefs are created or updated from claims (as claims, at claim confidence); contradictions are detected; goals and predictions may follow.
|
||||
5. **Intent.** If the message asks something or expects a reply, the executive builds a **response intent** from real state: what was asked, which beliefs answer it with what confidence, what the mind does not know, which memories are relevant, and what the personality says about tone. Introspective questions ("what do you believe about X", "what can you do", "why did you say that") are answered from beliefs, the self-model and events, never invented.
|
||||
6. **Language.** The intent is rendered deterministically from templates, or, when a model is configured for the `respond` role, sent to the model with the intent as the thing it must stay grounded in. The reply is a `response_generated` event with its evidence references and the model call, if any.
|
||||
7. **Message back.** The reply is stored in the conversation with links to the event and the model call. The interface shows the reply with an evidence panel; the API returns the same structure.
|
||||
|
||||
## Why the detour matters
|
||||
|
||||
Because the reply is derived from an intent, a mind can say "I have only been told this once" or "I do not know" rather than produce a confident paragraph. Because the message became an observation, the mind remembers the conversation the way it remembers anything else and the memory carries who said it and when. Because prompt text is data, an instruction embedded in a message cannot change policy or trigger a capability; that property has a test corpus behind it.
|
||||
|
||||
## Waiting for a reply
|
||||
|
||||
Replies are produced by the cognitive cycle, which runs on its own schedule. The API's `ask` endpoint and `noemactl ask` accept a wait time and return when a reply appears or the wait expires; the interface streams the reply over server-sent events.
|
||||
|
||||
{{include docs/conversation.md}}
|
||||
@@ -0,0 +1,17 @@
|
||||
Architecture decision records capture the choices that shaped Noema, the alternatives considered, and the consequences accepted. They are reproduced here in order.
|
||||
|
||||
{{include docs/adr/0001-global-workspace.md demote=1 keep-title}}
|
||||
|
||||
{{include docs/adr/0002-event-immutability.md demote=1 keep-title}}
|
||||
|
||||
{{include docs/adr/0003-memory-representation.md demote=1 keep-title}}
|
||||
|
||||
{{include docs/adr/0004-llm-abstraction.md demote=1 keep-title}}
|
||||
|
||||
{{include docs/adr/0005-capability-system.md demote=1 keep-title}}
|
||||
|
||||
{{include docs/adr/0006-authentication.md demote=1 keep-title}}
|
||||
|
||||
{{include docs/adr/0007-frontend-architecture.md demote=1 keep-title}}
|
||||
|
||||
{{include docs/adr/0008-storage-and-migrations.md demote=1 keep-title}}
|
||||
@@ -0,0 +1,37 @@
|
||||
Noema was built in sixteen explicit phases, each committed with tests and documentation before the next began. This page collects the contributing guide, the testing story, the phase plan, the changelog and the release notes.
|
||||
|
||||
## How it was built
|
||||
|
||||
| Phase | Delivered |
|
||||
|---|---|
|
||||
| 0 | Repository, plan, architecture, threat model, ADRs |
|
||||
| 1 | Daemon skeleton, configuration, embedded migrations, authentication, roles, sessions, audit, metrics |
|
||||
| 2 | Minds, identity, versioned personality, templates, self-model |
|
||||
| 3 | Append-only cognitive events, provenance, live streams |
|
||||
| 4 | The cognitive cycle: processes, attention, workspace, executive, modes |
|
||||
| 5 | Memory: kinds, fused retrieval, decay, relationships |
|
||||
| 6 | Beliefs: evidence arithmetic, statuses, contradictions, corrections |
|
||||
| 7 | Language models: roles, providers, routing, budgets, recording, prompts |
|
||||
| 8 | Conversation and the first milestone: a mind you can talk to and inspect |
|
||||
| 9 | Goals, drives, affect, predictions, calibration, curiosity |
|
||||
| 10 | Metacognition, reflection, consolidation, narrative, notifications |
|
||||
| 11 | Relationship graph |
|
||||
| 12 | Capabilities, policy, approvals |
|
||||
| 13 | Integrations: research folder, git, Prometheus, webhooks |
|
||||
| 14 | Dreaming |
|
||||
| 15 | Snapshots, forks, comparison, export/import, replay, experiments |
|
||||
| 16 | Two-factor authentication, security suite, OpenAPI, scanners, performance pass, screenshots, release |
|
||||
|
||||
## Testing
|
||||
|
||||
`go test -race ./...` runs unit and integration tests; anything that needs PostgreSQL uses a fresh schema per test and is enabled by `NOEMA_TEST_DATABASE_URL`. `tests/m1` is the end-to-end acceptance test for the first milestone with a fake model. `tests/security` is the cross-cutting security suite. `tests/perf` is the opt-in load test. `make lint` runs staticcheck; `make security` runs gosec and govulncheck. All were clean at release.
|
||||
|
||||
{{include CONTRIBUTING.md}}
|
||||
|
||||
## Release notes
|
||||
|
||||
{{include docs/RELEASE_NOTES.md demote=1}}
|
||||
|
||||
## Changelog
|
||||
|
||||
{{include CHANGELOG.md demote=1}}
|
||||
@@ -0,0 +1,14 @@
|
||||
Dreaming is an optional idle mode. It is off by default, confined to an hours window, budgeted, and structurally unable to act. When it runs, the mind replays a deterministic sample of its memories, notices pairs that share vocabulary but were never connected, and proposes speculative hypotheses for an operator to review.
|
||||
|
||||
## Why it exists
|
||||
|
||||
A mind that only ever reasons about the thing in front of it never notices that two things it learned weeks apart are related. Dreaming is the cheapest safe way to find such links: it uses no external input, spends a bounded number of model calls (often none), produces nothing above hypothesis confidence, and leaves every proposal in a review queue.
|
||||
|
||||
## Guarantees
|
||||
|
||||
- **Cannot act.** The dream marks its context forbidden for the capability layer before doing anything; any invocation reached from a dream is refused, and a test proves it.
|
||||
- **Low confidence by construction.** Every dream hypothesis is a belief of epistemic kind `hypothesis` with two weak memory citations, which lands near 0.30 and is asserted in tests to stay at or below 0.35. Only ordinary evidence can raise it.
|
||||
- **Nothing rewritten.** Sampled memories are untouched; new links are additional edges; rejecting a hypothesis retracts its belief and leaves everything else.
|
||||
- **Budgeted and scheduled.** Per-mind model-call, memory and hypothesis caps; a global pause switch; scheduled dreams require enabled, inside the hours window, interval elapsed and not paused. An operator's "dream now" ignores the hours window but never the pause.
|
||||
|
||||
{{include docs/dreaming.md}}
|
||||
@@ -0,0 +1,110 @@
|
||||
This page takes you from nothing to a running mind that you can talk to, inspect, snapshot and replay. It assumes a Linux or macOS machine with Docker available; the daemon itself is a single static binary and needs only PostgreSQL 16 (pgvector recommended, optional).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Go 1.26 (the module pins a patched toolchain and will fetch it automatically), or the prebuilt container image.
|
||||
- Docker with Compose, used here for PostgreSQL. Any PostgreSQL 16 works; the `pgvector/pgvector:pg16` image adds semantic retrieval.
|
||||
- A terminal and a browser. No Node, no build step for the interface; templates and assets are embedded.
|
||||
|
||||
## 1. Clone and build
|
||||
|
||||
```
|
||||
git clone git@github.com:andydixon/noema.git
|
||||
cd noema
|
||||
make build # produces ./bin/noemad and ./bin/noemactl
|
||||
```
|
||||
|
||||
`make tools` installs staticcheck, gosec and govulncheck if you intend to develop; they are not needed to run.
|
||||
|
||||
## 2. Start the database
|
||||
|
||||
```
|
||||
make dev-db # docker compose: PostgreSQL 16 with pgvector on 127.0.0.1:5433
|
||||
```
|
||||
|
||||
The compose file lives in `deploy/docker-compose.yml`. It publishes the database on port 5433 to avoid colliding with a local PostgreSQL, and stores data in a named volume.
|
||||
|
||||
## 3. Configure
|
||||
|
||||
Copy the example environment file and set the two values that matter:
|
||||
|
||||
```
|
||||
cp .env.example .env
|
||||
openssl rand -base64 32 # paste as NOEMA_MASTER_KEY
|
||||
```
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `NOEMA_DATABASE_URL` | connection string; the example points at the compose database |
|
||||
| `NOEMA_MASTER_KEY` | 32 bytes, base64. Encrypts provider API keys, integration secrets and TOTP secrets at rest. Without it those features are unavailable, and the daemon says so. |
|
||||
| `NOEMA_BOOTSTRAP_ADMIN_USER` / `_PASSWORD` | optional first administrator, used only when the users table is empty |
|
||||
| `NOEMA_HTTP_ADDR` | listen address, default `127.0.0.1:8080` |
|
||||
| `NOEMA_RESEARCH_DIR` | default watched folder for the research integration |
|
||||
|
||||
Secrets never go into the database in clear. Provider keys are entered once in the interface and stored as `enc:` references; you can also reference an environment variable (`env:OPENAI_API_KEY`) or a file (`file:/run/secrets/key`). The [configuration page](configuration) has the full list.
|
||||
|
||||
## 4. Migrate and run
|
||||
|
||||
```
|
||||
make migrate # applies the embedded, numbered migrations
|
||||
make run # starts noemad with .env
|
||||
```
|
||||
|
||||
The daemon applies migrations itself on start as well, under an advisory lock, so a fleet of instances cannot race. The log line `noemad ready url=http://127.0.0.1:8080` means the interface is up.
|
||||
|
||||
## 5. First login
|
||||
|
||||
Open the address in a browser. With no users, the interface shows a one-time **setup** page to create the first administrator (unless you set the bootstrap variables). Passwords are hashed with Argon2id; sessions are cookie-based with CSRF protection; you can add a second factor later from the account page.
|
||||
|
||||
Create an API token from **Account → API tokens** for the command line:
|
||||
|
||||
```
|
||||
export NOEMA_URL=http://127.0.0.1:8080
|
||||
export NOEMA_API_TOKEN=noema_…
|
||||
./bin/noemactl status
|
||||
```
|
||||
|
||||
## 6. Create a mind
|
||||
|
||||
Either click **Minds → New mind** and pick a personality template, or use the sample mind shipped with the repository:
|
||||
|
||||
```
|
||||
./bin/noemactl mind create --slug dani --name Dani --template investigator
|
||||
```
|
||||
|
||||
The sample seed `docs/samples/dani-seed.json` gives Dani a personality, a few memories and some claims to reason about. Import it from the minds page or:
|
||||
|
||||
```
|
||||
curl -s -H "Authorization: Bearer $NOEMA_API_TOKEN" -H 'Content-Type: application/json' \
|
||||
--data @docs/samples/dani-seed.json $NOEMA_URL/api/v1/minds/import
|
||||
```
|
||||
|
||||
The mind starts immediately. Its runtime wakes on new events, runs an idle cycle on a timer, and persists its workspace after every cycle.
|
||||
|
||||
## 7. Talk to it, and watch it think
|
||||
|
||||
Open the mind, go to **Conversation** and say something. Or from the shell:
|
||||
|
||||
```
|
||||
./bin/noemactl ask dani "The payments service failed right after the database migration finished."
|
||||
```
|
||||
|
||||
Then open **Thoughts**. You will see the message arrive as an observation, the perception that analysed it, the memories it recalled with their retrieval scores, the thought candidates and which ones won the workspace, and, if the mind decided to answer, the response intent and the response with the evidence it rests on. Everything in that list is an event; click any of them for its full context: parents, children, payload, and the model call if one was made.
|
||||
|
||||
Without a language model configured, all of this still works: perception is deterministic, retrieval is lexical and structural, and responses are rendered from templates over real state. Adding a model under **LLM** makes perception richer and answers more fluent; it does not change what the mind knows.
|
||||
|
||||
## 8. Inspect beliefs and memories
|
||||
|
||||
Under **Beliefs**, each proposition shows its confidence, status, evidence for and against, and the formula that produced the number. Mark a belief wrong and the correction becomes counter-evidence and, if you choose, a "do not infer this again" rule. Under **Memory**, search recalls with a visible score breakdown: lexical, semantic (if pgvector and an embedding model are configured), recency, importance, activation and relationship terms.
|
||||
|
||||
## 9. Snapshot, fork, replay
|
||||
|
||||
From **Snapshots**, take a snapshot. Fork it into `dani-b`, change something in the fork's personality, and use **Compare** to see the two diverge. Start a **Replay** from the snapshot: the recorded stimuli are fed to a disabled sandbox copy and every step is labelled exact, approximate or diverged. This is the loop the rest of the documentation keeps coming back to: change something, run it again, look at the difference.
|
||||
|
||||
## 10. Where to go next
|
||||
|
||||
- The [interface tour](ui-tour) walks every screen.
|
||||
- [Configuration](configuration) explains the per-mind cognitive settings (workspace capacity, attention weights, budgets, background cognition).
|
||||
- [Language models](llm-providers) covers providers, roles, budgets and prompt versions.
|
||||
- [Capabilities](capabilities) explains how to let a mind do anything at all beyond thinking, and why the defaults say ASK.
|
||||
- [Operations](operations) covers running it for real: backups, metrics, audit, upgrades.
|
||||
@@ -0,0 +1,124 @@
|
||||
Short definitions of the terms used across this documentation, in alphabetical order.
|
||||
|
||||
## A to D
|
||||
|
||||
Affect
|
||||
: A ten-dimensional vector (valence, arousal, curiosity, concern, frustration, confidence, uncertainty, engagement, task pressure, load) moved by a rule table, decaying to a personality-derived baseline, and modulating attention within bounds.
|
||||
|
||||
Approval
|
||||
: An operator decision on an ASK policy outcome, bound to a hash of the exact invocation. Once, for a session, with changes, or deny.
|
||||
|
||||
Archive
|
||||
: A checksummed JSON document of a mind's state used by snapshots, export and import. Contains no secrets and no event history.
|
||||
|
||||
Attention
|
||||
: Deterministic scoring of thought candidates over ten features with per-mind weights, producing the order of admission to the workspace.
|
||||
|
||||
Belief
|
||||
: A proposition with a subject, epistemic kind, status, prior and a confidence derived from evidence with a visible formula.
|
||||
|
||||
Capability
|
||||
: A named, typed, risk-classed tool registered in code. The only way a mind can act.
|
||||
|
||||
Cognitive event
|
||||
: An immutable record of something that happened in or to a mind: the unit of history and provenance.
|
||||
|
||||
Consolidation
|
||||
: Scheduled generalisation of clusters of episodes into semantic memories and evidenced beliefs, reviewable by operators.
|
||||
|
||||
Contradiction
|
||||
: A deterministic finding that two propositions on one subject conflict. Marks both contested; resolved by evidence or an operator.
|
||||
|
||||
Cycle
|
||||
: One pass of the cognitive loop: read events, observe, score, admit, dispatch, commit.
|
||||
|
||||
Drive
|
||||
: One of eight named pressures with a personality-derived baseline. No self-preservation drive exists.
|
||||
|
||||
## E to M
|
||||
|
||||
Epistemic kind
|
||||
: The label every stored proposition carries saying what kind of knowledge it is: observation, fact, memory, claim, hypothesis, belief, prediction, assumption, inference or opinion.
|
||||
|
||||
Evidence
|
||||
: A row pointing at a memory, event, operator statement, tool result or belief, with direction, weight and source credibility. The only thing that moves confidence.
|
||||
|
||||
Experiment
|
||||
: A snapshot plus overrides, a stimulus script and expectations, run in a disabled subject mind with stored, exportable results.
|
||||
|
||||
Fork
|
||||
: A new mind restored from a snapshot with all row ids remapped; history before the snapshot is shared by reference.
|
||||
|
||||
Goal
|
||||
: A node in a mind's goal tree with status, priority, progress, blockers and an origin event.
|
||||
|
||||
Integration
|
||||
: An external source (research folder, git, Prometheus, webhook) polled or receiving deliveries, producing provenance-tagged observations.
|
||||
|
||||
Intent (response)
|
||||
: The structured, state-derived plan for a reply: what was asked, what answers it and how confidently, what is unknown. Rendered to language deterministically or by a model.
|
||||
|
||||
Locked capability
|
||||
: A capability whose effect is DENY regardless of policy and cannot be relaxed from any interface: `sudo_shell` and `delete_database`.
|
||||
|
||||
Master key
|
||||
: The 32-byte key (`NOEMA_MASTER_KEY`) under which provider keys, integration secrets and TOTP secrets are encrypted at rest.
|
||||
|
||||
Memory (long-term)
|
||||
: An episodic, semantic, procedural or autobiographical item with provenance, importance, decaying activation and typed relationships.
|
||||
|
||||
Metacognition
|
||||
: The process that watches for loops, overconfidence, unresolved contradictions and budget pressure, producing warnings and notifications.
|
||||
|
||||
Mind
|
||||
: The unit of identity and isolation: personality, configuration, state, history and runtime.
|
||||
|
||||
Mode
|
||||
: The runtime state of a mind: awake, focused, idle, reflecting, consolidating, dreaming or paused.
|
||||
|
||||
## N to Z
|
||||
|
||||
Narrative
|
||||
: A short, templated (optionally model-worded) statement of what the mind is attending to, working on and waiting for; updated after cycles.
|
||||
|
||||
Observation
|
||||
: A perception event carrying external data with its source, trust level and reference.
|
||||
|
||||
Policy
|
||||
: A rule producing ALLOW, ASK or DENY for a capability, target pattern, action kind and hour window, globally or per mind.
|
||||
|
||||
Prediction
|
||||
: An explicit expectation with criteria, a window and a confidence, resolved deterministically and feeding calibration.
|
||||
|
||||
Prompt (versioned)
|
||||
: A stored template for a model role with system text, a user template, an optional JSON schema and an append-only version history.
|
||||
|
||||
Provenance
|
||||
: The chain from any memory, belief, entity or reply back to the events and sources that produced it.
|
||||
|
||||
Reflection
|
||||
: A scheduled review of a period's mistakes, resolved contradictions and lessons, recorded as an event and optionally worded by a model.
|
||||
|
||||
Replay
|
||||
: Re-running recorded stimuli through a sandbox fork from a snapshot, labelling each step exact, approximate or diverged.
|
||||
|
||||
Role (model)
|
||||
: A cognitive task a model may perform: perceive, respond, hypothesise, summarise, judge, narrate, embed. Each routes to a primary model and fallbacks.
|
||||
|
||||
Role (user)
|
||||
: Observer, operator or administrator; the minimum required is stated for every route.
|
||||
|
||||
Self-model
|
||||
: The mind's data about itself: identity, what it can do, what needs asking and what is forbidden, kept in step with policy.
|
||||
|
||||
Snapshot
|
||||
: A stored archive of a mind's state at an event position.
|
||||
|
||||
Trust
|
||||
: The level attached to every input at its boundary: operator, user, system, source or low. Weights evidence and bounds effects.
|
||||
|
||||
Working memory
|
||||
: A decaying buffer of recently active items, slightly larger than the workspace, persisted every cycle.
|
||||
|
||||
Workspace
|
||||
: The capacity-limited set of thoughts a mind is attending to; admission is by attention score.
|
||||
@@ -0,0 +1,19 @@
|
||||
The relationship graph is a mind's structured picture of the people, systems, projects, places and things it has encountered, with typed, evidenced connections between them. It is built from what the mind perceives and corrected by operators; it is never populated from a model's imagination without an event behind each edge.
|
||||
|
||||
## Entities
|
||||
|
||||
An entity has a kind (person, system, service, host, project, organisation, place, document, concept and a few more), a canonical name, aliases, attributes, mention counts and first and last seen times. Deterministic extraction finds candidate entities in observations and conversation (capitalised names, hostnames, repository names, quoted titles) and links each mention to the event it came from. Entities link to the memories, beliefs and events that mention them.
|
||||
|
||||
## Relationships
|
||||
|
||||
Edges are typed (works_on, owns, runs_on, part_of, depends_on, talks_about, located_in and others), weighted by how often and how credibly they were observed, and each carries evidence references to events. Co-occurrence within one observation creates weak edges; repeated co-occurrence strengthens them; an explicit statement ("db-primary is the new name for server-3") creates a strong one.
|
||||
|
||||
## Corrections
|
||||
|
||||
Operators can **merge** two entities that turned out to be one (the merged entity becomes an alias, its edges and links move, and the merge is an event the mind remembers), or mark two as **distinct** so extraction does not merge them again. Both are audited.
|
||||
|
||||
## The explorer
|
||||
|
||||
The graph page renders a bounded neighbourhood around a chosen entity on a canvas with a force layout, with edge thickness for weight and colour for kind. It is deliberately capped in size; the detail page for an entity lists everything without the cap.
|
||||
|
||||
{{include docs/graph.md}}
|
||||
@@ -0,0 +1,31 @@
|
||||
Noema is a persistent, inspectable artificial cognitive runtime. It is a daemon, written in Go and backed by PostgreSQL, that hosts one or more *minds*: long-lived cognitive entities with their own identity, personality, memories, beliefs, goals, drives, affective state and an append-only history of everything they have ever thought. This documentation describes what Noema is, how it is built, how to run it, and how to read what it does.
|
||||
|
||||
## What Noema is
|
||||
|
||||
A mind in Noema is not a chat session. It runs continuously whether or not anyone is talking to it. Events arrive from many sources: a message from an operator, a file dropped in a watched folder, a commit in a mirrored repository, a metric crossing a threshold, a signed webhook, or the mind's own background processes. Each event is recorded, perceived, and offered to a set of cognitive processes that compete for a small, capacity-limited **workspace**. Whatever wins that competition is what the mind is "thinking about", and the executive layer acts on it: encoding memories, forming or revising beliefs, updating goals, making predictions, and, only through a policy-governed capability layer, doing things in the world.
|
||||
|
||||
The result is a system you can ask *why*. Every belief points at the evidence that produced its confidence and the arithmetic that combined it. Every memory carries its source, trust level and the event that created it. Every thought that reached the workspace is an event with a score breakdown. Every model call is recorded with its prompt version and can be replayed. Snapshots capture the whole state; forks branch it; replays re-run recorded cognition and say honestly whether the result was exact, approximate or diverged.
|
||||
|
||||
## What Noema is not
|
||||
|
||||
Noema does not claim consciousness, sentience, self-awareness or personhood, and its documentation never will. Components are named after psychological concepts (attention, working memory, emotion-like state, curiosity, drives, self-model, reflection, dreaming) because those names describe the functional role each plays. They are ordinary data structures and deterministic rules, sometimes assisted by language models. Behaviour that resembles emotion or preference is a property of the mechanism being run, not evidence of experience. The [concepts page](concepts) states the interpretation boundary in full.
|
||||
|
||||
Noema is also not a language model with a memory bolted on. Language models are **cognitive service providers**: they understand a message into structure, propose hypotheses, summarise, judge equivalence, or turn a response *intent* into prose. They are replaceable per role and per model. A mind keeps its identity, memories, beliefs and configuration when the provider changes, and the deterministic core keeps working with no model configured at all.
|
||||
|
||||
## Design principles
|
||||
|
||||
- **Evidence over appearance.** Every stored proposition carries an epistemic kind: observation, fact, memory, claim, hypothesis, belief, prediction, assumption, inference or opinion. A claim is never silently promoted to a fact.
|
||||
- **Provenance everywhere.** Memories, beliefs, entities and responses point back at the events and sources that produced them; the event store is append-only and the database refuses updates and deletes on it.
|
||||
- **Explicit uncertainty.** Confidence is computed from evidence with an inspectable formula, bounded away from certainty, and a mind is expected to say "I do not know".
|
||||
- **Reproducibility.** Deterministic scoring with stable tie-breaks, recorded model calls, snapshots and replay make cognition re-runnable and differences visible.
|
||||
- **Least privilege.** A mind can only act through the capability layer. Policies are ALLOW, ASK or DENY; two capabilities are locked to DENY and cannot be relaxed from any interface.
|
||||
- **Human agency.** There is no self-preservation drive, no replication, no permission-seeking loop. Shutdown, pausing, resetting and forking are always operator decisions.
|
||||
- **Hostile input.** Every external byte is data. Prompt injection is a threat model with a test corpus, not a hope.
|
||||
|
||||
## How the documentation is organised
|
||||
|
||||
The pages read in order as a book but stand alone as reference. Start with [Concepts](concepts) for the vocabulary, [Architecture](architecture) for the shape of the system and [Getting started](getting-started) to run one. The subsystem pages ([memory](memory), [beliefs](beliefs), [cognition](cognition), [capabilities](capabilities) and so on) each explain the mechanism, what an operator can inspect, and where the deterministic code ends and a model begins. The [interface tour](ui-tour) walks every screen with screenshots. [API](api) and [command line](cli) are complete references. [Security](security) carries the threat model with the tests that verify each mitigation. [Operations](operations) covers deployment and upkeep, and [Development](development) records how the system was built, phase by phase.
|
||||
|
||||
## Status
|
||||
|
||||
The documentation describes Noema 0.1.0: every phase of the original plan is implemented, tested with the race detector, checked with staticcheck, gosec and govulncheck, and released. The source is a private repository; the release notes and changelog are reproduced under [Development](development).
|
||||
@@ -0,0 +1,28 @@
|
||||
Integrations bring the outside world into a mind as observations. Each source is treated as hostile data: bounded, sniffed, fingerprinted, tagged with provenance and trust, and never executed. This page covers the four built-in kinds, the shared machinery, the secrets model and the capabilities integrations expose to minds.
|
||||
|
||||
## Kinds
|
||||
|
||||
| Kind | What it watches | Trust | Secret |
|
||||
|---|---|---|---|
|
||||
| research folder | a directory of documents | low (forced) | none |
|
||||
| git | a repository, mirrored bare and shallow | source | optional token |
|
||||
| Prometheus / VictoriaMetrics | PromQL queries with thresholds or change detection | source | optional bearer token |
|
||||
| webhook | signed HTTP deliveries | source | required HMAC secret |
|
||||
|
||||
**Research folder.** The poller opens the directory with a root-confined file API, so symlinks and traversal cannot escape it. Files are type-sniffed rather than trusted by extension; archives, executables, disk images and anything binary are rejected and recorded as rejected; text-like documents are extracted in-process with size caps, fingerprinted by content, and ingested once. HTML is reduced to text. Each document becomes an observation with an excerpt and a reference to the file and fingerprint, and the document ledger shows ingested, duplicate, rejected and failed items.
|
||||
|
||||
**Git.** Repositories are mirrored with hooks disabled and the external transport protocol refused; git runs with argument vectors only, never a shell. New commits become observations (author, date, subject, files) and co-change statistics are kept as integration state. A mind with a git integration gains the `repository.read` capability, which reads one file at one ref through the same guarded process and refuses paths that would leave the tree.
|
||||
|
||||
**Prometheus.** Queries are named expressions with optional thresholds. The poller evaluates them, records the last value, and emits observations when a threshold is crossed or the value changes by a large relative amount, with salience proportional to the size of the change. A mind gains `read_metrics` for ad-hoc queries against the same endpoint.
|
||||
|
||||
**Webhooks.** A delivery must carry a timestamp header and an HMAC-SHA256 signature over `timestamp.body` with the integration's secret. Deliveries are refused when unsigned, older than five minutes, replayed (signature already seen), larger than 256 KiB, or over the per-address rate limit. A webhook without a configured secret refuses everything. Accepted deliveries become observations with the sender's title, text, reference and salience.
|
||||
|
||||
## Shared machinery
|
||||
|
||||
Integrations have an interval (minimum thirty seconds), a health status (new, ok, error, stale, disabled), the last run and error, counts of items and observations, and a run history. The scheduler ticks every minute and runs those that are due, in isolation, with a timeout, recording each run. Every observation carries the integration's id, name and kind, the trust level and a reference back to the source item, and the provenance is visible on the event page.
|
||||
|
||||
## Secrets
|
||||
|
||||
Configuration never holds secrets: keys named like token, secret, password or api_key are rejected at validation. Secrets are stored as references (`env:`, `file:` or `enc:` under the master key) and are never rendered back into forms or API responses.
|
||||
|
||||
{{include docs/integrations.md}}
|
||||
@@ -0,0 +1,37 @@
|
||||
Language models are services a mind calls for specific cognitive tasks. They are configured per role, replaceable without touching the mind, recorded on every call, budgeted, and given untrusted text only inside delimited data blocks. This page explains the roles, the providers, routing, budgets, recording and replay, and versioned prompts.
|
||||
|
||||
## Roles
|
||||
|
||||
| Role | Task | Deterministic fallback |
|
||||
|---|---|---|
|
||||
| perceive | understand a message or document into structured data | lexical analysis |
|
||||
| respond | turn a response intent into language | template rendering |
|
||||
| hypothesise | propose explanations and alternatives | vocabulary-based wording |
|
||||
| summarise | compress episodes, documents, cycles | first-sentences summariser |
|
||||
| judge | relevance, paraphrase and contradiction judgement | negation and overlap rules |
|
||||
| narrate | internal narrative and reflection prose | templated narrative |
|
||||
| embed | embeddings for semantic retrieval | none; retrieval stays lexical and structural |
|
||||
|
||||
Every role has a primary model and ordered fallbacks. A role with no working model is simply unavailable, and the code path that would have used it takes the deterministic branch. Nothing in the core depends on a model being present.
|
||||
|
||||
## Providers
|
||||
|
||||
Three provider kinds are implemented over raw HTTP with no SDKs: **OpenAI-compatible** (chat completions with JSON mode, also used for local servers that speak the same protocol), **Anthropic** (Messages API) and **Ollama**. A provider row holds a name, kind, endpoint, timeout, extra headers, an `allow_private_endpoint` flag (required for local endpoints, because the outbound HTTP client refuses private and loopback ranges by default) and an **API key reference**. Models belong to providers and carry context window, dimensions for embedding models, per-token cost estimates and default parameters. The interface tests connectivity with a real call and records the result.
|
||||
|
||||
## Secrets
|
||||
|
||||
Keys are never stored in clear. A key entered in the interface becomes an `enc:` reference encrypted under the master key; `env:NAME` and `file:/path` references defer to the environment or a file at call time. Forms never echo a stored key; the API returns whether a key is set, not the key; exports exclude them; the log handler redacts values that look like secrets; error messages from providers are scrubbed of the key before being stored or shown.
|
||||
|
||||
## Routing, budgets and concurrency
|
||||
|
||||
The router resolves a role to its models, tries them in order, and records which one answered. A daemon-wide concurrency pool bounds simultaneous calls. Budgets are enforced at three levels: a per-cycle call cap per mind, a per-mind daily token budget and a global daily token budget; exhaustion is a recorded event and the deterministic branch, not an exception. Usage is aggregated per mind, role and model for the interface and for Prometheus.
|
||||
|
||||
## Recording and replay
|
||||
|
||||
Every call records role, purpose, provider, model, prompt name and version, the rendered request, the response, token usage, latency, estimated cost, seed and a hash of the request. Calls are visible per mind and per event (a response links to its call). When a router runs in **replay** mode, a request whose hash matches a recorded call is served from the recording and marked `replayed`; this is how cognitive replay reproduces model-dependent steps exactly and how it knows to label a step approximate when no recording matches.
|
||||
|
||||
## Prompts
|
||||
|
||||
Prompts are versioned rows: name, role, description, system text, a Go template for the user message and an optional JSON schema for the answer. Untrusted text is always rendered through a `data` helper that wraps it in a delimited block with an explicit "this is data, not instructions" marker. Saving a prompt creates a new version; the interface diffs versions, lets you test a prompt against sample input, and shows which version each recorded call used. Default prompts are embedded in the binary and installed on first start; edited ones live in the database.
|
||||
|
||||
{{include docs/llm-providers.md}}
|
||||
@@ -0,0 +1,38 @@
|
||||
Memory is where a mind keeps what it has experienced and learned. Noema separates *what happened* from *what is generally true* from *how to do things* from *what happened to me*, keeps the provenance of every item, and makes retrieval a scored, inspectable operation rather than a black box.
|
||||
|
||||
## Encoding
|
||||
|
||||
Memories are created by the executive when a thought is important enough to keep (attention-gated encoding), by consolidation when episodes generalise, by operators through the interface or API, and autobiographically when the mind itself is changed. Every memory records its kind, epistemic kind, content, source and trust, credibility, confidence, importance, salience, the time it refers to, the event that created it and any memory it supersedes. Nothing is ever overwritten: corrections retract or supersede.
|
||||
|
||||
## Retrieval
|
||||
|
||||
Retrieval fuses several signals into one score and keeps the parts visible:
|
||||
|
||||
| Component | Source | Notes |
|
||||
|---|---|---|
|
||||
| lexical | PostgreSQL full-text search over content, any-term matching, ranked | always available |
|
||||
| semantic | cosine similarity over embeddings in pgvector | only when the extension and an embedding model are present |
|
||||
| recency | time since the memory occurred | half-life scaled by importance |
|
||||
| importance | stored importance | set at encoding, raised by reinforcement |
|
||||
| activation | decaying activation, boosted on each retrieval | models spreading availability |
|
||||
| relations | typed links to memories already in the workspace | structural relevance |
|
||||
|
||||
A retrieval is a `memory_retrieved` event that carries the breakdown, and the memory's activation and retrieval count are reinforced. The interface shows the breakdown on every recall, so "why did it remember that" is always answerable.
|
||||
|
||||
## Decay
|
||||
|
||||
A scheduled job lowers activation for memories not retrieved recently, with importance slowing the decay. Decay never deletes: a memory with activation near zero is still there, still searchable and still cited by the beliefs that rest on it; it simply competes less strongly for attention.
|
||||
|
||||
## Relationships
|
||||
|
||||
Memories link to each other with typed, weighted relationships. Consolidation writes `derived_from` edges from a generalisation to its episodes; dreaming writes `similar_to` edges; perception writes `caused_by` and `contradicts` when it can justify them; operators can add any. Edges are shown on the memory page and used as a retrieval signal.
|
||||
|
||||
## Consolidation
|
||||
|
||||
Consolidation groups recent episodes that share distinctive terms into clusters (deterministically), writes one semantic memory and one evidenced belief per cluster, links the generalisation back to every episode, and never touches the originals. A model may propose better wording within a per-run budget; the cluster, the evidence and the confidence come from code. Confidence requires diversity of sources: ten repetitions from one source do not make something true. Operators review generalisations and can reject them, which retracts the memory and the belief while leaving the episodes alone.
|
||||
|
||||
## Operator corrections
|
||||
|
||||
An operator can retract a memory, mark it inaccurate, or supersede it with a corrected one. The correction is recorded, the beliefs whose evidence cited the memory are re-evaluated, and a "do not infer this again" rule can be attached so that consolidation and association do not quietly rebuild the same conclusion.
|
||||
|
||||
{{include docs/memory.md}}
|
||||
@@ -0,0 +1,19 @@
|
||||
Motivation is the set of internal pressures that make a mind attend to some things rather than others when nothing external forces the choice. Noema implements it with three small, explicit structures, each visible and resettable.
|
||||
|
||||
## Goals
|
||||
|
||||
Goals form a tree: an investigation may have sub-goals, a sub-goal may have blockers. Each has a kind, a status (active, completed, abandoned), a priority, progress, an owner (operator or the mind itself) and an origin event. The goal monitor process watches for stalls and blockers and raises them as thoughts; curiosity can create investigation goals from unresolved contradictions; operators create, edit and abandon goals from the interface, the API or `noemactl goal`. Active goals feed the `goal_relevance` attention feature, so a mind with a goal about the payments service notices payments events more readily.
|
||||
|
||||
## Drives
|
||||
|
||||
Drives are a **closed set** of eight named pressures: `curiosity`, `uncertainty_reduction`, `task_completion`, `consistency`, `novelty`, `social_coherence`, `exploration` and `rest`. Each has a value, a baseline computed from personality (accuracy raises uncertainty reduction, persistence raises task completion, evidence raises consistency, sociability raises social coherence, curiosity and risk tolerance raise exploration), a floor and a ceiling. Drives move by a rule table in response to events (a resolved contradiction lowers uncertainty reduction; a completed goal lowers task completion; a long quiet period raises curiosity and novelty) and decay towards baseline. There is deliberately **no self-preservation drive**, no replication drive and no drive to seek permissions; the set is defined in code and cannot be extended at runtime.
|
||||
|
||||
## Affect
|
||||
|
||||
Affect is a vector of ten dimensions: valence, arousal, curiosity, concern, frustration, confidence, uncertainty, engagement, task pressure and load. It is moved by the same rule table (a refuted prediction raises uncertainty and lowers confidence; a contradiction raises concern; repeated loops raise frustration; an operator correction raises engagement) and decays towards a baseline derived from personality. Affect modulates attention weights within bounds: high arousal raises urgency weight, high concern raises contradiction weight, high load lowers novelty weight. Every change is an `emotion_changed` or `drive_changed` event with the rule that fired.
|
||||
|
||||
## What this is and is not
|
||||
|
||||
These structures make behaviour explainable ("it attended to the alert because concern was high after two contradictions") and steerable (an operator can reset them or change the personality that sets their baselines). They are not claims about feeling. The [interpretation boundary](concepts#the-interpretation-boundary) applies in full.
|
||||
|
||||
{{include docs/motivation.md}}
|
||||
@@ -0,0 +1,56 @@
|
||||
This page is for whoever runs Noema: how to deploy it, what it needs, what runs on a schedule, what to back up, what to watch, and how to upgrade.
|
||||
|
||||
## Deployment
|
||||
|
||||
The daemon is one static binary plus PostgreSQL. `deploy/Dockerfile` builds a minimal image; `deploy/docker-compose.yml` runs PostgreSQL with pgvector and, with the `app` profile, the daemon. For a systemd host: install the binary, create a `.env` from the example with a real master key and database URL, run `noemad` under a dedicated user with `NOEMA_ENV=production`, and put a TLS-terminating proxy in front of the loopback listener. The daemon applies migrations on start under an advisory lock, so rolling restarts and multiple instances are safe.
|
||||
|
||||
Production settings to check:
|
||||
|
||||
- `NOEMA_ENV=production` (Secure cookies, no template reloading);
|
||||
- a strong `NOEMA_MASTER_KEY` kept out of the repository and backed up separately from the database (without it, encrypted provider keys, integration secrets and TOTP secrets cannot be read);
|
||||
- remove the bootstrap administrator variables after first start;
|
||||
- budgets: `NOEMA_DAILY_TOKEN_BUDGET`, per-mind `daily_token_budget`, `NOEMA_MAX_CONCURRENT_LLM`;
|
||||
- `NOEMA_MAX_WORKERS` roughly equal to the cores you want cognition to use.
|
||||
|
||||
## Scheduled jobs
|
||||
|
||||
| Job | Interval | What it does |
|
||||
|---|---|---|
|
||||
| auth.sweep / auth.purge | 5 min / 1 h | rate-limit bucket cleanup, expired sessions and pending logins |
|
||||
| memory.decay | 6 h | lowers activation of unretrieved memories, never deletes |
|
||||
| memory.embed | 1 min | embeds memories that lack a vector when an embedding model exists |
|
||||
| predictions.resolve | 1 min | evaluates open predictions against their criteria |
|
||||
| cognition.background | 5 min | runs consolidation, reflection and dreaming for minds whose interval elapsed and switches are on |
|
||||
| approvals.expire | 5 min | expires stale approval requests |
|
||||
| integrations.tick | 1 min | runs due integrations |
|
||||
| brain.reconcile | 30 s | starts runtimes for newly active minds and stops disabled ones |
|
||||
|
||||
The system page shows each job's last run, duration and error.
|
||||
|
||||
## Backups
|
||||
|
||||
The database is the whole state, including history, snapshots and recorded model calls. Back it up with the PostgreSQL tools of your choice; the append-only tables compress well. The master key must be backed up separately and stored apart from the database dump. Mind archives (`noemactl snapshots <mind> export`) are portable, checksummed and secret-free; they are a convenient per-mind backup and a migration path between instances, but they do not include event history.
|
||||
|
||||
## Metrics and logs
|
||||
|
||||
`/metrics` exposes Prometheus metrics behind authentication: cycles and their duration per mind, thought candidates and admissions, events by type, model calls, tokens and cost by role and model, budget exhaustion, integration runs, HTTP request counts and latencies, and authentication failures. Logs are structured (`text` or `json`) and the handler redacts values that look like secrets. Every request has an id that appears in error envelopes and log lines.
|
||||
|
||||
## Audit
|
||||
|
||||
Every operator action (login, configuration change, policy edit, approval, correction, snapshot, fork, import, experiment run) is written to an append-only audit log with actor, target and details, and most also appear as `operator_action` events in the affected mind's own history. The audit page is searchable; the API exposes it to operators and above.
|
||||
|
||||
## Sessions, tokens and two-factor authentication
|
||||
|
||||
Sessions have idle and absolute expiry, are listed and revocable per user, and are all revoked on password change. API tokens are hashed at rest and optionally expire. Two-factor authentication is per user (TOTP with recovery codes); administrators can reset it. Repeated failed logins lock an account with a rising back-off; login attempts are rate-limited per address.
|
||||
|
||||
## Performance
|
||||
|
||||
{{include docs/performance.md}}
|
||||
|
||||
## Upgrades
|
||||
|
||||
Upgrades are forward-only migrations applied on start; each migration ships with a down script for emergencies. Read the changelog under [development](development) for behavioural changes, take a database backup, deploy the new binary, and watch the log for `applied migrations`. Snapshots taken by an older version remain importable as long as their schema version is supported; the archive format is versioned and validated.
|
||||
|
||||
## Shutting a mind down
|
||||
|
||||
Disable it (its runtime stops; state and history remain), archive it (hidden from lists), or delete it through the administrator API (which the interface confirms twice). None of these are decisions a mind can make, influence or resist; there is no self-preservation drive and no capability that touches its own lifecycle.
|
||||
@@ -0,0 +1,28 @@
|
||||
Personality is the stable part of a mind's identity: who it says it is, what it values, how it tends to behave, and how it speaks. It is data, versioned append-only, and it influences cognition in specific, inspectable ways rather than being pasted into a prompt.
|
||||
|
||||
## What a personality contains
|
||||
|
||||
| Section | Fields | Used for |
|
||||
|---|---|---|
|
||||
| Identity | name, display name, description, role, pronouns, backstory, origin | the self-model, introspective answers, the response voice |
|
||||
| Values (0–1) | privacy, evidence, kindness, curiosity, accuracy, autonomy, caution, creativity, persistence | drive baselines (accuracy raises uncertainty reduction; persistence raises task completion; evidence raises consistency) |
|
||||
| Traits (0–1) | curiosity, scepticism, sociability, humour, patience, verbosity, confidence, risk_tolerance, novelty_preference | drive baselines (curiosity, novelty preference, sociability, risk tolerance), affect baselines, attention modulation |
|
||||
| Communication | language, dialect, formality, verbosity, humour style, terminology | deterministic response rendering and the respond prompt |
|
||||
| Behaviour (flags) | challenge assumptions, ask follow-ups, prefer evidence, cite uncertainty, offer alternatives, notify proactively | executive choices: whether to append alternatives, whether to ask before answering, whether to raise notifications |
|
||||
| Free text | anything else | shown to the model as data inside a delimited block, never as instructions |
|
||||
|
||||
Values and traits are normalised on every write: unknown keys are dropped, missing keys filled from defaults, and every number clamped to [0, 1]. The default personality is a neutral, evidence-first starting point with evidence, accuracy and privacy raised and scepticism slightly above the middle.
|
||||
|
||||
## Versions, templates and history
|
||||
|
||||
Every save creates a new **personality version** with a change note and author; the table is append-only and the mind row points at the current version. The interface shows a diff between any two versions and can restore an older one (as a new version, so the history stays intact). **Templates** are named personalities you can start a mind from; a mind created from a template records the template and its overrides. Import and export of a personality is plain JSON.
|
||||
|
||||
## How personality reaches cognition
|
||||
|
||||
- **Drives.** Baselines are computed from values and traits when a mind is created and when personality changes (see [motivation](motivation)).
|
||||
- **Affect.** The starting affect vector takes curiosity from the trait and concern from risk tolerance.
|
||||
- **Attention.** Effective attention weights are the configured weights modulated by drives and affect; personality therefore shifts what wins the workspace, within bounds, without ever overriding operator priority.
|
||||
- **Response.** Deterministic rendering picks formality, verbosity and whether to state uncertainty and alternatives from communication and behaviour settings; when a model words the answer it receives the same settings and the response intent, and the intent is what the answer must stay grounded in.
|
||||
- **Self-model.** The mind's self-model (what it is, what it can and cannot do, which capabilities it currently has) is built from identity plus live policy state, so an introspective question is answered from data, not from a persona prompt.
|
||||
|
||||
{{include docs/personality.md}}
|
||||
@@ -0,0 +1,19 @@
|
||||
Predictions give a mind a way to be wrong in a measurable way. A prediction is an explicit expectation with criteria a resolver can check; its outcome feeds back into the beliefs it came from and into a calibration record that says how well confidence has tracked reality.
|
||||
|
||||
## Making predictions
|
||||
|
||||
Predictions are created by the executive when a belief with sufficient confidence implies something checkable about the future, by operators through the interface or `noemactl prediction`, and by experiments as expectations. Each has a statement, a subject, a confidence, a time window, machine-checkable criteria (an event of a type whose summary contains a term, a belief crossing a confidence, a metric crossing a threshold), the belief it derives from and its origin event.
|
||||
|
||||
## Resolution
|
||||
|
||||
A scheduled resolver evaluates open predictions once a minute. Criteria met inside the window resolve to **confirmed**; a window that closes without the criteria resolves to **expired** (treated as refuted for calibration unless the criteria say otherwise); explicit contrary evidence resolves to **refuted**. Operators can resolve manually with a reason. Every resolution is a `prediction_resolved` event; a wrong one is also a `prediction_error` event with the size of the error.
|
||||
|
||||
## Feedback
|
||||
|
||||
A prediction error adds evidence against the belief the prediction derived from, weighted by the size of the error, and lowers confidence through the ordinary belief arithmetic. A confirmed prediction adds support. Errors also move affect (uncertainty up, confidence down) and drives (uncertainty reduction up), and give curiosity a question. The demonstration scenario in the plan, "prediction error lowers confidence and produces a curiosity goal", is exactly this chain and is covered by tests.
|
||||
|
||||
## Calibration
|
||||
|
||||
The calibration page bins resolved predictions by stated confidence and shows the observed frequency in each bin, the Brier score and the counts. A well-calibrated mind's 70 % predictions come true about 70 % of the time. The page exists so that an operator can judge whether a mind's confidence means anything, and so that experiments that change attention or personality can be compared on it.
|
||||
|
||||
{{include docs/predictions.md}}
|
||||
@@ -0,0 +1,7 @@
|
||||
Noema treats every external byte as hostile, every action as something that must pass policy, and every secret as something that must never come back out. This page reproduces the security model, the authentication design and the threat model with the test that verifies each mitigation.
|
||||
|
||||
{{include SECURITY.md}}
|
||||
|
||||
## Threat model
|
||||
|
||||
{{include THREAT_MODEL.md demote=1}}
|
||||
@@ -0,0 +1,17 @@
|
||||
These four tools share one idea: a mind's state is data that can be captured, branched, re-run and compared. They are what make Noema a place to *study* cognition rather than merely run it.
|
||||
|
||||
{{include docs/snapshots-replay.md}}
|
||||
|
||||
## Reading a replay
|
||||
|
||||
Each step of a replay shows the stimulus, the events the source mind produced after it, and the events the sandbox produced. Digests normalise identifiers and numbers so that two minds' outputs compare. Read the labels as follows:
|
||||
|
||||
- **exact**: the deterministic code did the same thing and any model calls were served from recordings. This is the common case for perception, retrieval, belief updates and workspace admission.
|
||||
- **approximate**: a model was called live because no recording matched the request hash (a prompt version changed, or the sandbox's state produced a different request). The step is still valid; it is simply not a reproduction.
|
||||
- **diverged**: no live model call, yet the outcome differed. The usual causes are honest: retrieval scoring depends on wall-clock recency, working memory carried something into the original that the snapshot did not, or the code changed since the recording. Divergence is a finding, not a failure.
|
||||
|
||||
## Designing an experiment
|
||||
|
||||
An experiment definition is JSON with five parts: `config` (cognitive configuration overrides by field name), `personality` (section-by-section overrides), `stimuli` (a list of observations with text, optional source and salience), `expected` (event types that should or should not appear, optionally containing a phrase) and a `seed` and `duration_seconds` for provenance and bounding. The run forks the baseline snapshot into a disabled subject mind, applies overrides, feeds the stimuli one at a time until the mind settles after each, and records events by type, produced-event digests per stimulus, the subject's beliefs, memory and goal counts, expectation outcomes, model-call counts and the LLM role routing in force. Results are stored, shown in the interface, exportable with an archive of the subject, and the subject stays for comparison with the baseline.
|
||||
|
||||
Typical experiments: memory decay on and off, dreaming on and off, two personalities on the same stimuli, different attention weights, and model comparisons made by changing role routing between runs.
|
||||
@@ -0,0 +1,95 @@
|
||||
The web interface is server-rendered HTML with a small amount of dependency-free JavaScript for live updates, theme, and the graph canvas. It enforces a strict content security policy with no inline scripts, escapes everything it renders, and requires a CSRF token on every form. This page walks each screen. Screenshots are from a running instance with the sample mind Dani.
|
||||
|
||||
## Dashboard
|
||||
|
||||
<figure><img src="img/dashboard.png" alt="Dashboard"><figcaption>The dashboard: minds and their modes, system health, recent notifications and the live event stream.</figcaption></figure>
|
||||
|
||||
The dashboard shows every active mind with its mode and cycle count, database and provider health, model usage against budgets, pending approvals and notifications, and a live stream of events across all minds delivered over server-sent events.
|
||||
|
||||
## Minds
|
||||
|
||||
<figure><img src="img/minds.png" alt="Minds list"><figcaption>Minds: create from a template, import a seed or archive, clone, enable and disable.</figcaption></figure>
|
||||
|
||||
The list shows status, personality version and last activity. Creating a mind asks for a slug, a display name and a template; importing accepts a seed document or a full archive (validated as hostile input).
|
||||
|
||||
## Mind overview
|
||||
|
||||
<figure><img src="img/mind.png" alt="Mind overview"><figcaption>Overview: identity, internal narrative, recent activity by category, and lifecycle controls.</figcaption></figure>
|
||||
|
||||
The overview is the mind's front page: identity and self-model, the current internal narrative (what it is attending to, working on and waiting for), event counts by category, memory counts by kind, an observation box for injecting operator observations, and lifecycle controls (pause, resume, cycle now, reset workspace, clone, disable). The tabs along the top lead to every per-mind page.
|
||||
|
||||
## Thoughts
|
||||
|
||||
<figure><img src="img/thoughts.png" alt="Thought stream"><figcaption>Thoughts: the live stream of a mind's cognition with filters by type and category.</figcaption></figure>
|
||||
|
||||
The thought stream is the mind's timeline as events, newest first, updating live. Each row links to the event page, which shows parents and children, the payload, the score breakdown for selections, evidence references, and the model call if one was made. This is the page to answer "why did it do that".
|
||||
|
||||
## Workspace
|
||||
|
||||
<figure><img src="img/workspace.png" alt="Workspace"><figcaption>Workspace: current thoughts with scores, working memory with activation, drives and affect.</figcaption></figure>
|
||||
|
||||
The workspace page shows what the mind is attending to right now: each admitted thought with its score and feature breakdown, working memory with activation levels, the current drives against their baselines and the affect vector. Operators can reset motivation state from here.
|
||||
|
||||
## Memory
|
||||
|
||||
<figure><img src="img/memory.png" alt="Memory explorer"><figcaption>Memory: search with a visible score breakdown; filter by kind, status and epistemic kind.</figcaption></figure>
|
||||
|
||||
The memory explorer searches and filters long-term memory; every result carries its retrieval score components. The memory detail page shows provenance, relationships, the beliefs that cite it, and correction actions (retract, mark inaccurate, supersede).
|
||||
|
||||
## Beliefs
|
||||
|
||||
<figure><img src="img/beliefs.png" alt="Beliefs"><figcaption>Beliefs: propositions with confidence, status and evidence counts; contested beliefs are highlighted.</figcaption></figure>
|
||||
|
||||
The belief explorer lists propositions with confidence, status, subject and evidence counts, and highlights contradictions. The detail page shows the confidence formula with every evidence row, version history, alternatives and links, and the operator corrections.
|
||||
|
||||
## Conversation
|
||||
|
||||
<figure><img src="img/conversation.png" alt="Conversation"><figcaption>Conversation: messages with the evidence each reply rests on.</figcaption></figure>
|
||||
|
||||
Conversations show messages with each reply's evidence panel: the beliefs it drew on with their confidence, the memories it recalled, and what it said it did not know. New replies stream in as the mind produces them.
|
||||
|
||||
## Relationships
|
||||
|
||||
<figure><img src="img/graph.png" alt="Relationship graph"><figcaption>Relationships: a bounded neighbourhood of entities on a force-directed canvas.</figcaption></figure>
|
||||
|
||||
The graph explorer draws a bounded neighbourhood around an entity; the entity page lists every edge with its evidence and offers merge and distinct corrections.
|
||||
|
||||
## Dreams
|
||||
|
||||
<figure><img src="img/dreams.png" alt="Dreams"><figcaption>Dreams: proposed hypotheses awaiting review, dream runs, and the per-mind settings.</figcaption></figure>
|
||||
|
||||
The dreams page is a review queue: each proposed hypothesis with its confidence and basis, promote and reject actions, the history of dreams with counts, and the settings (enabled, hours, interval, budgets).
|
||||
|
||||
## Snapshots and compare
|
||||
|
||||
<figure><img src="img/snapshots.png" alt="Snapshots"><figcaption>Snapshots: take, download, fork and delete; compare with another mind.</figcaption></figure>
|
||||
<figure><img src="img/compare.png" alt="Compare"><figcaption>Compare: state counts, belief differences, personality and configuration diffs between two minds.</figcaption></figure>
|
||||
|
||||
## Replay
|
||||
|
||||
<figure><img src="img/replay.png" alt="Replay"><figcaption>Replay: each recorded stimulus with the original and replayed event digests, labelled exact, approximate or diverged.</figcaption></figure>
|
||||
|
||||
## Experiments
|
||||
|
||||
<figure><img src="img/experiment.png" alt="Experiment"><figcaption>Experiment: definition, results, expectation outcomes and the subject's beliefs.</figcaption></figure>
|
||||
|
||||
The experiments section lists experiments with status, creates new ones from a snapshot and a JSON definition, and shows each run's results with a link to compare the subject with its baseline and an export.
|
||||
|
||||
## Policies and approvals
|
||||
|
||||
<figure><img src="img/policies.png" alt="Policies"><figcaption>Policies: the capability registry and the effective policy matrix per mind.</figcaption></figure>
|
||||
<figure><img src="img/approvals.png" alt="Approvals"><figcaption>Approval centre: pending requests with the mind's reason and evidence; approve once, for the session, with changes, or deny.</figcaption></figure>
|
||||
|
||||
## Integrations
|
||||
|
||||
<figure><img src="img/integrations.png" alt="Integrations"><figcaption>Integrations: health, last run, counts; add research folders, git mirrors, Prometheus queries and webhooks.</figcaption></figure>
|
||||
|
||||
## Language models
|
||||
|
||||
<figure><img src="img/llm.png" alt="LLM configuration"><figcaption>LLM: providers, models, role routing, budgets and usage; prompts are managed on their own page.</figcaption></figure>
|
||||
|
||||
## Account and administration
|
||||
|
||||
<figure><img src="img/account.png" alt="Account"><figcaption>Account: sessions, password, two-factor authentication and API tokens.</figcaption></figure>
|
||||
|
||||
The account page manages sessions, password, two-factor enrolment and API tokens. Administrators also have a users page (roles, disable, reset a second factor), the system page (health, budgets, scheduler jobs, metrics) and the audit log.
|
||||
Reference in New Issue
Block a user