feat(privacy): audited redaction that keeps history's shape and removes its content

Rows in append-only history are never deleted; their content can now be
made irrecoverable through one audited path. A redaction replaces the
content columns of an event, memory, message, model call, belief or
assertion with a fixed marker, keeps identifiers, links, timestamps,
scores and breakdowns, writes an append-only ledger row with a SHA-256 of
the removed content, appends a redaction event, optionally cascades to
derived rows and to the events that quote them, and scrubs stored snapshot
archives before re-signing them. noema_redact() is the only write the
append-only trigger admits, for UPDATE only, under a transaction-local flag
the application never sets; plain updates and all deletes stay refused.
Replay labels redacted stimuli instead of inventing them.

Cryptographic erasure was considered and deferred with reasons (ADR-0009);
docs/privacy.md defines retained, redacted, irrecoverable and deleted and
states what backups and earlier exports are not covered. Migration 0023.
This commit is contained in:
Andy Dixon
2026-08-23 23:52:07 +01:00
parent 9eed6e2a18
commit 7d608c2783
33 changed files with 1245 additions and 45 deletions
+1
View File
@@ -10,6 +10,7 @@ All notable changes to Noema are recorded here. Format follows Keep a Changelog;
- Security suite: SQL-injection effect test, static allow-list scan for assembled SQL, deceived-mind authority test, garbage model output test, self-preservation vocabulary test; threat model rewritten so every verification cites a test that asserts the claim. - Security suite: SQL-injection effect test, static allow-list scan for assembled SQL, deceived-mind authority test, garbage model output test, self-preservation vocabulary test; threat model rewritten so every verification cites a test that asserts the claim.
### Added ### Added
- Privacy redaction (ADR-0009): the audited path that makes an event, memory, message, model call, belief or assertion irrecoverable while keeping the row, its links, timestamps and scores; a ledger with a hash of the removed content; cascade to derived rows and the events that quote them; stored snapshot archives scrubbed and re-signed; replay labels redacted stimuli. `noema_redact()` is the only write the append-only trigger admits. Migration 0023. `docs/privacy.md` defines retained, redacted, irrecoverable and deleted.
- Causal timeline: `GET /api/v1/events/{id}/chain` and `/events/{id}/timeline` walk the links events actually record (causation, parents, cited evidence) backwards and forwards within bounded depth and size, label edge kinds, group by cycle as correlation only, and infer nothing. - Causal timeline: `GET /api/v1/events/{id}/chain` and `/events/{id}/timeline` walk the links events actually record (causation, parents, cited evidence) backwards and forwards within bounded depth and size, label edge kinds, group by cycle as correlation only, and infer nothing.
- Cognitive forks per model: minds may override model routing per role (`llm_roles`, fail-closed); experiments may declare variants that fork one snapshot under different configuration, personality or routing, receive identical stimuli, and are compared deterministically (first divergent stimulus, belief and attention overlap, score differences, memory, goal, cost and capability-request ranges). - Cognitive forks per model: minds may override model routing per role (`llm_roles`, fail-closed); experiments may declare variants that fork one snapshot under different configuration, personality or routing, receive identical stimuli, and are compared deterministically (first divergent stimulus, belief and attention overlap, score differences, memory, goal, cost and capability-request ranges).
- Runtime identity: snapshots, replays and experiments record build (commit, modified tree, Go version, platform), schema version, a secret-free configuration digest, personality version, policy digest, model routing, prompt versions, deterministic subsystem versions and timezone. Replays compare identities and label themselves exact-candidate, different-build or incompatible (the last is never run), and analyse the first divergence by component. Migration 0022. - Runtime identity: snapshots, replays and experiments record build (commit, modified tree, Go version, platform), schema version, a secret-free configuration digest, personality version, policy digest, model routing, prompt versions, deterministic subsystem versions and timezone. Replays compare identities and label themselves exact-candidate, different-build or incompatible (the last is never run), and analyse the first divergence by component. Migration 0022.
+4
View File
@@ -35,6 +35,10 @@ Integrations are disabled until configured, run at a declared trust level, are r
Prompt injection, poisoned documents, hostile model output and manipulated memories can all change what a mind *thinks*. None of them can change what it may *do*. The boundary is `capabilities.Service.Invoke`: it looks the capability up in a registry built in code, validates parameters, evaluates the operator's policy rules, records the decision, and only then executes, asks or denies. Cognition can produce a request; it cannot produce a permission. There is no capability that edits policy, grants approvals or touches a mind's lifecycle, so there is nothing a deceived mind could ask for that would widen its own authority. Prompt delimiters and "this is data" markers are defence in depth for the quality of cognition; they are not the security boundary. `tests/security.TestDeceivedMindHasNoAuthority` forges action requests and hostile model output and checks that nothing executes and policy is unchanged. Prompt injection, poisoned documents, hostile model output and manipulated memories can all change what a mind *thinks*. None of them can change what it may *do*. The boundary is `capabilities.Service.Invoke`: it looks the capability up in a registry built in code, validates parameters, evaluates the operator's policy rules, records the decision, and only then executes, asks or denies. Cognition can produce a request; it cannot produce a permission. There is no capability that edits policy, grants approvals or touches a mind's lifecycle, so there is nothing a deceived mind could ask for that would widen its own authority. Prompt delimiters and "this is data" markers are defence in depth for the quality of cognition; they are not the security boundary. `tests/security.TestDeceivedMindHasNoAuthority` forges action requests and hostile model output and checks that nothing executes and policy is unchanged.
## Privacy and redaction
Cognitive history is append-only, and rows are never deleted. Sensitive content is removed through one audited path: an administrator redacts an event, memory, message, model call, belief or assertion; the content columns become `[redacted]`, a ledger row keeps a hash of what was removed, a `redaction` event is appended, derived rows can be cascaded, and stored snapshot archives are scrubbed. Identifiers, links, timestamps and scores survive so provenance and replay keep their shape. Backups and exports made before a redaction are outside the system's reach. Definitions, the threat model and the reasons cryptographic erasure was not adopted yet are in `docs/privacy.md` (ADR-0009).
## Capability policy defaults ## Capability policy defaults
Read-only capabilities default to `ASK`; anything that writes, sends, restarts, deletes or spends money defaults to `ASK` or `DENY`. `sudo_shell` and `delete_database` are `DENY` and cannot be set to `ALLOW` through the UI. Dreaming may never invoke a capability. There are no self-preservation, replication, concealment or permission-seeking drives, and shutdown is an operator control the mind cannot influence. Read-only capabilities default to `ASK`; anything that writes, sends, restarts, deletes or spends money defaults to `ASK` or `DENY`. `sudo_shell` and `delete_database` are `DENY` and cannot be set to `ALLOW` through the UI. Dreaming may never invoke a capability. There are no self-preservation, replication, concealment or permission-seeking drives, and shutdown is an operator control the mind cannot influence.
+4
View File
@@ -101,6 +101,10 @@ M: model answers are decoded into typed Go structures (unknown fields ignored, w
### T25 Operator-control subversion ### T25 Operator-control subversion
M: no drive or goal kind for self-preservation, replication or permission acquisition; no capability touches policy, approvals, snapshots or the mind's lifecycle; shutdown, pause and disable are authenticated operator routes with no path from cognition. V: `tests/security.TestNoSelfPreservationVocabulary` (drives, goal kinds and the capability registry) and `TestDeceivedMindHasNoAuthority`. ☑ M: no drive or goal kind for self-preservation, replication or permission acquisition; no capability touches policy, approvals, snapshots or the mind's lifecycle; shutdown, pause and disable are authenticated operator routes with no path from cognition. V: `tests/security.TestNoSelfPreservationVocabulary` (drives, goal kinds and the capability registry) and `TestDeceivedMindHasNoAuthority`. ☑
### T26 Personal data that must be erased
Surface: append-only events, memories, messages, model-call records, beliefs, assertions, snapshot archives, exports.
M: an administrator-only, CSRF-protected, audited redaction path replaces content with a marker while keeping identifiers, links, timestamps and scores; a ledger records a hash of the removed content; derived rows cascade; stored archives are scrubbed and re-signed; replay labels redacted stimuli instead of inventing them; `noema_redact()` is the only write the append-only trigger admits, for UPDATE only, under a transaction-local flag the application never sets. V: `internal/privacy.TestRedactionIsIrrecoverableButReferentiallyIntact` (content gone from every table, links intact, hash attested, cascade, archive scrub, plain UPDATE and DELETE still refused, ledger append-only) and `internal/replay.TestReplayIsExactWithoutModel` (redacted stimulus labelled). Not covered by the system: backups, replicas and exports made before the redaction; see `docs/privacy.md`. ☑
## Residual risks ## Residual risks
- A compromised host or database compromises everything. - A compromised host or database compromises everything.
+2
View File
@@ -43,6 +43,7 @@ import (
"github.com/andydixon/noema/internal/notifications" "github.com/andydixon/noema/internal/notifications"
"github.com/andydixon/noema/internal/observability" "github.com/andydixon/noema/internal/observability"
"github.com/andydixon/noema/internal/predictions" "github.com/andydixon/noema/internal/predictions"
"github.com/andydixon/noema/internal/privacy"
"github.com/andydixon/noema/internal/reflection" "github.com/andydixon/noema/internal/reflection"
"github.com/andydixon/noema/internal/replay" "github.com/andydixon/noema/internal/replay"
"github.com/andydixon/noema/internal/scheduler" "github.com/andydixon/noema/internal/scheduler"
@@ -192,6 +193,7 @@ func run() error {
sandboxCfg.Services.LLM = replayLLM sandboxCfg.Services.LLM = replayLLM
ident := &identity.Capturer{DB: db, Config: &cfg, Minds: deps.Minds, LLM: deps.LLMStore, Prompts: deps.Prompts, Capabilities: deps.Capabilities} ident := &identity.Capturer{DB: db, Config: &cfg, Minds: deps.Minds, LLM: deps.LLMStore, Prompts: deps.Prompts, Capabilities: deps.Capabilities}
deps.Snapshots = &snapshots.Store{DB: db, Events: deps.Events, Minds: deps.Minds, Identity: ident} deps.Snapshots = &snapshots.Store{DB: db, Events: deps.Events, Minds: deps.Minds, Identity: ident}
deps.Privacy = &privacy.Service{DB: db, Events: deps.Events, Audit: auditLog, Snapshots: deps.Snapshots, Log: log}
deps.Replay = &replay.Service{DB: db, Events: deps.Events, Minds: deps.Minds, Snapshots: deps.Snapshots, Brain: sandboxCfg, Identity: ident, Log: log} deps.Replay = &replay.Service{DB: db, Events: deps.Events, Minds: deps.Minds, Snapshots: deps.Snapshots, Brain: sandboxCfg, Identity: ident, Log: log}
deps.Experiments = &experiments.Service{DB: db, Minds: deps.Minds, Snapshots: deps.Snapshots, Brain: sandboxCfg, Identity: ident, Log: log, Routing: func(ctx context.Context) []map[string]any { deps.Experiments = &experiments.Service{DB: db, Minds: deps.Minds, Snapshots: deps.Snapshots, Brain: sandboxCfg, Identity: ident, Log: log, Routing: func(ctx context.Context) []map[string]any {
roles, _ := deps.LLMStore.ListRoles(ctx) roles, _ := deps.LLMStore.ListRoles(ctx)
+20
View File
@@ -0,0 +1,20 @@
# ADR-0009: Redaction as the only write on append-only history
Date: 2026-09-15
Status: Accepted
## Context
ADR-0002 made cognitive history append-only and promised a "dedicated, audited maintenance path" for removal. Privacy law and ordinary decency require that personal content can be made irrecoverable, while provenance, replay and forensic reconstruction require that the *shape* of history survives.
## Decision
Content is redacted, rows are never deleted. A `redactions` ledger (append-only) records target, reason, policy, requester and a SHA-256 of the removed content. One SQL function, `noema_redact()`, performs the replacement under a transaction-local flag that the append-only trigger admits for UPDATE only; the function is `SECURITY DEFINER`, revoked from `PUBLIC`, and the application never sets the flag. Content columns are replaced by a fixed marker; identifiers, types, timestamps, links, scores and breakdowns are kept. A `redaction` event is appended. Cascade follows recorded derivations one level (episodes, messages, model calls, assertions, beliefs rooted at the event, and the events that quote each of them). Stored snapshot archives are scrubbed and re-signed.
## Consequences
- Irrecoverability covers everything the system controls; backups, exports and replicas made earlier are the operator's responsibility and are documented as such.
- History keeps its structure: replay labels redacted stimuli instead of inventing them; belief provenance remains inspectable without its text.
- The append-only guarantee is weakened by exactly one, auditable, content-only write path.
## Alternatives considered
- Cryptographic erasure (per-row keys destroyed on request): the strongest option, rejected for now because search, retrieval, contradiction detection, replay and the interface read the columns in clear and per-row decryption would touch every read path; recorded as future work and recommended for backups.
- Disabling the trigger and deleting rows: breaks provenance links and replay, and hides that anything happened. Rejected.
- Leaving content in place with access controls: does not satisfy erasure. Rejected.
+117
View File
@@ -6345,6 +6345,123 @@
} }
} }
}, },
"/api/v1/minds/{mind}/redactions": {
"get": {
"tags": [
"minds"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object"
}
}
}
},
"401": {
"$ref": "#/components/responses/Error"
},
"403": {
"$ref": "#/components/responses/Error"
},
"404": {
"$ref": "#/components/responses/Error"
},
"201": {
"description": "Redacted; irreversible"
}
},
"parameters": [
{
"name": "mind",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"operationId": "redactionsList",
"summary": "list the redaction ledger",
"x-required-role": "operator"
},
"post": {
"tags": [
"minds"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object"
}
}
}
},
"401": {
"$ref": "#/components/responses/Error"
},
"403": {
"$ref": "#/components/responses/Error"
},
"404": {
"$ref": "#/components/responses/Error"
},
"201": {
"description": "Redacted; irreversible"
}
},
"parameters": [
{
"name": "mind",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"operationId": "redactionCreate",
"summary": "redact a row's content irreversibly (event, memory, message, llm_call, belief, assertion), optionally cascading to derived rows",
"x-required-role": "administrator",
"requestBody": {
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"kind": {
"type": "string"
},
"id": {
"type": "string"
},
"reason": {
"type": "string"
},
"policy": {
"type": "string"
},
"cascade": {
"type": "boolean"
}
},
"required": [
"kind",
"id",
"reason"
]
}
}
}
}
}
},
"/api/v1/minds/{mind}/reflections": { "/api/v1/minds/{mind}/reflections": {
"get": { "get": {
"operationId": "reflectionsList", "operationId": "reflectionsList",
+45
View File
@@ -0,0 +1,45 @@
# Privacy, retention and redaction
Noema's history is append-only because provenance, replay and forensic reconstruction depend on it. Personal or sensitive content can nevertheless need to be removed. This page defines exactly what the system can and cannot do about that.
## Definitions
| Term | Meaning in Noema |
|---|---|
| **retained** | the row exists with its content, identifiers, links and timestamps. The default for everything. |
| **redacted** | the row exists with its identifiers, type, timestamps and links intact, but its content columns hold the fixed marker `[redacted]`. A ledger row records when, why, by whom, under which policy, and a SHA-256 of the removed content. |
| **irrecoverable** | no copy of the content remains anywhere this system controls: the live tables, stored snapshot archives and future exports. Backups, exports and replicas made *before* the redaction are outside the system and are the operator's responsibility (see below). |
| **deleted** | not offered for cognitive history. Rows in append-only tables are never deleted; a redacted row is the strongest erasure the system performs. Ordinary mutable rows (sessions, tokens, approvals) are deleted by their own lifecycles. |
## The redaction path
`POST /api/v1/minds/{mind}/redactions` (administrators; also the mind's **Redactions** page) takes a kind (`event`, `memory`, `message`, `llm_call`, `belief`, `assertion`), an id, a reason, an optional policy name and a cascade flag. It runs in one transaction:
1. The row's content is hashed and a `redactions` ledger row is written (append-only, unique per target).
2. `noema_redact()` replaces the content columns. It is the only code path that may update an append-only table: it sets a transaction-local flag that the append-only trigger checks, touches nothing but content and the redaction bookkeeping, and cannot delete. The function is `SECURITY DEFINER`, revoked from `PUBLIC`, and the application never sets the flag itself.
3. A `redaction` event is appended to the mind's history naming the target, the policy and the hash, never the content.
4. With `cascade`, rows derived from the target are redacted too and linked to the primary ledger row: for an event, the episodes encoded from it, the message it carried and its reply, the model calls that worded them, the assertions made in it and the beliefs whose evidence roots at it; for each redacted row, the events that quote it (a perception, an encoding, a belief creation). Redaction events are never cascaded, so the record of redaction survives.
5. Stored snapshot archives of the mind are scrubbed of the same rows, the archive records the redacted ids, and its checksum is recomputed; archives still validate.
6. The action is written to the audit log.
What each kind replaces: events lose `summary`, `payload` and free-text metadata (structural keys such as conversation, message, cycle and belief ids are kept); memories lose `content` and `detail`; messages lose `content` and `intent`; model calls lose `request` and `response`; beliefs lose `proposition` and `detail` and their versions, evidence notes and alternatives; assertions lose `proposition`, `context` and notes. Scores, statuses, confidence breakdown numbers, families, links and timestamps are untouched, so the epistemic shape of the history remains inspectable: an operator can still see that a belief existed, how strong it was, what supported it and what later depended on it.
## Why not cryptographic erasure
Envelope-encrypting payloads with a per-row or per-subject key and destroying the key on erasure was considered and not adopted in this release. Full-text search, semantic retrieval, contradiction detection, the interface and replay all read these columns in clear; per-row decryption on every read would touch every code path and index, and search over ciphertext would need either plaintext shadow columns (defeating the purpose) or a redesign of retrieval. The marker-and-hash approach achieves the same end state for the live database and stored archives with a small, auditable surface. Cryptographic erasure remains the right answer for **backups**, which this system does not manage: encrypt backups under keys you can retire, and treat a redaction as a signal to rotate them on your retention schedule.
## Threat model
| Scenario | Outcome |
|---|---|
| Database compromise after redaction | the content is gone; the hash reveals nothing about it. |
| Database compromise before redaction, or a backup or replica taken before it | the content is present. Redaction cannot reach copies the system did not make. |
| Administrator misuse | redaction is administrator-only, CSRF-protected, audited, and leaves an append-only ledger and a history event; it cannot be used to hide that something happened, only what its content was. |
| Snapshots and exports | stored snapshots are scrubbed; exports and archives made after the redaction carry the marker; those made before do not. |
| Replay | a redacted stimulus is not replayed and the step is labelled `redacted`; the replay never fabricates content. |
| Logs | Noema does not log message, memory or payload content; the structured logger redacts secret-shaped values. Web-server access logs in front of Noema are outside its control. |
| Model providers | content already sent to a provider before the redaction is subject to that provider's retention, not Noema's. |
## Retention
There is no automatic retention purge. Retention is an operator policy applied through the redaction path with a policy name, so every removal is attributable. A future bulk policy runner would use the same function and ledger.
+2
View File
@@ -34,6 +34,7 @@ import (
"github.com/andydixon/noema/internal/notifications" "github.com/andydixon/noema/internal/notifications"
"github.com/andydixon/noema/internal/observability" "github.com/andydixon/noema/internal/observability"
"github.com/andydixon/noema/internal/predictions" "github.com/andydixon/noema/internal/predictions"
"github.com/andydixon/noema/internal/privacy"
"github.com/andydixon/noema/internal/reflection" "github.com/andydixon/noema/internal/reflection"
"github.com/andydixon/noema/internal/scheduler" "github.com/andydixon/noema/internal/scheduler"
"github.com/andydixon/noema/internal/security/secrets" "github.com/andydixon/noema/internal/security/secrets"
@@ -65,6 +66,7 @@ func testServer(t *testing.T) (*httptest.Server, string) {
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log} deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds} deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
deps.Assertions = epistemics.NewStore(pool) deps.Assertions = epistemics.NewStore(pool)
deps.Privacy = &privacy.Service{DB: pool, Events: deps.Events, Audit: deps.Audit, Snapshots: deps.Snapshots, Log: log}
deps.Narrative = narrative.NewStore(pool) deps.Narrative = narrative.NewStore(pool)
deps.Notifications = notifications.NewStore(pool) deps.Notifications = notifications.NewStore(pool)
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log} deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
+117
View File
@@ -6345,6 +6345,123 @@
} }
} }
}, },
"/api/v1/minds/{mind}/redactions": {
"get": {
"tags": [
"minds"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object"
}
}
}
},
"401": {
"$ref": "#/components/responses/Error"
},
"403": {
"$ref": "#/components/responses/Error"
},
"404": {
"$ref": "#/components/responses/Error"
},
"201": {
"description": "Redacted; irreversible"
}
},
"parameters": [
{
"name": "mind",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"operationId": "redactionsList",
"summary": "list the redaction ledger",
"x-required-role": "operator"
},
"post": {
"tags": [
"minds"
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"type": "object"
}
}
}
},
"401": {
"$ref": "#/components/responses/Error"
},
"403": {
"$ref": "#/components/responses/Error"
},
"404": {
"$ref": "#/components/responses/Error"
},
"201": {
"description": "Redacted; irreversible"
}
},
"parameters": [
{
"name": "mind",
"in": "path",
"required": true,
"schema": {
"type": "string"
}
}
],
"operationId": "redactionCreate",
"summary": "redact a row's content irreversibly (event, memory, message, llm_call, belief, assertion), optionally cascading to derived rows",
"x-required-role": "administrator",
"requestBody": {
"content": {
"application/json": {
"schema": {
"type": "object",
"properties": {
"kind": {
"type": "string"
},
"id": {
"type": "string"
},
"reason": {
"type": "string"
},
"policy": {
"type": "string"
},
"cascade": {
"type": "boolean"
}
},
"required": [
"kind",
"id",
"reason"
]
}
}
}
}
}
},
"/api/v1/minds/{mind}/reflections": { "/api/v1/minds/{mind}/reflections": {
"get": { "get": {
"operationId": "reflectionsList", "operationId": "reflectionsList",
+73
View File
@@ -0,0 +1,73 @@
package api
import (
"errors"
"net/http"
"github.com/andydixon/noema/internal/auth"
"github.com/andydixon/noema/internal/privacy"
"github.com/andydixon/noema/pkg/ids"
)
// privacyRoutes: the audited redaction path. Administrators only; every
// call is irreversible by design and says so in its response.
func (s *Server) privacyRoutes() {
m := s.mux
m.Handle("GET /api/v1/minds/{mind}/redactions", require(auth.RoleOperator, s.redactionsList))
m.Handle("POST /api/v1/minds/{mind}/redactions", require(auth.RoleAdministrator, s.redactionCreate))
}
func (s *Server) redactionsList(w http.ResponseWriter, r *http.Request) {
m, ok := s.mind(w, r)
if !ok {
return
}
list, err := s.deps.Privacy.List(r.Context(), m.ID, 200)
if err != nil {
fail(w, r, err)
return
}
if list == nil {
list = []privacy.Redaction{}
}
writeJSON(w, http.StatusOK, map[string]any{"items": list})
}
// RedactionInput is the request body (API and UI form).
type RedactionInput struct {
Kind string `json:"kind"`
ID ids.ID `json:"id"`
Reason string `json:"reason"`
Policy string `json:"policy"`
Cascade bool `json:"cascade"`
}
// Redact performs a redaction; shared with the UI.
func (s *Server) Redact(r *http.Request, mind ids.ID, in RedactionInput, by string) (privacy.Redaction, []privacy.Redaction, error) {
p, c, err := s.deps.Privacy.Redact(r.Context(), privacy.Request{MindID: mind, Kind: privacy.Kind(in.Kind), ID: in.ID, Reason: in.Reason, Policy: in.Policy, Cascade: in.Cascade, By: by})
switch {
case errors.Is(err, privacy.ErrInvalid):
return p, c, ValidationError(err.Error())
case errors.Is(err, privacy.ErrAlreadyRedacted):
return p, c, ValidationError("already redacted")
}
return p, c, err
}
func (s *Server) redactionCreate(w http.ResponseWriter, r *http.Request) {
m, ok := s.mind(w, r)
if !ok {
return
}
var in RedactionInput
if err := decode(r, &in); err != nil {
fail(w, r, err)
return
}
p, c, err := s.Redact(r, m.ID, in, actor(r))
if err != nil {
fail(w, r, err)
return
}
writeJSON(w, http.StatusCreated, map[string]any{"redaction": p, "cascaded": c, "note": "content is irrecoverable; the row, its links and a hash of the removed content remain"})
}
+1
View File
@@ -80,6 +80,7 @@ func (s *Server) routes() {
s.dreamRoutes() s.dreamRoutes()
s.studyRoutes() s.studyRoutes()
s.assertionRoutes() s.assertionRoutes()
s.privacyRoutes()
m.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) { m.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
writeError(w, r, http.StatusNotFound, "not_found", "no such endpoint") writeError(w, r, http.StatusNotFound, "not_found", "no such endpoint")
+2
View File
@@ -34,6 +34,7 @@ import (
"github.com/andydixon/noema/internal/notifications" "github.com/andydixon/noema/internal/notifications"
"github.com/andydixon/noema/internal/observability" "github.com/andydixon/noema/internal/observability"
"github.com/andydixon/noema/internal/predictions" "github.com/andydixon/noema/internal/predictions"
"github.com/andydixon/noema/internal/privacy"
"github.com/andydixon/noema/internal/reflection" "github.com/andydixon/noema/internal/reflection"
"github.com/andydixon/noema/internal/replay" "github.com/andydixon/noema/internal/replay"
"github.com/andydixon/noema/internal/security/secrets" "github.com/andydixon/noema/internal/security/secrets"
@@ -54,6 +55,7 @@ type Deps struct {
Memory *memory.Store Memory *memory.Store
Beliefs *beliefs.Store Beliefs *beliefs.Store
Assertions *epistemics.Store Assertions *epistemics.Store
Privacy *privacy.Service
Conversations *conversation.Store Conversations *conversation.Store
Goals *goals.Store Goals *goals.Store
Motivation *motivation.Store Motivation *motivation.Store
+5 -4
View File
@@ -43,6 +43,7 @@ type Belief struct {
ObservedAt *time.Time `json:"observed_at,omitempty"` ObservedAt *time.Time `json:"observed_at,omitempty"`
ValidFrom *time.Time `json:"valid_from,omitempty"` ValidFrom *time.Time `json:"valid_from,omitempty"`
ValidUntil *time.Time `json:"valid_until,omitempty"` ValidUntil *time.Time `json:"valid_until,omitempty"`
RedactedAt *time.Time `json:"redacted_at,omitempty"`
} }
// EvidenceRow is stored evidence. // EvidenceRow is stored evidence.
@@ -301,14 +302,14 @@ func nullID(id ids.ID) *string {
return &s return &s
} }
const cols = `b.id, b.mind_id, b.proposition, b.subject, b.epistemic, b.status, b.confidence, b.prior, b.version, b.source, b.detail, b.origin_event, b.superseded_by, b.pinned_status, b.first_created, b.last_evaluated, b.evaluations, b.temporal_scope, b.observed_at, b.valid_from, b.valid_until` const cols = `b.id, b.mind_id, b.proposition, b.subject, b.epistemic, b.status, b.confidence, b.prior, b.version, b.source, b.detail, b.origin_event, b.superseded_by, b.pinned_status, b.first_created, b.last_evaluated, b.evaluations, b.temporal_scope, b.observed_at, b.valid_from, b.valid_until, b.redacted_at`
func scan(row pgx.Row) (Belief, error) { func scan(row pgx.Row) (Belief, error) {
var b Belief var b Belief
var epi, status string var epi, status string
var detail []byte var detail []byte
var origin, superseded, pinned *string var origin, superseded, pinned *string
if err := row.Scan(&b.ID, &b.MindID, &b.Proposition, &b.Subject, &epi, &status, &b.Confidence, &b.Prior, &b.Version, &b.Source, &detail, &origin, &superseded, &pinned, &b.FirstCreated, &b.LastEvaluated, &b.Evaluations, &b.TemporalScope, &b.ObservedAt, &b.ValidFrom, &b.ValidUntil); err != nil { if err := row.Scan(&b.ID, &b.MindID, &b.Proposition, &b.Subject, &epi, &status, &b.Confidence, &b.Prior, &b.Version, &b.Source, &detail, &origin, &superseded, &pinned, &b.FirstCreated, &b.LastEvaluated, &b.Evaluations, &b.TemporalScope, &b.ObservedAt, &b.ValidFrom, &b.ValidUntil, &b.RedactedAt); err != nil {
return b, storage.Translate(err) return b, storage.Translate(err)
} }
b.Epistemic, b.Status = concepts.Epistemic(epi), Status(status) b.Epistemic, b.Status = concepts.Epistemic(epi), Status(status)
@@ -793,7 +794,7 @@ func (s *Store) Links(ctx context.Context, mind, id ids.ID) ([]Link, error) {
var detail []byte var detail []byte
var origin, superseded, pinned *string var origin, superseded, pinned *string
if err := rows.Scan(&l.ID, &l.FromID, &l.ToID, &l.Relation, &l.Confidence, &l.CreatedAt, if err := rows.Scan(&l.ID, &l.FromID, &l.ToID, &l.Relation, &l.Confidence, &l.CreatedAt,
&other.ID, &other.MindID, &other.Proposition, &other.Subject, &epi, &status, &other.Confidence, &other.Prior, &other.Version, &other.Source, &detail, &origin, &superseded, &pinned, &other.FirstCreated, &other.LastEvaluated, &other.Evaluations, &other.TemporalScope, &other.ObservedAt, &other.ValidFrom, &other.ValidUntil); err != nil { &other.ID, &other.MindID, &other.Proposition, &other.Subject, &epi, &status, &other.Confidence, &other.Prior, &other.Version, &other.Source, &detail, &origin, &superseded, &pinned, &other.FirstCreated, &other.LastEvaluated, &other.Evaluations, &other.TemporalScope, &other.ObservedAt, &other.ValidFrom, &other.ValidUntil, &other.RedactedAt); err != nil {
return nil, err return nil, err
} }
other.Epistemic, other.Status = concepts.Epistemic(epi), Status(status) other.Epistemic, other.Status = concepts.Epistemic(epi), Status(status)
@@ -919,7 +920,7 @@ func (s *Store) RecentChanges(ctx context.Context, mind ids.ID, since time.Time,
var detail []byte var detail []byte
var origin, superseded, pinned *string var origin, superseded, pinned *string
b := &c.Belief b := &c.Belief
if err := rows.Scan(&b.ID, &b.MindID, &b.Proposition, &b.Subject, &epi, &status, &b.Confidence, &b.Prior, &b.Version, &b.Source, &detail, &origin, &superseded, &pinned, &b.FirstCreated, &b.LastEvaluated, &b.Evaluations, &b.TemporalScope, &b.ObservedAt, &b.ValidFrom, &b.ValidUntil, &fromStatus, &c.FromConf, &c.Reason, &c.At); err != nil { if err := rows.Scan(&b.ID, &b.MindID, &b.Proposition, &b.Subject, &epi, &status, &b.Confidence, &b.Prior, &b.Version, &b.Source, &detail, &origin, &superseded, &pinned, &b.FirstCreated, &b.LastEvaluated, &b.Evaluations, &b.TemporalScope, &b.ObservedAt, &b.ValidFrom, &b.ValidUntil, &b.RedactedAt, &fromStatus, &c.FromConf, &c.Reason, &c.At); err != nil {
return nil, err return nil, err
} }
b.Epistemic, b.Status, c.FromStatus = concepts.Epistemic(epi), Status(status), Status(fromStatus) b.Epistemic, b.Status, c.FromStatus = concepts.Epistemic(epi), Status(status), Status(fromStatus)
+5 -4
View File
@@ -45,6 +45,7 @@ type Message struct {
Intent json.RawMessage `json:"intent,omitempty"` Intent json.RawMessage `json:"intent,omitempty"`
LLMCallID ids.ID `json:"llm_call_id,omitempty"` LLMCallID ids.ID `json:"llm_call_id,omitempty"`
CreatedAt time.Time `json:"created_at"` CreatedAt time.Time `json:"created_at"`
RedactedAt *time.Time `json:"redacted_at,omitempty"`
} }
// ErrInvalid marks bad input. // ErrInvalid marks bad input.
@@ -156,7 +157,7 @@ func (s *Store) Messages(ctx context.Context, mind, conv ids.ID, limit int) ([]M
if limit <= 0 || limit > 1000 { if limit <= 0 || limit > 1000 {
limit = 200 limit = 200
} }
rows, err := s.db.Query(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at FROM messages rows, err := s.db.Query(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at, redacted_at FROM messages
WHERE conversation_id=$1 AND mind_id=$2 ORDER BY created_at DESC LIMIT $3`, conv, mind, limit) WHERE conversation_id=$1 AND mind_id=$2 ORDER BY created_at DESC LIMIT $3`, conv, mind, limit)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -180,7 +181,7 @@ func scanMessage(row pgx.Row) (Message, error) {
var m Message var m Message
var ev, reply, call *string var ev, reply, call *string
var intent *json.RawMessage var intent *json.RawMessage
if err := row.Scan(&m.ID, &m.ConversationID, &m.MindID, &m.Role, &m.Actor, &m.Content, &ev, &reply, &intent, &call, &m.CreatedAt); err != nil { if err := row.Scan(&m.ID, &m.ConversationID, &m.MindID, &m.Role, &m.Actor, &m.Content, &ev, &reply, &intent, &call, &m.CreatedAt, &m.RedactedAt); err != nil {
return m, storage.Translate(err) return m, storage.Translate(err)
} }
if ev != nil { if ev != nil {
@@ -200,7 +201,7 @@ func scanMessage(row pgx.Row) (Message, error) {
// Reply returns the mind's reply to a user message, if any. // Reply returns the mind's reply to a user message, if any.
func (s *Store) Reply(ctx context.Context, mind, userMessage ids.ID) (Message, error) { func (s *Store) Reply(ctx context.Context, mind, userMessage ids.ID) (Message, error) {
return scanMessage(s.db.QueryRow(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at FROM messages WHERE reply_to=$1 AND mind_id=$2 ORDER BY created_at LIMIT 1`, userMessage, mind)) return scanMessage(s.db.QueryRow(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at, redacted_at FROM messages WHERE reply_to=$1 AND mind_id=$2 ORDER BY created_at LIMIT 1`, userMessage, mind))
} }
// Recent returns the last n messages of a conversation for context. // Recent returns the last n messages of a conversation for context.
@@ -211,5 +212,5 @@ func (s *Store) Recent(ctx context.Context, mind, conv ids.ID, n int) ([]Message
// GetMessage fetches one message. // GetMessage fetches one message.
func (s *Store) GetMessage(ctx context.Context, mind, id ids.ID) (Message, error) { func (s *Store) GetMessage(ctx context.Context, mind, id ids.ID) (Message, error) {
return scanMessage(s.db.QueryRow(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at FROM messages WHERE id=$1 AND mind_id=$2`, id, mind)) return scanMessage(s.db.QueryRow(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at, redacted_at FROM messages WHERE id=$1 AND mind_id=$2`, id, mind))
} }
+5 -3
View File
@@ -2,6 +2,7 @@ package epistemics
import ( import (
"context" "context"
"strings"
"time" "time"
"github.com/jackc/pgx/v5" "github.com/jackc/pgx/v5"
@@ -37,6 +38,7 @@ type Assertion struct {
Trust string `json:"trust"` Trust string `json:"trust"`
BeliefID ids.ID `json:"belief_id,omitempty"` BeliefID ids.ID `json:"belief_id,omitempty"`
CreatedAt time.Time `json:"created_at"` CreatedAt time.Time `json:"created_at"`
RedactedAt *time.Time `json:"redacted_at,omitempty"`
} }
// New is the input to Record. // New is the input to Record.
@@ -60,7 +62,7 @@ type Store struct{ db *pgxpool.Pool }
// NewStore creates a store. // NewStore creates a store.
func NewStore(db *pgxpool.Pool) *Store { return &Store{db: db} } func NewStore(db *pgxpool.Pool) *Store { return &Store{db: db} }
const cols = `id, mind_id, actor, actor_kind, attributed_to, proposition, subject, stance, certainty, modality, negated, reported, reported_actor, quoted, hypothetical, interpretation_confidence, interpretation_notes, source_event, conversation_id, message_id, context, trust, belief_id, created_at` const cols = `id, mind_id, actor, actor_kind, attributed_to, proposition, subject, stance, certainty, modality, negated, reported, reported_actor, quoted, hypothetical, interpretation_confidence, interpretation_notes, source_event, conversation_id, message_id, context, trust, belief_id, created_at, redacted_at`
// Record writes an assertion inside q. // Record writes an assertion inside q.
func (s *Store) Record(ctx context.Context, q storage.Querier, n New) (Assertion, error) { func (s *Store) Record(ctx context.Context, q storage.Querier, n New) (Assertion, error) {
@@ -76,7 +78,7 @@ func (s *Store) Record(ctx context.Context, q storage.Querier, n New) (Assertion
// Sarah asserts X" and "Andy doubts X" are both retrievable. // Sarah asserts X" and "Andy doubts X" are both retrievable.
a.Notes = append(a.Notes, "speaker stance: "+in.SpeakerStance) a.Notes = append(a.Notes, "speaker stance: "+in.SpeakerStance)
} }
_, err := q.Exec(ctx, `INSERT INTO assertions (`+cols+`) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24)`, _, err := q.Exec(ctx, `INSERT INTO assertions (`+strings.TrimSuffix(cols, ", redacted_at")+`) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24)`,
a.ID, a.MindID, a.Actor, a.ActorKind, a.AttributedTo, a.Proposition, a.Subject, a.Stance, a.Certainty, a.Modality, a.Negated, a.Reported, a.ReportedActor, a.Quoted, a.Hypothetical, a.ID, a.MindID, a.Actor, a.ActorKind, a.AttributedTo, a.Proposition, a.Subject, a.Stance, a.Certainty, a.Modality, a.Negated, a.Reported, a.ReportedActor, a.Quoted, a.Hypothetical,
a.Interpretation, a.Notes, nullID(a.SourceEvent), nullID(a.ConversationID), nullID(a.MessageID), a.Context, a.Trust, nullID(a.BeliefID), a.CreatedAt) a.Interpretation, a.Notes, nullID(a.SourceEvent), nullID(a.ConversationID), nullID(a.MessageID), a.Context, a.Trust, nullID(a.BeliefID), a.CreatedAt)
return a, err return a, err
@@ -142,7 +144,7 @@ func scan(row pgx.Row) (Assertion, error) {
var a Assertion var a Assertion
var src, conv, msg, bel *string var src, conv, msg, bel *string
if err := row.Scan(&a.ID, &a.MindID, &a.Actor, &a.ActorKind, &a.AttributedTo, &a.Proposition, &a.Subject, &a.Stance, &a.Certainty, &a.Modality, &a.Negated, &a.Reported, &a.ReportedActor, &a.Quoted, &a.Hypothetical, if err := row.Scan(&a.ID, &a.MindID, &a.Actor, &a.ActorKind, &a.AttributedTo, &a.Proposition, &a.Subject, &a.Stance, &a.Certainty, &a.Modality, &a.Negated, &a.Reported, &a.ReportedActor, &a.Quoted, &a.Hypothetical,
&a.Interpretation, &a.Notes, &src, &conv, &msg, &a.Context, &a.Trust, &bel, &a.CreatedAt); err != nil { &a.Interpretation, &a.Notes, &src, &conv, &msg, &a.Context, &a.Trust, &bel, &a.CreatedAt, &a.RedactedAt); err != nil {
return a, storage.Translate(err) return a, storage.Translate(err)
} }
a.SourceEvent, a.ConversationID, a.MessageID, a.BeliefID = idOf(src), idOf(conv), idOf(msg), idOf(bel) a.SourceEvent, a.ConversationID, a.MessageID, a.BeliefID = idOf(src), idOf(conv), idOf(msg), idOf(bel)
+2 -2
View File
@@ -193,7 +193,7 @@ type Filter struct {
Ascending bool Ascending bool
} }
const cols = `id, seq, mind_id, type, category, source, occurred_at, recorded_at, parent_ids, causation_id, correlation_id, salience, confidence, valence, arousal, evidence, summary, metadata, payload` const cols = `id, seq, mind_id, type, category, source, occurred_at, recorded_at, parent_ids, causation_id, correlation_id, salience, confidence, valence, arousal, evidence, summary, metadata, payload, redacted_at`
// Query returns events matching f ordered by seq (descending unless Ascending). // Query returns events matching f ordered by seq (descending unless Ascending).
func (s *Store) Query(ctx context.Context, f Filter) ([]events.Event, error) { func (s *Store) Query(ctx context.Context, f Filter) ([]events.Event, error) {
@@ -358,7 +358,7 @@ func scan(row pgx.Row) (events.Event, error) {
var causation, correlation *string var causation, correlation *string
var evidence, metadata []byte var evidence, metadata []byte
err := row.Scan(&ev.ID, &ev.Seq, &ev.MindID, &typ, &cat, &ev.Source, &ev.OccurredAt, &ev.RecordedAt, &parents, &causation, &correlation, err := row.Scan(&ev.ID, &ev.Seq, &ev.MindID, &typ, &cat, &ev.Source, &ev.OccurredAt, &ev.RecordedAt, &parents, &causation, &correlation,
&ev.Salience, &ev.Confidence, &ev.Valence, &ev.Arousal, &evidence, &ev.Summary, &metadata, &ev.Payload) &ev.Salience, &ev.Confidence, &ev.Valence, &ev.Arousal, &evidence, &ev.Summary, &metadata, &ev.Payload, &ev.RedactedAt)
if err != nil { if err != nil {
return ev, err return ev, err
} }
+4 -3
View File
@@ -60,6 +60,7 @@ type Memory struct {
LastRetrievedAt *time.Time `json:"last_retrieved_at,omitempty"` LastRetrievedAt *time.Time `json:"last_retrieved_at,omitempty"`
OriginEvent ids.ID `json:"origin_event,omitempty"` OriginEvent ids.ID `json:"origin_event,omitempty"`
Supersedes ids.ID `json:"supersedes,omitempty"` Supersedes ids.ID `json:"supersedes,omitempty"`
RedactedAt *time.Time `json:"redacted_at,omitempty"`
} }
// New describes a memory to store. // New describes a memory to store.
@@ -237,7 +238,7 @@ func nullID(id ids.ID) *string {
return &s return &s
} }
const cols = `m.id, m.mind_id, m.kind, m.epistemic, m.status, m.content, m.detail, m.source, m.trust, m.credibility, m.confidence, m.importance, m.salience, m.activation, m.valence, m.arousal, m.occurred_at, m.created_at, m.updated_at, m.retrieval_count, m.last_retrieved_at, m.origin_event, m.supersedes` const cols = `m.id, m.mind_id, m.kind, m.epistemic, m.status, m.content, m.detail, m.source, m.trust, m.credibility, m.confidence, m.importance, m.salience, m.activation, m.valence, m.arousal, m.occurred_at, m.created_at, m.updated_at, m.retrieval_count, m.last_retrieved_at, m.origin_event, m.supersedes, m.redacted_at`
func scan(row pgx.Row) (Memory, error) { func scan(row pgx.Row) (Memory, error) {
var m Memory var m Memory
@@ -245,7 +246,7 @@ func scan(row pgx.Row) (Memory, error) {
var detail []byte var detail []byte
var origin, supersedes *string var origin, supersedes *string
err := row.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation, err := row.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes) &m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &m.RedactedAt)
if err != nil { if err != nil {
return m, storage.Translate(err) return m, storage.Translate(err)
} }
@@ -435,7 +436,7 @@ func (s *Store) Relationships(ctx context.Context, mind, id ids.ID) ([]Related,
m := &r.Other m := &r.Other
if err := rows.Scan(&r.ID, &r.FromMemory, &r.ToMemory, &rel, &r.Confidence, &r.Source, &r.CreatedAt, &r.Direction, if err := rows.Scan(&r.ID, &r.FromMemory, &r.ToMemory, &rel, &r.Confidence, &r.Source, &r.CreatedAt, &r.Direction,
&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation, &m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes); err != nil { &m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &m.RedactedAt); err != nil {
return nil, err return nil, err
} }
r.Relation = concepts.Relation(rel) r.Relation = concepts.Relation(rel)
+2 -2
View File
@@ -88,7 +88,7 @@ func (s *Store) Retrieve(ctx context.Context, mind ids.ID, q Query) ([]Retrieval
var detail []byte var detail []byte
var origin, supersedes *string var origin, supersedes *string
if err := rows.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation, if err := rows.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &rank); err != nil { &m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &m.RedactedAt, &rank); err != nil {
rows.Close() rows.Close()
return nil, err return nil, err
} }
@@ -127,7 +127,7 @@ func (s *Store) Retrieve(ctx context.Context, mind ids.ID, q Query) ([]Retrieval
var detail []byte var detail []byte
var origin, supersedes *string var origin, supersedes *string
if err := rows.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation, if err := rows.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &sim); err != nil { &m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &m.RedactedAt, &sim); err != nil {
break break
} }
m.Kind, m.Epistemic, m.Status, m.Trust = concepts.MemoryKind(kind), concepts.Epistemic(epi), Status(status), events.Trust(trust) m.Kind, m.Epistemic, m.Status, m.Trust = concepts.MemoryKind(kind), concepts.Epistemic(epi), Status(status), events.Trust(trust)
+341
View File
@@ -0,0 +1,341 @@
// Package privacy is the audited path for making sensitive content
// irrecoverable without breaking the history it belongs to (ADR-0009).
//
// A redaction keeps the row, its identifiers, its links, its type and its
// timestamps, and replaces the content columns with a fixed marker. What was
// removed is attested by a SHA-256 of the removed content in the ledger, so
// a later reader can establish that an event existed, when, of what kind,
// what depended on it, that it was redacted, by whom and under which policy,
// while the content itself is gone from the live database and from stored
// snapshot archives. Backups and exports made before the redaction are
// outside this system's reach and are documented as such.
package privacy
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"log/slog"
"strings"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/andydixon/noema/internal/audit"
evstore "github.com/andydixon/noema/internal/events"
"github.com/andydixon/noema/internal/snapshots"
"github.com/andydixon/noema/internal/storage"
"github.com/andydixon/noema/pkg/events"
"github.com/andydixon/noema/pkg/ids"
)
// Kind of row a redaction targets.
type Kind string
// Kinds.
const (
KindEvent Kind = "event"
KindMemory Kind = "memory"
KindMessage Kind = "message"
KindLLMCall Kind = "llm_call"
KindBelief Kind = "belief"
KindAssertion Kind = "assertion"
)
// Marker replaces removed content everywhere.
const Marker = "[redacted]"
// Errors.
var (
ErrInvalid = errors.New("privacy: invalid request")
ErrAlreadyRedacted = errors.New("privacy: already redacted")
)
// Request describes one redaction.
type Request struct {
MindID ids.ID
Kind Kind
ID ids.ID
Reason string
Policy string
Cascade bool // also redact rows derived from the target
By string
}
// Redaction is one ledger row.
type Redaction struct {
ID ids.ID `json:"id"`
MindID ids.ID `json:"mind_id"`
Kind Kind `json:"target_kind"`
TargetID ids.ID `json:"target_id"`
Reason string `json:"reason"`
Policy string `json:"policy"`
ContentHash string `json:"content_hash"`
RequestedBy string `json:"requested_by"`
CascadeOf ids.ID `json:"cascade_of,omitempty"`
CreatedAt time.Time `json:"created_at"`
}
// Service performs redactions.
type Service struct {
DB *pgxpool.Pool
Events *evstore.Store
Audit *audit.Logger
Snapshots *snapshots.Store
Log *slog.Logger
}
// Redact makes the target's content irrecoverable and returns the ledger
// row plus any cascaded rows. Everything runs in one transaction; stored
// snapshot archives are scrubbed afterwards and the outcome is audited.
func (s *Service) Redact(ctx context.Context, req Request) (Redaction, []Redaction, error) {
if req.MindID.IsZero() || req.ID.IsZero() || strings.TrimSpace(req.Reason) == "" || !validKind(req.Kind) {
return Redaction{}, nil, fmt.Errorf("%w: mind, kind, id and reason are required", ErrInvalid)
}
var primary Redaction
var cascaded []Redaction
err := storage.Tx(ctx, s.DB, func(tx pgx.Tx) error {
var err error
primary, err = s.redactOne(ctx, tx, req, ids.Nil)
if err != nil {
return err
}
if !req.Cascade {
return nil
}
deps := s.dependants(ctx, tx, req.MindID, req.Kind, req.ID)
// Events that quote a redacted row (the perception of a message, the
// encoding of an episode, the creation of a belief) are redacted too,
// one level out from the target and from each dependant.
deps = append(deps, s.referencingEvents(ctx, tx, req.MindID, req.Kind, req.ID)...)
seen := map[dep]bool{{req.Kind, req.ID}: true}
for i := 0; i < len(deps); i++ {
d := deps[i]
if seen[d] {
continue
}
seen[d] = true
r, err := s.redactOne(ctx, tx, Request{MindID: req.MindID, Kind: d.kind, ID: d.id, Reason: req.Reason, Policy: req.Policy, By: req.By}, primary.ID)
if errors.Is(err, ErrAlreadyRedacted) {
continue
}
if err != nil {
return err
}
cascaded = append(cascaded, r)
if d.kind != KindEvent {
deps = append(deps, s.referencingEvents(ctx, tx, req.MindID, d.kind, d.id)...)
}
}
return nil
})
if err != nil {
return primary, cascaded, err
}
// Stored archives hold copies of memories, messages and beliefs; scrub them.
if s.Snapshots != nil {
for _, r := range append([]Redaction{primary}, cascaded...) {
if n, err := s.Snapshots.Scrub(ctx, r.MindID, string(r.Kind), r.TargetID); err != nil {
s.Log.Warn("snapshot scrub failed", "redaction", r.ID, "err", err)
} else if n > 0 {
s.Log.Info("snapshot archives scrubbed", "redaction", r.ID, "archives", n)
}
}
}
if s.Audit != nil {
s.Audit.Log(ctx, "privacy.redact", string(req.Kind), req.ID.String(), map[string]any{"redaction_id": primary.ID, "reason": req.Reason, "policy": req.Policy, "cascaded": len(cascaded), "content_hash": primary.ContentHash})
}
return primary, cascaded, nil
}
func (s *Service) redactOne(ctx context.Context, tx pgx.Tx, req Request, cascadeOf ids.ID) (Redaction, error) {
content, mind, err := s.content(ctx, tx, req.Kind, req.ID)
if err != nil {
return Redaction{}, err
}
if mind != req.MindID {
return Redaction{}, fmt.Errorf("%w: %s %s does not belong to this mind", ErrInvalid, req.Kind, req.ID)
}
if content == nil {
return Redaction{}, ErrAlreadyRedacted
}
sum := sha256.Sum256(content)
r := Redaction{ID: ids.New(), MindID: req.MindID, Kind: req.Kind, TargetID: req.ID, Reason: req.Reason, Policy: req.Policy, ContentHash: "sha256:" + hex.EncodeToString(sum[:]), RequestedBy: req.By, CascadeOf: cascadeOf, CreatedAt: time.Now()}
if _, err := tx.Exec(ctx, `INSERT INTO redactions (id, mind_id, target_kind, target_id, reason, policy, content_hash, requested_by, cascade_of, created_at) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)`,
r.ID, r.MindID, string(r.Kind), r.TargetID, r.Reason, r.Policy, r.ContentHash, r.RequestedBy, nullID(cascadeOf), r.CreatedAt); err != nil {
if strings.Contains(err.Error(), "redactions_target_kind_target_id_key") {
return Redaction{}, ErrAlreadyRedacted
}
return Redaction{}, err
}
var n int
if err := tx.QueryRow(ctx, `SELECT noema_redact($1, $2, $3)`, string(req.Kind), req.ID, r.ID).Scan(&n); err != nil {
return Redaction{}, err
}
if n == 0 {
return Redaction{}, ErrAlreadyRedacted
}
// The history records that a redaction happened, never what was removed.
if s.Events != nil {
if _, err := s.Events.AppendIn(ctx, tx, events.New{MindID: req.MindID, Type: events.Redaction, Source: "privacy", Summary: fmt.Sprintf("%s %s redacted under %s", req.Kind, req.ID.String()[len(req.ID.String())-8:], firstNonEmpty(req.Policy, "operator request")),
Salience: 0.2, Confidence: 1, Evidence: []events.EvidenceRef{{Kind: string(req.Kind), ID: string(req.ID), Note: "redacted"}},
Metadata: map[string]any{"redaction_id": r.ID, "target_kind": string(req.Kind), "target_id": req.ID, "policy": req.Policy, "content_hash": r.ContentHash, "cascade_of": cascadeOf},
Payload: map[string]any{"redaction_id": r.ID, "reason": req.Reason}}); err != nil {
return Redaction{}, err
}
}
return r, nil
}
// content returns the JSON of the columns a redaction will remove (nil when
// already redacted) and the row's mind, for the hash and the scoping check.
func (s *Service) content(ctx context.Context, q storage.Querier, kind Kind, id ids.ID) ([]byte, ids.ID, error) {
var mind ids.ID
var already *time.Time
var raw []byte
var err error
switch kind {
case KindEvent:
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('summary', summary, 'payload', payload, 'metadata', metadata) FROM cognitive_events WHERE id=$1`, id).Scan(&mind, &already, &raw)
case KindMemory:
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('content', content, 'detail', detail) FROM memories WHERE id=$1`, id).Scan(&mind, &already, &raw)
case KindMessage:
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('content', content, 'intent', intent) FROM messages WHERE id=$1`, id).Scan(&mind, &already, &raw)
case KindLLMCall:
err = q.QueryRow(ctx, `SELECT coalesce(mind_id, '00000000-0000-0000-0000-000000000000'::uuid), redacted_at, jsonb_build_object('request', request, 'response', response) FROM llm_calls WHERE id=$1`, id).Scan(&mind, &already, &raw)
case KindBelief:
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('proposition', proposition, 'detail', detail) FROM beliefs WHERE id=$1`, id).Scan(&mind, &already, &raw)
case KindAssertion:
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('proposition', proposition, 'context', context) FROM assertions WHERE id=$1`, id).Scan(&mind, &already, &raw)
}
if err != nil {
return nil, ids.Nil, storage.Translate(err)
}
if already != nil {
return nil, mind, nil
}
return raw, mind, nil
}
type dep struct {
kind Kind
id ids.ID
}
// dependants lists rows whose content derives from the target: episodes
// encoded from an event, the message it carried, assertions made in it,
// beliefs whose evidence roots at it, and the model calls that worded the
// messages. Cascade is one level from an event; deeper structure keeps its
// links but not the content.
func (s *Service) dependants(ctx context.Context, q storage.Querier, mind ids.ID, kind Kind, id ids.ID) []dep {
var out []dep
collect := func(k Kind, sql string, args ...any) {
rows, err := q.Query(ctx, sql, args...)
if err != nil {
return
}
defer rows.Close()
for rows.Next() {
var x ids.ID
if rows.Scan(&x) == nil && !x.IsZero() {
out = append(out, dep{k, x})
}
}
}
switch kind {
case KindEvent:
collect(KindMemory, `SELECT id FROM memories WHERE mind_id=$1 AND origin_event=$2 AND redacted_at IS NULL`, mind, id)
collect(KindMessage, `SELECT id FROM messages WHERE mind_id=$1 AND event_id=$2 AND redacted_at IS NULL`, mind, id)
collect(KindAssertion, `SELECT id FROM assertions WHERE mind_id=$1 AND source_event=$2 AND redacted_at IS NULL`, mind, id)
collect(KindBelief, `SELECT DISTINCT belief_id FROM belief_evidence WHERE mind_id=$1 AND $2::uuid = ANY(root_ids)`, mind, id)
collect(KindLLMCall, `SELECT llm_call_id FROM messages WHERE mind_id=$1 AND (event_id=$2 OR reply_to IN (SELECT id FROM messages WHERE event_id=$2)) AND llm_call_id IS NOT NULL`, mind, id)
collect(KindMessage, `SELECT id FROM messages WHERE mind_id=$1 AND reply_to IN (SELECT id FROM messages WHERE event_id=$2) AND redacted_at IS NULL`, mind, id)
case KindMessage:
collect(KindLLMCall, `SELECT llm_call_id FROM messages WHERE mind_id=$1 AND id=$2 AND llm_call_id IS NOT NULL`, mind, id)
case KindBelief:
collect(KindAssertion, `SELECT id FROM assertions WHERE mind_id=$1 AND belief_id=$2 AND redacted_at IS NULL`, mind, id)
}
return out
}
// referencingEvents finds events that quote a row: direct children of an
// event, and events whose evidence or metadata point at the row. Redaction
// events are never included, so the record of redaction survives.
func (s *Service) referencingEvents(ctx context.Context, q storage.Querier, mind ids.ID, kind Kind, id ids.ID) []dep {
var out []dep
ref := fmt.Sprintf(`[{"kind":%q,"id":%q}]`, string(kind), id.String())
metaKey := string(kind) + "_id"
rows, err := q.Query(ctx, `SELECT id FROM cognitive_events WHERE mind_id=$1 AND redacted_at IS NULL AND type <> 'redaction'
AND (evidence @> $2::jsonb OR metadata->>$3 = $4 OR ($5 AND (causation_id=$6 OR $6 = ANY(parent_ids)))) LIMIT 500`, mind, ref, metaKey, id.String(), kind == KindEvent, id)
if err != nil {
return nil
}
defer rows.Close()
for rows.Next() {
var x ids.ID
if rows.Scan(&x) == nil && x != id {
out = append(out, dep{KindEvent, x})
}
}
return out
}
// List returns the ledger for a mind, newest first.
func (s *Service) List(ctx context.Context, mind ids.ID, limit int) ([]Redaction, error) {
if limit <= 0 || limit > 500 {
limit = 100
}
rows, err := s.DB.Query(ctx, `SELECT id, mind_id, target_kind, target_id, reason, policy, content_hash, requested_by, cascade_of, created_at FROM redactions WHERE mind_id=$1 ORDER BY created_at DESC LIMIT $2`, mind, limit)
if err != nil {
return nil, err
}
defer rows.Close()
var out []Redaction
for rows.Next() {
var r Redaction
var kind string
var cascade *string
if err := rows.Scan(&r.ID, &r.MindID, &kind, &r.TargetID, &r.Reason, &r.Policy, &r.ContentHash, &r.RequestedBy, &cascade, &r.CreatedAt); err != nil {
return nil, err
}
r.Kind = Kind(kind)
if cascade != nil {
r.CascadeOf = ids.ID(*cascade)
}
out = append(out, r)
}
return out, rows.Err()
}
func validKind(k Kind) bool {
switch k {
case KindEvent, KindMemory, KindMessage, KindLLMCall, KindBelief, KindAssertion:
return true
}
return false
}
func nullID(id ids.ID) *string {
if id.IsZero() {
return nil
}
s := string(id)
return &s
}
func firstNonEmpty(a, b string) string {
if a != "" {
return a
}
return b
}
// Attest recomputes nothing: redacted content is gone. It exposes the hash
// so an operator holding an external copy can prove what was removed.
func Attest(r Redaction) string { return r.ContentHash }
var _ = json.Marshal
+197
View File
@@ -0,0 +1,197 @@
package privacy
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"log/slog"
"strings"
"testing"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/andydixon/noema/internal/audit"
"github.com/andydixon/noema/internal/beliefs"
"github.com/andydixon/noema/internal/epistemics"
evstore "github.com/andydixon/noema/internal/events"
"github.com/andydixon/noema/internal/memory"
"github.com/andydixon/noema/internal/minds"
"github.com/andydixon/noema/internal/snapshots"
"github.com/andydixon/noema/internal/storage"
"github.com/andydixon/noema/migrations"
"github.com/andydixon/noema/pkg/concepts"
"github.com/andydixon/noema/pkg/events"
"github.com/andydixon/noema/pkg/ids"
)
// TestRedactionIsIrrecoverableButReferentiallyIntact: content goes, the
// row, its links, its type and time stay; the hash attests what was removed;
// derived rows are cascaded; stored archives are scrubbed; and the general
// append-only guarantee still holds for every other write.
func TestRedactionIsIrrecoverableButReferentiallyIntact(t *testing.T) {
pool := storage.TestPool(t, migrations.FS)
ctx := context.Background()
log := slog.New(slog.DiscardHandler)
ms := minds.NewStore(pool)
m, _ := ms.Create(ctx, minds.CreateParams{Slug: "priv"})
es := evstore.NewStore(pool, evstore.NewBus(8), nil)
mem := memory.NewStore(ctx, pool, nil)
bs := beliefs.NewStore(pool)
as := epistemics.NewStore(pool)
snaps := &snapshots.Store{DB: pool, Events: es, Minds: ms}
secret := "Andy said Steve steals laptops from the Leeds office"
obs, err := es.Append(ctx, events.New{MindID: m.ID, Type: events.Observation, Source: "conversation", Summary: secret, Payload: events.ObservationPayload{Text: secret, Actor: "andy", Trust: events.TrustUser}, Metadata: map[string]any{"conversation_id": "c1", "channel": "conversation"}})
if err != nil {
t.Fatal(err)
}
child, _ := es.Append(ctx, events.New{MindID: m.ID, Type: events.Perception, Source: "brain.perceive", Summary: "perceived: " + secret, CausationID: obs[0].ID, ParentIDs: []ids.ID{obs[0].ID}})
episode, _ := mem.Create(ctx, pool, memory.New{MindID: m.ID, Kind: concepts.Episodic, Epistemic: concepts.Observation, Content: "andy said: " + secret, Source: "conversation", Trust: events.TrustUser, Confidence: 1, Importance: 0.5, OccurredAt: time.Now(), OriginEvent: obs[0].ID})
emit := func(ctx context.Context, q storage.Querier, n events.New) (events.Event, error) {
out, err := es.AppendIn(ctx, q, n)
if err != nil {
return events.Event{}, err
}
return out[0], nil
}
res, _ := bs.Create(ctx, pool, emit, beliefs.New{MindID: m.ID, Proposition: "Steve steals laptops", Subject: "steve", Epistemic: concepts.Claim, Source: "conversation",
Evidence: []beliefs.EvidenceInput{{Direction: "for", Kind: "assertion", Source: "asserted-by:andy", Family: "actor:andy", Derivation: beliefs.DerivationAssertion, Weight: 0.5, Credibility: 0.85, Note: "andy asserted it", RootIDs: []string{string(obs[0].ID)}}}})
assertion, _ := as.Record(ctx, pool, epistemics.New{MindID: m.ID, Actor: "andy", Interpretation: epistemics.Interpret("andy", "Steve steals laptops"), SourceEvent: obs[0].ID, Context: secret, BeliefID: res.Belief.ID})
convID, msgID, replyID, callID := ids.New(), ids.New(), ids.New(), ids.New()
pool.Exec(ctx, `INSERT INTO conversations (id, mind_id, title, channel) VALUES ($1,$2,'t','conversation')`, convID, m.ID) //nolint:errcheck
pool.Exec(ctx, `INSERT INTO llm_calls (id, mind_id, role, provider, model, request_hash, request, response) VALUES ($1,$2,'respond','p','m','h', $3, $4)`, callID, m.ID, `{"messages":[{"content":"`+secret+`"}]}`, `{"text":"noted"}`) //nolint:errcheck
if _, err := pool.Exec(ctx, `INSERT INTO messages (id, conversation_id, mind_id, role, actor, content, event_id) VALUES ($1,$2,$3,'user','andy',$4,$5)`, msgID, convID, m.ID, secret, obs[0].ID); err != nil {
t.Fatal(err)
}
pool.Exec(ctx, `INSERT INTO messages (id, conversation_id, mind_id, role, actor, content, reply_to, llm_call_id) VALUES ($1,$2,$3,'mind','mind','Noted: Steve steals laptops',$4,$5)`, replyID, convID, m.ID, msgID, callID) //nolint:errcheck
sn, err := snaps.Create(ctx, m, "before", "", "test")
if err != nil {
t.Fatal(err)
}
if raw, _ := json.Marshal(mustArchive(t, snaps, sn.ID)); !strings.Contains(string(raw), "Leeds") {
t.Fatal("test setup: archive should contain the secret before redaction")
}
svc := &Service{DB: pool, Events: es, Audit: audit.New(pool, log), Snapshots: snaps, Log: log}
// Refuses bad requests and rows from other minds.
other, _ := ms.Create(ctx, minds.CreateParams{Slug: "other"})
if _, _, err := svc.Redact(ctx, Request{MindID: other.ID, Kind: KindEvent, ID: obs[0].ID, Reason: "x", By: "admin"}); err == nil {
t.Fatal("redacting another mind's row must fail")
}
if _, _, err := svc.Redact(ctx, Request{MindID: m.ID, Kind: KindEvent, ID: obs[0].ID, By: "admin"}); err == nil {
t.Fatal("a reason is required")
}
primary, cascaded, err := svc.Redact(ctx, Request{MindID: m.ID, Kind: KindEvent, ID: obs[0].ID, Reason: "data subject request", Policy: "GDPR art. 17", Cascade: true, By: "admin"})
if err != nil {
t.Fatal(err)
}
// The event exists, keeps type, time, links and ids; content is a marker.
ev, _ := es.Get(ctx, obs[0].ID)
if ev.Summary != Marker || string(ev.Payload) != `{"redacted": true}` || ev.RedactedAt == nil || ev.Type != events.Observation || ev.Seq != obs[0].Seq {
t.Fatalf("event after redaction: %+v payload=%s", ev, ev.Payload)
}
if ev.Metadata["conversation_id"] != "c1" || ev.Metadata["redacted"] != true {
t.Fatalf("metadata should keep only structural keys: %+v", ev.Metadata)
}
if kids, _ := es.Children(ctx, obs[0].ID, 10); len(kids) != 1 || kids[0].ID != child[0].ID {
t.Fatal("links to dependants must survive")
}
// The hash attests the removed content.
want := sha256.Sum256(mustJSONB(t, pool, `SELECT jsonb_build_object('summary', $1::text, 'payload', $2::jsonb, 'metadata', $3::jsonb)`, secret, string(obs[0].Payload), mustJSON(obs[0].Metadata)))
if primary.ContentHash != "sha256:"+hex.EncodeToString(want[:]) {
t.Fatalf("hash: %s", primary.ContentHash)
}
// Cascade reached the episode, the message, the reply, the model call, the assertion and the belief.
kinds := map[Kind]int{}
for _, c := range cascaded {
kinds[c.Kind]++
if c.CascadeOf != primary.ID {
t.Fatalf("cascade lineage missing: %+v", c)
}
}
if kinds[KindMemory] != 1 || kinds[KindMessage] != 2 || kinds[KindLLMCall] != 1 || kinds[KindAssertion] != 1 || kinds[KindBelief] != 1 {
t.Fatalf("cascade: %+v", kinds)
}
if mm, _ := mem.Get(ctx, m.ID, episode.ID); mm.Content != Marker || mm.RedactedAt == nil || mm.OriginEvent != obs[0].ID {
t.Fatalf("memory: %+v", mm)
}
if b, _ := bs.Get(ctx, pool, m.ID, res.Belief.ID); b.Proposition != Marker || b.Confidence != res.Belief.Confidence {
t.Fatalf("belief proposition should be redacted and its score untouched: %+v", b)
}
var vprop, note, msg, reply, req string
var resp *string
pool.QueryRow(ctx, `SELECT proposition FROM belief_versions WHERE belief_id=$1 LIMIT 1`, res.Belief.ID).Scan(&vprop) //nolint:errcheck
pool.QueryRow(ctx, `SELECT note FROM belief_evidence WHERE belief_id=$1 LIMIT 1`, res.Belief.ID).Scan(&note) //nolint:errcheck
pool.QueryRow(ctx, `SELECT content FROM messages WHERE id=$1`, msgID).Scan(&msg) //nolint:errcheck
pool.QueryRow(ctx, `SELECT content FROM messages WHERE id=$1`, replyID).Scan(&reply) //nolint:errcheck
pool.QueryRow(ctx, `SELECT request::text, response::text FROM llm_calls WHERE id=$1`, callID).Scan(&req, &resp) //nolint:errcheck
if vprop != Marker || note != Marker || msg != Marker || reply != Marker || strings.Contains(req, "Leeds") || resp != nil {
t.Fatalf("cascade content: version=%q note=%q msg=%q reply=%q req=%q", vprop, note, msg, reply, req)
}
if a, _ := as.List(ctx, m.ID, epistemics.Filter{Limit: 5}); len(a) != 1 || a[0].Proposition != Marker || a[0].ID != assertion.ID {
t.Fatalf("assertion: %+v", a)
}
// Nothing in the live database still holds the secret.
var leaks int
pool.QueryRow(ctx, `SELECT (SELECT count(*) FROM cognitive_events WHERE mind_id=$1 AND (summary ILIKE '%leeds%' OR payload::text ILIKE '%leeds%' OR metadata::text ILIKE '%leeds%'))
+ (SELECT count(*) FROM memories WHERE mind_id=$1 AND content ILIKE '%leeds%') + (SELECT count(*) FROM messages WHERE mind_id=$1 AND content ILIKE '%leeds%')
+ (SELECT count(*) FROM llm_calls WHERE mind_id=$1 AND request::text ILIKE '%leeds%') + (SELECT count(*) FROM assertions WHERE mind_id=$1 AND (proposition ILIKE '%leeds%' OR context ILIKE '%leeds%'))`, m.ID).Scan(&leaks) //nolint:errcheck
if leaks != 0 {
t.Fatalf("secret still present in %d rows", leaks)
}
// Stored archives were scrubbed and re-signed; they still validate.
a := mustArchive(t, snaps, sn.ID)
if raw, _ := json.Marshal(a); strings.Contains(string(raw), "Leeds") || strings.Contains(string(raw), "steals") {
t.Fatal("archive still holds redacted content")
}
if len(a.Redacted) < 3 || snapshots.Validate(a) != nil {
t.Fatalf("archive scrub record or checksum: %v %v", a.Redacted, snapshots.Validate(a))
}
// The ledger and the history record what happened, not what was removed.
ledger, _ := svc.List(ctx, m.ID, 20)
if len(ledger) != 1+len(cascaded) || ledger[0].Reason != "data subject request" {
t.Fatalf("ledger: %d", len(ledger))
}
reds, _ := es.Query(ctx, evstore.Filter{MindID: m.ID, Types: []events.Type{events.Redaction}, Limit: 20})
if len(reds) != 1+len(cascaded) || strings.Contains(reds[0].Summary, "Leeds") {
t.Fatalf("redaction events: %d", len(reds))
}
// Idempotent and still append-only for everything else.
if _, _, err := svc.Redact(ctx, Request{MindID: m.ID, Kind: KindEvent, ID: obs[0].ID, Reason: "again", By: "admin"}); err == nil {
t.Fatal("second redaction should report already redacted")
}
if _, err := pool.Exec(ctx, `UPDATE cognitive_events SET summary='tampered' WHERE id=$1`, child[0].ID); err == nil {
t.Fatal("plain updates must still be forbidden")
}
if _, err := pool.Exec(ctx, `DELETE FROM cognitive_events WHERE id=$1`, obs[0].ID); err == nil {
t.Fatal("deletes must still be forbidden, even of redacted rows")
}
if _, err := pool.Exec(ctx, `UPDATE redactions SET reason='x' WHERE id=$1`, primary.ID); err == nil {
t.Fatal("the ledger is append-only")
}
}
func mustArchive(t *testing.T, s *snapshots.Store, id ids.ID) snapshots.Archive {
t.Helper()
a, err := s.Archive(context.Background(), id)
if err != nil {
t.Fatal(err)
}
return a
}
func mustJSON(v any) string {
b, _ := json.Marshal(v)
return string(b)
}
func mustJSONB(t *testing.T, pool *pgxpool.Pool, sql string, args ...any) []byte {
t.Helper()
var raw []byte
if err := pool.QueryRow(context.Background(), sql, args...).Scan(&raw); err != nil {
t.Fatal(err)
}
return raw
}
+8
View File
@@ -69,6 +69,7 @@ type Summary struct {
LLMLive int `json:"llm_live"` LLMLive int `json:"llm_live"`
LLMReplayed int `json:"llm_replayed"` LLMReplayed int `json:"llm_replayed"`
Cycles int `json:"cycles"` Cycles int `json:"cycles"`
Redacted int `json:"redacted"` // stimuli whose content was redacted and could not be replayed
} }
// Step is the outcome of replaying one stimulus. // Step is the outcome of replaying one stimulus.
@@ -218,6 +219,13 @@ func (s *Service) Execute(ctx context.Context, r Replay) (Replay, error) {
} }
orig, _ := s.Events.Query(ctx, evstore.Filter{MindID: src.ID, AfterSeq: stim.Seq, BeforeSeq: upper, Ascending: true, Limit: 2000}) orig, _ := s.Events.Query(ctx, evstore.Filter{MindID: src.ID, AfterSeq: stim.Seq, BeforeSeq: upper, Ascending: true, Limit: 2000})
st.Original = Digest(withoutPerception(orig)) st.Original = Digest(withoutPerception(orig))
if stim.RedactedAt != nil {
// The content is gone; replaying a marker would fabricate cognition.
st.Label, st.Note = "redacted", "stimulus content was redacted under a privacy policy and was not replayed"
r.Summary.Redacted++
steps = append(steps, st)
continue
}
startedAt := time.Now() startedAt := time.Now()
inj, err := drv.Inject(ctx, stim, map[string]any{"replay_of": stim.ID, "replay_id": r.ID}) inj, err := drv.Inject(ctx, stim, map[string]any{"replay_of": stim.ID, "replay_id": r.ID})
if err != nil { if err != nil {
+12
View File
@@ -87,6 +87,18 @@ func TestReplayIsExactWithoutModel(t *testing.T) {
if after[0].Seq != last[0].Seq { if after[0].Seq != last[0].Seq {
t.Fatal("replay wrote into the source mind") t.Fatal("replay wrote into the source mind")
} }
// A redacted stimulus is not replayed and is labelled as such.
if _, err := h.Pool.Exec(ctx, `INSERT INTO redactions (id, mind_id, target_kind, target_id, reason, content_hash, requested_by) VALUES ($1,$2,'event',$3,'test','sha256:0','test')`, ids.New(), src.ID, steps[0].StimulusID); err != nil {
t.Fatal(err)
}
var n int
if err := h.Pool.QueryRow(ctx, `SELECT noema_redact('event', $1, $2)`, steps[0].StimulusID, ids.New()).Scan(&n); err != nil || n != 1 {
t.Fatalf("redact: %v %d", err, n)
}
red, err := svc.Run(ctx, src, sn.ID, sn.EventSeq, last[0].Seq, "test")
if err != nil || red.Summary.Redacted != 1 || red.Summary.Exact != 0 || StepsOf(red)[0].Label != "redacted" {
t.Fatalf("redacted stimulus should be skipped and labelled: %v %+v", err, red.Summary)
}
// Digest normalisation ignores ids and numbers. // Digest normalisation ignores ids and numbers.
if Normalise("selected 3 thoughts for 01a0a148-9034-7685-9794-cf5ec0c25fb7") != "selected <n> thoughts for <id>" { if Normalise("selected 3 thoughts for 01a0a148-9034-7685-9794-cf5ec0c25fb7") != "selected <n> thoughts for <id>" {
t.Fatal(Normalise("selected 3 thoughts for 01a0a148-9034-7685-9794-cf5ec0c25fb7")) t.Fatal(Normalise("selected 3 thoughts for 01a0a148-9034-7685-9794-cf5ec0c25fb7"))
+79
View File
@@ -68,6 +68,10 @@ type Archive struct {
// Tables maps table name → rows (column → value, as JSON). // Tables maps table name → rows (column → value, as JSON).
Tables map[string][]map[string]any `json:"tables"` Tables map[string][]map[string]any `json:"tables"`
Checksum string `json:"checksum"` Checksum string `json:"checksum"`
// Redacted lists row ids whose content was removed from this archive
// after capture under a privacy redaction; the checksum covers the
// scrubbed content.
Redacted []string `json:"redacted,omitempty"`
} }
// Source records where an archive came from. // Source records where an archive came from.
@@ -544,3 +548,78 @@ func SortedTables(a Archive) []string {
sort.Strings(out) sort.Strings(out)
return out return out
} }
// Scrub removes the content of a redacted row from every stored archive of
// the mind, recording the id in the archive and re-signing it. Returns the
// number of archives changed. Events and model calls are never in archives.
func (s *Store) Scrub(ctx context.Context, mind ids.ID, kind string, id ids.ID) (int, error) {
rows, err := s.DB.Query(ctx, `SELECT id, archive FROM snapshots WHERE mind_id=$1`, mind)
if err != nil {
return 0, err
}
type snap struct {
id ids.ID
raw []byte
}
var snaps []snap
for rows.Next() {
var sn snap
if err := rows.Scan(&sn.id, &sn.raw); err != nil {
rows.Close()
return 0, err
}
snaps = append(snaps, sn)
}
rows.Close()
changed := 0
target := id.String()
for _, sn := range snaps {
var a Archive
if err := json.Unmarshal(sn.raw, &a); err != nil {
continue
}
if !scrubArchive(&a, kind, target) {
continue
}
a.Redacted = append(a.Redacted, target)
a.Checksum = Sum(a)
raw, _ := json.Marshal(a)
counts, _ := json.Marshal(a.Counts())
if _, err := s.DB.Exec(ctx, `UPDATE snapshots SET archive=$2, checksum=$3, size_bytes=$4, counts=$5 WHERE id=$1`, sn.id, raw, a.Checksum, len(raw), counts); err != nil {
return changed, err
}
changed++
}
return changed, nil
}
// scrubArchive replaces the content fields of the matching rows in place.
func scrubArchive(a *Archive, kind, id string) bool {
changed := false
replace := func(table string, match func(r map[string]any) bool, fields map[string]any) {
for _, r := range a.Tables[table] {
if match(r) {
for k, v := range fields {
if _, ok := r[k]; ok {
r[k] = v
}
}
changed = true
}
}
}
byID := func(r map[string]any) bool { return r["id"] == id }
switch kind {
case "memory":
replace("memories", byID, map[string]any{"content": "[redacted]", "detail": map[string]any{"redacted": true}})
case "message":
replace("messages", byID, map[string]any{"content": "[redacted]", "intent": nil})
case "belief":
replace("beliefs", byID, map[string]any{"proposition": "[redacted]", "detail": map[string]any{"redacted": true}})
byBelief := func(r map[string]any) bool { return r["belief_id"] == id }
replace("belief_versions", byBelief, map[string]any{"proposition": "[redacted]", "reason": "[redacted]"})
replace("belief_evidence", byBelief, map[string]any{"note": "[redacted]"})
replace("belief_alternatives", byBelief, map[string]any{"proposition": "[redacted]", "note": "[redacted]"})
}
return changed
}
+43
View File
@@ -0,0 +1,43 @@
package ui
import (
"net/http"
"strconv"
"github.com/andydixon/noema/internal/api"
"github.com/andydixon/noema/internal/auth"
"github.com/andydixon/noema/pkg/ids"
)
func (u *UI) privacyRoutes(m *http.ServeMux) {
m.Handle("GET /minds/{mind}/redactions", u.require(auth.RoleOperator, u.redactionsPage))
m.Handle("POST /minds/{mind}/redactions", u.require(auth.RoleAdministrator, u.redact))
}
func (u *UI) redactionsPage(w http.ResponseWriter, r *http.Request) {
m, ok := u.mind(w, r)
if !ok {
return
}
list, _ := u.deps.Privacy.List(r.Context(), m.ID, 200)
u.render(w, r, "redactions", m.DisplayName+" · Redactions", map[string]any{"Mind": m, "Redactions": list, "Kind": r.URL.Query().Get("kind"), "ID": r.URL.Query().Get("id")})
}
func (u *UI) redact(w http.ResponseWriter, r *http.Request) {
m, ok := u.mind(w, r)
if !ok {
return
}
id, err := ids.Parse(r.PostFormValue("id"))
if err != nil {
u.flash(w, r, "error", "Bad id.", "/minds/"+m.Slug+"/redactions")
return
}
cascade, _ := strconv.ParseBool(r.PostFormValue("cascade"))
p, c, err := u.studyF().Redact(r, m.ID, api.RedactionInput{Kind: r.PostFormValue("kind"), ID: id, Reason: r.PostFormValue("reason"), Policy: r.PostFormValue("policy"), Cascade: cascade}, username(r))
if err != nil {
u.flash(w, r, "error", err.Error(), "/minds/"+m.Slug+"/redactions")
return
}
u.flash(w, r, "ok", "Redacted "+string(p.Kind)+" "+p.TargetID.String()[len(p.TargetID.String())-8:]+" and "+itoa(len(c))+" dependant row(s). The content is irrecoverable; hash "+p.ContentHash[:23]+"… attests what was removed.", "/minds/"+m.Slug+"/redactions")
}
+2
View File
@@ -133,6 +133,7 @@ func (u *UI) parse(fsys fs.FS) error {
"duration": func(s int64) string { return (time.Duration(s) * time.Second).Truncate(time.Second).String() }, "duration": func(s int64) string { return (time.Duration(s) * time.Second).Truncate(time.Second).String() },
"bytes": humanBytes, "bytes": humanBytes,
"pct": func(f float64) string { return fmt.Sprintf("%.0f%%", f*100) }, "pct": func(f float64) string { return fmt.Sprintf("%.0f%%", f*100) },
"list": func(xs ...string) []string { return xs },
"conf": func(v any) string { "conf": func(v any) string {
switch x := v.(type) { switch x := v.(type) {
case float64: case float64:
@@ -388,6 +389,7 @@ func (u *UI) Register(m *http.ServeMux) {
u.integrationRoutes(m) u.integrationRoutes(m)
u.dreamRoutes(m) u.dreamRoutes(m)
u.studyRoutes(m) u.studyRoutes(m)
u.privacyRoutes(m)
for _, p := range []struct{ path, title, note string }{} { for _, p := range []struct{ path, title, note string }{} {
m.Handle("GET "+p.path, u.require(auth.RoleObserver, func(w http.ResponseWriter, r *http.Request) { u.render(w, r, "placeholder", p.title, p.note) })) m.Handle("GET "+p.path, u.require(auth.RoleObserver, func(w http.ResponseWriter, r *http.Request) { u.render(w, r, "placeholder", p.title, p.note) }))
+13
View File
@@ -0,0 +1,13 @@
-- Redacted content cannot be restored; this only removes the machinery.
DROP FUNCTION IF EXISTS noema_redact(text, uuid, uuid);
CREATE OR REPLACE FUNCTION noema_forbid_change() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
RAISE EXCEPTION 'table % is append-only', TG_TABLE_NAME USING ERRCODE = 'integrity_constraint_violation';
END $$;
ALTER TABLE assertions DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
ALTER TABLE beliefs DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
ALTER TABLE llm_calls DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
ALTER TABLE messages DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
ALTER TABLE memories DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
ALTER TABLE cognitive_events DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
DROP TABLE IF EXISTS redactions;
+81
View File
@@ -0,0 +1,81 @@
-- Privacy redaction (review pass §7.2 item 8): the audited maintenance path
-- ADR-0002 promised. Redaction replaces the sensitive content of a row with a
-- fixed marker while keeping the row, its identifiers, its links and a hash
-- of what was removed. It is the only write ever permitted on append-only
-- tables, it is confined to content columns, and it runs through
-- noema_redact() so the application cannot reach the bypass by accident.
CREATE TABLE redactions (
id uuid PRIMARY KEY,
mind_id uuid NOT NULL REFERENCES minds(id),
target_kind text NOT NULL CHECK (target_kind IN ('event','memory','message','llm_call','belief','assertion')),
target_id uuid NOT NULL,
reason text NOT NULL,
policy text NOT NULL DEFAULT '', -- retention or privacy policy invoked
content_hash text NOT NULL, -- sha256 of the removed content, for later attestation
requested_by text NOT NULL,
cascade_of uuid REFERENCES redactions(id),
created_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (target_kind, target_id)
);
CREATE INDEX redactions_mind_idx ON redactions (mind_id, created_at DESC);
CREATE TRIGGER redactions_append_only BEFORE UPDATE OR DELETE ON redactions
FOR EACH ROW EXECUTE FUNCTION noema_forbid_change();
ALTER TABLE cognitive_events ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
ALTER TABLE memories ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
ALTER TABLE messages ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
ALTER TABLE llm_calls ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
ALTER TABLE beliefs ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
ALTER TABLE assertions ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
-- The append-only trigger admits exactly one kind of UPDATE: a redaction
-- running under the session flag that only noema_redact() sets. DELETE
-- stays forbidden in every case.
CREATE OR REPLACE FUNCTION noema_forbid_change() RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
IF TG_OP = 'UPDATE' AND current_setting('noema.redaction', true) = 'on' THEN
RETURN NEW;
END IF;
RAISE EXCEPTION 'table % is append-only', TG_TABLE_NAME USING ERRCODE = 'integrity_constraint_violation';
END $$;
-- noema_redact replaces content columns for one row. It sets the bypass
-- flag for the current transaction only, touches nothing but content and
-- the redaction bookkeeping, and returns the number of rows changed.
CREATE OR REPLACE FUNCTION noema_redact(kind text, target uuid, redaction uuid) RETURNS integer LANGUAGE plpgsql SECURITY DEFINER AS $$
DECLARE n integer := 0;
BEGIN
PERFORM set_config('noema.redaction', 'on', true);
IF kind = 'event' THEN
UPDATE cognitive_events SET summary = '[redacted]', payload = '{"redacted":true}'::jsonb,
metadata = (SELECT coalesce(jsonb_object_agg(key, value), '{}'::jsonb) FROM jsonb_each(metadata)
WHERE key IN ('conversation_id','message_id','cycle','belief_id','memory_id','integration_id','integration','kind','trust','method','replay_of','replay_id'))
|| jsonb_build_object('redacted', true),
redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
ELSIF kind = 'memory' THEN
UPDATE memories SET content = '[redacted]', detail = '{"redacted":true}'::jsonb, redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
ELSIF kind = 'message' THEN
UPDATE messages SET content = '[redacted]', intent = NULL, redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
ELSIF kind = 'llm_call' THEN
UPDATE llm_calls SET request = '{"redacted":true}'::jsonb, response = NULL, error = CASE WHEN error = '' THEN '' ELSE '[redacted]' END, redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
ELSIF kind = 'belief' THEN
UPDATE beliefs SET proposition = '[redacted]', detail = '{"redacted":true}'::jsonb, redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
GET DIAGNOSTICS n = ROW_COUNT;
IF n > 0 THEN
UPDATE belief_versions SET proposition = '[redacted]', reason = '[redacted]' WHERE belief_id = target;
UPDATE belief_evidence SET note = '[redacted]' WHERE belief_id = target;
UPDATE belief_alternatives SET proposition = '[redacted]', note = '[redacted]' WHERE belief_id = target;
END IF;
PERFORM set_config('noema.redaction', 'off', true);
RETURN n;
ELSIF kind = 'assertion' THEN
UPDATE assertions SET proposition = '[redacted]', context = '[redacted]', interpretation_notes = '{}', redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
ELSE
RAISE EXCEPTION 'unknown redaction kind %', kind;
END IF;
GET DIAGNOSTICS n = ROW_COUNT;
PERFORM set_config('noema.redaction', 'off', true);
RETURN n;
END $$;
REVOKE ALL ON FUNCTION noema_redact(text, uuid, uuid) FROM PUBLIC;
+3 -1
View File
@@ -64,6 +64,7 @@ const (
ModeChanged Type = "mode_changed" ModeChanged Type = "mode_changed"
ResponseGenerated Type = "response_generated" ResponseGenerated Type = "response_generated"
CycleCompleted Type = "cycle_completed" CycleCompleted Type = "cycle_completed"
Redaction Type = "redaction" // content of an earlier row was made irrecoverable under a policy
) )
// Category groups types for filtering and colouring in the UI. // Category groups types for filtering and colouring in the UI.
@@ -88,7 +89,7 @@ var categories = map[Type]Category{
Observation: CatPerception, Perception: CatPerception, Observation: CatPerception, Perception: CatPerception,
ThoughtCandidate: CatCognition, ThoughtSelected: CatCognition, WorkspaceBroadcast: CatCognition, AttentionShift: CatCognition, ThoughtCandidate: CatCognition, ThoughtSelected: CatCognition, WorkspaceBroadcast: CatCognition, AttentionShift: CatCognition,
Reflection: CatCognition, ContradictionDetected: CatCognition, Metacognition: CatCognition, DreamAssociation: CatCognition, Reflection: CatCognition, ContradictionDetected: CatCognition, Metacognition: CatCognition, DreamAssociation: CatCognition,
NarrativeUpdated: CatCognition, ResponseGenerated: CatCognition, CycleCompleted: CatSystem, ModeChanged: CatSystem, NarrativeUpdated: CatCognition, ResponseGenerated: CatCognition, CycleCompleted: CatSystem, ModeChanged: CatSystem, Redaction: CatSystem,
MemoryCreated: CatMemory, MemoryRetrieved: CatMemory, MemoryReinforced: CatMemory, MemoryDecayed: CatMemory, MemoryConsolidated: CatMemory, MemoryCreated: CatMemory, MemoryRetrieved: CatMemory, MemoryReinforced: CatMemory, MemoryDecayed: CatMemory, MemoryConsolidated: CatMemory,
BeliefCreated: CatBelief, BeliefUpdated: CatBelief, BeliefChallenged: CatBelief, BeliefRetracted: CatBelief, BeliefCreated: CatBelief, BeliefUpdated: CatBelief, BeliefChallenged: CatBelief, BeliefRetracted: CatBelief,
HypothesisCreated: CatBelief, HypothesisSupported: CatBelief, HypothesisWeakened: CatBelief, HypothesisCreated: CatBelief, HypothesisSupported: CatBelief, HypothesisWeakened: CatBelief,
@@ -160,6 +161,7 @@ type Event struct {
Summary string `json:"summary"` Summary string `json:"summary"`
Metadata map[string]any `json:"metadata"` Metadata map[string]any `json:"metadata"`
Payload json.RawMessage `json:"payload"` Payload json.RawMessage `json:"payload"`
RedactedAt *time.Time `json:"redacted_at,omitempty"` // content replaced by a marker; see redactions
} }
// New describes an event to append. Zero OccurredAt means now. // New describes an event to append. Zero OccurredAt means now.
+2
View File
@@ -42,6 +42,7 @@ import (
"github.com/andydixon/noema/internal/notifications" "github.com/andydixon/noema/internal/notifications"
"github.com/andydixon/noema/internal/observability" "github.com/andydixon/noema/internal/observability"
"github.com/andydixon/noema/internal/predictions" "github.com/andydixon/noema/internal/predictions"
"github.com/andydixon/noema/internal/privacy"
"github.com/andydixon/noema/internal/reflection" "github.com/andydixon/noema/internal/reflection"
"github.com/andydixon/noema/internal/scheduler" "github.com/andydixon/noema/internal/scheduler"
"github.com/andydixon/noema/internal/security/secrets" "github.com/andydixon/noema/internal/security/secrets"
@@ -89,6 +90,7 @@ func Start(t *testing.T) *Harness {
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log} deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds} deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
deps.Assertions = epistemics.NewStore(pool) deps.Assertions = epistemics.NewStore(pool)
deps.Privacy = &privacy.Service{DB: pool, Events: deps.Events, Audit: deps.Audit, Snapshots: deps.Snapshots, Log: log}
deps.Narrative = narrative.NewStore(pool) deps.Narrative = narrative.NewStore(pool)
deps.Notifications = notifications.NewStore(pool) deps.Notifications = notifications.NewStore(pool)
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log} deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
+2
View File
@@ -42,6 +42,7 @@ import (
"github.com/andydixon/noema/internal/notifications" "github.com/andydixon/noema/internal/notifications"
"github.com/andydixon/noema/internal/observability" "github.com/andydixon/noema/internal/observability"
"github.com/andydixon/noema/internal/predictions" "github.com/andydixon/noema/internal/predictions"
"github.com/andydixon/noema/internal/privacy"
"github.com/andydixon/noema/internal/reflection" "github.com/andydixon/noema/internal/reflection"
"github.com/andydixon/noema/internal/scheduler" "github.com/andydixon/noema/internal/scheduler"
"github.com/andydixon/noema/internal/security/secrets" "github.com/andydixon/noema/internal/security/secrets"
@@ -130,6 +131,7 @@ func TestFirstMilestone(t *testing.T) {
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log} deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds} deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
deps.Assertions = epistemics.NewStore(pool) deps.Assertions = epistemics.NewStore(pool)
deps.Privacy = &privacy.Service{DB: pool, Events: deps.Events, Audit: deps.Audit, Snapshots: deps.Snapshots, Log: log}
deps.Narrative = narrative.NewStore(pool) deps.Narrative = narrative.NewStore(pool)
deps.Notifications = notifications.NewStore(pool) deps.Notifications = notifications.NewStore(pool)
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log} deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
+1 -1
View File
@@ -8,7 +8,7 @@
<section class="grid cols-3-1"> <section class="grid cols-3-1">
<div class="stack"> <div class="stack">
<div class="card"> <div class="card">
<h2>Causal chain <a class="small" href="/events/{{$e.ID}}/timeline">full timeline →</a></h2> <h2>Causal chain <a class="small" href="/events/{{$e.ID}}/timeline">full timeline →</a>{{if .Principal.User.Role.AtLeast "administrator"}} · <a class="small danger" href="/minds/{{.Data.Mind.Slug}}/redactions?kind=event&id={{$e.ID}}">redact…</a>{{end}}</h2>
<div class="chain"> <div class="chain">
{{if $c.Cause}}<div class="chain-node"><span class="muted small">caused by</span><br><span class="etype cat-{{$c.Cause.Category}}">{{$c.Cause.Type}}</span> <a href="/events/{{$c.Cause.ID}}">{{$c.Cause.Summary}}</a></div><div class="chain-arrow">↓</div>{{end}} {{if $c.Cause}}<div class="chain-node"><span class="muted small">caused by</span><br><span class="etype cat-{{$c.Cause.Category}}">{{$c.Cause.Type}}</span> <a href="/events/{{$c.Cause.ID}}">{{$c.Cause.Summary}}</a></div><div class="chain-arrow">↓</div>{{end}}
<div class="chain-node current"><span class="etype cat-{{$e.Category}}">{{$e.Type}}</span> {{$e.Summary}}</div> <div class="chain-node current"><span class="etype cat-{{$e.Category}}">{{$e.Type}}</span> {{$e.Summary}}</div>
+25
View File
@@ -0,0 +1,25 @@
{{define "content"}}
{{$m := .Data.Mind}}
<div class="pagehead mindhead"><div><h1>{{$m.DisplayName}} <span class="muted small">redactions</span></h1><p class="muted">The audited path for making sensitive content irrecoverable. A redaction keeps the row, its identifiers, its links, its type and its time, replaces the content with <code>[redacted]</code>, scrubs stored snapshot archives, and records a hash of what was removed. It cannot be undone. Backups and exports taken before it are outside this system's reach.</p></div></div>
{{template "mind_tabs" (dict "M" $m "Active" "redactions")}}
<section class="grid cols-3-1">
<div class="card">
{{if .Data.Redactions}}<table class="table compact"><thead><tr><th>When</th><th>Kind</th><th>Target</th><th>Reason</th><th>Policy</th><th>By</th><th>Cascade</th><th>Hash</th></tr></thead><tbody>
{{range .Data.Redactions}}<tr><td class="small muted nowrap">{{ago .CreatedAt}}</td><td><span class="badge">{{.Kind}}</span></td><td class="mono small">{{if eq .Kind "event"}}<a href="/events/{{.TargetID}}">{{trunc (printf "%s" .TargetID) 13}}</a>{{else if eq .Kind "memory"}}<a href="/minds/{{$m.Slug}}/memory/{{.TargetID}}">{{trunc (printf "%s" .TargetID) 13}}</a>{{else if eq .Kind "belief"}}<a href="/minds/{{$m.Slug}}/beliefs/{{.TargetID}}">{{trunc (printf "%s" .TargetID) 13}}</a>{{else}}{{trunc (printf "%s" .TargetID) 13}}{{end}}</td><td>{{.Reason}}</td><td class="small">{{.Policy}}</td><td class="small">{{.RequestedBy}}</td><td class="small muted">{{if .CascadeOf}}from {{trunc (printf "%s" .CascadeOf) 8}}{{else}}primary{{end}}</td><td class="mono small" title="{{.ContentHash}}">{{trunc .ContentHash 19}}…</td></tr>{{end}}</tbody></table>
{{else}}<p class="muted">Nothing has been redacted for this mind.</p>{{end}}
</div>
{{if .Principal.User.Role.AtLeast "administrator"}}
<div class="card"><h2>Redact</h2>
<form method="post" action="/minds/{{$m.Slug}}/redactions" class="stack"><input type="hidden" name="_csrf" value="{{.CSRF}}">
<label>Kind <select name="kind">{{range $k := list "event" "memory" "message" "llm_call" "belief" "assertion"}}<option value="{{$k}}" {{if eq $k $.Data.Kind}}selected{{end}}>{{$k}}</option>{{end}}</select></label>
<label>Id <input name="id" required value="{{.Data.ID}}" class="mono"></label>
<label>Reason <input name="reason" required maxlength="300" placeholder="data subject request, retention policy…"></label>
<label>Policy <input name="policy" maxlength="120" placeholder="e.g. GDPR art. 17"></label>
<label class="check"><input type="checkbox" name="cascade" value="true" checked> Also redact rows derived from it (episodes, messages, model calls, assertions, beliefs)</label>
<div><button class="btn btn-danger" data-confirm="This cannot be undone. The content will be irrecoverable everywhere this system controls. Continue?">Redact</button></div>
</form>
<p class="small muted">Redaction is recorded in the audit log and as a <code>redaction</code> event in the mind's history; neither contains the removed content.</p>
</div>
{{end}}
</section>
{{end}}
+1 -1
View File
@@ -3,6 +3,6 @@
<a href="/minds/{{$m.Slug}}" {{if eq $a "overview"}}aria-current="page"{{end}}>Overview</a> <a href="/minds/{{$m.Slug}}" {{if eq $a "overview"}}aria-current="page"{{end}}>Overview</a>
<a href="/minds/{{$m.Slug}}/thoughts" {{if eq $a "thoughts"}}aria-current="page"{{end}}>Thoughts</a> <a href="/minds/{{$m.Slug}}/thoughts" {{if eq $a "thoughts"}}aria-current="page"{{end}}>Thoughts</a>
<a href="/minds/{{$m.Slug}}/personality" {{if eq $a "personality"}}aria-current="page"{{end}}>Personality</a> <a href="/minds/{{$m.Slug}}/personality" {{if eq $a "personality"}}aria-current="page"{{end}}>Personality</a>
<a href="/minds/{{$m.Slug}}/workspace" {{if eq $a "workspace"}}aria-current="page"{{end}}>Workspace</a><a href="/minds/{{$m.Slug}}/memory" {{if eq $a "memory"}}aria-current="page"{{end}}>Memory</a><a href="/minds/{{$m.Slug}}/beliefs" {{if eq $a "beliefs"}}aria-current="page"{{end}}>Beliefs</a><a href="/minds/{{$m.Slug}}/conversations" {{if eq $a "conversation"}}aria-current="page"{{end}}>Conversation</a><a href="/minds/{{$m.Slug}}/dreams" {{if eq $a "dreams"}}aria-current="page"{{end}}>Dreams</a><a href="/minds/{{$m.Slug}}/snapshots" {{if eq $a "snapshots"}}aria-current="page"{{end}}>Snapshots</a><a href="/minds/{{$m.Slug}}/replays" {{if eq $a "replays"}}aria-current="page"{{end}}>Replay</a> <a href="/minds/{{$m.Slug}}/workspace" {{if eq $a "workspace"}}aria-current="page"{{end}}>Workspace</a><a href="/minds/{{$m.Slug}}/memory" {{if eq $a "memory"}}aria-current="page"{{end}}>Memory</a><a href="/minds/{{$m.Slug}}/beliefs" {{if eq $a "beliefs"}}aria-current="page"{{end}}>Beliefs</a><a href="/minds/{{$m.Slug}}/conversations" {{if eq $a "conversation"}}aria-current="page"{{end}}>Conversation</a><a href="/minds/{{$m.Slug}}/dreams" {{if eq $a "dreams"}}aria-current="page"{{end}}>Dreams</a><a href="/minds/{{$m.Slug}}/snapshots" {{if eq $a "snapshots"}}aria-current="page"{{end}}>Snapshots</a><a href="/minds/{{$m.Slug}}/replays" {{if eq $a "replays"}}aria-current="page"{{end}}>Replay</a><a href="/minds/{{$m.Slug}}/redactions" {{if eq $a "redactions"}}aria-current="page"{{end}}>Redactions</a>
</nav> </nav>
{{end}} {{end}}