feat(privacy): audited redaction that keeps history's shape and removes its content
Rows in append-only history are never deleted; their content can now be made irrecoverable through one audited path. A redaction replaces the content columns of an event, memory, message, model call, belief or assertion with a fixed marker, keeps identifiers, links, timestamps, scores and breakdowns, writes an append-only ledger row with a SHA-256 of the removed content, appends a redaction event, optionally cascades to derived rows and to the events that quote them, and scrubs stored snapshot archives before re-signing them. noema_redact() is the only write the append-only trigger admits, for UPDATE only, under a transaction-local flag the application never sets; plain updates and all deletes stay refused. Replay labels redacted stimuli instead of inventing them. Cryptographic erasure was considered and deferred with reasons (ADR-0009); docs/privacy.md defines retained, redacted, irrecoverable and deleted and states what backups and earlier exports are not covered. Migration 0023.
This commit is contained in:
@@ -10,6 +10,7 @@ All notable changes to Noema are recorded here. Format follows Keep a Changelog;
|
|||||||
- Security suite: SQL-injection effect test, static allow-list scan for assembled SQL, deceived-mind authority test, garbage model output test, self-preservation vocabulary test; threat model rewritten so every verification cites a test that asserts the claim.
|
- Security suite: SQL-injection effect test, static allow-list scan for assembled SQL, deceived-mind authority test, garbage model output test, self-preservation vocabulary test; threat model rewritten so every verification cites a test that asserts the claim.
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
- Privacy redaction (ADR-0009): the audited path that makes an event, memory, message, model call, belief or assertion irrecoverable while keeping the row, its links, timestamps and scores; a ledger with a hash of the removed content; cascade to derived rows and the events that quote them; stored snapshot archives scrubbed and re-signed; replay labels redacted stimuli. `noema_redact()` is the only write the append-only trigger admits. Migration 0023. `docs/privacy.md` defines retained, redacted, irrecoverable and deleted.
|
||||||
- Causal timeline: `GET /api/v1/events/{id}/chain` and `/events/{id}/timeline` walk the links events actually record (causation, parents, cited evidence) backwards and forwards within bounded depth and size, label edge kinds, group by cycle as correlation only, and infer nothing.
|
- Causal timeline: `GET /api/v1/events/{id}/chain` and `/events/{id}/timeline` walk the links events actually record (causation, parents, cited evidence) backwards and forwards within bounded depth and size, label edge kinds, group by cycle as correlation only, and infer nothing.
|
||||||
- Cognitive forks per model: minds may override model routing per role (`llm_roles`, fail-closed); experiments may declare variants that fork one snapshot under different configuration, personality or routing, receive identical stimuli, and are compared deterministically (first divergent stimulus, belief and attention overlap, score differences, memory, goal, cost and capability-request ranges).
|
- Cognitive forks per model: minds may override model routing per role (`llm_roles`, fail-closed); experiments may declare variants that fork one snapshot under different configuration, personality or routing, receive identical stimuli, and are compared deterministically (first divergent stimulus, belief and attention overlap, score differences, memory, goal, cost and capability-request ranges).
|
||||||
- Runtime identity: snapshots, replays and experiments record build (commit, modified tree, Go version, platform), schema version, a secret-free configuration digest, personality version, policy digest, model routing, prompt versions, deterministic subsystem versions and timezone. Replays compare identities and label themselves exact-candidate, different-build or incompatible (the last is never run), and analyse the first divergence by component. Migration 0022.
|
- Runtime identity: snapshots, replays and experiments record build (commit, modified tree, Go version, platform), schema version, a secret-free configuration digest, personality version, policy digest, model routing, prompt versions, deterministic subsystem versions and timezone. Replays compare identities and label themselves exact-candidate, different-build or incompatible (the last is never run), and analyse the first divergence by component. Migration 0022.
|
||||||
|
|||||||
@@ -35,6 +35,10 @@ Integrations are disabled until configured, run at a declared trust level, are r
|
|||||||
|
|
||||||
Prompt injection, poisoned documents, hostile model output and manipulated memories can all change what a mind *thinks*. None of them can change what it may *do*. The boundary is `capabilities.Service.Invoke`: it looks the capability up in a registry built in code, validates parameters, evaluates the operator's policy rules, records the decision, and only then executes, asks or denies. Cognition can produce a request; it cannot produce a permission. There is no capability that edits policy, grants approvals or touches a mind's lifecycle, so there is nothing a deceived mind could ask for that would widen its own authority. Prompt delimiters and "this is data" markers are defence in depth for the quality of cognition; they are not the security boundary. `tests/security.TestDeceivedMindHasNoAuthority` forges action requests and hostile model output and checks that nothing executes and policy is unchanged.
|
Prompt injection, poisoned documents, hostile model output and manipulated memories can all change what a mind *thinks*. None of them can change what it may *do*. The boundary is `capabilities.Service.Invoke`: it looks the capability up in a registry built in code, validates parameters, evaluates the operator's policy rules, records the decision, and only then executes, asks or denies. Cognition can produce a request; it cannot produce a permission. There is no capability that edits policy, grants approvals or touches a mind's lifecycle, so there is nothing a deceived mind could ask for that would widen its own authority. Prompt delimiters and "this is data" markers are defence in depth for the quality of cognition; they are not the security boundary. `tests/security.TestDeceivedMindHasNoAuthority` forges action requests and hostile model output and checks that nothing executes and policy is unchanged.
|
||||||
|
|
||||||
|
## Privacy and redaction
|
||||||
|
|
||||||
|
Cognitive history is append-only, and rows are never deleted. Sensitive content is removed through one audited path: an administrator redacts an event, memory, message, model call, belief or assertion; the content columns become `[redacted]`, a ledger row keeps a hash of what was removed, a `redaction` event is appended, derived rows can be cascaded, and stored snapshot archives are scrubbed. Identifiers, links, timestamps and scores survive so provenance and replay keep their shape. Backups and exports made before a redaction are outside the system's reach. Definitions, the threat model and the reasons cryptographic erasure was not adopted yet are in `docs/privacy.md` (ADR-0009).
|
||||||
|
|
||||||
## Capability policy defaults
|
## Capability policy defaults
|
||||||
|
|
||||||
Read-only capabilities default to `ASK`; anything that writes, sends, restarts, deletes or spends money defaults to `ASK` or `DENY`. `sudo_shell` and `delete_database` are `DENY` and cannot be set to `ALLOW` through the UI. Dreaming may never invoke a capability. There are no self-preservation, replication, concealment or permission-seeking drives, and shutdown is an operator control the mind cannot influence.
|
Read-only capabilities default to `ASK`; anything that writes, sends, restarts, deletes or spends money defaults to `ASK` or `DENY`. `sudo_shell` and `delete_database` are `DENY` and cannot be set to `ALLOW` through the UI. Dreaming may never invoke a capability. There are no self-preservation, replication, concealment or permission-seeking drives, and shutdown is an operator control the mind cannot influence.
|
||||||
|
|||||||
@@ -101,6 +101,10 @@ M: model answers are decoded into typed Go structures (unknown fields ignored, w
|
|||||||
### T25 Operator-control subversion
|
### T25 Operator-control subversion
|
||||||
M: no drive or goal kind for self-preservation, replication or permission acquisition; no capability touches policy, approvals, snapshots or the mind's lifecycle; shutdown, pause and disable are authenticated operator routes with no path from cognition. V: `tests/security.TestNoSelfPreservationVocabulary` (drives, goal kinds and the capability registry) and `TestDeceivedMindHasNoAuthority`. ☑
|
M: no drive or goal kind for self-preservation, replication or permission acquisition; no capability touches policy, approvals, snapshots or the mind's lifecycle; shutdown, pause and disable are authenticated operator routes with no path from cognition. V: `tests/security.TestNoSelfPreservationVocabulary` (drives, goal kinds and the capability registry) and `TestDeceivedMindHasNoAuthority`. ☑
|
||||||
|
|
||||||
|
### T26 Personal data that must be erased
|
||||||
|
Surface: append-only events, memories, messages, model-call records, beliefs, assertions, snapshot archives, exports.
|
||||||
|
M: an administrator-only, CSRF-protected, audited redaction path replaces content with a marker while keeping identifiers, links, timestamps and scores; a ledger records a hash of the removed content; derived rows cascade; stored archives are scrubbed and re-signed; replay labels redacted stimuli instead of inventing them; `noema_redact()` is the only write the append-only trigger admits, for UPDATE only, under a transaction-local flag the application never sets. V: `internal/privacy.TestRedactionIsIrrecoverableButReferentiallyIntact` (content gone from every table, links intact, hash attested, cascade, archive scrub, plain UPDATE and DELETE still refused, ledger append-only) and `internal/replay.TestReplayIsExactWithoutModel` (redacted stimulus labelled). Not covered by the system: backups, replicas and exports made before the redaction; see `docs/privacy.md`. ☑
|
||||||
|
|
||||||
## Residual risks
|
## Residual risks
|
||||||
|
|
||||||
- A compromised host or database compromises everything.
|
- A compromised host or database compromises everything.
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ import (
|
|||||||
"github.com/andydixon/noema/internal/notifications"
|
"github.com/andydixon/noema/internal/notifications"
|
||||||
"github.com/andydixon/noema/internal/observability"
|
"github.com/andydixon/noema/internal/observability"
|
||||||
"github.com/andydixon/noema/internal/predictions"
|
"github.com/andydixon/noema/internal/predictions"
|
||||||
|
"github.com/andydixon/noema/internal/privacy"
|
||||||
"github.com/andydixon/noema/internal/reflection"
|
"github.com/andydixon/noema/internal/reflection"
|
||||||
"github.com/andydixon/noema/internal/replay"
|
"github.com/andydixon/noema/internal/replay"
|
||||||
"github.com/andydixon/noema/internal/scheduler"
|
"github.com/andydixon/noema/internal/scheduler"
|
||||||
@@ -192,6 +193,7 @@ func run() error {
|
|||||||
sandboxCfg.Services.LLM = replayLLM
|
sandboxCfg.Services.LLM = replayLLM
|
||||||
ident := &identity.Capturer{DB: db, Config: &cfg, Minds: deps.Minds, LLM: deps.LLMStore, Prompts: deps.Prompts, Capabilities: deps.Capabilities}
|
ident := &identity.Capturer{DB: db, Config: &cfg, Minds: deps.Minds, LLM: deps.LLMStore, Prompts: deps.Prompts, Capabilities: deps.Capabilities}
|
||||||
deps.Snapshots = &snapshots.Store{DB: db, Events: deps.Events, Minds: deps.Minds, Identity: ident}
|
deps.Snapshots = &snapshots.Store{DB: db, Events: deps.Events, Minds: deps.Minds, Identity: ident}
|
||||||
|
deps.Privacy = &privacy.Service{DB: db, Events: deps.Events, Audit: auditLog, Snapshots: deps.Snapshots, Log: log}
|
||||||
deps.Replay = &replay.Service{DB: db, Events: deps.Events, Minds: deps.Minds, Snapshots: deps.Snapshots, Brain: sandboxCfg, Identity: ident, Log: log}
|
deps.Replay = &replay.Service{DB: db, Events: deps.Events, Minds: deps.Minds, Snapshots: deps.Snapshots, Brain: sandboxCfg, Identity: ident, Log: log}
|
||||||
deps.Experiments = &experiments.Service{DB: db, Minds: deps.Minds, Snapshots: deps.Snapshots, Brain: sandboxCfg, Identity: ident, Log: log, Routing: func(ctx context.Context) []map[string]any {
|
deps.Experiments = &experiments.Service{DB: db, Minds: deps.Minds, Snapshots: deps.Snapshots, Brain: sandboxCfg, Identity: ident, Log: log, Routing: func(ctx context.Context) []map[string]any {
|
||||||
roles, _ := deps.LLMStore.ListRoles(ctx)
|
roles, _ := deps.LLMStore.ListRoles(ctx)
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# ADR-0009: Redaction as the only write on append-only history
|
||||||
|
|
||||||
|
Date: 2026-09-15
|
||||||
|
Status: Accepted
|
||||||
|
|
||||||
|
## Context
|
||||||
|
ADR-0002 made cognitive history append-only and promised a "dedicated, audited maintenance path" for removal. Privacy law and ordinary decency require that personal content can be made irrecoverable, while provenance, replay and forensic reconstruction require that the *shape* of history survives.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
Content is redacted, rows are never deleted. A `redactions` ledger (append-only) records target, reason, policy, requester and a SHA-256 of the removed content. One SQL function, `noema_redact()`, performs the replacement under a transaction-local flag that the append-only trigger admits for UPDATE only; the function is `SECURITY DEFINER`, revoked from `PUBLIC`, and the application never sets the flag. Content columns are replaced by a fixed marker; identifiers, types, timestamps, links, scores and breakdowns are kept. A `redaction` event is appended. Cascade follows recorded derivations one level (episodes, messages, model calls, assertions, beliefs rooted at the event, and the events that quote each of them). Stored snapshot archives are scrubbed and re-signed.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
- Irrecoverability covers everything the system controls; backups, exports and replicas made earlier are the operator's responsibility and are documented as such.
|
||||||
|
- History keeps its structure: replay labels redacted stimuli instead of inventing them; belief provenance remains inspectable without its text.
|
||||||
|
- The append-only guarantee is weakened by exactly one, auditable, content-only write path.
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
- Cryptographic erasure (per-row keys destroyed on request): the strongest option, rejected for now because search, retrieval, contradiction detection, replay and the interface read the columns in clear and per-row decryption would touch every read path; recorded as future work and recommended for backups.
|
||||||
|
- Disabling the trigger and deleting rows: breaks provenance links and replay, and hides that anything happened. Rejected.
|
||||||
|
- Leaving content in place with access controls: does not satisfy erasure. Rejected.
|
||||||
@@ -6345,6 +6345,123 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"/api/v1/minds/{mind}/redactions": {
|
||||||
|
"get": {
|
||||||
|
"tags": [
|
||||||
|
"minds"
|
||||||
|
],
|
||||||
|
"responses": {
|
||||||
|
"200": {
|
||||||
|
"description": "OK",
|
||||||
|
"content": {
|
||||||
|
"application/json": {
|
||||||
|
"schema": {
|
||||||
|
"type": "object"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"401": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"403": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"404": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"201": {
|
||||||
|
"description": "Redacted; irreversible"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "mind",
|
||||||
|
"in": "path",
|
||||||
|
"required": true,
|
||||||
|
"schema": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"operationId": "redactionsList",
|
||||||
|
"summary": "list the redaction ledger",
|
||||||
|
"x-required-role": "operator"
|
||||||
|
},
|
||||||
|
"post": {
|
||||||
|
"tags": [
|
||||||
|
"minds"
|
||||||
|
],
|
||||||
|
"responses": {
|
||||||
|
"200": {
|
||||||
|
"description": "OK",
|
||||||
|
"content": {
|
||||||
|
"application/json": {
|
||||||
|
"schema": {
|
||||||
|
"type": "object"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"401": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"403": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"404": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"201": {
|
||||||
|
"description": "Redacted; irreversible"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "mind",
|
||||||
|
"in": "path",
|
||||||
|
"required": true,
|
||||||
|
"schema": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"operationId": "redactionCreate",
|
||||||
|
"summary": "redact a row's content irreversibly (event, memory, message, llm_call, belief, assertion), optionally cascading to derived rows",
|
||||||
|
"x-required-role": "administrator",
|
||||||
|
"requestBody": {
|
||||||
|
"content": {
|
||||||
|
"application/json": {
|
||||||
|
"schema": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"kind": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"id": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"reason": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"policy": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"cascade": {
|
||||||
|
"type": "boolean"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"kind",
|
||||||
|
"id",
|
||||||
|
"reason"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
"/api/v1/minds/{mind}/reflections": {
|
"/api/v1/minds/{mind}/reflections": {
|
||||||
"get": {
|
"get": {
|
||||||
"operationId": "reflectionsList",
|
"operationId": "reflectionsList",
|
||||||
|
|||||||
@@ -0,0 +1,45 @@
|
|||||||
|
# Privacy, retention and redaction
|
||||||
|
|
||||||
|
Noema's history is append-only because provenance, replay and forensic reconstruction depend on it. Personal or sensitive content can nevertheless need to be removed. This page defines exactly what the system can and cannot do about that.
|
||||||
|
|
||||||
|
## Definitions
|
||||||
|
|
||||||
|
| Term | Meaning in Noema |
|
||||||
|
|---|---|
|
||||||
|
| **retained** | the row exists with its content, identifiers, links and timestamps. The default for everything. |
|
||||||
|
| **redacted** | the row exists with its identifiers, type, timestamps and links intact, but its content columns hold the fixed marker `[redacted]`. A ledger row records when, why, by whom, under which policy, and a SHA-256 of the removed content. |
|
||||||
|
| **irrecoverable** | no copy of the content remains anywhere this system controls: the live tables, stored snapshot archives and future exports. Backups, exports and replicas made *before* the redaction are outside the system and are the operator's responsibility (see below). |
|
||||||
|
| **deleted** | not offered for cognitive history. Rows in append-only tables are never deleted; a redacted row is the strongest erasure the system performs. Ordinary mutable rows (sessions, tokens, approvals) are deleted by their own lifecycles. |
|
||||||
|
|
||||||
|
## The redaction path
|
||||||
|
|
||||||
|
`POST /api/v1/minds/{mind}/redactions` (administrators; also the mind's **Redactions** page) takes a kind (`event`, `memory`, `message`, `llm_call`, `belief`, `assertion`), an id, a reason, an optional policy name and a cascade flag. It runs in one transaction:
|
||||||
|
|
||||||
|
1. The row's content is hashed and a `redactions` ledger row is written (append-only, unique per target).
|
||||||
|
2. `noema_redact()` replaces the content columns. It is the only code path that may update an append-only table: it sets a transaction-local flag that the append-only trigger checks, touches nothing but content and the redaction bookkeeping, and cannot delete. The function is `SECURITY DEFINER`, revoked from `PUBLIC`, and the application never sets the flag itself.
|
||||||
|
3. A `redaction` event is appended to the mind's history naming the target, the policy and the hash, never the content.
|
||||||
|
4. With `cascade`, rows derived from the target are redacted too and linked to the primary ledger row: for an event, the episodes encoded from it, the message it carried and its reply, the model calls that worded them, the assertions made in it and the beliefs whose evidence roots at it; for each redacted row, the events that quote it (a perception, an encoding, a belief creation). Redaction events are never cascaded, so the record of redaction survives.
|
||||||
|
5. Stored snapshot archives of the mind are scrubbed of the same rows, the archive records the redacted ids, and its checksum is recomputed; archives still validate.
|
||||||
|
6. The action is written to the audit log.
|
||||||
|
|
||||||
|
What each kind replaces: events lose `summary`, `payload` and free-text metadata (structural keys such as conversation, message, cycle and belief ids are kept); memories lose `content` and `detail`; messages lose `content` and `intent`; model calls lose `request` and `response`; beliefs lose `proposition` and `detail` and their versions, evidence notes and alternatives; assertions lose `proposition`, `context` and notes. Scores, statuses, confidence breakdown numbers, families, links and timestamps are untouched, so the epistemic shape of the history remains inspectable: an operator can still see that a belief existed, how strong it was, what supported it and what later depended on it.
|
||||||
|
|
||||||
|
## Why not cryptographic erasure
|
||||||
|
|
||||||
|
Envelope-encrypting payloads with a per-row or per-subject key and destroying the key on erasure was considered and not adopted in this release. Full-text search, semantic retrieval, contradiction detection, the interface and replay all read these columns in clear; per-row decryption on every read would touch every code path and index, and search over ciphertext would need either plaintext shadow columns (defeating the purpose) or a redesign of retrieval. The marker-and-hash approach achieves the same end state for the live database and stored archives with a small, auditable surface. Cryptographic erasure remains the right answer for **backups**, which this system does not manage: encrypt backups under keys you can retire, and treat a redaction as a signal to rotate them on your retention schedule.
|
||||||
|
|
||||||
|
## Threat model
|
||||||
|
|
||||||
|
| Scenario | Outcome |
|
||||||
|
|---|---|
|
||||||
|
| Database compromise after redaction | the content is gone; the hash reveals nothing about it. |
|
||||||
|
| Database compromise before redaction, or a backup or replica taken before it | the content is present. Redaction cannot reach copies the system did not make. |
|
||||||
|
| Administrator misuse | redaction is administrator-only, CSRF-protected, audited, and leaves an append-only ledger and a history event; it cannot be used to hide that something happened, only what its content was. |
|
||||||
|
| Snapshots and exports | stored snapshots are scrubbed; exports and archives made after the redaction carry the marker; those made before do not. |
|
||||||
|
| Replay | a redacted stimulus is not replayed and the step is labelled `redacted`; the replay never fabricates content. |
|
||||||
|
| Logs | Noema does not log message, memory or payload content; the structured logger redacts secret-shaped values. Web-server access logs in front of Noema are outside its control. |
|
||||||
|
| Model providers | content already sent to a provider before the redaction is subject to that provider's retention, not Noema's. |
|
||||||
|
|
||||||
|
## Retention
|
||||||
|
|
||||||
|
There is no automatic retention purge. Retention is an operator policy applied through the redaction path with a policy name, so every removal is attributable. A future bulk policy runner would use the same function and ledger.
|
||||||
@@ -34,6 +34,7 @@ import (
|
|||||||
"github.com/andydixon/noema/internal/notifications"
|
"github.com/andydixon/noema/internal/notifications"
|
||||||
"github.com/andydixon/noema/internal/observability"
|
"github.com/andydixon/noema/internal/observability"
|
||||||
"github.com/andydixon/noema/internal/predictions"
|
"github.com/andydixon/noema/internal/predictions"
|
||||||
|
"github.com/andydixon/noema/internal/privacy"
|
||||||
"github.com/andydixon/noema/internal/reflection"
|
"github.com/andydixon/noema/internal/reflection"
|
||||||
"github.com/andydixon/noema/internal/scheduler"
|
"github.com/andydixon/noema/internal/scheduler"
|
||||||
"github.com/andydixon/noema/internal/security/secrets"
|
"github.com/andydixon/noema/internal/security/secrets"
|
||||||
@@ -65,6 +66,7 @@ func testServer(t *testing.T) (*httptest.Server, string) {
|
|||||||
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
|
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
|
||||||
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
|
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
|
||||||
deps.Assertions = epistemics.NewStore(pool)
|
deps.Assertions = epistemics.NewStore(pool)
|
||||||
|
deps.Privacy = &privacy.Service{DB: pool, Events: deps.Events, Audit: deps.Audit, Snapshots: deps.Snapshots, Log: log}
|
||||||
deps.Narrative = narrative.NewStore(pool)
|
deps.Narrative = narrative.NewStore(pool)
|
||||||
deps.Notifications = notifications.NewStore(pool)
|
deps.Notifications = notifications.NewStore(pool)
|
||||||
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
|
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
|
||||||
|
|||||||
@@ -6345,6 +6345,123 @@
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"/api/v1/minds/{mind}/redactions": {
|
||||||
|
"get": {
|
||||||
|
"tags": [
|
||||||
|
"minds"
|
||||||
|
],
|
||||||
|
"responses": {
|
||||||
|
"200": {
|
||||||
|
"description": "OK",
|
||||||
|
"content": {
|
||||||
|
"application/json": {
|
||||||
|
"schema": {
|
||||||
|
"type": "object"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"401": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"403": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"404": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"201": {
|
||||||
|
"description": "Redacted; irreversible"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "mind",
|
||||||
|
"in": "path",
|
||||||
|
"required": true,
|
||||||
|
"schema": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"operationId": "redactionsList",
|
||||||
|
"summary": "list the redaction ledger",
|
||||||
|
"x-required-role": "operator"
|
||||||
|
},
|
||||||
|
"post": {
|
||||||
|
"tags": [
|
||||||
|
"minds"
|
||||||
|
],
|
||||||
|
"responses": {
|
||||||
|
"200": {
|
||||||
|
"description": "OK",
|
||||||
|
"content": {
|
||||||
|
"application/json": {
|
||||||
|
"schema": {
|
||||||
|
"type": "object"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"401": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"403": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"404": {
|
||||||
|
"$ref": "#/components/responses/Error"
|
||||||
|
},
|
||||||
|
"201": {
|
||||||
|
"description": "Redacted; irreversible"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"parameters": [
|
||||||
|
{
|
||||||
|
"name": "mind",
|
||||||
|
"in": "path",
|
||||||
|
"required": true,
|
||||||
|
"schema": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"operationId": "redactionCreate",
|
||||||
|
"summary": "redact a row's content irreversibly (event, memory, message, llm_call, belief, assertion), optionally cascading to derived rows",
|
||||||
|
"x-required-role": "administrator",
|
||||||
|
"requestBody": {
|
||||||
|
"content": {
|
||||||
|
"application/json": {
|
||||||
|
"schema": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"kind": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"id": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"reason": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"policy": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"cascade": {
|
||||||
|
"type": "boolean"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required": [
|
||||||
|
"kind",
|
||||||
|
"id",
|
||||||
|
"reason"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
"/api/v1/minds/{mind}/reflections": {
|
"/api/v1/minds/{mind}/reflections": {
|
||||||
"get": {
|
"get": {
|
||||||
"operationId": "reflectionsList",
|
"operationId": "reflectionsList",
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
package api
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
|
||||||
|
"github.com/andydixon/noema/internal/auth"
|
||||||
|
"github.com/andydixon/noema/internal/privacy"
|
||||||
|
"github.com/andydixon/noema/pkg/ids"
|
||||||
|
)
|
||||||
|
|
||||||
|
// privacyRoutes: the audited redaction path. Administrators only; every
|
||||||
|
// call is irreversible by design and says so in its response.
|
||||||
|
func (s *Server) privacyRoutes() {
|
||||||
|
m := s.mux
|
||||||
|
m.Handle("GET /api/v1/minds/{mind}/redactions", require(auth.RoleOperator, s.redactionsList))
|
||||||
|
m.Handle("POST /api/v1/minds/{mind}/redactions", require(auth.RoleAdministrator, s.redactionCreate))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) redactionsList(w http.ResponseWriter, r *http.Request) {
|
||||||
|
m, ok := s.mind(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
list, err := s.deps.Privacy.List(r.Context(), m.ID, 200)
|
||||||
|
if err != nil {
|
||||||
|
fail(w, r, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if list == nil {
|
||||||
|
list = []privacy.Redaction{}
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusOK, map[string]any{"items": list})
|
||||||
|
}
|
||||||
|
|
||||||
|
// RedactionInput is the request body (API and UI form).
|
||||||
|
type RedactionInput struct {
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
ID ids.ID `json:"id"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
Policy string `json:"policy"`
|
||||||
|
Cascade bool `json:"cascade"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Redact performs a redaction; shared with the UI.
|
||||||
|
func (s *Server) Redact(r *http.Request, mind ids.ID, in RedactionInput, by string) (privacy.Redaction, []privacy.Redaction, error) {
|
||||||
|
p, c, err := s.deps.Privacy.Redact(r.Context(), privacy.Request{MindID: mind, Kind: privacy.Kind(in.Kind), ID: in.ID, Reason: in.Reason, Policy: in.Policy, Cascade: in.Cascade, By: by})
|
||||||
|
switch {
|
||||||
|
case errors.Is(err, privacy.ErrInvalid):
|
||||||
|
return p, c, ValidationError(err.Error())
|
||||||
|
case errors.Is(err, privacy.ErrAlreadyRedacted):
|
||||||
|
return p, c, ValidationError("already redacted")
|
||||||
|
}
|
||||||
|
return p, c, err
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Server) redactionCreate(w http.ResponseWriter, r *http.Request) {
|
||||||
|
m, ok := s.mind(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
var in RedactionInput
|
||||||
|
if err := decode(r, &in); err != nil {
|
||||||
|
fail(w, r, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
p, c, err := s.Redact(r, m.ID, in, actor(r))
|
||||||
|
if err != nil {
|
||||||
|
fail(w, r, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writeJSON(w, http.StatusCreated, map[string]any{"redaction": p, "cascaded": c, "note": "content is irrecoverable; the row, its links and a hash of the removed content remain"})
|
||||||
|
}
|
||||||
@@ -80,6 +80,7 @@ func (s *Server) routes() {
|
|||||||
s.dreamRoutes()
|
s.dreamRoutes()
|
||||||
s.studyRoutes()
|
s.studyRoutes()
|
||||||
s.assertionRoutes()
|
s.assertionRoutes()
|
||||||
|
s.privacyRoutes()
|
||||||
|
|
||||||
m.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
|
m.HandleFunc("/api/", func(w http.ResponseWriter, r *http.Request) {
|
||||||
writeError(w, r, http.StatusNotFound, "not_found", "no such endpoint")
|
writeError(w, r, http.StatusNotFound, "not_found", "no such endpoint")
|
||||||
|
|||||||
@@ -34,6 +34,7 @@ import (
|
|||||||
"github.com/andydixon/noema/internal/notifications"
|
"github.com/andydixon/noema/internal/notifications"
|
||||||
"github.com/andydixon/noema/internal/observability"
|
"github.com/andydixon/noema/internal/observability"
|
||||||
"github.com/andydixon/noema/internal/predictions"
|
"github.com/andydixon/noema/internal/predictions"
|
||||||
|
"github.com/andydixon/noema/internal/privacy"
|
||||||
"github.com/andydixon/noema/internal/reflection"
|
"github.com/andydixon/noema/internal/reflection"
|
||||||
"github.com/andydixon/noema/internal/replay"
|
"github.com/andydixon/noema/internal/replay"
|
||||||
"github.com/andydixon/noema/internal/security/secrets"
|
"github.com/andydixon/noema/internal/security/secrets"
|
||||||
@@ -54,6 +55,7 @@ type Deps struct {
|
|||||||
Memory *memory.Store
|
Memory *memory.Store
|
||||||
Beliefs *beliefs.Store
|
Beliefs *beliefs.Store
|
||||||
Assertions *epistemics.Store
|
Assertions *epistemics.Store
|
||||||
|
Privacy *privacy.Service
|
||||||
Conversations *conversation.Store
|
Conversations *conversation.Store
|
||||||
Goals *goals.Store
|
Goals *goals.Store
|
||||||
Motivation *motivation.Store
|
Motivation *motivation.Store
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ type Belief struct {
|
|||||||
ObservedAt *time.Time `json:"observed_at,omitempty"`
|
ObservedAt *time.Time `json:"observed_at,omitempty"`
|
||||||
ValidFrom *time.Time `json:"valid_from,omitempty"`
|
ValidFrom *time.Time `json:"valid_from,omitempty"`
|
||||||
ValidUntil *time.Time `json:"valid_until,omitempty"`
|
ValidUntil *time.Time `json:"valid_until,omitempty"`
|
||||||
|
RedactedAt *time.Time `json:"redacted_at,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// EvidenceRow is stored evidence.
|
// EvidenceRow is stored evidence.
|
||||||
@@ -301,14 +302,14 @@ func nullID(id ids.ID) *string {
|
|||||||
return &s
|
return &s
|
||||||
}
|
}
|
||||||
|
|
||||||
const cols = `b.id, b.mind_id, b.proposition, b.subject, b.epistemic, b.status, b.confidence, b.prior, b.version, b.source, b.detail, b.origin_event, b.superseded_by, b.pinned_status, b.first_created, b.last_evaluated, b.evaluations, b.temporal_scope, b.observed_at, b.valid_from, b.valid_until`
|
const cols = `b.id, b.mind_id, b.proposition, b.subject, b.epistemic, b.status, b.confidence, b.prior, b.version, b.source, b.detail, b.origin_event, b.superseded_by, b.pinned_status, b.first_created, b.last_evaluated, b.evaluations, b.temporal_scope, b.observed_at, b.valid_from, b.valid_until, b.redacted_at`
|
||||||
|
|
||||||
func scan(row pgx.Row) (Belief, error) {
|
func scan(row pgx.Row) (Belief, error) {
|
||||||
var b Belief
|
var b Belief
|
||||||
var epi, status string
|
var epi, status string
|
||||||
var detail []byte
|
var detail []byte
|
||||||
var origin, superseded, pinned *string
|
var origin, superseded, pinned *string
|
||||||
if err := row.Scan(&b.ID, &b.MindID, &b.Proposition, &b.Subject, &epi, &status, &b.Confidence, &b.Prior, &b.Version, &b.Source, &detail, &origin, &superseded, &pinned, &b.FirstCreated, &b.LastEvaluated, &b.Evaluations, &b.TemporalScope, &b.ObservedAt, &b.ValidFrom, &b.ValidUntil); err != nil {
|
if err := row.Scan(&b.ID, &b.MindID, &b.Proposition, &b.Subject, &epi, &status, &b.Confidence, &b.Prior, &b.Version, &b.Source, &detail, &origin, &superseded, &pinned, &b.FirstCreated, &b.LastEvaluated, &b.Evaluations, &b.TemporalScope, &b.ObservedAt, &b.ValidFrom, &b.ValidUntil, &b.RedactedAt); err != nil {
|
||||||
return b, storage.Translate(err)
|
return b, storage.Translate(err)
|
||||||
}
|
}
|
||||||
b.Epistemic, b.Status = concepts.Epistemic(epi), Status(status)
|
b.Epistemic, b.Status = concepts.Epistemic(epi), Status(status)
|
||||||
@@ -793,7 +794,7 @@ func (s *Store) Links(ctx context.Context, mind, id ids.ID) ([]Link, error) {
|
|||||||
var detail []byte
|
var detail []byte
|
||||||
var origin, superseded, pinned *string
|
var origin, superseded, pinned *string
|
||||||
if err := rows.Scan(&l.ID, &l.FromID, &l.ToID, &l.Relation, &l.Confidence, &l.CreatedAt,
|
if err := rows.Scan(&l.ID, &l.FromID, &l.ToID, &l.Relation, &l.Confidence, &l.CreatedAt,
|
||||||
&other.ID, &other.MindID, &other.Proposition, &other.Subject, &epi, &status, &other.Confidence, &other.Prior, &other.Version, &other.Source, &detail, &origin, &superseded, &pinned, &other.FirstCreated, &other.LastEvaluated, &other.Evaluations, &other.TemporalScope, &other.ObservedAt, &other.ValidFrom, &other.ValidUntil); err != nil {
|
&other.ID, &other.MindID, &other.Proposition, &other.Subject, &epi, &status, &other.Confidence, &other.Prior, &other.Version, &other.Source, &detail, &origin, &superseded, &pinned, &other.FirstCreated, &other.LastEvaluated, &other.Evaluations, &other.TemporalScope, &other.ObservedAt, &other.ValidFrom, &other.ValidUntil, &other.RedactedAt); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
other.Epistemic, other.Status = concepts.Epistemic(epi), Status(status)
|
other.Epistemic, other.Status = concepts.Epistemic(epi), Status(status)
|
||||||
@@ -919,7 +920,7 @@ func (s *Store) RecentChanges(ctx context.Context, mind ids.ID, since time.Time,
|
|||||||
var detail []byte
|
var detail []byte
|
||||||
var origin, superseded, pinned *string
|
var origin, superseded, pinned *string
|
||||||
b := &c.Belief
|
b := &c.Belief
|
||||||
if err := rows.Scan(&b.ID, &b.MindID, &b.Proposition, &b.Subject, &epi, &status, &b.Confidence, &b.Prior, &b.Version, &b.Source, &detail, &origin, &superseded, &pinned, &b.FirstCreated, &b.LastEvaluated, &b.Evaluations, &b.TemporalScope, &b.ObservedAt, &b.ValidFrom, &b.ValidUntil, &fromStatus, &c.FromConf, &c.Reason, &c.At); err != nil {
|
if err := rows.Scan(&b.ID, &b.MindID, &b.Proposition, &b.Subject, &epi, &status, &b.Confidence, &b.Prior, &b.Version, &b.Source, &detail, &origin, &superseded, &pinned, &b.FirstCreated, &b.LastEvaluated, &b.Evaluations, &b.TemporalScope, &b.ObservedAt, &b.ValidFrom, &b.ValidUntil, &b.RedactedAt, &fromStatus, &c.FromConf, &c.Reason, &c.At); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
b.Epistemic, b.Status, c.FromStatus = concepts.Epistemic(epi), Status(status), Status(fromStatus)
|
b.Epistemic, b.Status, c.FromStatus = concepts.Epistemic(epi), Status(status), Status(fromStatus)
|
||||||
|
|||||||
@@ -45,6 +45,7 @@ type Message struct {
|
|||||||
Intent json.RawMessage `json:"intent,omitempty"`
|
Intent json.RawMessage `json:"intent,omitempty"`
|
||||||
LLMCallID ids.ID `json:"llm_call_id,omitempty"`
|
LLMCallID ids.ID `json:"llm_call_id,omitempty"`
|
||||||
CreatedAt time.Time `json:"created_at"`
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
RedactedAt *time.Time `json:"redacted_at,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// ErrInvalid marks bad input.
|
// ErrInvalid marks bad input.
|
||||||
@@ -156,7 +157,7 @@ func (s *Store) Messages(ctx context.Context, mind, conv ids.ID, limit int) ([]M
|
|||||||
if limit <= 0 || limit > 1000 {
|
if limit <= 0 || limit > 1000 {
|
||||||
limit = 200
|
limit = 200
|
||||||
}
|
}
|
||||||
rows, err := s.db.Query(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at FROM messages
|
rows, err := s.db.Query(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at, redacted_at FROM messages
|
||||||
WHERE conversation_id=$1 AND mind_id=$2 ORDER BY created_at DESC LIMIT $3`, conv, mind, limit)
|
WHERE conversation_id=$1 AND mind_id=$2 ORDER BY created_at DESC LIMIT $3`, conv, mind, limit)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
@@ -180,7 +181,7 @@ func scanMessage(row pgx.Row) (Message, error) {
|
|||||||
var m Message
|
var m Message
|
||||||
var ev, reply, call *string
|
var ev, reply, call *string
|
||||||
var intent *json.RawMessage
|
var intent *json.RawMessage
|
||||||
if err := row.Scan(&m.ID, &m.ConversationID, &m.MindID, &m.Role, &m.Actor, &m.Content, &ev, &reply, &intent, &call, &m.CreatedAt); err != nil {
|
if err := row.Scan(&m.ID, &m.ConversationID, &m.MindID, &m.Role, &m.Actor, &m.Content, &ev, &reply, &intent, &call, &m.CreatedAt, &m.RedactedAt); err != nil {
|
||||||
return m, storage.Translate(err)
|
return m, storage.Translate(err)
|
||||||
}
|
}
|
||||||
if ev != nil {
|
if ev != nil {
|
||||||
@@ -200,7 +201,7 @@ func scanMessage(row pgx.Row) (Message, error) {
|
|||||||
|
|
||||||
// Reply returns the mind's reply to a user message, if any.
|
// Reply returns the mind's reply to a user message, if any.
|
||||||
func (s *Store) Reply(ctx context.Context, mind, userMessage ids.ID) (Message, error) {
|
func (s *Store) Reply(ctx context.Context, mind, userMessage ids.ID) (Message, error) {
|
||||||
return scanMessage(s.db.QueryRow(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at FROM messages WHERE reply_to=$1 AND mind_id=$2 ORDER BY created_at LIMIT 1`, userMessage, mind))
|
return scanMessage(s.db.QueryRow(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at, redacted_at FROM messages WHERE reply_to=$1 AND mind_id=$2 ORDER BY created_at LIMIT 1`, userMessage, mind))
|
||||||
}
|
}
|
||||||
|
|
||||||
// Recent returns the last n messages of a conversation for context.
|
// Recent returns the last n messages of a conversation for context.
|
||||||
@@ -211,5 +212,5 @@ func (s *Store) Recent(ctx context.Context, mind, conv ids.ID, n int) ([]Message
|
|||||||
|
|
||||||
// GetMessage fetches one message.
|
// GetMessage fetches one message.
|
||||||
func (s *Store) GetMessage(ctx context.Context, mind, id ids.ID) (Message, error) {
|
func (s *Store) GetMessage(ctx context.Context, mind, id ids.ID) (Message, error) {
|
||||||
return scanMessage(s.db.QueryRow(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at FROM messages WHERE id=$1 AND mind_id=$2`, id, mind))
|
return scanMessage(s.db.QueryRow(ctx, `SELECT id, conversation_id, mind_id, role, actor, content, event_id, reply_to, intent, llm_call_id, created_at, redacted_at FROM messages WHERE id=$1 AND mind_id=$2`, id, mind))
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package epistemics
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/jackc/pgx/v5"
|
"github.com/jackc/pgx/v5"
|
||||||
@@ -37,6 +38,7 @@ type Assertion struct {
|
|||||||
Trust string `json:"trust"`
|
Trust string `json:"trust"`
|
||||||
BeliefID ids.ID `json:"belief_id,omitempty"`
|
BeliefID ids.ID `json:"belief_id,omitempty"`
|
||||||
CreatedAt time.Time `json:"created_at"`
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
RedactedAt *time.Time `json:"redacted_at,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// New is the input to Record.
|
// New is the input to Record.
|
||||||
@@ -60,7 +62,7 @@ type Store struct{ db *pgxpool.Pool }
|
|||||||
// NewStore creates a store.
|
// NewStore creates a store.
|
||||||
func NewStore(db *pgxpool.Pool) *Store { return &Store{db: db} }
|
func NewStore(db *pgxpool.Pool) *Store { return &Store{db: db} }
|
||||||
|
|
||||||
const cols = `id, mind_id, actor, actor_kind, attributed_to, proposition, subject, stance, certainty, modality, negated, reported, reported_actor, quoted, hypothetical, interpretation_confidence, interpretation_notes, source_event, conversation_id, message_id, context, trust, belief_id, created_at`
|
const cols = `id, mind_id, actor, actor_kind, attributed_to, proposition, subject, stance, certainty, modality, negated, reported, reported_actor, quoted, hypothetical, interpretation_confidence, interpretation_notes, source_event, conversation_id, message_id, context, trust, belief_id, created_at, redacted_at`
|
||||||
|
|
||||||
// Record writes an assertion inside q.
|
// Record writes an assertion inside q.
|
||||||
func (s *Store) Record(ctx context.Context, q storage.Querier, n New) (Assertion, error) {
|
func (s *Store) Record(ctx context.Context, q storage.Querier, n New) (Assertion, error) {
|
||||||
@@ -76,7 +78,7 @@ func (s *Store) Record(ctx context.Context, q storage.Querier, n New) (Assertion
|
|||||||
// Sarah asserts X" and "Andy doubts X" are both retrievable.
|
// Sarah asserts X" and "Andy doubts X" are both retrievable.
|
||||||
a.Notes = append(a.Notes, "speaker stance: "+in.SpeakerStance)
|
a.Notes = append(a.Notes, "speaker stance: "+in.SpeakerStance)
|
||||||
}
|
}
|
||||||
_, err := q.Exec(ctx, `INSERT INTO assertions (`+cols+`) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24)`,
|
_, err := q.Exec(ctx, `INSERT INTO assertions (`+strings.TrimSuffix(cols, ", redacted_at")+`) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24)`,
|
||||||
a.ID, a.MindID, a.Actor, a.ActorKind, a.AttributedTo, a.Proposition, a.Subject, a.Stance, a.Certainty, a.Modality, a.Negated, a.Reported, a.ReportedActor, a.Quoted, a.Hypothetical,
|
a.ID, a.MindID, a.Actor, a.ActorKind, a.AttributedTo, a.Proposition, a.Subject, a.Stance, a.Certainty, a.Modality, a.Negated, a.Reported, a.ReportedActor, a.Quoted, a.Hypothetical,
|
||||||
a.Interpretation, a.Notes, nullID(a.SourceEvent), nullID(a.ConversationID), nullID(a.MessageID), a.Context, a.Trust, nullID(a.BeliefID), a.CreatedAt)
|
a.Interpretation, a.Notes, nullID(a.SourceEvent), nullID(a.ConversationID), nullID(a.MessageID), a.Context, a.Trust, nullID(a.BeliefID), a.CreatedAt)
|
||||||
return a, err
|
return a, err
|
||||||
@@ -142,7 +144,7 @@ func scan(row pgx.Row) (Assertion, error) {
|
|||||||
var a Assertion
|
var a Assertion
|
||||||
var src, conv, msg, bel *string
|
var src, conv, msg, bel *string
|
||||||
if err := row.Scan(&a.ID, &a.MindID, &a.Actor, &a.ActorKind, &a.AttributedTo, &a.Proposition, &a.Subject, &a.Stance, &a.Certainty, &a.Modality, &a.Negated, &a.Reported, &a.ReportedActor, &a.Quoted, &a.Hypothetical,
|
if err := row.Scan(&a.ID, &a.MindID, &a.Actor, &a.ActorKind, &a.AttributedTo, &a.Proposition, &a.Subject, &a.Stance, &a.Certainty, &a.Modality, &a.Negated, &a.Reported, &a.ReportedActor, &a.Quoted, &a.Hypothetical,
|
||||||
&a.Interpretation, &a.Notes, &src, &conv, &msg, &a.Context, &a.Trust, &bel, &a.CreatedAt); err != nil {
|
&a.Interpretation, &a.Notes, &src, &conv, &msg, &a.Context, &a.Trust, &bel, &a.CreatedAt, &a.RedactedAt); err != nil {
|
||||||
return a, storage.Translate(err)
|
return a, storage.Translate(err)
|
||||||
}
|
}
|
||||||
a.SourceEvent, a.ConversationID, a.MessageID, a.BeliefID = idOf(src), idOf(conv), idOf(msg), idOf(bel)
|
a.SourceEvent, a.ConversationID, a.MessageID, a.BeliefID = idOf(src), idOf(conv), idOf(msg), idOf(bel)
|
||||||
|
|||||||
@@ -193,7 +193,7 @@ type Filter struct {
|
|||||||
Ascending bool
|
Ascending bool
|
||||||
}
|
}
|
||||||
|
|
||||||
const cols = `id, seq, mind_id, type, category, source, occurred_at, recorded_at, parent_ids, causation_id, correlation_id, salience, confidence, valence, arousal, evidence, summary, metadata, payload`
|
const cols = `id, seq, mind_id, type, category, source, occurred_at, recorded_at, parent_ids, causation_id, correlation_id, salience, confidence, valence, arousal, evidence, summary, metadata, payload, redacted_at`
|
||||||
|
|
||||||
// Query returns events matching f ordered by seq (descending unless Ascending).
|
// Query returns events matching f ordered by seq (descending unless Ascending).
|
||||||
func (s *Store) Query(ctx context.Context, f Filter) ([]events.Event, error) {
|
func (s *Store) Query(ctx context.Context, f Filter) ([]events.Event, error) {
|
||||||
@@ -358,7 +358,7 @@ func scan(row pgx.Row) (events.Event, error) {
|
|||||||
var causation, correlation *string
|
var causation, correlation *string
|
||||||
var evidence, metadata []byte
|
var evidence, metadata []byte
|
||||||
err := row.Scan(&ev.ID, &ev.Seq, &ev.MindID, &typ, &cat, &ev.Source, &ev.OccurredAt, &ev.RecordedAt, &parents, &causation, &correlation,
|
err := row.Scan(&ev.ID, &ev.Seq, &ev.MindID, &typ, &cat, &ev.Source, &ev.OccurredAt, &ev.RecordedAt, &parents, &causation, &correlation,
|
||||||
&ev.Salience, &ev.Confidence, &ev.Valence, &ev.Arousal, &evidence, &ev.Summary, &metadata, &ev.Payload)
|
&ev.Salience, &ev.Confidence, &ev.Valence, &ev.Arousal, &evidence, &ev.Summary, &metadata, &ev.Payload, &ev.RedactedAt)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ev, err
|
return ev, err
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -60,6 +60,7 @@ type Memory struct {
|
|||||||
LastRetrievedAt *time.Time `json:"last_retrieved_at,omitempty"`
|
LastRetrievedAt *time.Time `json:"last_retrieved_at,omitempty"`
|
||||||
OriginEvent ids.ID `json:"origin_event,omitempty"`
|
OriginEvent ids.ID `json:"origin_event,omitempty"`
|
||||||
Supersedes ids.ID `json:"supersedes,omitempty"`
|
Supersedes ids.ID `json:"supersedes,omitempty"`
|
||||||
|
RedactedAt *time.Time `json:"redacted_at,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// New describes a memory to store.
|
// New describes a memory to store.
|
||||||
@@ -237,7 +238,7 @@ func nullID(id ids.ID) *string {
|
|||||||
return &s
|
return &s
|
||||||
}
|
}
|
||||||
|
|
||||||
const cols = `m.id, m.mind_id, m.kind, m.epistemic, m.status, m.content, m.detail, m.source, m.trust, m.credibility, m.confidence, m.importance, m.salience, m.activation, m.valence, m.arousal, m.occurred_at, m.created_at, m.updated_at, m.retrieval_count, m.last_retrieved_at, m.origin_event, m.supersedes`
|
const cols = `m.id, m.mind_id, m.kind, m.epistemic, m.status, m.content, m.detail, m.source, m.trust, m.credibility, m.confidence, m.importance, m.salience, m.activation, m.valence, m.arousal, m.occurred_at, m.created_at, m.updated_at, m.retrieval_count, m.last_retrieved_at, m.origin_event, m.supersedes, m.redacted_at`
|
||||||
|
|
||||||
func scan(row pgx.Row) (Memory, error) {
|
func scan(row pgx.Row) (Memory, error) {
|
||||||
var m Memory
|
var m Memory
|
||||||
@@ -245,7 +246,7 @@ func scan(row pgx.Row) (Memory, error) {
|
|||||||
var detail []byte
|
var detail []byte
|
||||||
var origin, supersedes *string
|
var origin, supersedes *string
|
||||||
err := row.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
|
err := row.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
|
||||||
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes)
|
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &m.RedactedAt)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return m, storage.Translate(err)
|
return m, storage.Translate(err)
|
||||||
}
|
}
|
||||||
@@ -435,7 +436,7 @@ func (s *Store) Relationships(ctx context.Context, mind, id ids.ID) ([]Related,
|
|||||||
m := &r.Other
|
m := &r.Other
|
||||||
if err := rows.Scan(&r.ID, &r.FromMemory, &r.ToMemory, &rel, &r.Confidence, &r.Source, &r.CreatedAt, &r.Direction,
|
if err := rows.Scan(&r.ID, &r.FromMemory, &r.ToMemory, &rel, &r.Confidence, &r.Source, &r.CreatedAt, &r.Direction,
|
||||||
&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
|
&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
|
||||||
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes); err != nil {
|
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &m.RedactedAt); err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
r.Relation = concepts.Relation(rel)
|
r.Relation = concepts.Relation(rel)
|
||||||
|
|||||||
@@ -88,7 +88,7 @@ func (s *Store) Retrieve(ctx context.Context, mind ids.ID, q Query) ([]Retrieval
|
|||||||
var detail []byte
|
var detail []byte
|
||||||
var origin, supersedes *string
|
var origin, supersedes *string
|
||||||
if err := rows.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
|
if err := rows.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
|
||||||
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &rank); err != nil {
|
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &m.RedactedAt, &rank); err != nil {
|
||||||
rows.Close()
|
rows.Close()
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -127,7 +127,7 @@ func (s *Store) Retrieve(ctx context.Context, mind ids.ID, q Query) ([]Retrieval
|
|||||||
var detail []byte
|
var detail []byte
|
||||||
var origin, supersedes *string
|
var origin, supersedes *string
|
||||||
if err := rows.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
|
if err := rows.Scan(&m.ID, &m.MindID, &kind, &epi, &status, &m.Content, &detail, &m.Source, &trust, &m.Credibility, &m.Confidence, &m.Importance, &m.Salience, &m.Activation,
|
||||||
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &sim); err != nil {
|
&m.Valence, &m.Arousal, &m.OccurredAt, &m.CreatedAt, &m.UpdatedAt, &m.RetrievalCount, &m.LastRetrievedAt, &origin, &supersedes, &m.RedactedAt, &sim); err != nil {
|
||||||
break
|
break
|
||||||
}
|
}
|
||||||
m.Kind, m.Epistemic, m.Status, m.Trust = concepts.MemoryKind(kind), concepts.Epistemic(epi), Status(status), events.Trust(trust)
|
m.Kind, m.Epistemic, m.Status, m.Trust = concepts.MemoryKind(kind), concepts.Epistemic(epi), Status(status), events.Trust(trust)
|
||||||
|
|||||||
@@ -0,0 +1,341 @@
|
|||||||
|
// Package privacy is the audited path for making sensitive content
|
||||||
|
// irrecoverable without breaking the history it belongs to (ADR-0009).
|
||||||
|
//
|
||||||
|
// A redaction keeps the row, its identifiers, its links, its type and its
|
||||||
|
// timestamps, and replaces the content columns with a fixed marker. What was
|
||||||
|
// removed is attested by a SHA-256 of the removed content in the ledger, so
|
||||||
|
// a later reader can establish that an event existed, when, of what kind,
|
||||||
|
// what depended on it, that it was redacted, by whom and under which policy,
|
||||||
|
// while the content itself is gone from the live database and from stored
|
||||||
|
// snapshot archives. Backups and exports made before the redaction are
|
||||||
|
// outside this system's reach and are documented as such.
|
||||||
|
package privacy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"log/slog"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5"
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"github.com/andydixon/noema/internal/audit"
|
||||||
|
evstore "github.com/andydixon/noema/internal/events"
|
||||||
|
"github.com/andydixon/noema/internal/snapshots"
|
||||||
|
"github.com/andydixon/noema/internal/storage"
|
||||||
|
"github.com/andydixon/noema/pkg/events"
|
||||||
|
"github.com/andydixon/noema/pkg/ids"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Kind of row a redaction targets.
|
||||||
|
type Kind string
|
||||||
|
|
||||||
|
// Kinds.
|
||||||
|
const (
|
||||||
|
KindEvent Kind = "event"
|
||||||
|
KindMemory Kind = "memory"
|
||||||
|
KindMessage Kind = "message"
|
||||||
|
KindLLMCall Kind = "llm_call"
|
||||||
|
KindBelief Kind = "belief"
|
||||||
|
KindAssertion Kind = "assertion"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Marker replaces removed content everywhere.
|
||||||
|
const Marker = "[redacted]"
|
||||||
|
|
||||||
|
// Errors.
|
||||||
|
var (
|
||||||
|
ErrInvalid = errors.New("privacy: invalid request")
|
||||||
|
ErrAlreadyRedacted = errors.New("privacy: already redacted")
|
||||||
|
)
|
||||||
|
|
||||||
|
// Request describes one redaction.
|
||||||
|
type Request struct {
|
||||||
|
MindID ids.ID
|
||||||
|
Kind Kind
|
||||||
|
ID ids.ID
|
||||||
|
Reason string
|
||||||
|
Policy string
|
||||||
|
Cascade bool // also redact rows derived from the target
|
||||||
|
By string
|
||||||
|
}
|
||||||
|
|
||||||
|
// Redaction is one ledger row.
|
||||||
|
type Redaction struct {
|
||||||
|
ID ids.ID `json:"id"`
|
||||||
|
MindID ids.ID `json:"mind_id"`
|
||||||
|
Kind Kind `json:"target_kind"`
|
||||||
|
TargetID ids.ID `json:"target_id"`
|
||||||
|
Reason string `json:"reason"`
|
||||||
|
Policy string `json:"policy"`
|
||||||
|
ContentHash string `json:"content_hash"`
|
||||||
|
RequestedBy string `json:"requested_by"`
|
||||||
|
CascadeOf ids.ID `json:"cascade_of,omitempty"`
|
||||||
|
CreatedAt time.Time `json:"created_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Service performs redactions.
|
||||||
|
type Service struct {
|
||||||
|
DB *pgxpool.Pool
|
||||||
|
Events *evstore.Store
|
||||||
|
Audit *audit.Logger
|
||||||
|
Snapshots *snapshots.Store
|
||||||
|
Log *slog.Logger
|
||||||
|
}
|
||||||
|
|
||||||
|
// Redact makes the target's content irrecoverable and returns the ledger
|
||||||
|
// row plus any cascaded rows. Everything runs in one transaction; stored
|
||||||
|
// snapshot archives are scrubbed afterwards and the outcome is audited.
|
||||||
|
func (s *Service) Redact(ctx context.Context, req Request) (Redaction, []Redaction, error) {
|
||||||
|
if req.MindID.IsZero() || req.ID.IsZero() || strings.TrimSpace(req.Reason) == "" || !validKind(req.Kind) {
|
||||||
|
return Redaction{}, nil, fmt.Errorf("%w: mind, kind, id and reason are required", ErrInvalid)
|
||||||
|
}
|
||||||
|
var primary Redaction
|
||||||
|
var cascaded []Redaction
|
||||||
|
err := storage.Tx(ctx, s.DB, func(tx pgx.Tx) error {
|
||||||
|
var err error
|
||||||
|
primary, err = s.redactOne(ctx, tx, req, ids.Nil)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !req.Cascade {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
deps := s.dependants(ctx, tx, req.MindID, req.Kind, req.ID)
|
||||||
|
// Events that quote a redacted row (the perception of a message, the
|
||||||
|
// encoding of an episode, the creation of a belief) are redacted too,
|
||||||
|
// one level out from the target and from each dependant.
|
||||||
|
deps = append(deps, s.referencingEvents(ctx, tx, req.MindID, req.Kind, req.ID)...)
|
||||||
|
seen := map[dep]bool{{req.Kind, req.ID}: true}
|
||||||
|
for i := 0; i < len(deps); i++ {
|
||||||
|
d := deps[i]
|
||||||
|
if seen[d] {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[d] = true
|
||||||
|
r, err := s.redactOne(ctx, tx, Request{MindID: req.MindID, Kind: d.kind, ID: d.id, Reason: req.Reason, Policy: req.Policy, By: req.By}, primary.ID)
|
||||||
|
if errors.Is(err, ErrAlreadyRedacted) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
cascaded = append(cascaded, r)
|
||||||
|
if d.kind != KindEvent {
|
||||||
|
deps = append(deps, s.referencingEvents(ctx, tx, req.MindID, d.kind, d.id)...)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return primary, cascaded, err
|
||||||
|
}
|
||||||
|
// Stored archives hold copies of memories, messages and beliefs; scrub them.
|
||||||
|
if s.Snapshots != nil {
|
||||||
|
for _, r := range append([]Redaction{primary}, cascaded...) {
|
||||||
|
if n, err := s.Snapshots.Scrub(ctx, r.MindID, string(r.Kind), r.TargetID); err != nil {
|
||||||
|
s.Log.Warn("snapshot scrub failed", "redaction", r.ID, "err", err)
|
||||||
|
} else if n > 0 {
|
||||||
|
s.Log.Info("snapshot archives scrubbed", "redaction", r.ID, "archives", n)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if s.Audit != nil {
|
||||||
|
s.Audit.Log(ctx, "privacy.redact", string(req.Kind), req.ID.String(), map[string]any{"redaction_id": primary.ID, "reason": req.Reason, "policy": req.Policy, "cascaded": len(cascaded), "content_hash": primary.ContentHash})
|
||||||
|
}
|
||||||
|
return primary, cascaded, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *Service) redactOne(ctx context.Context, tx pgx.Tx, req Request, cascadeOf ids.ID) (Redaction, error) {
|
||||||
|
content, mind, err := s.content(ctx, tx, req.Kind, req.ID)
|
||||||
|
if err != nil {
|
||||||
|
return Redaction{}, err
|
||||||
|
}
|
||||||
|
if mind != req.MindID {
|
||||||
|
return Redaction{}, fmt.Errorf("%w: %s %s does not belong to this mind", ErrInvalid, req.Kind, req.ID)
|
||||||
|
}
|
||||||
|
if content == nil {
|
||||||
|
return Redaction{}, ErrAlreadyRedacted
|
||||||
|
}
|
||||||
|
sum := sha256.Sum256(content)
|
||||||
|
r := Redaction{ID: ids.New(), MindID: req.MindID, Kind: req.Kind, TargetID: req.ID, Reason: req.Reason, Policy: req.Policy, ContentHash: "sha256:" + hex.EncodeToString(sum[:]), RequestedBy: req.By, CascadeOf: cascadeOf, CreatedAt: time.Now()}
|
||||||
|
if _, err := tx.Exec(ctx, `INSERT INTO redactions (id, mind_id, target_kind, target_id, reason, policy, content_hash, requested_by, cascade_of, created_at) VALUES ($1,$2,$3,$4,$5,$6,$7,$8,$9,$10)`,
|
||||||
|
r.ID, r.MindID, string(r.Kind), r.TargetID, r.Reason, r.Policy, r.ContentHash, r.RequestedBy, nullID(cascadeOf), r.CreatedAt); err != nil {
|
||||||
|
if strings.Contains(err.Error(), "redactions_target_kind_target_id_key") {
|
||||||
|
return Redaction{}, ErrAlreadyRedacted
|
||||||
|
}
|
||||||
|
return Redaction{}, err
|
||||||
|
}
|
||||||
|
var n int
|
||||||
|
if err := tx.QueryRow(ctx, `SELECT noema_redact($1, $2, $3)`, string(req.Kind), req.ID, r.ID).Scan(&n); err != nil {
|
||||||
|
return Redaction{}, err
|
||||||
|
}
|
||||||
|
if n == 0 {
|
||||||
|
return Redaction{}, ErrAlreadyRedacted
|
||||||
|
}
|
||||||
|
// The history records that a redaction happened, never what was removed.
|
||||||
|
if s.Events != nil {
|
||||||
|
if _, err := s.Events.AppendIn(ctx, tx, events.New{MindID: req.MindID, Type: events.Redaction, Source: "privacy", Summary: fmt.Sprintf("%s %s redacted under %s", req.Kind, req.ID.String()[len(req.ID.String())-8:], firstNonEmpty(req.Policy, "operator request")),
|
||||||
|
Salience: 0.2, Confidence: 1, Evidence: []events.EvidenceRef{{Kind: string(req.Kind), ID: string(req.ID), Note: "redacted"}},
|
||||||
|
Metadata: map[string]any{"redaction_id": r.ID, "target_kind": string(req.Kind), "target_id": req.ID, "policy": req.Policy, "content_hash": r.ContentHash, "cascade_of": cascadeOf},
|
||||||
|
Payload: map[string]any{"redaction_id": r.ID, "reason": req.Reason}}); err != nil {
|
||||||
|
return Redaction{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return r, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// content returns the JSON of the columns a redaction will remove (nil when
|
||||||
|
// already redacted) and the row's mind, for the hash and the scoping check.
|
||||||
|
func (s *Service) content(ctx context.Context, q storage.Querier, kind Kind, id ids.ID) ([]byte, ids.ID, error) {
|
||||||
|
var mind ids.ID
|
||||||
|
var already *time.Time
|
||||||
|
var raw []byte
|
||||||
|
var err error
|
||||||
|
switch kind {
|
||||||
|
case KindEvent:
|
||||||
|
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('summary', summary, 'payload', payload, 'metadata', metadata) FROM cognitive_events WHERE id=$1`, id).Scan(&mind, &already, &raw)
|
||||||
|
case KindMemory:
|
||||||
|
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('content', content, 'detail', detail) FROM memories WHERE id=$1`, id).Scan(&mind, &already, &raw)
|
||||||
|
case KindMessage:
|
||||||
|
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('content', content, 'intent', intent) FROM messages WHERE id=$1`, id).Scan(&mind, &already, &raw)
|
||||||
|
case KindLLMCall:
|
||||||
|
err = q.QueryRow(ctx, `SELECT coalesce(mind_id, '00000000-0000-0000-0000-000000000000'::uuid), redacted_at, jsonb_build_object('request', request, 'response', response) FROM llm_calls WHERE id=$1`, id).Scan(&mind, &already, &raw)
|
||||||
|
case KindBelief:
|
||||||
|
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('proposition', proposition, 'detail', detail) FROM beliefs WHERE id=$1`, id).Scan(&mind, &already, &raw)
|
||||||
|
case KindAssertion:
|
||||||
|
err = q.QueryRow(ctx, `SELECT mind_id, redacted_at, jsonb_build_object('proposition', proposition, 'context', context) FROM assertions WHERE id=$1`, id).Scan(&mind, &already, &raw)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return nil, ids.Nil, storage.Translate(err)
|
||||||
|
}
|
||||||
|
if already != nil {
|
||||||
|
return nil, mind, nil
|
||||||
|
}
|
||||||
|
return raw, mind, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type dep struct {
|
||||||
|
kind Kind
|
||||||
|
id ids.ID
|
||||||
|
}
|
||||||
|
|
||||||
|
// dependants lists rows whose content derives from the target: episodes
|
||||||
|
// encoded from an event, the message it carried, assertions made in it,
|
||||||
|
// beliefs whose evidence roots at it, and the model calls that worded the
|
||||||
|
// messages. Cascade is one level from an event; deeper structure keeps its
|
||||||
|
// links but not the content.
|
||||||
|
func (s *Service) dependants(ctx context.Context, q storage.Querier, mind ids.ID, kind Kind, id ids.ID) []dep {
|
||||||
|
var out []dep
|
||||||
|
collect := func(k Kind, sql string, args ...any) {
|
||||||
|
rows, err := q.Query(ctx, sql, args...)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
for rows.Next() {
|
||||||
|
var x ids.ID
|
||||||
|
if rows.Scan(&x) == nil && !x.IsZero() {
|
||||||
|
out = append(out, dep{k, x})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
switch kind {
|
||||||
|
case KindEvent:
|
||||||
|
collect(KindMemory, `SELECT id FROM memories WHERE mind_id=$1 AND origin_event=$2 AND redacted_at IS NULL`, mind, id)
|
||||||
|
collect(KindMessage, `SELECT id FROM messages WHERE mind_id=$1 AND event_id=$2 AND redacted_at IS NULL`, mind, id)
|
||||||
|
collect(KindAssertion, `SELECT id FROM assertions WHERE mind_id=$1 AND source_event=$2 AND redacted_at IS NULL`, mind, id)
|
||||||
|
collect(KindBelief, `SELECT DISTINCT belief_id FROM belief_evidence WHERE mind_id=$1 AND $2::uuid = ANY(root_ids)`, mind, id)
|
||||||
|
collect(KindLLMCall, `SELECT llm_call_id FROM messages WHERE mind_id=$1 AND (event_id=$2 OR reply_to IN (SELECT id FROM messages WHERE event_id=$2)) AND llm_call_id IS NOT NULL`, mind, id)
|
||||||
|
collect(KindMessage, `SELECT id FROM messages WHERE mind_id=$1 AND reply_to IN (SELECT id FROM messages WHERE event_id=$2) AND redacted_at IS NULL`, mind, id)
|
||||||
|
case KindMessage:
|
||||||
|
collect(KindLLMCall, `SELECT llm_call_id FROM messages WHERE mind_id=$1 AND id=$2 AND llm_call_id IS NOT NULL`, mind, id)
|
||||||
|
case KindBelief:
|
||||||
|
collect(KindAssertion, `SELECT id FROM assertions WHERE mind_id=$1 AND belief_id=$2 AND redacted_at IS NULL`, mind, id)
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// referencingEvents finds events that quote a row: direct children of an
|
||||||
|
// event, and events whose evidence or metadata point at the row. Redaction
|
||||||
|
// events are never included, so the record of redaction survives.
|
||||||
|
func (s *Service) referencingEvents(ctx context.Context, q storage.Querier, mind ids.ID, kind Kind, id ids.ID) []dep {
|
||||||
|
var out []dep
|
||||||
|
ref := fmt.Sprintf(`[{"kind":%q,"id":%q}]`, string(kind), id.String())
|
||||||
|
metaKey := string(kind) + "_id"
|
||||||
|
rows, err := q.Query(ctx, `SELECT id FROM cognitive_events WHERE mind_id=$1 AND redacted_at IS NULL AND type <> 'redaction'
|
||||||
|
AND (evidence @> $2::jsonb OR metadata->>$3 = $4 OR ($5 AND (causation_id=$6 OR $6 = ANY(parent_ids)))) LIMIT 500`, mind, ref, metaKey, id.String(), kind == KindEvent, id)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
for rows.Next() {
|
||||||
|
var x ids.ID
|
||||||
|
if rows.Scan(&x) == nil && x != id {
|
||||||
|
out = append(out, dep{KindEvent, x})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// List returns the ledger for a mind, newest first.
|
||||||
|
func (s *Service) List(ctx context.Context, mind ids.ID, limit int) ([]Redaction, error) {
|
||||||
|
if limit <= 0 || limit > 500 {
|
||||||
|
limit = 100
|
||||||
|
}
|
||||||
|
rows, err := s.DB.Query(ctx, `SELECT id, mind_id, target_kind, target_id, reason, policy, content_hash, requested_by, cascade_of, created_at FROM redactions WHERE mind_id=$1 ORDER BY created_at DESC LIMIT $2`, mind, limit)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer rows.Close()
|
||||||
|
var out []Redaction
|
||||||
|
for rows.Next() {
|
||||||
|
var r Redaction
|
||||||
|
var kind string
|
||||||
|
var cascade *string
|
||||||
|
if err := rows.Scan(&r.ID, &r.MindID, &kind, &r.TargetID, &r.Reason, &r.Policy, &r.ContentHash, &r.RequestedBy, &cascade, &r.CreatedAt); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
r.Kind = Kind(kind)
|
||||||
|
if cascade != nil {
|
||||||
|
r.CascadeOf = ids.ID(*cascade)
|
||||||
|
}
|
||||||
|
out = append(out, r)
|
||||||
|
}
|
||||||
|
return out, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
func validKind(k Kind) bool {
|
||||||
|
switch k {
|
||||||
|
case KindEvent, KindMemory, KindMessage, KindLLMCall, KindBelief, KindAssertion:
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func nullID(id ids.ID) *string {
|
||||||
|
if id.IsZero() {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
s := string(id)
|
||||||
|
return &s
|
||||||
|
}
|
||||||
|
|
||||||
|
func firstNonEmpty(a, b string) string {
|
||||||
|
if a != "" {
|
||||||
|
return a
|
||||||
|
}
|
||||||
|
return b
|
||||||
|
}
|
||||||
|
|
||||||
|
// Attest recomputes nothing: redacted content is gone. It exposes the hash
|
||||||
|
// so an operator holding an external copy can prove what was removed.
|
||||||
|
func Attest(r Redaction) string { return r.ContentHash }
|
||||||
|
|
||||||
|
var _ = json.Marshal
|
||||||
@@ -0,0 +1,197 @@
|
|||||||
|
package privacy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"log/slog"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/jackc/pgx/v5/pgxpool"
|
||||||
|
|
||||||
|
"github.com/andydixon/noema/internal/audit"
|
||||||
|
"github.com/andydixon/noema/internal/beliefs"
|
||||||
|
"github.com/andydixon/noema/internal/epistemics"
|
||||||
|
evstore "github.com/andydixon/noema/internal/events"
|
||||||
|
"github.com/andydixon/noema/internal/memory"
|
||||||
|
"github.com/andydixon/noema/internal/minds"
|
||||||
|
"github.com/andydixon/noema/internal/snapshots"
|
||||||
|
"github.com/andydixon/noema/internal/storage"
|
||||||
|
"github.com/andydixon/noema/migrations"
|
||||||
|
"github.com/andydixon/noema/pkg/concepts"
|
||||||
|
"github.com/andydixon/noema/pkg/events"
|
||||||
|
"github.com/andydixon/noema/pkg/ids"
|
||||||
|
)
|
||||||
|
|
||||||
|
// TestRedactionIsIrrecoverableButReferentiallyIntact: content goes, the
|
||||||
|
// row, its links, its type and time stay; the hash attests what was removed;
|
||||||
|
// derived rows are cascaded; stored archives are scrubbed; and the general
|
||||||
|
// append-only guarantee still holds for every other write.
|
||||||
|
func TestRedactionIsIrrecoverableButReferentiallyIntact(t *testing.T) {
|
||||||
|
pool := storage.TestPool(t, migrations.FS)
|
||||||
|
ctx := context.Background()
|
||||||
|
log := slog.New(slog.DiscardHandler)
|
||||||
|
ms := minds.NewStore(pool)
|
||||||
|
m, _ := ms.Create(ctx, minds.CreateParams{Slug: "priv"})
|
||||||
|
es := evstore.NewStore(pool, evstore.NewBus(8), nil)
|
||||||
|
mem := memory.NewStore(ctx, pool, nil)
|
||||||
|
bs := beliefs.NewStore(pool)
|
||||||
|
as := epistemics.NewStore(pool)
|
||||||
|
snaps := &snapshots.Store{DB: pool, Events: es, Minds: ms}
|
||||||
|
secret := "Andy said Steve steals laptops from the Leeds office"
|
||||||
|
|
||||||
|
obs, err := es.Append(ctx, events.New{MindID: m.ID, Type: events.Observation, Source: "conversation", Summary: secret, Payload: events.ObservationPayload{Text: secret, Actor: "andy", Trust: events.TrustUser}, Metadata: map[string]any{"conversation_id": "c1", "channel": "conversation"}})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
child, _ := es.Append(ctx, events.New{MindID: m.ID, Type: events.Perception, Source: "brain.perceive", Summary: "perceived: " + secret, CausationID: obs[0].ID, ParentIDs: []ids.ID{obs[0].ID}})
|
||||||
|
episode, _ := mem.Create(ctx, pool, memory.New{MindID: m.ID, Kind: concepts.Episodic, Epistemic: concepts.Observation, Content: "andy said: " + secret, Source: "conversation", Trust: events.TrustUser, Confidence: 1, Importance: 0.5, OccurredAt: time.Now(), OriginEvent: obs[0].ID})
|
||||||
|
emit := func(ctx context.Context, q storage.Querier, n events.New) (events.Event, error) {
|
||||||
|
out, err := es.AppendIn(ctx, q, n)
|
||||||
|
if err != nil {
|
||||||
|
return events.Event{}, err
|
||||||
|
}
|
||||||
|
return out[0], nil
|
||||||
|
}
|
||||||
|
res, _ := bs.Create(ctx, pool, emit, beliefs.New{MindID: m.ID, Proposition: "Steve steals laptops", Subject: "steve", Epistemic: concepts.Claim, Source: "conversation",
|
||||||
|
Evidence: []beliefs.EvidenceInput{{Direction: "for", Kind: "assertion", Source: "asserted-by:andy", Family: "actor:andy", Derivation: beliefs.DerivationAssertion, Weight: 0.5, Credibility: 0.85, Note: "andy asserted it", RootIDs: []string{string(obs[0].ID)}}}})
|
||||||
|
assertion, _ := as.Record(ctx, pool, epistemics.New{MindID: m.ID, Actor: "andy", Interpretation: epistemics.Interpret("andy", "Steve steals laptops"), SourceEvent: obs[0].ID, Context: secret, BeliefID: res.Belief.ID})
|
||||||
|
convID, msgID, replyID, callID := ids.New(), ids.New(), ids.New(), ids.New()
|
||||||
|
pool.Exec(ctx, `INSERT INTO conversations (id, mind_id, title, channel) VALUES ($1,$2,'t','conversation')`, convID, m.ID) //nolint:errcheck
|
||||||
|
pool.Exec(ctx, `INSERT INTO llm_calls (id, mind_id, role, provider, model, request_hash, request, response) VALUES ($1,$2,'respond','p','m','h', $3, $4)`, callID, m.ID, `{"messages":[{"content":"`+secret+`"}]}`, `{"text":"noted"}`) //nolint:errcheck
|
||||||
|
if _, err := pool.Exec(ctx, `INSERT INTO messages (id, conversation_id, mind_id, role, actor, content, event_id) VALUES ($1,$2,$3,'user','andy',$4,$5)`, msgID, convID, m.ID, secret, obs[0].ID); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
pool.Exec(ctx, `INSERT INTO messages (id, conversation_id, mind_id, role, actor, content, reply_to, llm_call_id) VALUES ($1,$2,$3,'mind','mind','Noted: Steve steals laptops',$4,$5)`, replyID, convID, m.ID, msgID, callID) //nolint:errcheck
|
||||||
|
sn, err := snaps.Create(ctx, m, "before", "", "test")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if raw, _ := json.Marshal(mustArchive(t, snaps, sn.ID)); !strings.Contains(string(raw), "Leeds") {
|
||||||
|
t.Fatal("test setup: archive should contain the secret before redaction")
|
||||||
|
}
|
||||||
|
|
||||||
|
svc := &Service{DB: pool, Events: es, Audit: audit.New(pool, log), Snapshots: snaps, Log: log}
|
||||||
|
// Refuses bad requests and rows from other minds.
|
||||||
|
other, _ := ms.Create(ctx, minds.CreateParams{Slug: "other"})
|
||||||
|
if _, _, err := svc.Redact(ctx, Request{MindID: other.ID, Kind: KindEvent, ID: obs[0].ID, Reason: "x", By: "admin"}); err == nil {
|
||||||
|
t.Fatal("redacting another mind's row must fail")
|
||||||
|
}
|
||||||
|
if _, _, err := svc.Redact(ctx, Request{MindID: m.ID, Kind: KindEvent, ID: obs[0].ID, By: "admin"}); err == nil {
|
||||||
|
t.Fatal("a reason is required")
|
||||||
|
}
|
||||||
|
primary, cascaded, err := svc.Redact(ctx, Request{MindID: m.ID, Kind: KindEvent, ID: obs[0].ID, Reason: "data subject request", Policy: "GDPR art. 17", Cascade: true, By: "admin"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
// The event exists, keeps type, time, links and ids; content is a marker.
|
||||||
|
ev, _ := es.Get(ctx, obs[0].ID)
|
||||||
|
if ev.Summary != Marker || string(ev.Payload) != `{"redacted": true}` || ev.RedactedAt == nil || ev.Type != events.Observation || ev.Seq != obs[0].Seq {
|
||||||
|
t.Fatalf("event after redaction: %+v payload=%s", ev, ev.Payload)
|
||||||
|
}
|
||||||
|
if ev.Metadata["conversation_id"] != "c1" || ev.Metadata["redacted"] != true {
|
||||||
|
t.Fatalf("metadata should keep only structural keys: %+v", ev.Metadata)
|
||||||
|
}
|
||||||
|
if kids, _ := es.Children(ctx, obs[0].ID, 10); len(kids) != 1 || kids[0].ID != child[0].ID {
|
||||||
|
t.Fatal("links to dependants must survive")
|
||||||
|
}
|
||||||
|
// The hash attests the removed content.
|
||||||
|
want := sha256.Sum256(mustJSONB(t, pool, `SELECT jsonb_build_object('summary', $1::text, 'payload', $2::jsonb, 'metadata', $3::jsonb)`, secret, string(obs[0].Payload), mustJSON(obs[0].Metadata)))
|
||||||
|
if primary.ContentHash != "sha256:"+hex.EncodeToString(want[:]) {
|
||||||
|
t.Fatalf("hash: %s", primary.ContentHash)
|
||||||
|
}
|
||||||
|
// Cascade reached the episode, the message, the reply, the model call, the assertion and the belief.
|
||||||
|
kinds := map[Kind]int{}
|
||||||
|
for _, c := range cascaded {
|
||||||
|
kinds[c.Kind]++
|
||||||
|
if c.CascadeOf != primary.ID {
|
||||||
|
t.Fatalf("cascade lineage missing: %+v", c)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if kinds[KindMemory] != 1 || kinds[KindMessage] != 2 || kinds[KindLLMCall] != 1 || kinds[KindAssertion] != 1 || kinds[KindBelief] != 1 {
|
||||||
|
t.Fatalf("cascade: %+v", kinds)
|
||||||
|
}
|
||||||
|
if mm, _ := mem.Get(ctx, m.ID, episode.ID); mm.Content != Marker || mm.RedactedAt == nil || mm.OriginEvent != obs[0].ID {
|
||||||
|
t.Fatalf("memory: %+v", mm)
|
||||||
|
}
|
||||||
|
if b, _ := bs.Get(ctx, pool, m.ID, res.Belief.ID); b.Proposition != Marker || b.Confidence != res.Belief.Confidence {
|
||||||
|
t.Fatalf("belief proposition should be redacted and its score untouched: %+v", b)
|
||||||
|
}
|
||||||
|
var vprop, note, msg, reply, req string
|
||||||
|
var resp *string
|
||||||
|
pool.QueryRow(ctx, `SELECT proposition FROM belief_versions WHERE belief_id=$1 LIMIT 1`, res.Belief.ID).Scan(&vprop) //nolint:errcheck
|
||||||
|
pool.QueryRow(ctx, `SELECT note FROM belief_evidence WHERE belief_id=$1 LIMIT 1`, res.Belief.ID).Scan(¬e) //nolint:errcheck
|
||||||
|
pool.QueryRow(ctx, `SELECT content FROM messages WHERE id=$1`, msgID).Scan(&msg) //nolint:errcheck
|
||||||
|
pool.QueryRow(ctx, `SELECT content FROM messages WHERE id=$1`, replyID).Scan(&reply) //nolint:errcheck
|
||||||
|
pool.QueryRow(ctx, `SELECT request::text, response::text FROM llm_calls WHERE id=$1`, callID).Scan(&req, &resp) //nolint:errcheck
|
||||||
|
if vprop != Marker || note != Marker || msg != Marker || reply != Marker || strings.Contains(req, "Leeds") || resp != nil {
|
||||||
|
t.Fatalf("cascade content: version=%q note=%q msg=%q reply=%q req=%q", vprop, note, msg, reply, req)
|
||||||
|
}
|
||||||
|
if a, _ := as.List(ctx, m.ID, epistemics.Filter{Limit: 5}); len(a) != 1 || a[0].Proposition != Marker || a[0].ID != assertion.ID {
|
||||||
|
t.Fatalf("assertion: %+v", a)
|
||||||
|
}
|
||||||
|
// Nothing in the live database still holds the secret.
|
||||||
|
var leaks int
|
||||||
|
pool.QueryRow(ctx, `SELECT (SELECT count(*) FROM cognitive_events WHERE mind_id=$1 AND (summary ILIKE '%leeds%' OR payload::text ILIKE '%leeds%' OR metadata::text ILIKE '%leeds%'))
|
||||||
|
+ (SELECT count(*) FROM memories WHERE mind_id=$1 AND content ILIKE '%leeds%') + (SELECT count(*) FROM messages WHERE mind_id=$1 AND content ILIKE '%leeds%')
|
||||||
|
+ (SELECT count(*) FROM llm_calls WHERE mind_id=$1 AND request::text ILIKE '%leeds%') + (SELECT count(*) FROM assertions WHERE mind_id=$1 AND (proposition ILIKE '%leeds%' OR context ILIKE '%leeds%'))`, m.ID).Scan(&leaks) //nolint:errcheck
|
||||||
|
if leaks != 0 {
|
||||||
|
t.Fatalf("secret still present in %d rows", leaks)
|
||||||
|
}
|
||||||
|
// Stored archives were scrubbed and re-signed; they still validate.
|
||||||
|
a := mustArchive(t, snaps, sn.ID)
|
||||||
|
if raw, _ := json.Marshal(a); strings.Contains(string(raw), "Leeds") || strings.Contains(string(raw), "steals") {
|
||||||
|
t.Fatal("archive still holds redacted content")
|
||||||
|
}
|
||||||
|
if len(a.Redacted) < 3 || snapshots.Validate(a) != nil {
|
||||||
|
t.Fatalf("archive scrub record or checksum: %v %v", a.Redacted, snapshots.Validate(a))
|
||||||
|
}
|
||||||
|
// The ledger and the history record what happened, not what was removed.
|
||||||
|
ledger, _ := svc.List(ctx, m.ID, 20)
|
||||||
|
if len(ledger) != 1+len(cascaded) || ledger[0].Reason != "data subject request" {
|
||||||
|
t.Fatalf("ledger: %d", len(ledger))
|
||||||
|
}
|
||||||
|
reds, _ := es.Query(ctx, evstore.Filter{MindID: m.ID, Types: []events.Type{events.Redaction}, Limit: 20})
|
||||||
|
if len(reds) != 1+len(cascaded) || strings.Contains(reds[0].Summary, "Leeds") {
|
||||||
|
t.Fatalf("redaction events: %d", len(reds))
|
||||||
|
}
|
||||||
|
// Idempotent and still append-only for everything else.
|
||||||
|
if _, _, err := svc.Redact(ctx, Request{MindID: m.ID, Kind: KindEvent, ID: obs[0].ID, Reason: "again", By: "admin"}); err == nil {
|
||||||
|
t.Fatal("second redaction should report already redacted")
|
||||||
|
}
|
||||||
|
if _, err := pool.Exec(ctx, `UPDATE cognitive_events SET summary='tampered' WHERE id=$1`, child[0].ID); err == nil {
|
||||||
|
t.Fatal("plain updates must still be forbidden")
|
||||||
|
}
|
||||||
|
if _, err := pool.Exec(ctx, `DELETE FROM cognitive_events WHERE id=$1`, obs[0].ID); err == nil {
|
||||||
|
t.Fatal("deletes must still be forbidden, even of redacted rows")
|
||||||
|
}
|
||||||
|
if _, err := pool.Exec(ctx, `UPDATE redactions SET reason='x' WHERE id=$1`, primary.ID); err == nil {
|
||||||
|
t.Fatal("the ledger is append-only")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustArchive(t *testing.T, s *snapshots.Store, id ids.ID) snapshots.Archive {
|
||||||
|
t.Helper()
|
||||||
|
a, err := s.Archive(context.Background(), id)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return a
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustJSON(v any) string {
|
||||||
|
b, _ := json.Marshal(v)
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustJSONB(t *testing.T, pool *pgxpool.Pool, sql string, args ...any) []byte {
|
||||||
|
t.Helper()
|
||||||
|
var raw []byte
|
||||||
|
if err := pool.QueryRow(context.Background(), sql, args...).Scan(&raw); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return raw
|
||||||
|
}
|
||||||
@@ -69,6 +69,7 @@ type Summary struct {
|
|||||||
LLMLive int `json:"llm_live"`
|
LLMLive int `json:"llm_live"`
|
||||||
LLMReplayed int `json:"llm_replayed"`
|
LLMReplayed int `json:"llm_replayed"`
|
||||||
Cycles int `json:"cycles"`
|
Cycles int `json:"cycles"`
|
||||||
|
Redacted int `json:"redacted"` // stimuli whose content was redacted and could not be replayed
|
||||||
}
|
}
|
||||||
|
|
||||||
// Step is the outcome of replaying one stimulus.
|
// Step is the outcome of replaying one stimulus.
|
||||||
@@ -218,6 +219,13 @@ func (s *Service) Execute(ctx context.Context, r Replay) (Replay, error) {
|
|||||||
}
|
}
|
||||||
orig, _ := s.Events.Query(ctx, evstore.Filter{MindID: src.ID, AfterSeq: stim.Seq, BeforeSeq: upper, Ascending: true, Limit: 2000})
|
orig, _ := s.Events.Query(ctx, evstore.Filter{MindID: src.ID, AfterSeq: stim.Seq, BeforeSeq: upper, Ascending: true, Limit: 2000})
|
||||||
st.Original = Digest(withoutPerception(orig))
|
st.Original = Digest(withoutPerception(orig))
|
||||||
|
if stim.RedactedAt != nil {
|
||||||
|
// The content is gone; replaying a marker would fabricate cognition.
|
||||||
|
st.Label, st.Note = "redacted", "stimulus content was redacted under a privacy policy and was not replayed"
|
||||||
|
r.Summary.Redacted++
|
||||||
|
steps = append(steps, st)
|
||||||
|
continue
|
||||||
|
}
|
||||||
startedAt := time.Now()
|
startedAt := time.Now()
|
||||||
inj, err := drv.Inject(ctx, stim, map[string]any{"replay_of": stim.ID, "replay_id": r.ID})
|
inj, err := drv.Inject(ctx, stim, map[string]any{"replay_of": stim.ID, "replay_id": r.ID})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -87,6 +87,18 @@ func TestReplayIsExactWithoutModel(t *testing.T) {
|
|||||||
if after[0].Seq != last[0].Seq {
|
if after[0].Seq != last[0].Seq {
|
||||||
t.Fatal("replay wrote into the source mind")
|
t.Fatal("replay wrote into the source mind")
|
||||||
}
|
}
|
||||||
|
// A redacted stimulus is not replayed and is labelled as such.
|
||||||
|
if _, err := h.Pool.Exec(ctx, `INSERT INTO redactions (id, mind_id, target_kind, target_id, reason, content_hash, requested_by) VALUES ($1,$2,'event',$3,'test','sha256:0','test')`, ids.New(), src.ID, steps[0].StimulusID); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var n int
|
||||||
|
if err := h.Pool.QueryRow(ctx, `SELECT noema_redact('event', $1, $2)`, steps[0].StimulusID, ids.New()).Scan(&n); err != nil || n != 1 {
|
||||||
|
t.Fatalf("redact: %v %d", err, n)
|
||||||
|
}
|
||||||
|
red, err := svc.Run(ctx, src, sn.ID, sn.EventSeq, last[0].Seq, "test")
|
||||||
|
if err != nil || red.Summary.Redacted != 1 || red.Summary.Exact != 0 || StepsOf(red)[0].Label != "redacted" {
|
||||||
|
t.Fatalf("redacted stimulus should be skipped and labelled: %v %+v", err, red.Summary)
|
||||||
|
}
|
||||||
// Digest normalisation ignores ids and numbers.
|
// Digest normalisation ignores ids and numbers.
|
||||||
if Normalise("selected 3 thoughts for 01a0a148-9034-7685-9794-cf5ec0c25fb7") != "selected <n> thoughts for <id>" {
|
if Normalise("selected 3 thoughts for 01a0a148-9034-7685-9794-cf5ec0c25fb7") != "selected <n> thoughts for <id>" {
|
||||||
t.Fatal(Normalise("selected 3 thoughts for 01a0a148-9034-7685-9794-cf5ec0c25fb7"))
|
t.Fatal(Normalise("selected 3 thoughts for 01a0a148-9034-7685-9794-cf5ec0c25fb7"))
|
||||||
|
|||||||
@@ -68,6 +68,10 @@ type Archive struct {
|
|||||||
// Tables maps table name → rows (column → value, as JSON).
|
// Tables maps table name → rows (column → value, as JSON).
|
||||||
Tables map[string][]map[string]any `json:"tables"`
|
Tables map[string][]map[string]any `json:"tables"`
|
||||||
Checksum string `json:"checksum"`
|
Checksum string `json:"checksum"`
|
||||||
|
// Redacted lists row ids whose content was removed from this archive
|
||||||
|
// after capture under a privacy redaction; the checksum covers the
|
||||||
|
// scrubbed content.
|
||||||
|
Redacted []string `json:"redacted,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// Source records where an archive came from.
|
// Source records where an archive came from.
|
||||||
@@ -544,3 +548,78 @@ func SortedTables(a Archive) []string {
|
|||||||
sort.Strings(out)
|
sort.Strings(out)
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Scrub removes the content of a redacted row from every stored archive of
|
||||||
|
// the mind, recording the id in the archive and re-signing it. Returns the
|
||||||
|
// number of archives changed. Events and model calls are never in archives.
|
||||||
|
func (s *Store) Scrub(ctx context.Context, mind ids.ID, kind string, id ids.ID) (int, error) {
|
||||||
|
rows, err := s.DB.Query(ctx, `SELECT id, archive FROM snapshots WHERE mind_id=$1`, mind)
|
||||||
|
if err != nil {
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
type snap struct {
|
||||||
|
id ids.ID
|
||||||
|
raw []byte
|
||||||
|
}
|
||||||
|
var snaps []snap
|
||||||
|
for rows.Next() {
|
||||||
|
var sn snap
|
||||||
|
if err := rows.Scan(&sn.id, &sn.raw); err != nil {
|
||||||
|
rows.Close()
|
||||||
|
return 0, err
|
||||||
|
}
|
||||||
|
snaps = append(snaps, sn)
|
||||||
|
}
|
||||||
|
rows.Close()
|
||||||
|
changed := 0
|
||||||
|
target := id.String()
|
||||||
|
for _, sn := range snaps {
|
||||||
|
var a Archive
|
||||||
|
if err := json.Unmarshal(sn.raw, &a); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if !scrubArchive(&a, kind, target) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
a.Redacted = append(a.Redacted, target)
|
||||||
|
a.Checksum = Sum(a)
|
||||||
|
raw, _ := json.Marshal(a)
|
||||||
|
counts, _ := json.Marshal(a.Counts())
|
||||||
|
if _, err := s.DB.Exec(ctx, `UPDATE snapshots SET archive=$2, checksum=$3, size_bytes=$4, counts=$5 WHERE id=$1`, sn.id, raw, a.Checksum, len(raw), counts); err != nil {
|
||||||
|
return changed, err
|
||||||
|
}
|
||||||
|
changed++
|
||||||
|
}
|
||||||
|
return changed, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// scrubArchive replaces the content fields of the matching rows in place.
|
||||||
|
func scrubArchive(a *Archive, kind, id string) bool {
|
||||||
|
changed := false
|
||||||
|
replace := func(table string, match func(r map[string]any) bool, fields map[string]any) {
|
||||||
|
for _, r := range a.Tables[table] {
|
||||||
|
if match(r) {
|
||||||
|
for k, v := range fields {
|
||||||
|
if _, ok := r[k]; ok {
|
||||||
|
r[k] = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
changed = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
byID := func(r map[string]any) bool { return r["id"] == id }
|
||||||
|
switch kind {
|
||||||
|
case "memory":
|
||||||
|
replace("memories", byID, map[string]any{"content": "[redacted]", "detail": map[string]any{"redacted": true}})
|
||||||
|
case "message":
|
||||||
|
replace("messages", byID, map[string]any{"content": "[redacted]", "intent": nil})
|
||||||
|
case "belief":
|
||||||
|
replace("beliefs", byID, map[string]any{"proposition": "[redacted]", "detail": map[string]any{"redacted": true}})
|
||||||
|
byBelief := func(r map[string]any) bool { return r["belief_id"] == id }
|
||||||
|
replace("belief_versions", byBelief, map[string]any{"proposition": "[redacted]", "reason": "[redacted]"})
|
||||||
|
replace("belief_evidence", byBelief, map[string]any{"note": "[redacted]"})
|
||||||
|
replace("belief_alternatives", byBelief, map[string]any{"proposition": "[redacted]", "note": "[redacted]"})
|
||||||
|
}
|
||||||
|
return changed
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,43 @@
|
|||||||
|
package ui
|
||||||
|
|
||||||
|
import (
|
||||||
|
"net/http"
|
||||||
|
"strconv"
|
||||||
|
|
||||||
|
"github.com/andydixon/noema/internal/api"
|
||||||
|
"github.com/andydixon/noema/internal/auth"
|
||||||
|
"github.com/andydixon/noema/pkg/ids"
|
||||||
|
)
|
||||||
|
|
||||||
|
func (u *UI) privacyRoutes(m *http.ServeMux) {
|
||||||
|
m.Handle("GET /minds/{mind}/redactions", u.require(auth.RoleOperator, u.redactionsPage))
|
||||||
|
m.Handle("POST /minds/{mind}/redactions", u.require(auth.RoleAdministrator, u.redact))
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *UI) redactionsPage(w http.ResponseWriter, r *http.Request) {
|
||||||
|
m, ok := u.mind(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
list, _ := u.deps.Privacy.List(r.Context(), m.ID, 200)
|
||||||
|
u.render(w, r, "redactions", m.DisplayName+" · Redactions", map[string]any{"Mind": m, "Redactions": list, "Kind": r.URL.Query().Get("kind"), "ID": r.URL.Query().Get("id")})
|
||||||
|
}
|
||||||
|
|
||||||
|
func (u *UI) redact(w http.ResponseWriter, r *http.Request) {
|
||||||
|
m, ok := u.mind(w, r)
|
||||||
|
if !ok {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
id, err := ids.Parse(r.PostFormValue("id"))
|
||||||
|
if err != nil {
|
||||||
|
u.flash(w, r, "error", "Bad id.", "/minds/"+m.Slug+"/redactions")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
cascade, _ := strconv.ParseBool(r.PostFormValue("cascade"))
|
||||||
|
p, c, err := u.studyF().Redact(r, m.ID, api.RedactionInput{Kind: r.PostFormValue("kind"), ID: id, Reason: r.PostFormValue("reason"), Policy: r.PostFormValue("policy"), Cascade: cascade}, username(r))
|
||||||
|
if err != nil {
|
||||||
|
u.flash(w, r, "error", err.Error(), "/minds/"+m.Slug+"/redactions")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
u.flash(w, r, "ok", "Redacted "+string(p.Kind)+" "+p.TargetID.String()[len(p.TargetID.String())-8:]+" and "+itoa(len(c))+" dependant row(s). The content is irrecoverable; hash "+p.ContentHash[:23]+"… attests what was removed.", "/minds/"+m.Slug+"/redactions")
|
||||||
|
}
|
||||||
@@ -133,6 +133,7 @@ func (u *UI) parse(fsys fs.FS) error {
|
|||||||
"duration": func(s int64) string { return (time.Duration(s) * time.Second).Truncate(time.Second).String() },
|
"duration": func(s int64) string { return (time.Duration(s) * time.Second).Truncate(time.Second).String() },
|
||||||
"bytes": humanBytes,
|
"bytes": humanBytes,
|
||||||
"pct": func(f float64) string { return fmt.Sprintf("%.0f%%", f*100) },
|
"pct": func(f float64) string { return fmt.Sprintf("%.0f%%", f*100) },
|
||||||
|
"list": func(xs ...string) []string { return xs },
|
||||||
"conf": func(v any) string {
|
"conf": func(v any) string {
|
||||||
switch x := v.(type) {
|
switch x := v.(type) {
|
||||||
case float64:
|
case float64:
|
||||||
@@ -388,6 +389,7 @@ func (u *UI) Register(m *http.ServeMux) {
|
|||||||
u.integrationRoutes(m)
|
u.integrationRoutes(m)
|
||||||
u.dreamRoutes(m)
|
u.dreamRoutes(m)
|
||||||
u.studyRoutes(m)
|
u.studyRoutes(m)
|
||||||
|
u.privacyRoutes(m)
|
||||||
|
|
||||||
for _, p := range []struct{ path, title, note string }{} {
|
for _, p := range []struct{ path, title, note string }{} {
|
||||||
m.Handle("GET "+p.path, u.require(auth.RoleObserver, func(w http.ResponseWriter, r *http.Request) { u.render(w, r, "placeholder", p.title, p.note) }))
|
m.Handle("GET "+p.path, u.require(auth.RoleObserver, func(w http.ResponseWriter, r *http.Request) { u.render(w, r, "placeholder", p.title, p.note) }))
|
||||||
|
|||||||
@@ -0,0 +1,13 @@
|
|||||||
|
-- Redacted content cannot be restored; this only removes the machinery.
|
||||||
|
DROP FUNCTION IF EXISTS noema_redact(text, uuid, uuid);
|
||||||
|
CREATE OR REPLACE FUNCTION noema_forbid_change() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
BEGIN
|
||||||
|
RAISE EXCEPTION 'table % is append-only', TG_TABLE_NAME USING ERRCODE = 'integrity_constraint_violation';
|
||||||
|
END $$;
|
||||||
|
ALTER TABLE assertions DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
|
||||||
|
ALTER TABLE beliefs DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
|
||||||
|
ALTER TABLE llm_calls DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
|
||||||
|
ALTER TABLE messages DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
|
||||||
|
ALTER TABLE memories DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
|
||||||
|
ALTER TABLE cognitive_events DROP COLUMN IF EXISTS redaction_id, DROP COLUMN IF EXISTS redacted_at;
|
||||||
|
DROP TABLE IF EXISTS redactions;
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
-- Privacy redaction (review pass §7.2 item 8): the audited maintenance path
|
||||||
|
-- ADR-0002 promised. Redaction replaces the sensitive content of a row with a
|
||||||
|
-- fixed marker while keeping the row, its identifiers, its links and a hash
|
||||||
|
-- of what was removed. It is the only write ever permitted on append-only
|
||||||
|
-- tables, it is confined to content columns, and it runs through
|
||||||
|
-- noema_redact() so the application cannot reach the bypass by accident.
|
||||||
|
|
||||||
|
CREATE TABLE redactions (
|
||||||
|
id uuid PRIMARY KEY,
|
||||||
|
mind_id uuid NOT NULL REFERENCES minds(id),
|
||||||
|
target_kind text NOT NULL CHECK (target_kind IN ('event','memory','message','llm_call','belief','assertion')),
|
||||||
|
target_id uuid NOT NULL,
|
||||||
|
reason text NOT NULL,
|
||||||
|
policy text NOT NULL DEFAULT '', -- retention or privacy policy invoked
|
||||||
|
content_hash text NOT NULL, -- sha256 of the removed content, for later attestation
|
||||||
|
requested_by text NOT NULL,
|
||||||
|
cascade_of uuid REFERENCES redactions(id),
|
||||||
|
created_at timestamptz NOT NULL DEFAULT now(),
|
||||||
|
UNIQUE (target_kind, target_id)
|
||||||
|
);
|
||||||
|
CREATE INDEX redactions_mind_idx ON redactions (mind_id, created_at DESC);
|
||||||
|
CREATE TRIGGER redactions_append_only BEFORE UPDATE OR DELETE ON redactions
|
||||||
|
FOR EACH ROW EXECUTE FUNCTION noema_forbid_change();
|
||||||
|
|
||||||
|
ALTER TABLE cognitive_events ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
|
||||||
|
ALTER TABLE memories ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
|
||||||
|
ALTER TABLE messages ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
|
||||||
|
ALTER TABLE llm_calls ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
|
||||||
|
ALTER TABLE beliefs ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
|
||||||
|
ALTER TABLE assertions ADD COLUMN IF NOT EXISTS redacted_at timestamptz, ADD COLUMN IF NOT EXISTS redaction_id uuid;
|
||||||
|
|
||||||
|
-- The append-only trigger admits exactly one kind of UPDATE: a redaction
|
||||||
|
-- running under the session flag that only noema_redact() sets. DELETE
|
||||||
|
-- stays forbidden in every case.
|
||||||
|
CREATE OR REPLACE FUNCTION noema_forbid_change() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||||
|
BEGIN
|
||||||
|
IF TG_OP = 'UPDATE' AND current_setting('noema.redaction', true) = 'on' THEN
|
||||||
|
RETURN NEW;
|
||||||
|
END IF;
|
||||||
|
RAISE EXCEPTION 'table % is append-only', TG_TABLE_NAME USING ERRCODE = 'integrity_constraint_violation';
|
||||||
|
END $$;
|
||||||
|
|
||||||
|
-- noema_redact replaces content columns for one row. It sets the bypass
|
||||||
|
-- flag for the current transaction only, touches nothing but content and
|
||||||
|
-- the redaction bookkeeping, and returns the number of rows changed.
|
||||||
|
CREATE OR REPLACE FUNCTION noema_redact(kind text, target uuid, redaction uuid) RETURNS integer LANGUAGE plpgsql SECURITY DEFINER AS $$
|
||||||
|
DECLARE n integer := 0;
|
||||||
|
BEGIN
|
||||||
|
PERFORM set_config('noema.redaction', 'on', true);
|
||||||
|
IF kind = 'event' THEN
|
||||||
|
UPDATE cognitive_events SET summary = '[redacted]', payload = '{"redacted":true}'::jsonb,
|
||||||
|
metadata = (SELECT coalesce(jsonb_object_agg(key, value), '{}'::jsonb) FROM jsonb_each(metadata)
|
||||||
|
WHERE key IN ('conversation_id','message_id','cycle','belief_id','memory_id','integration_id','integration','kind','trust','method','replay_of','replay_id'))
|
||||||
|
|| jsonb_build_object('redacted', true),
|
||||||
|
redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
|
||||||
|
ELSIF kind = 'memory' THEN
|
||||||
|
UPDATE memories SET content = '[redacted]', detail = '{"redacted":true}'::jsonb, redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
|
||||||
|
ELSIF kind = 'message' THEN
|
||||||
|
UPDATE messages SET content = '[redacted]', intent = NULL, redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
|
||||||
|
ELSIF kind = 'llm_call' THEN
|
||||||
|
UPDATE llm_calls SET request = '{"redacted":true}'::jsonb, response = NULL, error = CASE WHEN error = '' THEN '' ELSE '[redacted]' END, redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
|
||||||
|
ELSIF kind = 'belief' THEN
|
||||||
|
UPDATE beliefs SET proposition = '[redacted]', detail = '{"redacted":true}'::jsonb, redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
|
||||||
|
GET DIAGNOSTICS n = ROW_COUNT;
|
||||||
|
IF n > 0 THEN
|
||||||
|
UPDATE belief_versions SET proposition = '[redacted]', reason = '[redacted]' WHERE belief_id = target;
|
||||||
|
UPDATE belief_evidence SET note = '[redacted]' WHERE belief_id = target;
|
||||||
|
UPDATE belief_alternatives SET proposition = '[redacted]', note = '[redacted]' WHERE belief_id = target;
|
||||||
|
END IF;
|
||||||
|
PERFORM set_config('noema.redaction', 'off', true);
|
||||||
|
RETURN n;
|
||||||
|
ELSIF kind = 'assertion' THEN
|
||||||
|
UPDATE assertions SET proposition = '[redacted]', context = '[redacted]', interpretation_notes = '{}', redacted_at = now(), redaction_id = redaction WHERE id = target AND redacted_at IS NULL;
|
||||||
|
ELSE
|
||||||
|
RAISE EXCEPTION 'unknown redaction kind %', kind;
|
||||||
|
END IF;
|
||||||
|
GET DIAGNOSTICS n = ROW_COUNT;
|
||||||
|
PERFORM set_config('noema.redaction', 'off', true);
|
||||||
|
RETURN n;
|
||||||
|
END $$;
|
||||||
|
REVOKE ALL ON FUNCTION noema_redact(text, uuid, uuid) FROM PUBLIC;
|
||||||
@@ -64,6 +64,7 @@ const (
|
|||||||
ModeChanged Type = "mode_changed"
|
ModeChanged Type = "mode_changed"
|
||||||
ResponseGenerated Type = "response_generated"
|
ResponseGenerated Type = "response_generated"
|
||||||
CycleCompleted Type = "cycle_completed"
|
CycleCompleted Type = "cycle_completed"
|
||||||
|
Redaction Type = "redaction" // content of an earlier row was made irrecoverable under a policy
|
||||||
)
|
)
|
||||||
|
|
||||||
// Category groups types for filtering and colouring in the UI.
|
// Category groups types for filtering and colouring in the UI.
|
||||||
@@ -88,7 +89,7 @@ var categories = map[Type]Category{
|
|||||||
Observation: CatPerception, Perception: CatPerception,
|
Observation: CatPerception, Perception: CatPerception,
|
||||||
ThoughtCandidate: CatCognition, ThoughtSelected: CatCognition, WorkspaceBroadcast: CatCognition, AttentionShift: CatCognition,
|
ThoughtCandidate: CatCognition, ThoughtSelected: CatCognition, WorkspaceBroadcast: CatCognition, AttentionShift: CatCognition,
|
||||||
Reflection: CatCognition, ContradictionDetected: CatCognition, Metacognition: CatCognition, DreamAssociation: CatCognition,
|
Reflection: CatCognition, ContradictionDetected: CatCognition, Metacognition: CatCognition, DreamAssociation: CatCognition,
|
||||||
NarrativeUpdated: CatCognition, ResponseGenerated: CatCognition, CycleCompleted: CatSystem, ModeChanged: CatSystem,
|
NarrativeUpdated: CatCognition, ResponseGenerated: CatCognition, CycleCompleted: CatSystem, ModeChanged: CatSystem, Redaction: CatSystem,
|
||||||
MemoryCreated: CatMemory, MemoryRetrieved: CatMemory, MemoryReinforced: CatMemory, MemoryDecayed: CatMemory, MemoryConsolidated: CatMemory,
|
MemoryCreated: CatMemory, MemoryRetrieved: CatMemory, MemoryReinforced: CatMemory, MemoryDecayed: CatMemory, MemoryConsolidated: CatMemory,
|
||||||
BeliefCreated: CatBelief, BeliefUpdated: CatBelief, BeliefChallenged: CatBelief, BeliefRetracted: CatBelief,
|
BeliefCreated: CatBelief, BeliefUpdated: CatBelief, BeliefChallenged: CatBelief, BeliefRetracted: CatBelief,
|
||||||
HypothesisCreated: CatBelief, HypothesisSupported: CatBelief, HypothesisWeakened: CatBelief,
|
HypothesisCreated: CatBelief, HypothesisSupported: CatBelief, HypothesisWeakened: CatBelief,
|
||||||
@@ -160,6 +161,7 @@ type Event struct {
|
|||||||
Summary string `json:"summary"`
|
Summary string `json:"summary"`
|
||||||
Metadata map[string]any `json:"metadata"`
|
Metadata map[string]any `json:"metadata"`
|
||||||
Payload json.RawMessage `json:"payload"`
|
Payload json.RawMessage `json:"payload"`
|
||||||
|
RedactedAt *time.Time `json:"redacted_at,omitempty"` // content replaced by a marker; see redactions
|
||||||
}
|
}
|
||||||
|
|
||||||
// New describes an event to append. Zero OccurredAt means now.
|
// New describes an event to append. Zero OccurredAt means now.
|
||||||
|
|||||||
@@ -42,6 +42,7 @@ import (
|
|||||||
"github.com/andydixon/noema/internal/notifications"
|
"github.com/andydixon/noema/internal/notifications"
|
||||||
"github.com/andydixon/noema/internal/observability"
|
"github.com/andydixon/noema/internal/observability"
|
||||||
"github.com/andydixon/noema/internal/predictions"
|
"github.com/andydixon/noema/internal/predictions"
|
||||||
|
"github.com/andydixon/noema/internal/privacy"
|
||||||
"github.com/andydixon/noema/internal/reflection"
|
"github.com/andydixon/noema/internal/reflection"
|
||||||
"github.com/andydixon/noema/internal/scheduler"
|
"github.com/andydixon/noema/internal/scheduler"
|
||||||
"github.com/andydixon/noema/internal/security/secrets"
|
"github.com/andydixon/noema/internal/security/secrets"
|
||||||
@@ -89,6 +90,7 @@ func Start(t *testing.T) *Harness {
|
|||||||
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
|
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
|
||||||
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
|
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
|
||||||
deps.Assertions = epistemics.NewStore(pool)
|
deps.Assertions = epistemics.NewStore(pool)
|
||||||
|
deps.Privacy = &privacy.Service{DB: pool, Events: deps.Events, Audit: deps.Audit, Snapshots: deps.Snapshots, Log: log}
|
||||||
deps.Narrative = narrative.NewStore(pool)
|
deps.Narrative = narrative.NewStore(pool)
|
||||||
deps.Notifications = notifications.NewStore(pool)
|
deps.Notifications = notifications.NewStore(pool)
|
||||||
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
|
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
|
||||||
|
|||||||
@@ -42,6 +42,7 @@ import (
|
|||||||
"github.com/andydixon/noema/internal/notifications"
|
"github.com/andydixon/noema/internal/notifications"
|
||||||
"github.com/andydixon/noema/internal/observability"
|
"github.com/andydixon/noema/internal/observability"
|
||||||
"github.com/andydixon/noema/internal/predictions"
|
"github.com/andydixon/noema/internal/predictions"
|
||||||
|
"github.com/andydixon/noema/internal/privacy"
|
||||||
"github.com/andydixon/noema/internal/reflection"
|
"github.com/andydixon/noema/internal/reflection"
|
||||||
"github.com/andydixon/noema/internal/scheduler"
|
"github.com/andydixon/noema/internal/scheduler"
|
||||||
"github.com/andydixon/noema/internal/security/secrets"
|
"github.com/andydixon/noema/internal/security/secrets"
|
||||||
@@ -130,6 +131,7 @@ func TestFirstMilestone(t *testing.T) {
|
|||||||
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
|
deps.Dreaming = &dreaming.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Log: log}
|
||||||
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
|
deps.Snapshots = &snapshots.Store{DB: pool, Events: deps.Events, Minds: deps.Minds}
|
||||||
deps.Assertions = epistemics.NewStore(pool)
|
deps.Assertions = epistemics.NewStore(pool)
|
||||||
|
deps.Privacy = &privacy.Service{DB: pool, Events: deps.Events, Audit: deps.Audit, Snapshots: deps.Snapshots, Log: log}
|
||||||
deps.Narrative = narrative.NewStore(pool)
|
deps.Narrative = narrative.NewStore(pool)
|
||||||
deps.Notifications = notifications.NewStore(pool)
|
deps.Notifications = notifications.NewStore(pool)
|
||||||
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
|
deps.Consolidation = &consolidation.Service{DB: pool, Memory: deps.Memory, Beliefs: deps.Beliefs, Events: deps.Events, Minds: deps.Minds, Log: log}
|
||||||
|
|||||||
@@ -8,7 +8,7 @@
|
|||||||
<section class="grid cols-3-1">
|
<section class="grid cols-3-1">
|
||||||
<div class="stack">
|
<div class="stack">
|
||||||
<div class="card">
|
<div class="card">
|
||||||
<h2>Causal chain <a class="small" href="/events/{{$e.ID}}/timeline">full timeline →</a></h2>
|
<h2>Causal chain <a class="small" href="/events/{{$e.ID}}/timeline">full timeline →</a>{{if .Principal.User.Role.AtLeast "administrator"}} · <a class="small danger" href="/minds/{{.Data.Mind.Slug}}/redactions?kind=event&id={{$e.ID}}">redact…</a>{{end}}</h2>
|
||||||
<div class="chain">
|
<div class="chain">
|
||||||
{{if $c.Cause}}<div class="chain-node"><span class="muted small">caused by</span><br><span class="etype cat-{{$c.Cause.Category}}">{{$c.Cause.Type}}</span> <a href="/events/{{$c.Cause.ID}}">{{$c.Cause.Summary}}</a></div><div class="chain-arrow">↓</div>{{end}}
|
{{if $c.Cause}}<div class="chain-node"><span class="muted small">caused by</span><br><span class="etype cat-{{$c.Cause.Category}}">{{$c.Cause.Type}}</span> <a href="/events/{{$c.Cause.ID}}">{{$c.Cause.Summary}}</a></div><div class="chain-arrow">↓</div>{{end}}
|
||||||
<div class="chain-node current"><span class="etype cat-{{$e.Category}}">{{$e.Type}}</span> {{$e.Summary}}</div>
|
<div class="chain-node current"><span class="etype cat-{{$e.Category}}">{{$e.Type}}</span> {{$e.Summary}}</div>
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
{{define "content"}}
|
||||||
|
{{$m := .Data.Mind}}
|
||||||
|
<div class="pagehead mindhead"><div><h1>{{$m.DisplayName}} <span class="muted small">redactions</span></h1><p class="muted">The audited path for making sensitive content irrecoverable. A redaction keeps the row, its identifiers, its links, its type and its time, replaces the content with <code>[redacted]</code>, scrubs stored snapshot archives, and records a hash of what was removed. It cannot be undone. Backups and exports taken before it are outside this system's reach.</p></div></div>
|
||||||
|
{{template "mind_tabs" (dict "M" $m "Active" "redactions")}}
|
||||||
|
<section class="grid cols-3-1">
|
||||||
|
<div class="card">
|
||||||
|
{{if .Data.Redactions}}<table class="table compact"><thead><tr><th>When</th><th>Kind</th><th>Target</th><th>Reason</th><th>Policy</th><th>By</th><th>Cascade</th><th>Hash</th></tr></thead><tbody>
|
||||||
|
{{range .Data.Redactions}}<tr><td class="small muted nowrap">{{ago .CreatedAt}}</td><td><span class="badge">{{.Kind}}</span></td><td class="mono small">{{if eq .Kind "event"}}<a href="/events/{{.TargetID}}">{{trunc (printf "%s" .TargetID) 13}}</a>{{else if eq .Kind "memory"}}<a href="/minds/{{$m.Slug}}/memory/{{.TargetID}}">{{trunc (printf "%s" .TargetID) 13}}</a>{{else if eq .Kind "belief"}}<a href="/minds/{{$m.Slug}}/beliefs/{{.TargetID}}">{{trunc (printf "%s" .TargetID) 13}}</a>{{else}}{{trunc (printf "%s" .TargetID) 13}}{{end}}</td><td>{{.Reason}}</td><td class="small">{{.Policy}}</td><td class="small">{{.RequestedBy}}</td><td class="small muted">{{if .CascadeOf}}from {{trunc (printf "%s" .CascadeOf) 8}}{{else}}primary{{end}}</td><td class="mono small" title="{{.ContentHash}}">{{trunc .ContentHash 19}}…</td></tr>{{end}}</tbody></table>
|
||||||
|
{{else}}<p class="muted">Nothing has been redacted for this mind.</p>{{end}}
|
||||||
|
</div>
|
||||||
|
{{if .Principal.User.Role.AtLeast "administrator"}}
|
||||||
|
<div class="card"><h2>Redact</h2>
|
||||||
|
<form method="post" action="/minds/{{$m.Slug}}/redactions" class="stack"><input type="hidden" name="_csrf" value="{{.CSRF}}">
|
||||||
|
<label>Kind <select name="kind">{{range $k := list "event" "memory" "message" "llm_call" "belief" "assertion"}}<option value="{{$k}}" {{if eq $k $.Data.Kind}}selected{{end}}>{{$k}}</option>{{end}}</select></label>
|
||||||
|
<label>Id <input name="id" required value="{{.Data.ID}}" class="mono"></label>
|
||||||
|
<label>Reason <input name="reason" required maxlength="300" placeholder="data subject request, retention policy…"></label>
|
||||||
|
<label>Policy <input name="policy" maxlength="120" placeholder="e.g. GDPR art. 17"></label>
|
||||||
|
<label class="check"><input type="checkbox" name="cascade" value="true" checked> Also redact rows derived from it (episodes, messages, model calls, assertions, beliefs)</label>
|
||||||
|
<div><button class="btn btn-danger" data-confirm="This cannot be undone. The content will be irrecoverable everywhere this system controls. Continue?">Redact</button></div>
|
||||||
|
</form>
|
||||||
|
<p class="small muted">Redaction is recorded in the audit log and as a <code>redaction</code> event in the mind's history; neither contains the removed content.</p>
|
||||||
|
</div>
|
||||||
|
{{end}}
|
||||||
|
</section>
|
||||||
|
{{end}}
|
||||||
@@ -3,6 +3,6 @@
|
|||||||
<a href="/minds/{{$m.Slug}}" {{if eq $a "overview"}}aria-current="page"{{end}}>Overview</a>
|
<a href="/minds/{{$m.Slug}}" {{if eq $a "overview"}}aria-current="page"{{end}}>Overview</a>
|
||||||
<a href="/minds/{{$m.Slug}}/thoughts" {{if eq $a "thoughts"}}aria-current="page"{{end}}>Thoughts</a>
|
<a href="/minds/{{$m.Slug}}/thoughts" {{if eq $a "thoughts"}}aria-current="page"{{end}}>Thoughts</a>
|
||||||
<a href="/minds/{{$m.Slug}}/personality" {{if eq $a "personality"}}aria-current="page"{{end}}>Personality</a>
|
<a href="/minds/{{$m.Slug}}/personality" {{if eq $a "personality"}}aria-current="page"{{end}}>Personality</a>
|
||||||
<a href="/minds/{{$m.Slug}}/workspace" {{if eq $a "workspace"}}aria-current="page"{{end}}>Workspace</a><a href="/minds/{{$m.Slug}}/memory" {{if eq $a "memory"}}aria-current="page"{{end}}>Memory</a><a href="/minds/{{$m.Slug}}/beliefs" {{if eq $a "beliefs"}}aria-current="page"{{end}}>Beliefs</a><a href="/minds/{{$m.Slug}}/conversations" {{if eq $a "conversation"}}aria-current="page"{{end}}>Conversation</a><a href="/minds/{{$m.Slug}}/dreams" {{if eq $a "dreams"}}aria-current="page"{{end}}>Dreams</a><a href="/minds/{{$m.Slug}}/snapshots" {{if eq $a "snapshots"}}aria-current="page"{{end}}>Snapshots</a><a href="/minds/{{$m.Slug}}/replays" {{if eq $a "replays"}}aria-current="page"{{end}}>Replay</a>
|
<a href="/minds/{{$m.Slug}}/workspace" {{if eq $a "workspace"}}aria-current="page"{{end}}>Workspace</a><a href="/minds/{{$m.Slug}}/memory" {{if eq $a "memory"}}aria-current="page"{{end}}>Memory</a><a href="/minds/{{$m.Slug}}/beliefs" {{if eq $a "beliefs"}}aria-current="page"{{end}}>Beliefs</a><a href="/minds/{{$m.Slug}}/conversations" {{if eq $a "conversation"}}aria-current="page"{{end}}>Conversation</a><a href="/minds/{{$m.Slug}}/dreams" {{if eq $a "dreams"}}aria-current="page"{{end}}>Dreams</a><a href="/minds/{{$m.Slug}}/snapshots" {{if eq $a "snapshots"}}aria-current="page"{{end}}>Snapshots</a><a href="/minds/{{$m.Slug}}/replays" {{if eq $a "replays"}}aria-current="page"{{end}}>Replay</a><a href="/minds/{{$m.Slug}}/redactions" {{if eq $a "redactions"}}aria-current="page"{{end}}>Redactions</a>
|
||||||
</nav>
|
</nav>
|
||||||
{{end}}
|
{{end}}
|
||||||
|
|||||||
Reference in New Issue
Block a user