feat(beliefs): source reliability by domain, never global

A source's track record is kept per (evidence family, belief subject) and
updated only by outcomes: predictions derived from a belief resolving, and
operators marking a belief wrong. With three or more outcomes in a domain
the record adjusts the credibility of that source's new evidence in that
domain, and the adjustment is written on the evidence row. Standing in one
domain never moves standing in another, and the trust-level credibility
constants are never rewritten. Migration 0021 is additive; belief pages
show each cited family's record in the belief's domain.

CHANGELOG.md

@@ -10,6 +10,7 @@ All notable changes to Noema are recorded here. Format follows Keep a Changelog;
 - Security suite: SQL-injection effect test, static allow-list scan for assembled SQL, deceived-mind authority test, garbage model output test, self-preservation vocabulary test; threat model rewritten so every verification cites a test that asserts the claim.
 
 ### Added
+- Contextual source reliability: a per-(source, domain) record updated only by prediction outcomes and operator corrections, adjusting evidence credibility in that domain once three outcomes exist; never global. Shown on belief pages. Migration 0021.
 - Temporal beliefs: every belief has a temporal scope (persistent, transient, unknown, classified deterministically), an observation time and, for transient states, a validity interval. A later observation that a transient state changed closes the earlier belief (valid_until, superseded, score unchanged) instead of contesting it; simultaneous conflicts and persistent propositions still contradict. Introspection reports what was true and when. Migration 0020.
 - Confidence semantics: every breakdown states that the number is an uncalibrated epistemic score; belief detail adds a calibrated estimate from the mind's resolved predictions when at least twenty were decided in the same range (otherwise says why not) and names the primary uncertainty deterministically. `docs/beliefs.md` defines every epistemic number.
 - Explicit `unknown` belief status: no live evidence either way is neither tentative belief nor falsity. Migration 0019.

docs/beliefs.md

@@ -72,6 +72,10 @@ Derived from the breakdown unless an operator pinned it:
 - `rejected` below 0.15; `hypothesis` for hypotheses without support or below 0.5; `tentative` below 0.7; `likely` below 0.85; `accepted` above;
 - `superseded` when replaced by a newer proposition.
 
+## Source reliability by domain
+
+A source's track record is kept per (family, domain), where the domain is the belief's subject. It is updated only by outcomes: a prediction derived from a belief resolving (confirmed counts as right, refuted or expired as wrong, for every primary supporting family of that belief) and an operator marking a belief wrong. With at least three outcomes in a domain the record adjusts the credibility of that source's new evidence *in that domain*: neutral leaves it alone, a perfect record raises it by half, a hopeless one halves it; the adjustment is written into the evidence note. A person wrong three times about database internals keeps their full credibility about their own diary. The trust-level credibility (operator, user, system, tool, source, low) is never rewritten.
+
 ## Temporal scope
 
 Every belief has a `temporal_scope`: `persistent` (a fact without an interval: "atlas is the primary host", "Andy prefers Go"), `transient` (a state that can change: "the door is open", "server A is down") or `unknown` when the deterministic classifier (`ClassifyScope`) cannot tell. Transient beliefs record `observed_at` and `valid_from`.

internal/api/beliefs.go

@@ -143,6 +143,7 @@ type BeliefDetail struct {
 	// outcomes allow one, and the strongest deterministic reason to doubt it.
 	Calibration        beliefs.Calibrated          `json:"calibration"`
 	PrimaryUncertainty string                      `json:"primary_uncertainty"`
+	Reliability        []beliefs.SourceReliability `json:"reliability"` // each cited family\'s record in this belief\'s domain
 }
 
 // BeliefDetail assembles the detail; shared with the UI.
@@ -174,6 +175,10 @@ func (s *Server) BeliefDetail(r *http.Request, mind, id ids.ID) (BeliefDetail, e
 		d.Calibration = beliefs.CalibrateWith(b.Confidence, nil)
 	}
 	d.PrimaryUncertainty = beliefs.PrimaryUncertainty(d.Breakdown, d.Evidence)
+	d.Reliability = s.deps.Beliefs.ReliabilityFor(ctx, mind, b, d.Evidence)
+	if d.Reliability == nil {
+		d.Reliability = []beliefs.SourceReliability{}
+	}
 	d.Events, _ = s.deps.Events.DerivedFrom(ctx, "belief", string(id), 100)
 	if !b.OriginEvent.IsZero() {
 		if e, err := s.deps.Events.Get(ctx, b.OriginEvent); err == nil {

internal/beliefs/reliability.go

@@ -0,0 +1,146 @@
+package beliefs
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"math"
+
+	"github.com/jackc/pgx/v5"
+
+	"github.com/andydixon/noema/internal/storage"
+	"github.com/andydixon/noema/pkg/ids"
+)
+
+// SourceReliability is a source's recorded track record in one domain.
+// The score is a Laplace estimate (correct+1)/(correct+wrong+2), so a
+// source with no history sits at 0.5 and is left alone.
+type SourceReliability struct {
+	Source  string  `json:"source"`
+	Domain  string  `json:"domain"`
+	Correct int     `json:"correct"`
+	Wrong   int     `json:"wrong"`
+	Score   float64 `json:"score"`
+	Applied bool    `json:"applied"` // enough history to adjust credibility
+}
+
+// MinReliabilityObservations is the fewest outcomes in a domain before a
+// source's record adjusts the credibility of its evidence there.
+const MinReliabilityObservations = 3
+
+func (r SourceReliability) score() float64 {
+	return float64(r.Correct+1) / float64(r.Correct+r.Wrong+2)
+}
+
+// Adjust scales a trust-level credibility by the source's record in the
+// domain: neutral (0.5) leaves it alone, a perfect record raises it by
+// half, a hopeless one halves it. Applied only with enough history.
+func (r SourceReliability) Adjust(credibility float64) float64 {
+	if !r.Applied {
+		return credibility
+	}
+	return math.Min(1, math.Max(0.05, credibility*(0.5+r.Score)))
+}
+
+// RecordOutcome adds one correct or wrong outcome for source in domain.
+func (s *Store) RecordOutcome(ctx context.Context, q storage.Querier, mind ids.ID, source, domain string, correct bool) error {
+	if source == "" {
+		return nil
+	}
+	if domain == "" {
+		domain = "*"
+	}
+	col := "wrong"
+	if correct {
+		col = "correct"
+	}
+	_, err := q.Exec(ctx, fmt.Sprintf(`INSERT INTO source_reliability (mind_id, source, domain, %s) VALUES ($1,$2,$3,1)
+		ON CONFLICT (mind_id, source, domain) DO UPDATE SET %s = source_reliability.%s + 1, updated_at = now()`, col, col, col), mind, source, domain) // #nosec G202 -- col is one of two literals
+	return err
+}
+
+// Reliability returns the record for a source in a domain.
+func (s *Store) Reliability(ctx context.Context, q storage.Querier, mind ids.ID, source, domain string) (SourceReliability, error) {
+	if q == nil {
+		q = s.db
+	}
+	if domain == "" {
+		domain = "*"
+	}
+	r := SourceReliability{Source: source, Domain: domain}
+	err := q.QueryRow(ctx, `SELECT correct, wrong FROM source_reliability WHERE mind_id=$1 AND source=$2 AND domain=$3`, mind, source, domain).Scan(&r.Correct, &r.Wrong)
+	if err != nil && !isNoRows(err) {
+		return r, err
+	}
+	r.Score = round4(r.score())
+	r.Applied = r.Correct+r.Wrong >= MinReliabilityObservations
+	return r, nil
+}
+
+// ReliabilityFor lists the records of every family cited by a belief in its
+// domain, for display.
+func (s *Store) ReliabilityFor(ctx context.Context, mind ids.ID, b Belief, rows []EvidenceRow) []SourceReliability {
+	seen := map[string]bool{}
+	var out []SourceReliability
+	for _, e := range rows {
+		fam := FamilyOf(e.Evidence)
+		if fam == "" || seen[fam] {
+			continue
+		}
+		seen[fam] = true
+		if r, err := s.Reliability(ctx, nil, mind, fam, b.Subject); err == nil {
+			out = append(out, r)
+		}
+	}
+	return out
+}
+
+// applyReliability adjusts the credibility of new evidence by the source's
+// record in the belief's domain, noting the adjustment on the row so the
+// arithmetic stays inspectable. Trust-level credibility is never changed.
+func (s *Store) applyReliability(ctx context.Context, q storage.Querier, mind ids.ID, domain string, e Evidence, note string) (Evidence, string) {
+	fam := FamilyOf(e)
+	if fam == "" || e.Direction != "for" {
+		return e, note
+	}
+	r, err := s.Reliability(ctx, q, mind, fam, domain)
+	if err != nil || !r.Applied {
+		return e, note
+	}
+	before := e.Credibility
+	e.Credibility = round4(r.Adjust(e.Credibility))
+	if e.Credibility != before {
+		note += fmt.Sprintf(" [credibility %.2f→%.2f: %s has been right %d and wrong %d time(s) about %s]", before, e.Credibility, fam, r.Correct, r.Wrong, domain)
+	}
+	return e, note
+}
+
+// RecordBeliefOutcome records an outcome for every primary supporting family
+// of a belief, in the belief's domain only.
+func (s *Store) RecordBeliefOutcome(ctx context.Context, q storage.Querier, mind, beliefID ids.ID, correct bool) error {
+	b, err := s.Get(ctx, q, mind, beliefID)
+	if err != nil {
+		return err
+	}
+	rows, err := s.EvidenceFor(ctx, q, mind, beliefID)
+	if err != nil {
+		return err
+	}
+	seen := map[string]bool{}
+	for _, e := range rows {
+		if e.Retracted || e.Direction != "for" || Derived(e.Derivation) {
+			continue
+		}
+		fam := FamilyOf(e.Evidence)
+		if fam == "" || seen[fam] || fam == "prediction:"+"" {
+			continue
+		}
+		seen[fam] = true
+		if err := s.RecordOutcome(ctx, q, mind, fam, b.Subject, correct); err != nil {
+			return err
+		}
+	}
+	return nil
+}
+
+func isNoRows(err error) bool { return errors.Is(err, pgx.ErrNoRows) }

internal/beliefs/store.go

@@ -175,8 +175,12 @@ func (s *Store) Create(ctx context.Context, q storage.Querier, emit Emitter, n N
 		return Result{}, fmt.Errorf("%w: %s", ErrBlocked, c.Statement)
 	}
 	var ev []Evidence
+	var notes []string
 	for _, e := range n.Evidence {
-		ev = append(ev, normalise(e))
+		ne := normalise(e)
+		ne, note := s.applyReliability(ctx, q, n.MindID, strings.TrimSpace(n.Subject), ne, e.Note)
+		ev = append(ev, ne)
+		notes = append(notes, note)
 	}
 	prior := PriorFor(n.Epistemic)
 	bd := Compute(prior, ev)
@@ -220,8 +224,8 @@ func (s *Store) Create(ctx context.Context, q storage.Querier, emit Emitter, n N
 		return Result{}, err
 	}
 	res := Result{Belief: b, Breakdown: bd, Events: []events.Event{created}}
-	for _, e := range n.Evidence {
+	for i, e := range ev {
-		if err := s.insertEvidence(ctx, q, b, normalise(e), e.Note, created.ID); err != nil {
+		if err := s.insertEvidence(ctx, q, b, e, notes[i], created.ID); err != nil {
 			return Result{}, err
 		}
 	}
@@ -425,7 +429,8 @@ func (s *Store) AddEvidence(ctx context.Context, q storage.Querier, emit Emitter
 		return Result{}, fmt.Errorf("%w: belief is superseded", ErrInvalid)
 	}
 	e := normalise(in)
-	if err := s.insertEvidence(ctx, q, b, e, in.Note, ids.Nil); err != nil {
+	e, note := s.applyReliability(ctx, q, mind, b.Subject, e, in.Note)
+	if err := s.insertEvidence(ctx, q, b, e, note, ids.Nil); err != nil {
 		return Result{}, err
 	}
 	return s.reevaluate(ctx, q, emit, b, reason, causation, correlation, &e, in)
@@ -617,6 +622,11 @@ func (s *Store) Retract(ctx context.Context, emit Emitter, mind, id ids.ID, reas
 		if err != nil {
 			return err
 		}
+		// An operator saying "wrong" is an outcome for the sources that backed
+		// it, in this belief's domain only: nobody's standing elsewhere moves.
+		if err := s.RecordBeliefOutcome(ctx, tx, mind, id, false); err != nil {
+			return err
+		}
 		ev, err := emit(ctx, tx, events.New{MindID: mind, Type: events.BeliefRetracted, Source: "operator", Summary: fmt.Sprintf("operator %s rejected belief: %s (%s)", by, b.Proposition, reason),
 			Salience: 0.7, Confidence: 1, Evidence: []events.EvidenceRef{{Kind: "belief", ID: string(b.ID)}},
 			Metadata: map[string]any{"belief_id": string(b.ID), "actor": by, "do_not_infer": doNotInfer}, Payload: map[string]any{"belief_id": b.ID, "proposition": b.Proposition, "reason": reason, "previous": map[string]any{"confidence": b.Confidence, "status": b.Status}}})

internal/beliefs/store_test.go

@@ -3,6 +3,7 @@ package beliefs
 import (
 	"context"
 	"errors"
+	"strings"
 	"testing"
 	"time"
 
@@ -229,3 +230,67 @@ func TestTransientStatesSupersedeInsteadOfContesting(t *testing.T) {
 		t.Fatalf("persistent fact wrongly treated as a state change: %+v", p1b)
 	}
 }
+
+// TestReliabilityIsPerDomain: being wrong about databases three times lowers a
+// source's credibility for database claims and nothing else; the trust-level
+// credibility itself never changes.
+func TestReliabilityIsPerDomain(t *testing.T) {
+	pool := storage.TestPool(t, migrations.FS)
+	ctx := context.Background()
+	m, _ := minds.NewStore(pool).Create(ctx, minds.CreateParams{Slug: "rel"})
+	es := evstore.NewStore(pool, evstore.NewBus(8), nil)
+	emit := func(ctx context.Context, q storage.Querier, n events.New) (events.Event, error) {
+		out, err := es.AppendIn(ctx, q, n)
+		if err != nil {
+			return events.Event{}, err
+		}
+		return out[0], nil
+	}
+	s := NewStore(pool)
+	andy := func(subject, prop string) Result {
+		r, err := s.Create(ctx, pool, emit, New{MindID: m.ID, Proposition: prop, Subject: subject, Epistemic: concepts.Claim, Source: "conversation",
+			Evidence: []EvidenceInput{{Direction: "for", Kind: "assertion", Source: "asserted-by:andy", Family: "actor:andy", Derivation: DerivationAssertion, Weight: 0.5, Credibility: 0.85}}})
+		if err != nil {
+			t.Fatal(err)
+		}
+		return r
+	}
+	baseline := andy("postgres", "postgres corrupts data under load")
+	// Three operator corrections in the database domain.
+	for i, prop := range []string{"postgres loses commits", "postgres ignores fsync", "postgres drops indexes"} {
+		r := andy("postgres", prop)
+		if _, err := s.Retract(ctx, emit, m.ID, r.Belief.ID, "wrong", "operator", false); err != nil {
+			t.Fatal(err)
+		}
+		rec, _ := s.Reliability(ctx, nil, m.ID, "actor:andy", "postgres")
+		if rec.Wrong != i+1 {
+			t.Fatalf("wrong count after %d corrections: %+v", i+1, rec)
+		}
+	}
+	rec, _ := s.Reliability(ctx, nil, m.ID, "actor:andy", "postgres")
+	if !rec.Applied || rec.Score >= 0.5 {
+		t.Fatalf("reliability should now apply and be low: %+v", rec)
+	}
+	// New database claims from Andy are weighed down, with the reason on the row.
+	after := andy("postgres", "postgres cannot handle json")
+	if after.Belief.Confidence >= baseline.Belief.Confidence {
+		t.Fatalf("credibility not lowered in domain: %.3f vs %.3f", after.Belief.Confidence, baseline.Belief.Confidence)
+	}
+	rows, _ := s.EvidenceFor(ctx, pool, m.ID, after.Belief.ID)
+	if len(rows) != 1 || rows[0].Credibility >= 0.85 || !strings.Contains(rows[0].Note, "credibility 0.85→") {
+		t.Fatalf("adjustment not recorded: %+v", rows)
+	}
+	// A different domain is untouched: same credibility, same score as the baseline.
+	diary := andy("andy", "andy was born in march")
+	if diary.Belief.Confidence != baseline.Belief.Confidence {
+		t.Fatalf("unrelated domain affected: %.3f vs %.3f", diary.Belief.Confidence, baseline.Belief.Confidence)
+	}
+	other, _ := s.Reliability(ctx, nil, m.ID, "actor:andy", "andy")
+	if other.Applied || other.Wrong != 0 {
+		t.Fatalf("global standing leaked: %+v", other)
+	}
+	// Trust-level credibility is a constant; it was never rewritten.
+	if events.TrustUser.Credibility() != 0.85 {
+		t.Fatal("trust credibility changed")
+	}
+}

internal/predictions/resolver.go

@@ -128,6 +128,9 @@ func (r *Resolver) ResolveIn(ctx context.Context, q storage.Querier, p Predictio
 			evs = append(evs, out[0])
 			return out[0], nil
 		}
+		if err := r.Beliefs.RecordBeliefOutcome(ctx, q, p.MindID, p.BeliefID, o.Status == StatusConfirmed); err != nil {
+			r.Log.Warn("reliability update failed", "belief", p.BeliefID, "err", err)
+		}
 		if _, err := r.Beliefs.AddEvidence(ctx, q, emit, p.MindID, p.BeliefID, beliefs.EvidenceInput{Direction: dir, Kind: "prediction", RefID: string(p.ID), Source: "prediction:" + string(p.ID), Weight: weight, Credibility: 0.9, Note: "prediction " + string(o.Status) + ": " + p.Statement,
 			Derivation: beliefs.DerivationPrediction, Family: "prediction:" + string(p.ID), RootIDs: refIDs(o.Evidence)}, "prediction "+string(o.Status), evs[0].ID, ids.Nil); err != nil {
 			r.Log.Warn("belief feedback failed", "belief", p.BeliefID, "err", err)

migrations/0021_source_reliability.down.sql

@@ -0,0 +1 @@
+DROP TABLE IF EXISTS source_reliability;

migrations/0021_source_reliability.up.sql

@@ -0,0 +1,13 @@
+-- Contextual source reliability (review pass §7.2 item 14). Reliability is
+-- recorded per (source family, domain) from prediction outcomes and operator
+-- corrections; a source's standing in one domain never moves its standing
+-- in another, and the trust-level credibility itself is never rewritten.
+CREATE TABLE source_reliability (
+  mind_id    uuid NOT NULL REFERENCES minds(id),
+  source     text NOT NULL,             -- evidence family, e.g. actor:andy, source:prometheus:prod
+  domain     text NOT NULL,             -- belief subject the outcome concerned
+  correct    integer NOT NULL DEFAULT 0,
+  wrong      integer NOT NULL DEFAULT 0,
+  updated_at timestamptz NOT NULL DEFAULT now(),
+  PRIMARY KEY (mind_id, source, domain)
+);

web/templates/pages/belief_detail.html

@@ -27,7 +27,8 @@
       <div class="card">
         <h2>Evidence for</h2>
         <ul class="plain ev">{{range $d.Evidence}}{{if eq .Direction "for"}}<li class="{{if .Retracted}}retracted{{end}}"><div><strong>{{conf .Weight}}</strong> × cred {{conf .Credibility}} <span class="badge">{{.Kind}}</span> {{if .Retracted}}<span class="badge danger">retracted</span>{{end}}</div><div>{{.Note}}</div><div class="muted small">{{.Source}} · {{ago .CreatedAt}} {{if and (eq .Kind "event") .RefID}}· <a href="/events/{{.RefID}}">event</a>{{else if and (eq .Kind "memory") .RefID}}· <a href="/minds/{{$m.Slug}}/memory/{{.RefID}}">memory</a>{{else if and (eq .Kind "belief") .RefID}}· <a href="/minds/{{$m.Slug}}/beliefs/{{.RefID}}">belief</a>{{end}}</div><div class="muted small">provenance: <span class="badge small">{{.Derivation}}</span> family <code>{{.Family}}</code>{{if eq .ProvenanceStatus "legacy_unknown"}} <span class="badge warn small" title="written before ancestry was recorded; family is the source string">legacy</span>{{end}}{{range .RootIDs}} · <a href="/events/{{.}}">root</a>{{end}}</div></li>{{end}}{{else}}<li class="muted">None.</li>{{end}}</ul>
-        {{if $bd.Families}}<h3 class="small muted">Independent sources</h3><table class="table compact"><thead><tr><th>Family</th><th></th><th class="num">Items</th><th class="num">Primary</th><th class="num">Counts as</th></tr></thead><tbody>{{range $bd.Families}}<tr><td><code>{{.Family}}</code></td><td class="small">{{.Direction}}</td><td class="num">{{.Items}}</td><td class="num">{{.Primary}}</td><td class="num">{{printf "%.2f" .Contribution}}</td></tr>{{end}}</tbody></table><p class="small muted">Within a family the strongest primary item counts in full and further items add a quarter each (capped at 1.5×); derived items count only when the family has no primary item. Diversity counts families: {{$bd.SourcesFor}}.</p>{{end}}
+        {{if $bd.Families}}<h3 class="small muted">Independent sources</h3><table class="table compact"><thead><tr><th>Family</th><th></th><th class="num">Items</th><th class="num">Primary</th><th class="num">Counts as</th></tr></thead><tbody>{{range $bd.Families}}<tr><td><code>{{.Family}}</code></td><td class="small">{{.Direction}}</td><td class="num">{{.Items}}</td><td class="num">{{.Primary}}</td><td class="num">{{printf "%.2f" .Contribution}}</td></tr>{{end}}</tbody></table>{{if $d.Reliability}}<h3 class="small muted">Source reliability in this domain{{with $b.Subject}} ({{.}}){{end}}</h3><table class="table compact"><thead><tr><th>Family</th><th class="num">Right</th><th class="num">Wrong</th><th class="num">Score</th><th></th></tr></thead><tbody>{{range $d.Reliability}}<tr><td><code>{{.Source}}</code></td><td class="num">{{.Correct}}</td><td class="num">{{.Wrong}}</td><td class="num">{{printf "%.2f" .Score}}</td><td class="small muted">{{if .Applied}}applied to credibility{{else}}too little history to apply{{end}}</td></tr>{{end}}</tbody></table>{{end}}
+        <p class="small muted">Within a family the strongest primary item counts in full and further items add a quarter each (capped at 1.5×); derived items count only when the family has no primary item. Diversity counts families: {{$bd.SourcesFor}}.</p>{{end}}
       </div>
       <div class="card">
         {{if $d.Assertions}}<h2>Who said so</h2>