Docs: encryption/TLS guide, development guide, troubleshooting, CHANGELOG; API reference for Phase 3/5 endpoints

CHANGELOG.md

@@ -0,0 +1,47 @@
+# Changelog
+
+All notable changes. Dates are when the work was done on the reference Pi 5.
+
+## 0.3.1 — 2026-10-09
+- Benchmark harness (`scripts/benchmark.sh`, python3-only) and measured figures
+  in `docs/benchmarks.md`.
+- `MemoryAccounting=yes` on `pilens.service`.
+
+## 0.3.0 — 2026-10-09 (Phase 5: hardening)
+- **Findings** diagnostics page: observable TCP/DNS/TLS/ICMP events and capture
+  drops, each with evidence packets, a reproducing filter and benign-cause notes.
+  Explicitly a forensic aid, not an IDS.
+- Pinned Go toolchain 1.27.2 — 0 called standard-library vulnerabilities
+  (govulncheck); `npm audit` clean.
+
+## 0.2.1 — 2026-10-09
+- TLS Explorer reports the true negotiated version from `supported_versions`
+  (and labels QUIC as TLS 1.3), instead of the compatibility record version.
+  Captures the QUIC server port.
+
+## 0.2.0 — 2026-10-09 (Phase 3: analysis)
+- Conversations (TCP/UDP, cross-segment, per-direction bytes) from the index.
+- Follow TCP Stream via tshark reassembly, labelled as transport bytes.
+- DNS Explorer (cleartext), TLS Explorer (metadata only), Statistics (SVG charts).
+
+## 0.1.3 — 2026-10-09
+- Reindex interrupted sessions on restart and correct packet counts.
+- Actionable access-point activation errors.
+
+## 0.1.2 — 2026-10-09
+- `KillMode=mixed` so shutdown/upgrade finalises a running capture as *stopped*.
+- `dumpcap -F pcapng`; installer AP-mode check runs after `iw` is installed.
+
+## 0.1.1 — 2026-10-09
+- `rpi_volt` under-voltage fallback where `vcgencmd` is root-only.
+- dnsmasq bound to the AP interface only (`except-interface=lo`).
+
+## 0.1.0 — 2026-10-09 (Phases 1–2)
+- Installer/uninstaller/repair/diagnose, systemd units, transactional network
+  helper with rollback, Argon2id auth + TOTP + CSRF + source allow-list.
+- Access point, DHCP/DNS, nftables firewall (default-deny, IPv6 blocked for
+  clients), management isolation.
+- dumpcap capture to hashed pcapng segments, tshark dissection with verified
+  byte ranges, Wireshark display filters, export/import, crash recovery.
+- React analyser (packet list / protocol tree / hex), setup wizard, device,
+  health, library and settings pages.

README.md

@@ -61,9 +61,12 @@ make test-docker        # tshark/dumpcap tests + network-namespace integration s
 - [Architecture and design decisions](docs/architecture.md)
 - [Installation, first run, upgrade and recovery](docs/operations.md)
 - [Network topology and firewall policy](docs/network.md)
+- [Encryption: what PiLens can and cannot see](docs/tls-and-encryption.md)
 - [Security and privacy model](docs/security.md)
 - [Forensic evidence handling](docs/forensics.md)
 - [API](docs/api.md)
+- [Development guide](docs/development.md)
+- [Troubleshooting](docs/troubleshooting.md)
 - [Benchmarks](docs/benchmarks.md)
 - [Testing and hardware acceptance checklist](docs/testing.md)
 - [Known limitations](docs/limitations.md)

docs/api.md

@@ -39,3 +39,20 @@ to 1 MiB, and unknown fields are rejected.
 Tree nodes: `{label, field, show, pos, size, verified, children}`. `verified`
 means the frame bytes at `pos..pos+size` were checked against tshark's raw
 value, so highlighting them is accurate.
+
+## Analysis (Phase 3) and findings (Phase 5)
+
+| Method & path | Purpose |
+|---|---|
+| `GET /captures/{id}/conversations?transport=tcp\|udp` | Conversations from the index: canonical endpoint pair, per-direction bytes, protocols, seg+stream for Follow |
+| `GET /captures/{id}/endpoints?limit=` | Top talkers (per-address tx/rx bytes, packets) |
+| `GET /captures/{id}/statistics?buckets=` | Totals, protocol hierarchy, size distribution, timeline, TCP stream count |
+| `GET /captures/{id}/follow?seg=&stream=` | Follow TCP stream: reassembled payload chunks per direction (hex), via tshark. Transport bytes, not decrypted content. |
+| `GET /captures/{id}/dns?limit=` | Cleartext DNS messages: client, resolver, name, type, rcode, answers |
+| `GET /captures/{id}/tls?limit=` | `{handshakes[], certificates[], note}`. Handshakes carry SNI, ALPN, negotiated version, cipher. Metadata only. |
+| `GET /captures/{id}/findings?min=` | Observed diagnostics: rule, title, severity, count, flows, example packets, reproducing filter, benign-explanation note |
+
+All are GET, authenticated, and derived from already-captured data; none change
+state or touch the network. Conversations and statistics are computed from the
+SQLite index (fast, cross-segment); follow/dns/tls/findings call tshark over the
+segment files on demand.

docs/development.md

@@ -0,0 +1,67 @@
+# Development guide
+
+## Layout
+
+```
+cmd/pilens         service + CLI (serve, init, setup, reset-password, rlimit-exec)
+cmd/pilens-helper  the only root component: typed network operations over a socket
+internal/config    configuration + validation (shared by core and helper)
+internal/netgen    renders NetworkManager / dnsmasq / nftables (pure, unit-tested)
+internal/helper    core↔helper protocol + parsers
+internal/store     SQLite open + migrations
+internal/capture   capture engine, pcap/pcapng I/O, tshark, analysis, findings
+internal/pcapfile  bounds-checked pcap/pcapng reader/writer
+internal/sysinfo   unprivileged host/interface/health probes
+internal/auth      Argon2id, TOTP, session tokens, login throttling
+internal/api       HTTP API + static UI serving + SSE
+web/               React + TypeScript + Vite UI (embedded via go:embed)
+migrations/        ordered SQL, embedded
+systemd/ scripts/ packaging/ docs/ tests/
+```
+
+## Prerequisites
+
+Go ≥ 1.27 (the module pins toolchain 1.27.2), Node ≥ 20, Docker for the
+integration suite. On a dev box you do not need tshark/dnsmasq/nftables unless
+you run those tests directly.
+
+## Common tasks
+
+```sh
+make test          # go vet + go test + tsc --noEmit (no external tools needed)
+make test-docker   # tshark/dumpcap tests + netns integration suite (needs Docker)
+make package       # cross-compile arm64, bundle UI + docs + scripts -> dist/*.tar.gz
+cd web && npm run dev   # UI against a running appliance: PILENS_API=https://<pi>:8443 npm run dev
+```
+
+The dev loop used to build this on real hardware: `make package VERSION=x.y.z`,
+`scp` the tarball to the Pi, run `install.sh --yes` (it upgrades in place,
+keeping data), then drive the API or UI. `install.sh` backs up and rolls back.
+
+## Design rules (that reviewers should enforce)
+
+- **The core never runs as root.** Privileged actions go through the helper,
+  which has a fixed set of typed operations and re-validates everything. No
+  generic command execution, ever.
+- **Never interpolate user input into a shell or SQL.** All execs use argv
+  lists; all queries use bound parameters. tshark children run via
+  `rlimit-exec` with resource limits and `no_new_privs`.
+- **Original captures are immutable.** Packet blocks are copied byte-for-byte;
+  derived data (summaries, dissection) is regenerable and clearly separated.
+- **Be honest in the UI.** No fabricated data, no controls that do nothing, no
+  claims the evidence doesn't support (e.g. TLS version, interception status,
+  drop vs. UI-drop). A feature that isn't built is absent, not faked.
+- **Validation lives in `internal/config` and runs twice** — in the core and
+  again in the root helper, which does not trust the core.
+
+## Adding a migration
+
+Drop `migrations/NNN_name.sql`; it is applied in order on next start and
+recorded in `schema_migrations`. Migrations run inside a transaction.
+
+## Releasing
+
+1. `make test && make test-docker` green.
+2. `go run golang.org/x/vuln/cmd/govulncheck ./...` and `cd web && npm audit`.
+3. `make package VERSION=x.y.z`; the tarball carries its own SHA-256.
+4. Record hardware results in `docs/testing.md` only after actually running them.

docs/tls-and-encryption.md

@@ -0,0 +1,71 @@
+# Encryption: what PiLens can and cannot see
+
+Short version: in capture-only mode PiLens shows you **packets and metadata**,
+never decrypted application content. This page explains why, because it is the
+single most common misunderstanding about a tool like this.
+
+## Two completely separate kinds of TLS
+
+1. **Management TLS** — the HTTPS you use to reach the PiLens web UI on :8443.
+   PiLens generates its own self-signed certificate for this on first start
+   (ECDSA P-256, 5-year). It protects *your* connection to the appliance and is
+   unrelated to anything captured. The installer prints its SHA-256 fingerprint;
+   verify it the first time your browser warns about it.
+
+2. **Interception CA** — a separate authority that *would* let PiLens decrypt a
+   test device's HTTPS, by having that device explicitly trust it. **This is not
+   implemented in this version** (Phase 4). When it is, it will be opt-in,
+   per-device, and require you to install and trust the CA on the client
+   yourself. There is no way to do it silently.
+
+The two are kept independent on purpose: compromising or distributing an
+interception CA must never affect the management channel.
+
+## Why HTTPS content is unavailable without client trust
+
+TLS exists precisely to stop a device in the middle from reading traffic. To
+decrypt it, the middle device must present a certificate the client trusts —
+which means the client must have been configured, by its owner, to trust
+PiLens's CA. Without that explicit step, PiLens sees only:
+
+- The **TLS handshake** metadata sent in clear: SNI (the server name, unless
+  Encrypted ClientHello is used), offered/negotiated version, cipher, ALPN.
+- The **certificate** the server sends — but only for TLS 1.2 and earlier. TLS
+  1.3 encrypts the certificate, so the TLS Explorer's certificate tab is usually
+  empty for modern servers. That is expected.
+- Everything after the handshake as **ciphertext**. "Follow TCP stream" will
+  show it, clearly labelled as encrypted transport bytes, not content.
+
+## Cannot be sent via DHCP
+
+A trusted root CA **cannot** be delivered or installed through DHCP, a captive
+portal, or any network mechanism. No mainstream OS will trust a CA handed to it
+by the network — doing so would defeat TLS entirely. Trust is always a manual,
+explicit action by the device owner. This is a feature, not a limitation.
+
+## QUIC and HTTP/3
+
+QUIC carries HTTP/3 inside TLS 1.3 over UDP. PiLens captures it and reads the
+clear handshake fields (SNI, ALPN), and labels it `TLS 1.3 (QUIC)`. The payload
+is encrypted. Many apps prefer QUIC, so a lot of "web" traffic appears as QUIC
+rather than TCP/TLS.
+
+## Encrypted DNS (DoH / DoT)
+
+If a client uses DNS-over-HTTPS or DNS-over-TLS, its lookups are inside TLS to a
+resolver and do **not** appear in the DNS Explorer — they look like HTTPS/QUIC to
+that resolver's address. Only cleartext DNS through the appliance's resolver is
+observable. PiLens never presents encrypted DNS as if it were cleartext.
+
+## Certificate pinning
+
+Some apps pin to a specific certificate or key and will refuse *any* substitute,
+including a trusted interception CA. When Phase 4 lands, such connections will be
+reported honestly as failed/pinned rather than silently bypassed — PiLens will
+never weaken a client's TLS to force a connection through.
+
+## IPv6
+
+IPv6 is not inspected, so it is disabled on the AP and dropped in the firewall
+(see [network.md](network.md)). Clients fall back to IPv4, which is fully
+captured. This avoids an uninspected escape path.

docs/troubleshooting.md

@@ -0,0 +1,22 @@
+# Troubleshooting
+
+Run `sudo /opt/pilens/diagnose.sh` first — it bundles logs and state with
+secrets redacted. Then:
+
+| Symptom | Check | Fix |
+|---|---|---|
+| Web UI won't load | `systemctl status pilens`; is your source IP in the management subnet? | `sudo /opt/pilens/repair.sh`; widen the subnet from the console if locked out (see operations.md) |
+| Browser certificate warning | Expected (self-signed). Compare the fingerprint the installer printed. | Trust it for this host, or import `/var/lib/pilens/tls/mgmt.crt` |
+| Can't log in | — | `sudo -u pilens /opt/pilens/bin/pilens reset-password -user admin` |
+| Locked out by `429` | Login throttling after failures | Wait the stated time, or restart `pilens` to clear the limiter |
+| No clients get an address | `systemctl status pilens-dnsmasq`; another DHCP server on the AP link? | `repair.sh`; ensure no stray `dnsmasq`/`hostapd` is running |
+| AP won't start | `journalctl -u NetworkManager`; radio AP-mode and band/country support | Try WPA2 / 2.4 GHz / a permitted channel; failed applies roll back automatically |
+| Clients associate but no internet | Dashboard "forwarding" card; Ethernet default route up? | Check the uplink; `forwarding` must read "forwarding", not "blocked" |
+| Capture shows drops | System Health / Findings; CPU or SD-card bound | Use a capture filter; capture to an SSD (operations.md) |
+| "interrupted" sessions after power loss | Expected: segments are hashed and re-indexed on restart | Verify hashes in Capture Library → Integrity |
+| High CPU / slow filters | Large capture; tshark re-reads segments | Filter narrows after the first run (cached); smaller segment size helps |
+| Under-voltage warning on dashboard | `rpi_volt` alarm | Use the official PSU; under-voltage corrupts storage and drops packets |
+
+If a network change ever leaves the appliance unreachable, every apply keeps a
+backup and rolls back on failure; the last-known-good firewall is also restored
+at boot before the AP starts.