Docs: encryption/TLS guide, development guide, troubleshooting, CHANGELOG; API reference for Phase 3/5 endpoints
This commit is contained in:
Andy Dixon
@@ -0,0 +1,47 @@
|
||||
# Changelog
|
||||
|
||||
All notable changes. Dates are when the work was done on the reference Pi 5.
|
||||
|
||||
## 0.3.1 — 2026-10-09
|
||||
- Benchmark harness (`scripts/benchmark.sh`, python3-only) and measured figures
|
||||
in `docs/benchmarks.md`.
|
||||
- `MemoryAccounting=yes` on `pilens.service`.
|
||||
|
||||
## 0.3.0 — 2026-10-09 (Phase 5: hardening)
|
||||
- **Findings** diagnostics page: observable TCP/DNS/TLS/ICMP events and capture
|
||||
drops, each with evidence packets, a reproducing filter and benign-cause notes.
|
||||
Explicitly a forensic aid, not an IDS.
|
||||
- Pinned Go toolchain 1.27.2 — 0 called standard-library vulnerabilities
|
||||
(govulncheck); `npm audit` clean.
|
||||
|
||||
## 0.2.1 — 2026-10-09
|
||||
- TLS Explorer reports the true negotiated version from `supported_versions`
|
||||
(and labels QUIC as TLS 1.3), instead of the compatibility record version.
|
||||
Captures the QUIC server port.
|
||||
|
||||
## 0.2.0 — 2026-10-09 (Phase 3: analysis)
|
||||
- Conversations (TCP/UDP, cross-segment, per-direction bytes) from the index.
|
||||
- Follow TCP Stream via tshark reassembly, labelled as transport bytes.
|
||||
- DNS Explorer (cleartext), TLS Explorer (metadata only), Statistics (SVG charts).
|
||||
|
||||
## 0.1.3 — 2026-10-09
|
||||
- Reindex interrupted sessions on restart and correct packet counts.
|
||||
- Actionable access-point activation errors.
|
||||
|
||||
## 0.1.2 — 2026-10-09
|
||||
- `KillMode=mixed` so shutdown/upgrade finalises a running capture as *stopped*.
|
||||
- `dumpcap -F pcapng`; installer AP-mode check runs after `iw` is installed.
|
||||
|
||||
## 0.1.1 — 2026-10-09
|
||||
- `rpi_volt` under-voltage fallback where `vcgencmd` is root-only.
|
||||
- dnsmasq bound to the AP interface only (`except-interface=lo`).
|
||||
|
||||
## 0.1.0 — 2026-10-09 (Phases 1–2)
|
||||
- Installer/uninstaller/repair/diagnose, systemd units, transactional network
|
||||
helper with rollback, Argon2id auth + TOTP + CSRF + source allow-list.
|
||||
- Access point, DHCP/DNS, nftables firewall (default-deny, IPv6 blocked for
|
||||
clients), management isolation.
|
||||
- dumpcap capture to hashed pcapng segments, tshark dissection with verified
|
||||
byte ranges, Wireshark display filters, export/import, crash recovery.
|
||||
- React analyser (packet list / protocol tree / hex), setup wizard, device,
|
||||
health, library and settings pages.
|
||||
@@ -61,9 +61,12 @@ make test-docker # tshark/dumpcap tests + network-namespace integration s
|
||||
- [Architecture and design decisions](docs/architecture.md)
|
||||
- [Installation, first run, upgrade and recovery](docs/operations.md)
|
||||
- [Network topology and firewall policy](docs/network.md)
|
||||
- [Encryption: what PiLens can and cannot see](docs/tls-and-encryption.md)
|
||||
- [Security and privacy model](docs/security.md)
|
||||
- [Forensic evidence handling](docs/forensics.md)
|
||||
- [API](docs/api.md)
|
||||
- [Development guide](docs/development.md)
|
||||
- [Troubleshooting](docs/troubleshooting.md)
|
||||
- [Benchmarks](docs/benchmarks.md)
|
||||
- [Testing and hardware acceptance checklist](docs/testing.md)
|
||||
- [Known limitations](docs/limitations.md)
|
||||
|
||||
+17
@@ -39,3 +39,20 @@ to 1 MiB, and unknown fields are rejected.
|
||||
Tree nodes: `{label, field, show, pos, size, verified, children}`. `verified`
|
||||
means the frame bytes at `pos..pos+size` were checked against tshark's raw
|
||||
value, so highlighting them is accurate.
|
||||
|
||||
## Analysis (Phase 3) and findings (Phase 5)
|
||||
|
||||
| Method & path | Purpose |
|
||||
|---|---|
|
||||
| `GET /captures/{id}/conversations?transport=tcp\|udp` | Conversations from the index: canonical endpoint pair, per-direction bytes, protocols, seg+stream for Follow |
|
||||
| `GET /captures/{id}/endpoints?limit=` | Top talkers (per-address tx/rx bytes, packets) |
|
||||
| `GET /captures/{id}/statistics?buckets=` | Totals, protocol hierarchy, size distribution, timeline, TCP stream count |
|
||||
| `GET /captures/{id}/follow?seg=&stream=` | Follow TCP stream: reassembled payload chunks per direction (hex), via tshark. Transport bytes, not decrypted content. |
|
||||
| `GET /captures/{id}/dns?limit=` | Cleartext DNS messages: client, resolver, name, type, rcode, answers |
|
||||
| `GET /captures/{id}/tls?limit=` | `{handshakes[], certificates[], note}`. Handshakes carry SNI, ALPN, negotiated version, cipher. Metadata only. |
|
||||
| `GET /captures/{id}/findings?min=` | Observed diagnostics: rule, title, severity, count, flows, example packets, reproducing filter, benign-explanation note |
|
||||
|
||||
All are GET, authenticated, and derived from already-captured data; none change
|
||||
state or touch the network. Conversations and statistics are computed from the
|
||||
SQLite index (fast, cross-segment); follow/dns/tls/findings call tshark over the
|
||||
segment files on demand.
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# Development guide
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
cmd/pilens service + CLI (serve, init, setup, reset-password, rlimit-exec)
|
||||
cmd/pilens-helper the only root component: typed network operations over a socket
|
||||
internal/config configuration + validation (shared by core and helper)
|
||||
internal/netgen renders NetworkManager / dnsmasq / nftables (pure, unit-tested)
|
||||
internal/helper core↔helper protocol + parsers
|
||||
internal/store SQLite open + migrations
|
||||
internal/capture capture engine, pcap/pcapng I/O, tshark, analysis, findings
|
||||
internal/pcapfile bounds-checked pcap/pcapng reader/writer
|
||||
internal/sysinfo unprivileged host/interface/health probes
|
||||
internal/auth Argon2id, TOTP, session tokens, login throttling
|
||||
internal/api HTTP API + static UI serving + SSE
|
||||
web/ React + TypeScript + Vite UI (embedded via go:embed)
|
||||
migrations/ ordered SQL, embedded
|
||||
systemd/ scripts/ packaging/ docs/ tests/
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Go ≥ 1.27 (the module pins toolchain 1.27.2), Node ≥ 20, Docker for the
|
||||
integration suite. On a dev box you do not need tshark/dnsmasq/nftables unless
|
||||
you run those tests directly.
|
||||
|
||||
## Common tasks
|
||||
|
||||
```sh
|
||||
make test # go vet + go test + tsc --noEmit (no external tools needed)
|
||||
make test-docker # tshark/dumpcap tests + netns integration suite (needs Docker)
|
||||
make package # cross-compile arm64, bundle UI + docs + scripts -> dist/*.tar.gz
|
||||
cd web && npm run dev # UI against a running appliance: PILENS_API=https://<pi>:8443 npm run dev
|
||||
```
|
||||
|
||||
The dev loop used to build this on real hardware: `make package VERSION=x.y.z`,
|
||||
`scp` the tarball to the Pi, run `install.sh --yes` (it upgrades in place,
|
||||
keeping data), then drive the API or UI. `install.sh` backs up and rolls back.
|
||||
|
||||
## Design rules (that reviewers should enforce)
|
||||
|
||||
- **The core never runs as root.** Privileged actions go through the helper,
|
||||
which has a fixed set of typed operations and re-validates everything. No
|
||||
generic command execution, ever.
|
||||
- **Never interpolate user input into a shell or SQL.** All execs use argv
|
||||
lists; all queries use bound parameters. tshark children run via
|
||||
`rlimit-exec` with resource limits and `no_new_privs`.
|
||||
- **Original captures are immutable.** Packet blocks are copied byte-for-byte;
|
||||
derived data (summaries, dissection) is regenerable and clearly separated.
|
||||
- **Be honest in the UI.** No fabricated data, no controls that do nothing, no
|
||||
claims the evidence doesn't support (e.g. TLS version, interception status,
|
||||
drop vs. UI-drop). A feature that isn't built is absent, not faked.
|
||||
- **Validation lives in `internal/config` and runs twice** — in the core and
|
||||
again in the root helper, which does not trust the core.
|
||||
|
||||
## Adding a migration
|
||||
|
||||
Drop `migrations/NNN_name.sql`; it is applied in order on next start and
|
||||
recorded in `schema_migrations`. Migrations run inside a transaction.
|
||||
|
||||
## Releasing
|
||||
|
||||
1. `make test && make test-docker` green.
|
||||
2. `go run golang.org/x/vuln/cmd/govulncheck ./...` and `cd web && npm audit`.
|
||||
3. `make package VERSION=x.y.z`; the tarball carries its own SHA-256.
|
||||
4. Record hardware results in `docs/testing.md` only after actually running them.
|
||||
@@ -0,0 +1,71 @@
|
||||
# Encryption: what PiLens can and cannot see
|
||||
|
||||
Short version: in capture-only mode PiLens shows you **packets and metadata**,
|
||||
never decrypted application content. This page explains why, because it is the
|
||||
single most common misunderstanding about a tool like this.
|
||||
|
||||
## Two completely separate kinds of TLS
|
||||
|
||||
1. **Management TLS** — the HTTPS you use to reach the PiLens web UI on :8443.
|
||||
PiLens generates its own self-signed certificate for this on first start
|
||||
(ECDSA P-256, 5-year). It protects *your* connection to the appliance and is
|
||||
unrelated to anything captured. The installer prints its SHA-256 fingerprint;
|
||||
verify it the first time your browser warns about it.
|
||||
|
||||
2. **Interception CA** — a separate authority that *would* let PiLens decrypt a
|
||||
test device's HTTPS, by having that device explicitly trust it. **This is not
|
||||
implemented in this version** (Phase 4). When it is, it will be opt-in,
|
||||
per-device, and require you to install and trust the CA on the client
|
||||
yourself. There is no way to do it silently.
|
||||
|
||||
The two are kept independent on purpose: compromising or distributing an
|
||||
interception CA must never affect the management channel.
|
||||
|
||||
## Why HTTPS content is unavailable without client trust
|
||||
|
||||
TLS exists precisely to stop a device in the middle from reading traffic. To
|
||||
decrypt it, the middle device must present a certificate the client trusts —
|
||||
which means the client must have been configured, by its owner, to trust
|
||||
PiLens's CA. Without that explicit step, PiLens sees only:
|
||||
|
||||
- The **TLS handshake** metadata sent in clear: SNI (the server name, unless
|
||||
Encrypted ClientHello is used), offered/negotiated version, cipher, ALPN.
|
||||
- The **certificate** the server sends — but only for TLS 1.2 and earlier. TLS
|
||||
1.3 encrypts the certificate, so the TLS Explorer's certificate tab is usually
|
||||
empty for modern servers. That is expected.
|
||||
- Everything after the handshake as **ciphertext**. "Follow TCP stream" will
|
||||
show it, clearly labelled as encrypted transport bytes, not content.
|
||||
|
||||
## Cannot be sent via DHCP
|
||||
|
||||
A trusted root CA **cannot** be delivered or installed through DHCP, a captive
|
||||
portal, or any network mechanism. No mainstream OS will trust a CA handed to it
|
||||
by the network — doing so would defeat TLS entirely. Trust is always a manual,
|
||||
explicit action by the device owner. This is a feature, not a limitation.
|
||||
|
||||
## QUIC and HTTP/3
|
||||
|
||||
QUIC carries HTTP/3 inside TLS 1.3 over UDP. PiLens captures it and reads the
|
||||
clear handshake fields (SNI, ALPN), and labels it `TLS 1.3 (QUIC)`. The payload
|
||||
is encrypted. Many apps prefer QUIC, so a lot of "web" traffic appears as QUIC
|
||||
rather than TCP/TLS.
|
||||
|
||||
## Encrypted DNS (DoH / DoT)
|
||||
|
||||
If a client uses DNS-over-HTTPS or DNS-over-TLS, its lookups are inside TLS to a
|
||||
resolver and do **not** appear in the DNS Explorer — they look like HTTPS/QUIC to
|
||||
that resolver's address. Only cleartext DNS through the appliance's resolver is
|
||||
observable. PiLens never presents encrypted DNS as if it were cleartext.
|
||||
|
||||
## Certificate pinning
|
||||
|
||||
Some apps pin to a specific certificate or key and will refuse *any* substitute,
|
||||
including a trusted interception CA. When Phase 4 lands, such connections will be
|
||||
reported honestly as failed/pinned rather than silently bypassed — PiLens will
|
||||
never weaken a client's TLS to force a connection through.
|
||||
|
||||
## IPv6
|
||||
|
||||
IPv6 is not inspected, so it is disabled on the AP and dropped in the firewall
|
||||
(see [network.md](network.md)). Clients fall back to IPv4, which is fully
|
||||
captured. This avoids an uninspected escape path.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Troubleshooting
|
||||
|
||||
Run `sudo /opt/pilens/diagnose.sh` first — it bundles logs and state with
|
||||
secrets redacted. Then:
|
||||
|
||||
| Symptom | Check | Fix |
|
||||
|---|---|---|
|
||||
| Web UI won't load | `systemctl status pilens`; is your source IP in the management subnet? | `sudo /opt/pilens/repair.sh`; widen the subnet from the console if locked out (see operations.md) |
|
||||
| Browser certificate warning | Expected (self-signed). Compare the fingerprint the installer printed. | Trust it for this host, or import `/var/lib/pilens/tls/mgmt.crt` |
|
||||
| Can't log in | — | `sudo -u pilens /opt/pilens/bin/pilens reset-password -user admin` |
|
||||
| Locked out by `429` | Login throttling after failures | Wait the stated time, or restart `pilens` to clear the limiter |
|
||||
| No clients get an address | `systemctl status pilens-dnsmasq`; another DHCP server on the AP link? | `repair.sh`; ensure no stray `dnsmasq`/`hostapd` is running |
|
||||
| AP won't start | `journalctl -u NetworkManager`; radio AP-mode and band/country support | Try WPA2 / 2.4 GHz / a permitted channel; failed applies roll back automatically |
|
||||
| Clients associate but no internet | Dashboard "forwarding" card; Ethernet default route up? | Check the uplink; `forwarding` must read "forwarding", not "blocked" |
|
||||
| Capture shows drops | System Health / Findings; CPU or SD-card bound | Use a capture filter; capture to an SSD (operations.md) |
|
||||
| "interrupted" sessions after power loss | Expected: segments are hashed and re-indexed on restart | Verify hashes in Capture Library → Integrity |
|
||||
| High CPU / slow filters | Large capture; tshark re-reads segments | Filter narrows after the first run (cached); smaller segment size helps |
|
||||
| Under-voltage warning on dashboard | `rpi_volt` alarm | Use the official PSU; under-voltage corrupts storage and drops packets |
|
||||
|
||||
If a network change ever leaves the appliance unreachable, every apply keeps a
|
||||
backup and rolls back on failure; the last-known-good firewall is also restored
|
||||
at boot before the AP starts.
|
||||
Reference in New Issue
Block a user