From 1c31e3b3d542b6955afa5c7d93fa65c31df4c497 Mon Sep 17 00:00:00 2001 From: Andy Dixon Date: Fri, 9 Oct 2026 17:58:58 +0100 Subject: [PATCH] Documentation: network policy, security model, forensics, operations, API, testing, limitations --- README.md | 74 ++++++++++++++++++++++++++++++++++ docs/api.md | 41 +++++++++++++++++++ docs/forensics.md | 50 +++++++++++++++++++++++ docs/limitations.md | 32 +++++++++++++++ docs/network.md | 86 ++++++++++++++++++++++++++++++++++++++++ docs/operations.md | 82 ++++++++++++++++++++++++++++++++++++++ docs/security.md | 71 +++++++++++++++++++++++++++++++++ docs/testing.md | 50 +++++++++++++++++++++++ tests/integration/run.sh | 2 +- 9 files changed, 487 insertions(+), 1 deletion(-) create mode 100644 README.md create mode 100644 docs/api.md create mode 100644 docs/forensics.md create mode 100644 docs/limitations.md create mode 100644 docs/network.md create mode 100644 docs/operations.md create mode 100644 docs/security.md create mode 100644 docs/testing.md diff --git a/README.md b/README.md new file mode 100644 index 0000000..ae9cdf0 --- /dev/null +++ b/README.md @@ -0,0 +1,74 @@ +# PiLens + +PiLens turns a Raspberry Pi into a self-contained Wi-Fi test network with packet +capture and Wireshark-grade analysis in the browser. You connect an **authorised** +test device to the Pi's access point and watch its traffic: live packets, a +protocol tree from TShark, and the original bytes with field highlighting. The +original captures are kept unmodified, with SHA-256 integrity hashes. + +No cloud service, account or internet connection is needed for management, +capture or analysis. + +> **Use it only on devices and networks you own or are explicitly authorised to +> inspect.** PiLens has no covert, impersonation or credential-harvesting +> features. Encrypted traffic (HTTPS, QUIC, VPNs, encrypted DNS) stays encrypted: +> in this version PiLens shows packets and metadata, not decrypted content. + +## Status + +| Phase | Scope | State | +|---|---|---| +| 1 | Installer, services, auth, AP / DHCP / DNS / firewall, health | implemented | +| 2 | dumpcap capture, pcapng storage, live analyser, filters, export / import | implemented | +| 3 | Conversations, follow TCP stream, DNS/TLS explorers, statistics | not started | +| 4 | Optional HTTPS inspection (mitmproxy, client-trusted CA) | not started | +| 5 | Hardening, advanced UI, benchmarks | partial | + +Pages for later phases are not in the UI. PiLens never shows controls that do +nothing. For what was tested and how, see [docs/testing.md](docs/testing.md). + +## Quick start + +On a Raspberry Pi 4/5 with Raspberry Pi OS Lite (64-bit, Bookworm or Trixie), +connected by **Ethernet**: + +```sh +tar xzf pilens--linux-arm64.tar.gz +cd pilens--linux-arm64 +sudo ./install.sh +``` + +The installer prints the management URL, the certificate fingerprint and a +one-time setup token. Open the URL from a machine on the management network and +complete the setup wizard. The wizard creates the access point and turns the +Wi-Fi interface into the AP, so **manage the Pi over Ethernet**. + +For unattended installation: `sudo ./install.sh --non-interactive --config setup.yaml` +(see `setup.example.yaml`). + +## Building + +Requirements: Go ≥ 1.24, Node ≥ 20. Docker is used only for the test suite. + +```sh +make package # dist/pilens--linux-arm64.tar.gz (cross-compiled, static) +make test # unit tests + TypeScript typecheck +make test-docker # tshark/dumpcap tests + network-namespace integration suite +``` + +## Documentation + +- [Architecture and design decisions](docs/architecture.md) +- [Installation, first run, upgrade and recovery](docs/operations.md) +- [Network topology and firewall policy](docs/network.md) +- [Security and privacy model](docs/security.md) +- [Forensic evidence handling](docs/forensics.md) +- [API](docs/api.md) +- [Testing and hardware acceptance checklist](docs/testing.md) +- [Known limitations](docs/limitations.md) + +## Licence + +AGPL-3.0-or-later. Bundled JavaScript dependencies (React, MIT) are compatible. +TShark/dumpcap (GPL-2.0+), dnsmasq (GPL-2.0/3.0) and nftables (GPL-2.0) are +installed from Debian packages and run as separate programs. diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..797b927 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,41 @@ +# API + +All endpoints are under `https://:8443/api/`. Except `GET /health`, +`GET /setup/status`, `POST /setup/admin` and `POST /auth/login`, every request +needs the session cookie. Every non-GET request also needs the +`X-CSRF-Token` header (value from `/auth/session` or the login response). +Unauthenticated POSTs need `X-Requested-With: pilens`. Bodies are JSON, limited +to 1 MiB, and unknown fields are rejected. + +| Method & path | Purpose | +|---|---| +| `GET /health` | Liveness `{ok:true}` | +| `GET /setup/status` | `{needs_admin, setup_complete, version}` | +| `POST /setup/admin` | `{token, username, password, authorised:true}` → session | +| `POST /auth/login` | `{username, password, totp?}` → `{user, csrf}`; 401 `totp_required` | +| `POST /auth/logout` · `GET /auth/session` · `POST /auth/password {Current, New}` | | +| `POST /auth/totp/begin` · `/enable {Code}` · `/disable {Password}` | TOTP lifecycle | +| `GET /audit` | Last 500 audit events | +| `GET /events` | SSE stream: `packets` `{session, rows}`, `capture` (session object, every 1 s while capturing), `resync` (`{session}` or `{ui_dropped}`), `ping` | +| `GET /system/interfaces` | Detected interfaces, Wi-Fi capabilities, bootstrap management, caller address | +| `GET /system/health` | System, helper, capture, storage, upstream, forwarding state | +| `GET /network` | Current config (passphrase redacted) | +| `POST /network/validate` | `{valid, errors[]}`, including lock-out detection | +| `POST /network/apply` | Transactional apply. `{result:{applied, rolled_back, steps[], error}}`. 502 on failure (rolled back). | +| `POST /setup/complete` | Marks first-run done | +| `GET /ap/clients` · `PUT /devices/{mac} {Label, Notes}` | AP clients (leases + stations + labels) | +| `GET /captures` · `POST /captures {name, interfaces[], capture_filter, ring_segments}` | List / start | +| `POST /captures/live/stop` · `POST /captures/live/pause {Paused}` | | +| `POST /captures/import?name=` | Raw pcap/pcapng body | +| `GET /captures/{id}` · `PATCH {Name, Description, Tags, Notes, Preserved}` · `DELETE ?confirm={id}` | | +| `GET /captures/{id}/packets?offset&limit&filter` | Page of summary rows. With `filter`, `nums[]` gives packet numbers per position. | +| `GET /captures/{id}/packets/{n}` | `{detail:{row, bytes(hex), tree[], segment, note}, error?}` | +| `GET /captures/{id}/export?filter=` | pcapng (or pcap for pcap imports), original records | +| `GET /captures/{id}/manifest` · `POST /captures/{id}/verify` | Integrity | +| `POST /filters/validate {Kind: display|capture, Filter, Interface}` | tshark / dumpcap compile check | +| `GET/POST /filters/saved` · `DELETE /filters/saved/{id}` | Saved filters | +| `GET /storage` · `PUT /storage {max_bytes, reserve_bytes, segment_bytes, import_bytes}` | | + +Tree nodes: `{label, field, show, pos, size, verified, children}`. `verified` +means the frame bytes at `pos..pos+size` were checked against tshark's raw +value, so highlighting them is accurate. diff --git a/docs/forensics.md b/docs/forensics.md new file mode 100644 index 0000000..518fb92 --- /dev/null +++ b/docs/forensics.md @@ -0,0 +1,50 @@ +# Forensic evidence handling + +## What is evidence and what is derived + +| Data | Status | +|---|---| +| `captures//seg-NNNNNN.pcapng` (live) | **Original**. dumpcap's pcapng output, written byte for byte. Each segment starts with the section and interface headers copied verbatim, so it opens on its own. | +| `captures//original.pcap[ng]` (import) | **Original as received**. The uploaded file, unchanged. | +| SHA-256 per segment | Computed while writing, and over the whole file for imports. Stored in SQLite. | +| Packet summary rows (SQLite) | Derived: tshark output, regenerable | +| Protocol trees, filters, exports with a filter | Derived. A filtered export copies the original packet records verbatim, but as a selection it is a new file. | + +PiLens never labels reconstructed or synthetic data as an original capture. + +## Recorded metadata (per session) + +Session `meta` and the manifest record: + +- UTC start/stop (milliseconds), capture interfaces and capture filter +- PiLens and TShark versions, hostname +- `clock_synchronised` from `timedatectl` at start. If this is not `yes`, + absolute timestamps may be wrong. A Pi has no battery-backed clock, so make + sure NTP is working before you capture evidence. +- Segment size, ring-buffer setting, pause/resume log, packets discarded while + paused +- dumpcap's own "received/dropped" report, verbatim (kernel and pcap drops) + +## Verifying + +- UI: Capture Library → *Integrity* → *Verify hashes now*. This recomputes + SHA-256 for every complete segment. +- Manifest: *Download manifest* gives JSON with every file, its hash, packet + count and the metadata above. +- Offline: `sha256sum seg-*.pcapng` and compare with the manifest. + +## Caveats + +- **Capture loss** happens under load. Check the drop report. A capture with + drops is incomplete. +- **Segment boundaries**: summaries and dissections are computed per segment, + so TCP stream indices and reassembly restart at each boundary. The packet + bytes are unaffected. +- **Interrupted captures** (power loss): on restart, the last segment is + scanned, counted and hashed as found, and the session is marked + `interrupted`. A truncated final packet record may be present. +- **Pause capture** discards packets while paused. The pause log in the + session metadata records when. +- **Randomised MAC addresses** do not identify hardware. PiLens never merges + device identities automatically. +- DNS answers do not prove which host a later connection went to. diff --git a/docs/limitations.md b/docs/limitations.md new file mode 100644 index 0000000..c5f37c2 --- /dev/null +++ b/docs/limitations.md @@ -0,0 +1,32 @@ +# Known limitations (v0.1) + +**Encryption.** PiLens captures packets; it does not decrypt them. HTTPS, QUIC/HTTP-3, TLS 1.3 with +Encrypted ClientHello, DNS-over-HTTPS/TLS, and VPN traffic are visible only as encrypted +packets plus whatever metadata the protocol exposes in clear (for TLS ≤1.3 without ECH: SNI, +versions, ALPN; for TLS 1.2: certificates). The optional HTTPS inspection mode (client-installed +CA via mitmproxy) is **not implemented** in this version, and even when it is, it will not work +against certificate pinning, private trust stores, mutual TLS, QUIC, or apps that bypass proxies. + +**Not yet implemented:** conversations view, follow TCP stream, DNS/TLS/HTTP explorers, +statistics charts, findings/diagnostics rules, capture scheduling, the strict inspection mode, +WireGuard, packet comparison. They are not shown in the UI. + +**IPv6** is blocked for AP clients rather than routed (see network.md). + +**AP features.** Max-client limits are not available through NetworkManager. WPA2/WPA3 +transition mode is not offered. WPA3 (SAE) in AP mode depends on the Wi-Fi driver and +firmware. On the Raspberry Pi's brcmfmac it may fail, in which case the change is +rolled back. + +**Capture.** +- One live capture at a time. +- Display filters on large captures run tshark over every segment. That takes seconds per GiB + on a Pi 4/5, and results are cached per filter. +- The packet table's scroll height is limited by browser element-size limits (about 1.5 million + rows at compact density). Use a display filter for larger captures. +- Capture drop counts are reported by dumpcap when the capture stops, not live. + +**Performance figures** have not yet been measured on hardware, so none are claimed. +`docs/testing.md` lists the benchmarks to run. + +**Single administrator role.** Every account has full rights. diff --git a/docs/network.md b/docs/network.md new file mode 100644 index 0000000..d2df333 --- /dev/null +++ b/docs/network.md @@ -0,0 +1,86 @@ +# Network topology and policy + +``` + upstream LAN / internet + │ eth0 (uplink + management, DHCP client as before) + ┌──────┴───────────────────────────────┐ + │ Raspberry Pi │ + │ nft table inet pilens │ + │ NAT (masquerade) AP subnet → uplink │ + │ pilens-dnsmasq: DHCP+DNS on wlan0 │ + └──────┬───────────────────────────────┘ + │ wlan0 AP "PiLens-Lab", 192.168.50.1/24 (static, NM keyfile) + test device(s) 192.168.50.100–199 +``` + +The Pi **routes**: it does not bridge. AP clients are always on their own subnet +behind NAT, so every packet they send crosses the Pi's IP stack. That is what +makes capture and policy enforcement complete. + +## Interface discovery + +Interface names are discovered, not assumed. The installer picks: + +- **Uplink**: the interface holding the IPv4 default route. +- **Management**: the interface your SSH session arrived on, falling back to the + uplink. The default allowed sources are that interface's connected subnet, + plus your SSH client's /32 if it lies outside that subnet (for example when + you connect through NAT or a VPN). +- **AP**: the first interface with `/sys/class/net//wireless`. + +All three can be changed in the wizard or under Network Settings. Validation +rejects an AP interface that is also used for management or uplink, and any AP +subnet that overlaps a management subnet. + +## Firewall rules, in words + +| Traffic | Decision | Why | +|---|---|---| +| AP client → Pi UDP 67 (DHCP) | accept | Address assignment | +| AP client → Pi gateway :53 (DNS) | accept | Resolution through the Pi, so DNS is observable | +| AP client → Pi ICMP echo / errors | accept | Basic diagnostics | +| AP client → Pi anything else (SSH, :8443, …) | **drop** | The AP network is untrusted | +| AP client IPv6 (to the Pi or forwarded) | **drop** | IPv6 is not inspected, so it is closed rather than left open | +| AP client → private / link-local / CGNAT / multicast / management nets | **drop** | Clients cannot pivot into your LAN | +| AP client → explicit `allow_cidrs` | accept | An opt-in exception, e.g. a lab server | +| AP client → public addresses via uplink | accept when mode = `internet` | The purpose of the gateway | +| AP client → anything, mode = `deny` | **drop** | Observe-only lab mode (DHCP and DNS still work) | +| Anything → AP subnet, not a reply | **drop** | No inbound connections to clients | +| Any interface → :8443 from a non-allowed source | **drop** | Management isolation | +| Everything else | not touched by PiLens | Host SSH and other services keep their policy | + +Multicast, broadcast and mDNS from clients are not forwarded. DHCP broadcasts +reach dnsmasq because they are delivered locally, not forwarded. With **client +isolation** enabled, the Wi-Fi driver stops clients talking to each other +directly. Without it, client-to-client frames on the same radio never reach the +IP layer, but they are still visible to a capture on the AP interface. + +Rules are applied atomically: one `nft -f` both deletes and recreates the +table. A ruleset that fails `nft -c` is never loaded. + +## Failure behaviour + +| Event | Result | +|---|---| +| Ethernet unplugged / upstream gateway unreachable | Clients keep DHCP and DNS (answers from cache or overrides; other lookups fail). Forwarded traffic has no route and fails. The dashboard shows "no default route". | +| DNS upstream fails | dnsmasq returns SERVFAIL. Visible in captures. | +| Capture engine crashes | Forwarding is unaffected in capture-only mode. Interrupted segments are hashed and closed on restart, and the session is marked *failed / interrupted*. | +| Storage exhausted | The oldest unpreserved sessions are deleted. If none are left, capture stops with an explicit error. Forwarding continues. | +| Network apply fails | Automatic rollback to the previous NM keyfile, dnsmasq config and nft table, and the failure is reported with each step. | +| Firewall restore fails at boot | `pilens-firewall.service` fails, but **IPv4 forwarding stays disabled** (it is never persisted), so clients cannot be routed. | +| Reboot | `pilens-firewall` loads the last good table before NetworkManager starts the AP. Forwarding is enabled only after that. | +| Helper unavailable | The UI shows forwarding state "unknown". The last loaded table stays in force. | + +TLS-proxy related behaviour (mode B/C) belongs to a later phase. + +## IPv6 + +IPv6 forwarding and inspection are not implemented. The AP profile sets +`ipv6.method=disabled`, so clients get no router advertisements from the Pi, +and the firewall drops any IPv6 a client sends. Clients therefore fall back to +IPv4, which is fully routed and captured. + +## Regulatory domain + +The country code is applied with `iw reg set` on every apply and at boot. Set +it correctly: it determines the legal channels and transmit power. diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..13d5a85 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,82 @@ +# Operations: install, first run, upgrade, recovery + +## Requirements + +- Raspberry Pi 4 or 5 (other arm64 Debian systems with NetworkManager and an AP-capable radio + may work but are untested) +- Raspberry Pi OS Lite 64-bit, Bookworm or Trixie, with NetworkManager (the default) +- Ethernet connected to the network you will manage it from +- ≥ 1 GiB free disk. An SSD (USB 3 or NVMe) is strongly recommended for captures: SD cards + are slow and wear out. + +## Install + +```sh +tar xzf pilens--linux-arm64.tar.gz && cd pilens--linux-arm64 +sudo ./install.sh # explains every change and asks first +``` + +What it does, in order: checks the platform; detects interfaces and your SSH path; checks AP +support, disk, RAM and conflicting services; shows the plan; backs up +`/etc/NetworkManager/system-connections` and `nft list ruleset` to +`/var/backups/pilens/