diff --git a/CHANGELOG.md b/CHANGELOG.md index bd86fe2..4741ef3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,16 @@ # Changelog +## 0.4.0 — 2026-10-10 (Phase 4: authorised HTTPS inspection, Mode B) +- Toggleable transparent HTTPS interception via mitmdump in a venv. Off by default. +- Own RSA-2048 CA generated in-app (`PiLens Interception CA`), downloadable public + cert with per-OS install guide + fingerprint; private key never leaves the box. +- nftables redirect accepts only dnat'd connections (never an open proxy); the + proxy's own upstream never loops. +- HTTP Explorer: transactions with headers and opt-in body viewers (JSON/text/hex/ + image, sandboxed). Bodies off by default. Pinned/untrusting = reported errors. +- Verified on the Pi 5: a CA-trusting client's HTTPS decrypted (TLS 1.3), an + untrusting client honestly reported as a TLS failure. + All notable changes. Dates are when the work was done on the reference Pi 5. ## 0.3.1 — 2026-10-09 diff --git a/README.md b/README.md index 1e57d1a..ef8471c 100644 --- a/README.md +++ b/README.md @@ -21,10 +21,10 @@ capture or analysis. | 1 | Installer, services, auth, AP / DHCP / DNS / firewall, health | implemented | | 2 | dumpcap capture, pcapng storage, live analyser, filters, export / import | implemented | | 3 | Conversations, follow TCP stream, DNS/TLS explorers, statistics | implemented (HTTP explorer deferred) | -| 4 | Optional HTTPS inspection (mitmproxy, client-trusted CA) | not started | +| 4 | Optional HTTPS inspection (mitmproxy, client-trusted CA) | implemented | | 5 | Hardening, Findings, benchmarks, dependency audits | implemented | -HTTP Explorer (Phase 3 in the spec) is deferred with Phase 4, since useful HTTP content requires interception. Pages for later phases are not in the UI. PiLens never shows controls that do +Pages for later phases are not in the UI. PiLens never shows controls that do nothing. For what was tested and how, see [docs/testing.md](docs/testing.md). ## Quick start diff --git a/docs/api.md b/docs/api.md index 9093f81..8c0e163 100644 --- a/docs/api.md +++ b/docs/api.md @@ -56,3 +56,19 @@ All are GET, authenticated, and derived from already-captured data; none change state or touch the network. Conversations and statistics are computed from the SQLite index (fast, cross-segment); follow/dns/tls/findings call tshark over the segment files on demand. + +## Interception (Phase 4, Mode B) + +| Method & path | Purpose | +|---|---| +| `GET /interception` | `{config, ca, running, port, note}` | +| `PUT /interception {enabled, capture_bodies, max_body_kb, exclude_hosts[]}` | Toggle/configure; applies firewall redirect + proxy via helper | +| `GET /interception/ca.crt` | Download the CA **public** certificate (never the key) | +| `POST /interception/ca/rotate` | Regenerate the CA (only while disabled) | +| `POST /interception/clear` | Delete all intercepted HTTP flows + retained bodies | +| `GET /http?q=&client=&limit=` | Decrypted HTTP transactions, newest first | +| `GET /http/{id}` | One transaction (headers, timing, TLS, flags) | +| `GET /http/{id}/body/{req\|resp}` | Stored body as an opaque download (nosniff, sandbox CSP); only if body capture was on | + +Interception is off by default. Live transactions also arrive on `/events` as +`http` events. diff --git a/docs/architecture.md b/docs/architecture.md index 1124478..29092f8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -167,7 +167,19 @@ dumpcap -i IF [-f BPF] -w - ──pcapng──▶ splitter ──▶ seg-000001 ## Not yet implemented (phase 3+) -Implemented since Phase 2: conversations, follow TCP stream, DNS explorer, TLS -explorer (metadata only), statistics. Not in the navigation until they work: -HTTP explorer and TLS interception (modes B and C, deferred together), findings, -and capture scheduling. +Implemented: conversations, follow TCP stream, DNS/TLS explorers, statistics, +findings, and Mode B HTTPS interception (CA + mitmdump + HTTP explorer). Not yet +built: Mode C strict fail-closed policy, and capture scheduling. + +## Interception (Mode B) + +`pilens-proxy.service` runs `mitmdump` (in `/opt/pilens/proxy-venv`) as the +unprivileged `pilens` user, in transparent mode. An in-app RSA CA +(`internal/proxy`) is written to the mitmproxy confdir; the public cert is +downloadable, the key never is. When the operator toggles interception on, the +core writes `proxy.env`, then the root helper re-renders the nft table with a +`prerouting` redirect (`iifname tcp dport {80,443} redirect`) plus an +`input` rule accepting only `ct status dnat` connections, and starts the service. +A mitmdump addon writes each transaction as NDJSON; the core tails it into SQLite +and the HTTP Explorer. Toggling off removes the redirect and stops the service. +Interception is off by default and the appliance is pure capture-only until enabled. diff --git a/docs/guide/fig/certificates.png b/docs/guide/fig/certificates.png new file mode 100644 index 0000000..31d39ad Binary files /dev/null and b/docs/guide/fig/certificates.png differ diff --git a/docs/guide/fig/http.png b/docs/guide/fig/http.png new file mode 100644 index 0000000..b2a625d Binary files /dev/null and b/docs/guide/fig/http.png differ diff --git a/docs/guide/pilens-guide.pdf b/docs/guide/pilens-guide.pdf index 62ca3f0..2952706 100644 Binary files a/docs/guide/pilens-guide.pdf and b/docs/guide/pilens-guide.pdf differ diff --git a/docs/guide/pilens-guide.tex b/docs/guide/pilens-guide.tex index 84d98c3..046ddc7 100644 --- a/docs/guide/pilens-guide.tex +++ b/docs/guide/pilens-guide.tex @@ -189,8 +189,7 @@ ports, DNS, TLS handshakes, timing, the lot. scrambled. PiLens shows you \emph{who} talked to \emph{whom}, \emph{when}, over \emph{what} --- the server name, the TLS version, the certificate when it is sent in the clear --- but \textbf{not the contents}. That is TLS doing its job, not -PiLens failing. Decryption needs a separate, explicit step on each device (a -future feature), and \textbf{a trusted certificate can never be pushed down the +PiLens failing. Decryption needs a separate, explicit step on each device (Mode B, \S3.9), and \textbf{a trusted certificate can never be pushed down the network} --- the device's owner must install it by hand. Anything that claimed otherwise would be lying to you. \end{truth} @@ -201,7 +200,8 @@ otherwise would be lying to you. self-signed, so your browser warns once. Check the fingerprint the installer printed, trust it, move on. \item \textbf{The interception padlock} (decrypting a device's HTTPS) is a - \emph{separate}, opt-in, per-device thing that is not part of this version. + \emph{separate}, opt-in, per-device thing: a toggle on the Certificates page. + See \S3.9. \end{itemize} % ======================= 3. THE SCREENS ======================= @@ -327,6 +327,43 @@ spells out the innocent explanations. PiLens will never tell you ``you've been hacked'' from a retransmission. \end{truth} +\subsection{HTTPS inspection (Mode B) --- the opt-in one} +\begin{tldr} +A toggle on \textbf{Certificates}. Download the appliance's CA, install and trust +it on a device you own, flip the switch, and that device's HTTPS shows up decrypted +in the \textbf{HTTP Explorer}. Off by default. +\end{tldr} + +\screen{certificates}{Certificates \& HTTPS Inspection --- the toggle, the CA to download, per-device install steps, and the privacy options.} + +\begin{truth} +This is the only part that can see inside HTTPS, and it only works \emph{with +consent}: you must install the CA on each device by hand and turn trust on. It +\textbf{cannot} be pushed over the network or via DHCP. Pinned apps, QUIC and +private trust stores still can't be read --- they show up as honest errors, never +silently bypassed. The CA's private key never leaves the Pi and is not +downloadable. +\end{truth} + +\textbf{To use it:} Certificates \textgreater\ \emph{Download CA certificate} +\textgreater\ install it on your test device (the page has per-OS steps; iOS needs +the extra ``Certificate Trust Settings'' toggle) \textgreater\ verify the +fingerprint matches \textgreater\ flip \emph{Interception} on. + +\begin{heads} +Turning interception on while a device has \emph{not} installed the CA will break +that device's HTTPS (it correctly refuses the unknown certificate). Install first, +toggle second --- and toggle it back off when you're done. +\end{heads} + +\begin{tip} +\textbf{Bodies are off by default.} The Explorer shows headers, status and timing +for everything; request/response bodies are only stored if you opt in on the +Certificates page, for a named investigation. Clear them when done. +\end{tip} + +\screen{http}{HTTP Explorer --- decrypted transactions from trusting devices, with headers and (opt-in) body viewers.} + \subsection{The housekeeping screens} \begin{description}[leftmargin=0pt, style=nextline, itemsep=3pt] \item[\textcolor{amber}{Capture Library}] Every saved session. Rename, tag, @@ -334,6 +371,8 @@ hacked'' from a retransmission. integrity (SHA-256 per file + a manifest). \item[\textcolor{amber}{Devices}] Who has been on the Wi-Fi. Label them, add notes, jump to their packets. \mn{Randomised MACs show a badge; PiLens never pretends they identify hardware.} + \item[\textcolor{amber}{Certificates}] The HTTPS-inspection toggle, CA + download and device install guide (\S3.9). \item[\textcolor{amber}{Network Settings}] Change the access point, DHCP, DNS, forwarding and management access. Applied as one transaction with automatic rollback. diff --git a/docs/limitations.md b/docs/limitations.md index e8891f6..69595ed 100644 --- a/docs/limitations.md +++ b/docs/limitations.md @@ -3,12 +3,14 @@ **Encryption.** PiLens captures packets; it does not decrypt them. HTTPS, QUIC/HTTP-3, TLS 1.3 with Encrypted ClientHello, DNS-over-HTTPS/TLS, and VPN traffic are visible only as encrypted packets plus whatever metadata the protocol exposes in clear (for TLS ≤1.3 without ECH: SNI, -versions, ALPN; for TLS 1.2: certificates). The optional HTTPS inspection mode (client-installed -CA via mitmproxy) is **not implemented** in this version, and even when it is, it will not work -against certificate pinning, private trust stores, mutual TLS, QUIC, or apps that bypass proxies. +versions, ALPN; for TLS 1.2: certificates). The optional HTTPS inspection mode (Mode B) **is** available: enable it on the Certificates page, +install the appliance CA on each authorised test device, and decrypted transactions appear in the +HTTP Explorer. It does **not** work against certificate pinning, private trust stores, mutual TLS, +QUIC/HTTP-3, or apps that bypass the proxy — these are reported honestly as not intercepted, never +bypassed. It is off by default; while off, PiLens is pure capture-only. -**Not yet implemented:** HTTP explorer, findings/diagnostics rules, capture scheduling, -the strict inspection mode, WireGuard, packet comparison. They are not shown in the UI. +**Not yet implemented:** the strict inspection mode (Mode C fail-closed policy), capture scheduling, +WireGuard, packet comparison. They are not shown in the UI. **TLS explorer shows metadata only.** SNI, ALPN, negotiated version and cipher, plus any certificate sent in the clear (TLS ≤1.2). TLS 1.3 encrypts the certificate, so the certificate diff --git a/docs/security.md b/docs/security.md index fe5066f..415081a 100644 --- a/docs/security.md +++ b/docs/security.md @@ -28,12 +28,17 @@ sensitive system. and `systemctl` with fixed argv; and nothing else. There is no generic command execution. - **pilens-dnsmasq**: Debian's dnsmasq, which drops privileges itself. +- **pilens-proxy** (Mode B, optional): `mitmdump` as the unprivileged `pilens` + user, sandboxed (`ProtectSystem=strict`, `NoNewPrivileges`). Started/stopped by + the helper on toggle. The interception CA key is 0600 and never downloadable; + only the root helper adds the firewall redirect. ## Secrets | Secret | Where | Exposure | |---|---|---| | Admin password | Argon2id hash in SQLite (0640 pilens) | Never returned | +| Interception CA key | `proxy/ca/mitmproxy-ca.pem` (0600 pilens) | Never downloadable; only the public cert is served. Separate from the management CA. | | Session tokens | Only the SHA-256 is stored | Cookie only | | Wi-Fi passphrase | SQLite + helper state (0600 root) + NM keyfile (0600 root) | Write-only in the API. Never in a process argv. Redacted in diagnostics. | | Setup token | `/var/lib/pilens/setup-token` (0600) | Deleted when the first admin is created | @@ -41,7 +46,9 @@ sensitive system. ## Data protection -- PiLens currently stores no decrypted payloads, because it decrypts nothing. +- With interception **off** (the default) PiLens decrypts nothing and stores no + payloads. With it on, HTTP metadata is stored; bodies are retained only when the + operator explicitly opts in, size-limited, and clearable, with views audited. - Captures are deleted by retention (oldest unpreserved first) or explicitly by an admin, who must confirm. Deleting a session removes its files and all index rows. diff --git a/docs/testing.md b/docs/testing.md index 2d579aa..6f33ba9 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -67,6 +67,7 @@ Only tests that were actually run are listed. | H15 | **pass** | `setcap -r dumpcap` → `repair.sh` restored `cap_net_admin,cap_net_raw=eip`, all services ✓ | | H16 | **partial** | Loopback worst-case: 108 MB/s / 4,000 pkt/s with 14.7% drops honestly reported. Real Wi-Fi iperf3 not run. See [benchmarks.md](benchmarks.md). | | H17 | **pass** | 263 MB import indexed in 3.2 s (~3,200 pkt/s); filters ~1.2 s; UI responsive. | +| Interception (Mode B) | **pass** | On the Pi 5: CA generated + downloaded (public only, 0 key lines). Toggle on → proxy active on 8081, nft redirect + `ct status dnat` accept present. A CA-trusting client (curl `--cacert`) decrypted https://example.com and api.github.com (TLS 1.3); an untrusting client was refused and logged as a TLS failure. HTTP Explorer showed all three. Toggle off → redirect removed, proxy stopped, masquerade intact. Full device trust (phone) is the operator's manual step. | | Findings | **pass** | On the iPhone capture: 18 retransmissions/7 flows, 1 SYN retry, 7 resets, 17 dup-ACKs, 3 NXDOMAIN, 1 ICMP unreachable — each linking to evidence packets. | | H9, H10, H14 | not run | Need physical cable/client actions. | diff --git a/docs/tls-and-encryption.md b/docs/tls-and-encryption.md index 7dfe02e..a923983 100644 --- a/docs/tls-and-encryption.md +++ b/docs/tls-and-encryption.md @@ -12,15 +12,36 @@ single most common misunderstanding about a tool like this. unrelated to anything captured. The installer prints its SHA-256 fingerprint; verify it the first time your browser warns about it. -2. **Interception CA** — a separate authority that *would* let PiLens decrypt a - test device's HTTPS, by having that device explicitly trust it. **This is not - implemented in this version** (Phase 4). When it is, it will be opt-in, - per-device, and require you to install and trust the CA on the client - yourself. There is no way to do it silently. +2. **Interception CA** — a separate authority that lets PiLens decrypt a test + device's HTTPS, *after* that device's owner explicitly installs and trusts it. + Generated in-app (RSA-2048, `PiLens Interception CA`), downloadable from the + Certificates page (public cert only — the key never leaves the appliance). + Interception is a toggle, **off by default**. There is no way to make a device + trust it silently or over the network. The two are kept independent on purpose: compromising or distributing an interception CA must never affect the management channel. +## Mode B: authorised interception (optional, off by default) + +Turn it on from **Certificates & HTTPS Inspection**. PiLens then transparently +redirects AP clients' TCP 80/443 to a local mitmdump, which presents certificates +signed by the appliance CA. A device that has installed and trusts that CA sees a +valid chain, so its HTTPS is decrypted and the transactions appear in the **HTTP +Explorer**. A device that has *not* installed it sees an untrusted certificate and +its connections fail — which is why interception is useless (and harmless) until +you do the manual install step on each device you own. + +- Redirect is `iifname ` only, TCP 80/443. The proxy accepts only + connections the redirect created (`ct status dnat`), so it is never an open + proxy for clients. +- **Excluded hosts** (pinned apps, privacy-sensitive sites) are passed straight + through, still encrypted. +- **Bodies are not retained by default.** Enable it per-investigation; the data + is on the appliance disk until cleared, and views/exports are audited. +- Pinned apps, private trust stores, mutual TLS and QUIC cannot be intercepted + and are reported as such — never bypassed, never downgraded. + ## Why HTTPS content is unavailable without client trust TLS exists precisely to stop a device in the middle from reading traffic. To