From 53968f8822d3b91efe271f6bfd2ad00256046612 Mon Sep 17 00:00:00 2001 From: Andy Dixon Date: Thu, 8 Oct 2026 11:50:55 +0100 Subject: [PATCH] Docs for the Rust version; benchmarks; build.sh fixes README, CLAUDE.md, man pages, the user guide and its screenshot script now describe the Rust build and the intentional behaviour changes, which docs/MIGRATION.md lists. Go and Rust micro-benchmarks for the same operations, and scripts/bench.sh for end-to-end comparison. build.sh exports its tree root to the per-target jobs and keeps logs out of the output directory; riscv64 musl builds use getrandom's raw syscall backend, since zig's musl getrandom never reaches the kernel there. Co-Authored-By: Claude --- CLAUDE.md | 110 +++++++++++------- Cargo.toml | 4 + README.md | 50 +++++--- benches/micro.rs | 79 +++++++++++++ build.sh | 12 +- docs/MIGRATION.md | 95 +++++++++++++++ docs/guide/shots.py | 9 +- docs/guide/vault-guide.tex | 53 ++++++--- docs/vault-add.1 | 23 +++- docs/vault-get.1 | 13 ++- docs/vault-migrate.1 | 7 +- docs/vault-mirror.1 | 77 +++++++++--- docs/vault-unlock.1 | 8 ++ docs/vault.1 | 25 +++- .../go/internal/pqcrypt/oracle_bench_test.go | 57 +++++++++ .../go/internal/stream/oracle_bench_test.go | 27 +++++ oracle/go/oracle_bench_test.go | 23 ++++ scripts/bench.sh | 96 +++++++++++++++ 18 files changed, 659 insertions(+), 109 deletions(-) create mode 100644 benches/micro.rs create mode 100644 docs/MIGRATION.md create mode 100644 oracle/go/internal/pqcrypt/oracle_bench_test.go create mode 100644 oracle/go/internal/stream/oracle_bench_test.go create mode 100644 oracle/go/oracle_bench_test.go create mode 100755 scripts/bench.sh diff --git a/CLAUDE.md b/CLAUDE.md index b73e585..0442c15 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,71 +4,93 @@ This file provides guidance to Claude Code when working with this repository. ## Overview -vault is a post-quantum encrypted file vault: one `package main` at the repo -root that builds a single `vault` binary. Module `github.com/andydixon/vault`. +vault is a post-quantum encrypted file vault: one Rust package (`Cargo.toml` +at the root, toolchain pinned in `rust-toolchain.toml`) that builds a single +`vault` binary. The library (`src/lib.rs`) exists so tests and benchmarks can +reach the same code; `src/main.rs` only calls `cli::main`. -- `main.go` - CLI: `genkey`, `init`, `add` (file, directory - or URL), `migrate` (add, then delete originals), `mirror`, `absorb` (tar - on stdin, `absorb.go`), `ls` (one folder level), `list` (recursive), - `get` (`-r` for a folder), `mv`/`cp` (files or folders; `transfers` - resolves names like mv(1), copies share objects), `rm` (`-r` for a - folder, asks unless `-f`), `unlock`/`lock`. -- `agent*.go` - `vault unlock` re-executes itself as a hidden `__agent` - process holding the suite ID and seed (pipe in, mlocked, undumpable) and - serving them on a Unix socket in a 0700 per-user dir until a deadline - (fixed, or with `--idle` reset on each use). `keyIdentity` asks it first - unless `$VAULT_PASSPHRASE` is set. -- `keyfile.go` - key file with two passphrase slots (Argon2id + - XChaCha20-Poly1305 sealing a suite ID and 32-byte master seed). The second - slot is either a decoy identity or random bytes; slot order is random. -- `store.go` - on-disk vault: `records/` holds fixed-size encrypted catalogue - records, `objects/` holds file bodies cut into 4 MiB chunks (last one - bucket-padded) whose IDs are HMAC'd from a per-record seed. Both are - sharded into 256 `xx/` subdirectories by ID prefix. Records a key cannot - open are skipped, not treated as errors. -- `lock*.go` - vault lock on `DIR/.lock`: fcntl on unix, LockFileEx on - Windows, an exclusive lock file elsewhere. Writers lock exclusively, - readers shared, taken after the passphrase prompt. -- `docs/guide` - LaTeX user guide (`build-pdf.sh` via Tectonic; `shots.py` - re-shoots screenshots from the real binary). Keep it in step with the CLI. -- `packaging/repo.sh [TAG]` - builds dixon-vault .deb/.rpm/Arch packages - from a tag with nfpm and publishes them, signed, to the apt/dnf/pacman - repo at repo.dixon.cx (same script, other project block, in ../mapsize). -- `build.sh` - cross-compiles every `go tool dist list` target except - iOS and android/386, amd64, arm (cgo-only). -- `internal/pqcrypt` - file encryption (header + STREAM ChaCha20-Poly1305), - suite registry, hybrid FrodoKEM-640 + ML-KEM-768 default suite. -- `internal/format`, `internal/stream`, `internal/term` - header encoding, - chunked payload encryption, terminal passphrase prompts. +- `src/cli.rs` - commands and Go-`flag`-compatible parsing: `genkey`, + `init`, `add` (file, directory or URL), `migrate`, `mirror`, `absorb`, + `ls`/`list`, `get` (`-r`), `mv`/`cp` (`transfers` resolves names like + mv(1)), `rm` (`-r`, asks unless `-f`), `unlock`/`lock`, hidden `__agent`. +- `src/agent.rs` - the unlock agent: `unlock` re-executes the binary as + `__agent` (passphrase variables stripped, secret over a pipe); it hardens + itself (`platform::memory`) before reading the key, serves it on a Unix + socket in a 0700 per-user dir until a deadline (fixed or `--idle`). + Wire-compatible with the Go agent. Not available on Windows. +- `src/keyfile.rs` - key file: two Argon2id/XChaCha20-Poly1305 slots, decoy + or random bytes in the second, random order, both always tried. +- `src/crypto/` - `frodo.rs` (FrodoKEM-640-SHAKE ported from CIRCL v1.6.4), + `kem.rs` (hybrid suite 0x0001 with RustCrypto ml-kem, `Identity`), + `header.rs` (Fury header format), `fury.rs` (file encryption), `stream.rs` + (STREAM, including random access). +- `src/store/` - on-disk vault: `records/`, `objects/` sharded by ID prefix, + catalogue, add/remove/copy/extract, filler; `types.rs` holds the validated + types (`VaultName`, `Id`, `ObjectSeed`, `Record`) and record JSON. +- `src/pipeline.rs` - the one bounded worker pool and the eight reusable + object buffers; coordinators run on the caller's thread. +- `src/platform/` - all `unsafe` and OS specifics: `fs.rs` (no-clobber + publish, `Dir` confined extraction, epoch times, dir sync), `lock.rs` + (fcntl / LockFileEx / lock file), `memory.rs` (mlock, undumpable), + `term.rs` (prompts). +- `src/net/` - `add URL` and `mirror` with limits; `url.rs` (Go `net/url` + semantics), `html.rs` (tokenizer and link rewriting). +- `src/absorb.rs` - tar (optionally gzipped) from stdin, streamed in. +- `src/secret.rs`, `src/rng.rs`, `src/util.rs`, `src/error.rs` - secret + buffers, randomness (seeded only under `cfg(test)`), Go-compatible + parsing/printing helpers, the error type. +- `oracle/go` - the frozen Go implementation (test-only reference, see + `oracle/README.md`); `oracle/gen-vectors.sh` writes `tests/vectors`. +- `tests/` - CLI and interoperability tests against the real binaries; + `fuzz/` - cargo-fuzz targets; `benches/micro.rs` - micro-benchmarks. +- `docs/` - man pages, `COMPATIBILITY.md`, `MIGRATION.md`, `PLATFORMS.md`, + `BENCHMARKS.md`, and the LaTeX guide (`docs/guide`, `build-pdf.sh` via + Tectonic; `shots.py` re-shoots screenshots from the real binary). Keep + them in step with the CLI. +- `build.sh` - cross-builds every supported target (see `PLATFORMS.md`). +- `packaging/package.sh` builds and inspects dixon-vault packages locally; + `packaging/repo.sh [TAG]` signs and publishes them to repo.dixon.cx. ## Commands ```sh -go test ./... -go build . -./build.sh # all platforms, into ./dist -TARGETS="linux/amd64" ./build.sh # a subset +cargo test --locked # needs go on PATH for interop tests +cargo clippy --locked --all-targets --all-features -- -D warnings +cargo fmt --check +cargo build --release --locked +./build.sh # all buildable targets, into ./dist +TARGETS="linux/amd64" ./build.sh # a subset +cargo deny --locked check # advisories and licences +(cd fuzz && cargo +nightly fuzz run header) ``` End-to-end check: ```sh export VAULT_PASSPHRASE=test VAULT_KEY=$PWD/k VAULT_DIR=$PWD/s -./vault genkey && ./vault init && ./vault add README.md docs && ./vault list +target/release/vault genkey && target/release/vault init && target/release/vault add README.md docs && target/release/vault list ``` ## Invariants +- The on-disk formats are frozen and shared with the Go version; check any + change against `docs/COMPATIBILITY.md` and the oracle vectors. - Never write a seed or private key unencrypted (the agent gets it by pipe). -- Keep suite IDs, stanza tags, HKDF info strings and `fileKeySize` (32) stable. +- Keep suite IDs, stanza tags, HKDF info strings and the 32-byte file key stable. - Every ciphertext's and shard directory's mtime is pinned to the epoch; records stay one fixed size; objects stay at most 4 MiB and bucket-padded. These hide metadata from the storage host. - An object's plaintext starts with its own ID, binding it to its record and its position in the file. -- `get` publishes output only after every object has authenticated. +- `get` publishes output only after every object has authenticated, with a + no-replace primitive; never fall back to check-then-rename. - Decoy deniability: both key slots stay the same size and are both always tried; `init` always writes filler; foreign records are skipped silently. -- Every command that changes the vault holds the exclusive lock. -- Keep dependencies pure Go and CGO_ENABLED=0 builds working. +- Every command that changes the vault holds the exclusive lock (POSIX + fcntl locks, compatible with Go's). +- Crash-safe order: objects before records, new records before removing old + ones, record removal (synced) before object removal. +- Secrets live in `SecretArray`/`SecretVec`/`LockedBuf`: no Clone, Debug or + serialisation. Production randomness is the OS only. +- `unsafe` only in `src/platform`, each block with a SAFETY comment. - Diagnostics go to standard error, lower-case, no trailing full stops. diff --git a/Cargo.toml b/Cargo.toml index 67e6588..5a47ebf 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -40,3 +40,7 @@ windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Storage codegen-units = 1 lto = true strip = true + +[[bench]] +name = "micro" +harness = false diff --git a/README.md b/README.md index 3e6f2de..72aa72f 100644 --- a/README.md +++ b/README.md @@ -23,16 +23,36 @@ Releases are packaged and published with `packaging/repo.sh [TAG]`. ## Build +vault is written in Rust (the toolchain is pinned in `rust-toolchain.toml`; +`rustup` installs it on first use). Releases up to 0.5 were Go; the Go code +lives on, frozen, in `oracle/go` as the test reference. Nothing at run time +needs Go. + ```sh -go build . # produces ./vault -go test ./... -./build.sh # every OS and CPU architecture, into ./dist +cargo build --release --locked # produces target/release/vault +cargo test --locked # unit, CLI and Go-interoperability tests +./build.sh # every target this host can build, into ./dist +TARGETS="linux/amd64 windows/amd64" ./build.sh ``` -`build.sh` cross-compiles static binaries for every target in -`go tool dist list` except iOS and 32-bit or x86 Android, which need cgo. It builds 32-bit ARM Linux for -both armv6 and armv7, and writes `dist/SHA256SUMS`. Set `TARGETS` to build a -subset, for example `TARGETS="linux/amd64 darwin/arm64" ./build.sh`. +`build.sh` cross-compiles with `cargo-zigbuild` and `zig` (Linux, statically +linked with musl, and Windows), on a Mac for macOS, and with `cross` and +Docker for the BSDs, illumos and Solaris. It names binaries as before +(`vault-linux-arm64`, `vault-windows-amd64.exe`, …), builds 32-bit ARM Linux +for both armv6 and armv7, and writes `dist/SHA256SUMS`. `docs/PLATFORMS.md` +lists every target, how it is built and tested, and the Go targets Rust +cannot build. + +The interoperability tests build the Go oracle when `go` is installed and +are skipped (and say so) otherwise; set `VAULT_REQUIRE_ORACLE=1` to make a +missing oracle an error, as CI does. `oracle/gen-vectors.sh` regenerates the +Go test vectors. Packages are built and inspected locally, unsigned, with +`packaging/package.sh`, and published with `packaging/repo.sh [TAG]`. + +Further reading: `SECURITY.md` (what is and is not protected, memory and +filesystem guarantees), `docs/COMPATIBILITY.md` (the formats), +`docs/MIGRATION.md` (what changed from the Go version) and +`docs/BENCHMARKS.md`. ## User guide @@ -82,12 +102,15 @@ vault lock # forget it now Commands that change the vault take an exclusive lock on `DIR/.lock`, and commands that read it take a shared one. A second process waits, and says so, until the first finishes. The operating system releases the lock if a -process dies. Plan 9 and WebAssembly have no file locks, so there the lock -file itself is the lock. After a crash on those platforms you remove it by -hand; the error message names it. +process dies; the locks are the same ones the Go version used, so old and +new versions exclude each other. WebAssembly (WASI) has no file locks, so +there the lock file itself is the lock. After a crash there you remove it +by hand; the error message names it. -Adding a name that already exists replaces it. `get` refuses to overwrite an -existing file and writes nothing until the whole object has authenticated. +Adding a name that already exists replaces it. `get` never replaces an +existing file (or symlink) and writes nothing until the whole file has +authenticated. `add URL` and `mirror` have size and time limits; see +`vault(1)`. `--key PATH` and `--vault DIR` work with every command. They default to `$VAULT_KEY` and `$VAULT_DIR`, then to `~/.vault/key` and `~/.vault/store`. @@ -103,7 +126,8 @@ over a pipe and never touches the disk; the helper's memory is locked against swapping, it cannot be core-dumped (or, on Linux, debugged), and it answers only on a socket in a directory private to your user. While it is unlocked, anything running as you can use the vault, decoy or not, so lock -before handing the machine over. +before handing the machine over. If the helper cannot protect its memory it +refuses the key. `unlock` is not available on Windows. ## Decoy passphrase diff --git a/benches/micro.rs b/benches/micro.rs new file mode 100644 index 0000000..7b3939c --- /dev/null +++ b/benches/micro.rs @@ -0,0 +1,79 @@ +// SPDX-License-Identifier: GPL-3.0-or-later + +//! The Rust side of scripts/bench.sh: the same operations as the Go +//! oracle's Benchmark* functions, timed as the median of repeated runs. +//! Run with `cargo bench --bench micro`. + +use std::hint::black_box; +use std::io::{Read, Write}; +use std::time::{Duration, Instant}; +use vault::crypto::{fury, kem, stream}; + +fn pattern(n: usize, salt: u8) -> Vec { + (0..n).map(|i| (i.wrapping_mul(7) as u8) ^ salt).collect() +} + +/// Runs `f` until about two seconds have passed (at least 5 times) and +/// prints the median and the spread. +fn bench(name: &str, bytes: usize, mut f: impl FnMut()) { + f(); // warm up + let mut times = Vec::new(); + let start = Instant::now(); + while times.len() < 5 || (start.elapsed() < Duration::from_secs(2) && times.len() < 10_000) { + let t = Instant::now(); + f(); + times.push(t.elapsed()); + } + times.sort(); + let med = times[times.len() / 2]; + let (lo, hi) = (times[times.len() / 10], times[times.len() * 9 / 10]); + let rate = if bytes > 0 { + format!(" {:.0} MB/s", bytes as f64 / med.as_secs_f64() / 1e6) + } else { + String::new() + }; + println!( + "{name:<24} median {:>10.3} ms (p10 {:.3}, p90 {:.3}, n={}){rate}", + med.as_secs_f64() * 1e3, + lo.as_secs_f64() * 1e3, + hi.as_secs_f64() * 1e3, + times.len() + ); +} + +fn main() { + let seed = pattern(32, 1); + bench("derive", 0, || { + black_box(kem::Identity::from_suite_seed(kem::SUITE_ID, &seed).unwrap()); + }); + let id = kem::Identity::from_suite_seed(kem::SUITE_ID, &seed).unwrap(); + let r = id.recipient(); + let fk: [u8; 32] = pattern(32, 2).try_into().unwrap(); + bench("wrap", 0, || { + black_box(r.wrap(&fk, &mut vault::rng::os())); + }); + let st = r.wrap(&fk, &mut vault::rng::os()); + bench("unwrap", 0, || { + black_box(id.unwrap(std::slice::from_ref(&st)).unwrap()); + }); + let pt = pattern(4 << 20, 3); + bench("object 4MiB", pt.len(), || { + let mut ct = Vec::new(); + fury::encrypt_to(&mut ct, &r, &pt, &mut vault::rng::os()); + black_box(fury::decrypt_bounded(&ct[..], &id, pt.len()).unwrap()); + }); + let key = [0u8; 32]; + let big = vec![0u8; 64 << 20]; + bench("stream 64MiB", big.len(), || { + let mut w = stream::StreamWriter::new(&key, Vec::with_capacity(big.len() + (1 << 16))); + w.write_all(&big).unwrap(); + let ct = w.finish().unwrap(); + let mut out = Vec::with_capacity(big.len()); + stream::StreamReader::new(&key, &ct[..]).read_to_end(&mut out).unwrap(); + black_box(out); + }); + let body = vault::keyfile::new_body(b"bench", b"", &mut vault::rng::os()); + bench("unlock", 0, || { + black_box(vault::keyfile::unlock_body(&body, b"bench").unwrap()); + }); +} diff --git a/build.sh b/build.sh index 1594179..87a2d09 100755 --- a/build.sh +++ b/build.sh @@ -73,11 +73,14 @@ out=$(cd -- "$out" && pwd -P) case $out in / | "$(cd ~ && pwd -P)" | "$root") echo "refusing OUT=$out: use a dedicated directory" >&2; exit 2 ;; esac -rm -f -- "$out"/vault-* "$out"/SHA256SUMS "$out"/*.log +rm -f -- "$out"/vault-* "$out"/SHA256SUMS export CARGO_TARGET_DIR_BASE=$root/target/build-sh export RUSTFLAGS="${RUSTFLAGS:-} --remap-path-prefix=$root=vault --remap-path-prefix=${CARGO_HOME:-$HOME/.cargo}=cargo" -export OUT_DIR=$out JOBS_N=$jobs HOST_OS=$(uname -s) +export OUT_DIR=$out JOBS_N=$jobs HOST_OS=$(uname -s) root +logs=$root/target/build-sh-logs +mkdir -p -- "$logs" +export logs build_one() { local want=$1 row os triple method name ext="" tdir bin @@ -115,15 +118,14 @@ build_one() { # zig's musl getrandom() never reaches the kernel on riscv64 (it spins); # make the getrandom crate issue the system call itself. [[ $triple == riscv64*-musl ]] && flags+=' --cfg getrandom_backend="linux_raw"' - if ! RUSTFLAGS=$flags CARGO_TARGET_DIR=$tdir "${cmd[@]}" --release --locked --target "$triple" >"$OUT_DIR/$name.log" 2>&1; then + if ! RUSTFLAGS=$flags CARGO_TARGET_DIR=$tdir "${cmd[@]}" --release --locked --target "$triple" >"$logs/$name.log" 2>&1; then echo "FAILED $want ($triple)" >&2 - tail -20 "$OUT_DIR/$name.log" >&2 + tail -20 "$logs/$name.log" >&2 return 1 fi bin=$tdir/$triple/release/vault$ext [[ $os == wasip1 ]] && bin=$tdir/$triple/release/vault.wasm cp -- "$bin" "$OUT_DIR/$name" - rm -f -- "$OUT_DIR/$name.log" if [[ $triple == *-musl* ]] && command -v file >/dev/null; then file -L "$OUT_DIR/$name" | grep -q 'static' || { echo "FAILED $want: not statically linked" >&2; return 1; } fi diff --git a/docs/MIGRATION.md b/docs/MIGRATION.md new file mode 100644 index 0000000..f547fa2 --- /dev/null +++ b/docs/MIGRATION.md @@ -0,0 +1,95 @@ +# Migrating from the Go implementation + +vault 0.6 is a rewrite in Rust of the Go vault (0.5 and earlier). It is an +implementation migration only: key files, vaults, the lock and the unlock +agent are the same formats, byte for byte where the format is +deterministic (see `COMPATIBILITY.md`). There is nothing to convert. Go and +Rust builds can be used on the same vault, even at the same time: they +honour each other's locks, and either can use an agent the other started. + +To go back, install 0.5: everything 0.6 writes, 0.5 reads. + +## Intentional behaviour changes + +Safety and correctness: + +- **`get` never replaces anything.** Output is published with an atomic + no-replace rename or link, so a file, a symlink (even a dangling one) or a + file created while `get` runs is left alone. Go checked first and then + renamed, which a concurrent creator could race. +- **`get -r` does not follow symlinks inside the output directory.** Each + directory below the chosen one is opened without following links; a link + or non-directory in the way stops the run. (Windows: checked just before + use, see `SECURITY.md`.) +- **Extracted files are synced** to disk before they appear under their + name, and their directory afterwards. +- **A source file that changes while being added is not stored.** If it + grows, shrinks, is modified or replaced during `add`/`migrate`, the + command fails and nothing is stored. Go stored whatever it read (for a + file that grew, a truncated copy). +- **`migrate` deletes an original only if it is still the file stored** + (same inode, size and time), after the record is durable. +- **Deletes are durable in order.** Removing a file syncs the removal of its + record before deleting its objects, so a crash cannot leave a record + pointing at deleted objects. A failed sync after a record is already in + place no longer deletes its objects. +- **Lock errors are reported.** Go let a reader continue without the lock on + any error; Rust does so only on a read-only filesystem. +- **Stricter records.** A record must have every field, with exactly the + field names Go writes, and nothing after the JSON. Go also accepted + differently-cased names, missing fields (as zero) and trailing bytes; + Go never writes such records, so no existing vault is affected. +- **Key files over 64 KiB** are rejected without reading them whole. + +Agent (`unlock`): + +- The agent **refuses the key if it cannot protect it**: locking its memory, + disabling core dumps and (Linux) becoming undumpable must all succeed. Go + ignored failures there. +- Its environment is stripped of `VAULT_PASSPHRASE` and + `VAULT_DECOY_PASSPHRASE`; it runs in `/` with standard error closed. +- Expiry no longer waits for the request in progress beyond its deadline, + and a key is never handed out after expiry. +- `unlock` waits at most ten seconds for the agent to start and kills it on + failure; the agent removes its socket only if it is still its own. +- **Windows: `unlock` is not available** (Go's agent there had no memory + protection). Every command asks for the passphrase. + +Network (`add URL`, `mirror`): + +- Explicit limits, all adjustable: connect 30 s, response headers 60 s, + whole request 2 h (`add`) or 5 min (`mirror`, per request), download size + 16 GiB (`add`) or 4 GiB in total (`mirror`), 10 redirects, 10 000 mirror + pages, 16 MiB per HTML or CSS page. Bytes are counted after decompression. + New flags `--max-bytes`, `--timeout`, `--max-pages`. Go had no limits. +- Redirects go to http or https only; a `mirror` redirect to another host is + skipped (Go followed it). +- `mirror` resolves links against the final address after redirects and the + page's ``; it parses HTML instead of matching a pattern, so it + rewrites only `href`, `src`, `action` and `poster` on real tags, plus + `url(...)` in CSS, `