Docs for the Rust version; benchmarks; build.sh fixes

README, CLAUDE.md, man pages, the user guide and its screenshot script
now describe the Rust build and the intentional behaviour changes, which
docs/MIGRATION.md lists. Go and Rust micro-benchmarks for the same
operations, and scripts/bench.sh for end-to-end comparison. build.sh
exports its tree root to the per-target jobs and keeps logs out of the
output directory; riscv64 musl builds use getrandom's raw syscall
backend, since zig's musl getrandom never reaches the kernel there.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Andy Dixon
2026-10-08 11:50:55 +01:00
co-authored by Claude
parent cddde5ca52
commit 53968f8822
18 changed files with 659 additions and 109 deletions
+65 -43
View File
@@ -4,71 +4,93 @@ This file provides guidance to Claude Code when working with this repository.
## Overview ## Overview
vault is a post-quantum encrypted file vault: one `package main` at the repo vault is a post-quantum encrypted file vault: one Rust package (`Cargo.toml`
root that builds a single `vault` binary. Module `github.com/andydixon/vault`. at the root, toolchain pinned in `rust-toolchain.toml`) that builds a single
`vault` binary. The library (`src/lib.rs`) exists so tests and benchmarks can
reach the same code; `src/main.rs` only calls `cli::main`.
- `main.go` - CLI: `genkey`, `init`, `add` (file, directory - `src/cli.rs` - commands and Go-`flag`-compatible parsing: `genkey`,
or URL), `migrate` (add, then delete originals), `mirror`, `absorb` (tar `init`, `add` (file, directory or URL), `migrate`, `mirror`, `absorb`,
on stdin, `absorb.go`), `ls` (one folder level), `list` (recursive), `ls`/`list`, `get` (`-r`), `mv`/`cp` (`transfers` resolves names like
`get` (`-r` for a folder), `mv`/`cp` (files or folders; `transfers` mv(1)), `rm` (`-r`, asks unless `-f`), `unlock`/`lock`, hidden `__agent`.
resolves names like mv(1), copies share objects), `rm` (`-r` for a - `src/agent.rs` - the unlock agent: `unlock` re-executes the binary as
folder, asks unless `-f`), `unlock`/`lock`. `__agent` (passphrase variables stripped, secret over a pipe); it hardens
- `agent*.go` - `vault unlock` re-executes itself as a hidden `__agent` itself (`platform::memory`) before reading the key, serves it on a Unix
process holding the suite ID and seed (pipe in, mlocked, undumpable) and socket in a 0700 per-user dir until a deadline (fixed or `--idle`).
serving them on a Unix socket in a 0700 per-user dir until a deadline Wire-compatible with the Go agent. Not available on Windows.
(fixed, or with `--idle` reset on each use). `keyIdentity` asks it first - `src/keyfile.rs` - key file: two Argon2id/XChaCha20-Poly1305 slots, decoy
unless `$VAULT_PASSPHRASE` is set. or random bytes in the second, random order, both always tried.
- `keyfile.go` - key file with two passphrase slots (Argon2id + - `src/crypto/` - `frodo.rs` (FrodoKEM-640-SHAKE ported from CIRCL v1.6.4),
XChaCha20-Poly1305 sealing a suite ID and 32-byte master seed). The second `kem.rs` (hybrid suite 0x0001 with RustCrypto ml-kem, `Identity`),
slot is either a decoy identity or random bytes; slot order is random. `header.rs` (Fury header format), `fury.rs` (file encryption), `stream.rs`
- `store.go` - on-disk vault: `records/` holds fixed-size encrypted catalogue (STREAM, including random access).
records, `objects/` holds file bodies cut into 4 MiB chunks (last one - `src/store/` - on-disk vault: `records/`, `objects/` sharded by ID prefix,
bucket-padded) whose IDs are HMAC'd from a per-record seed. Both are catalogue, add/remove/copy/extract, filler; `types.rs` holds the validated
sharded into 256 `xx/` subdirectories by ID prefix. Records a key cannot types (`VaultName`, `Id`, `ObjectSeed`, `Record`) and record JSON.
open are skipped, not treated as errors. - `src/pipeline.rs` - the one bounded worker pool and the eight reusable
- `lock*.go` - vault lock on `DIR/.lock`: fcntl on unix, LockFileEx on object buffers; coordinators run on the caller's thread.
Windows, an exclusive lock file elsewhere. Writers lock exclusively, - `src/platform/` - all `unsafe` and OS specifics: `fs.rs` (no-clobber
readers shared, taken after the passphrase prompt. publish, `Dir` confined extraction, epoch times, dir sync), `lock.rs`
- `docs/guide` - LaTeX user guide (`build-pdf.sh` via Tectonic; `shots.py` (fcntl / LockFileEx / lock file), `memory.rs` (mlock, undumpable),
re-shoots screenshots from the real binary). Keep it in step with the CLI. `term.rs` (prompts).
- `packaging/repo.sh [TAG]` - builds dixon-vault .deb/.rpm/Arch packages - `src/net/` - `add URL` and `mirror` with limits; `url.rs` (Go `net/url`
from a tag with nfpm and publishes them, signed, to the apt/dnf/pacman semantics), `html.rs` (tokenizer and link rewriting).
repo at repo.dixon.cx (same script, other project block, in ../mapsize). - `src/absorb.rs` - tar (optionally gzipped) from stdin, streamed in.
- `build.sh` - cross-compiles every `go tool dist list` target except - `src/secret.rs`, `src/rng.rs`, `src/util.rs`, `src/error.rs` - secret
iOS and android/386, amd64, arm (cgo-only). buffers, randomness (seeded only under `cfg(test)`), Go-compatible
- `internal/pqcrypt` - file encryption (header + STREAM ChaCha20-Poly1305), parsing/printing helpers, the error type.
suite registry, hybrid FrodoKEM-640 + ML-KEM-768 default suite. - `oracle/go` - the frozen Go implementation (test-only reference, see
- `internal/format`, `internal/stream`, `internal/term` - header encoding, `oracle/README.md`); `oracle/gen-vectors.sh` writes `tests/vectors`.
chunked payload encryption, terminal passphrase prompts. - `tests/` - CLI and interoperability tests against the real binaries;
`fuzz/` - cargo-fuzz targets; `benches/micro.rs` - micro-benchmarks.
- `docs/` - man pages, `COMPATIBILITY.md`, `MIGRATION.md`, `PLATFORMS.md`,
`BENCHMARKS.md`, and the LaTeX guide (`docs/guide`, `build-pdf.sh` via
Tectonic; `shots.py` re-shoots screenshots from the real binary). Keep
them in step with the CLI.
- `build.sh` - cross-builds every supported target (see `PLATFORMS.md`).
- `packaging/package.sh` builds and inspects dixon-vault packages locally;
`packaging/repo.sh [TAG]` signs and publishes them to repo.dixon.cx.
## Commands ## Commands
```sh ```sh
go test ./... cargo test --locked # needs go on PATH for interop tests
go build . cargo clippy --locked --all-targets --all-features -- -D warnings
./build.sh # all platforms, into ./dist cargo fmt --check
cargo build --release --locked
./build.sh # all buildable targets, into ./dist
TARGETS="linux/amd64" ./build.sh # a subset TARGETS="linux/amd64" ./build.sh # a subset
cargo deny --locked check # advisories and licences
(cd fuzz && cargo +nightly fuzz run header)
``` ```
End-to-end check: End-to-end check:
```sh ```sh
export VAULT_PASSPHRASE=test VAULT_KEY=$PWD/k VAULT_DIR=$PWD/s export VAULT_PASSPHRASE=test VAULT_KEY=$PWD/k VAULT_DIR=$PWD/s
./vault genkey && ./vault init && ./vault add README.md docs && ./vault list target/release/vault genkey && target/release/vault init && target/release/vault add README.md docs && target/release/vault list
``` ```
## Invariants ## Invariants
- The on-disk formats are frozen and shared with the Go version; check any
change against `docs/COMPATIBILITY.md` and the oracle vectors.
- Never write a seed or private key unencrypted (the agent gets it by pipe). - Never write a seed or private key unencrypted (the agent gets it by pipe).
- Keep suite IDs, stanza tags, HKDF info strings and `fileKeySize` (32) stable. - Keep suite IDs, stanza tags, HKDF info strings and the 32-byte file key stable.
- Every ciphertext's and shard directory's mtime is pinned to the epoch; - Every ciphertext's and shard directory's mtime is pinned to the epoch;
records stay one fixed size; objects stay at most 4 MiB and bucket-padded. records stay one fixed size; objects stay at most 4 MiB and bucket-padded.
These hide metadata from the storage host. These hide metadata from the storage host.
- An object's plaintext starts with its own ID, binding it to its record - An object's plaintext starts with its own ID, binding it to its record
and its position in the file. and its position in the file.
- `get` publishes output only after every object has authenticated. - `get` publishes output only after every object has authenticated, with a
no-replace primitive; never fall back to check-then-rename.
- Decoy deniability: both key slots stay the same size and are both always - Decoy deniability: both key slots stay the same size and are both always
tried; `init` always writes filler; foreign records are skipped silently. tried; `init` always writes filler; foreign records are skipped silently.
- Every command that changes the vault holds the exclusive lock. - Every command that changes the vault holds the exclusive lock (POSIX
- Keep dependencies pure Go and CGO_ENABLED=0 builds working. fcntl locks, compatible with Go's).
- Crash-safe order: objects before records, new records before removing old
ones, record removal (synced) before object removal.
- Secrets live in `SecretArray`/`SecretVec`/`LockedBuf`: no Clone, Debug or
serialisation. Production randomness is the OS only.
- `unsafe` only in `src/platform`, each block with a SAFETY comment.
- Diagnostics go to standard error, lower-case, no trailing full stops. - Diagnostics go to standard error, lower-case, no trailing full stops.
+4
View File
@@ -40,3 +40,7 @@ windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Storage
codegen-units = 1 codegen-units = 1
lto = true lto = true
strip = true strip = true
[[bench]]
name = "micro"
harness = false
+37 -13
View File
@@ -23,16 +23,36 @@ Releases are packaged and published with `packaging/repo.sh [TAG]`.
## Build ## Build
vault is written in Rust (the toolchain is pinned in `rust-toolchain.toml`;
`rustup` installs it on first use). Releases up to 0.5 were Go; the Go code
lives on, frozen, in `oracle/go` as the test reference. Nothing at run time
needs Go.
```sh ```sh
go build . # produces ./vault cargo build --release --locked # produces target/release/vault
go test ./... cargo test --locked # unit, CLI and Go-interoperability tests
./build.sh # every OS and CPU architecture, into ./dist ./build.sh # every target this host can build, into ./dist
TARGETS="linux/amd64 windows/amd64" ./build.sh
``` ```
`build.sh` cross-compiles static binaries for every target in `build.sh` cross-compiles with `cargo-zigbuild` and `zig` (Linux, statically
`go tool dist list` except iOS and 32-bit or x86 Android, which need cgo. It builds 32-bit ARM Linux for linked with musl, and Windows), on a Mac for macOS, and with `cross` and
both armv6 and armv7, and writes `dist/SHA256SUMS`. Set `TARGETS` to build a Docker for the BSDs, illumos and Solaris. It names binaries as before
subset, for example `TARGETS="linux/amd64 darwin/arm64" ./build.sh`. (`vault-linux-arm64`, `vault-windows-amd64.exe`, …), builds 32-bit ARM Linux
for both armv6 and armv7, and writes `dist/SHA256SUMS`. `docs/PLATFORMS.md`
lists every target, how it is built and tested, and the Go targets Rust
cannot build.
The interoperability tests build the Go oracle when `go` is installed and
are skipped (and say so) otherwise; set `VAULT_REQUIRE_ORACLE=1` to make a
missing oracle an error, as CI does. `oracle/gen-vectors.sh` regenerates the
Go test vectors. Packages are built and inspected locally, unsigned, with
`packaging/package.sh`, and published with `packaging/repo.sh [TAG]`.
Further reading: `SECURITY.md` (what is and is not protected, memory and
filesystem guarantees), `docs/COMPATIBILITY.md` (the formats),
`docs/MIGRATION.md` (what changed from the Go version) and
`docs/BENCHMARKS.md`.
## User guide ## User guide
@@ -82,12 +102,15 @@ vault lock # forget it now
Commands that change the vault take an exclusive lock on `DIR/.lock`, and Commands that change the vault take an exclusive lock on `DIR/.lock`, and
commands that read it take a shared one. A second process waits, and says commands that read it take a shared one. A second process waits, and says
so, until the first finishes. The operating system releases the lock if a so, until the first finishes. The operating system releases the lock if a
process dies. Plan 9 and WebAssembly have no file locks, so there the lock process dies; the locks are the same ones the Go version used, so old and
file itself is the lock. After a crash on those platforms you remove it by new versions exclude each other. WebAssembly (WASI) has no file locks, so
hand; the error message names it. there the lock file itself is the lock. After a crash there you remove it
by hand; the error message names it.
Adding a name that already exists replaces it. `get` refuses to overwrite an Adding a name that already exists replaces it. `get` never replaces an
existing file and writes nothing until the whole object has authenticated. existing file (or symlink) and writes nothing until the whole file has
authenticated. `add URL` and `mirror` have size and time limits; see
`vault(1)`.
`--key PATH` and `--vault DIR` work with every command. They default to `--key PATH` and `--vault DIR` work with every command. They default to
`$VAULT_KEY` and `$VAULT_DIR`, then to `~/.vault/key` and `~/.vault/store`. `$VAULT_KEY` and `$VAULT_DIR`, then to `~/.vault/key` and `~/.vault/store`.
@@ -103,7 +126,8 @@ over a pipe and never touches the disk; the helper's memory is locked
against swapping, it cannot be core-dumped (or, on Linux, debugged), and it against swapping, it cannot be core-dumped (or, on Linux, debugged), and it
answers only on a socket in a directory private to your user. While it is answers only on a socket in a directory private to your user. While it is
unlocked, anything running as you can use the vault, decoy or not, so lock unlocked, anything running as you can use the vault, decoy or not, so lock
before handing the machine over. before handing the machine over. If the helper cannot protect its memory it
refuses the key. `unlock` is not available on Windows.
## Decoy passphrase ## Decoy passphrase
+79
View File
@@ -0,0 +1,79 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! The Rust side of scripts/bench.sh: the same operations as the Go
//! oracle's Benchmark* functions, timed as the median of repeated runs.
//! Run with `cargo bench --bench micro`.
use std::hint::black_box;
use std::io::{Read, Write};
use std::time::{Duration, Instant};
use vault::crypto::{fury, kem, stream};
fn pattern(n: usize, salt: u8) -> Vec<u8> {
(0..n).map(|i| (i.wrapping_mul(7) as u8) ^ salt).collect()
}
/// Runs `f` until about two seconds have passed (at least 5 times) and
/// prints the median and the spread.
fn bench(name: &str, bytes: usize, mut f: impl FnMut()) {
f(); // warm up
let mut times = Vec::new();
let start = Instant::now();
while times.len() < 5 || (start.elapsed() < Duration::from_secs(2) && times.len() < 10_000) {
let t = Instant::now();
f();
times.push(t.elapsed());
}
times.sort();
let med = times[times.len() / 2];
let (lo, hi) = (times[times.len() / 10], times[times.len() * 9 / 10]);
let rate = if bytes > 0 {
format!(" {:.0} MB/s", bytes as f64 / med.as_secs_f64() / 1e6)
} else {
String::new()
};
println!(
"{name:<24} median {:>10.3} ms (p10 {:.3}, p90 {:.3}, n={}){rate}",
med.as_secs_f64() * 1e3,
lo.as_secs_f64() * 1e3,
hi.as_secs_f64() * 1e3,
times.len()
);
}
fn main() {
let seed = pattern(32, 1);
bench("derive", 0, || {
black_box(kem::Identity::from_suite_seed(kem::SUITE_ID, &seed).unwrap());
});
let id = kem::Identity::from_suite_seed(kem::SUITE_ID, &seed).unwrap();
let r = id.recipient();
let fk: [u8; 32] = pattern(32, 2).try_into().unwrap();
bench("wrap", 0, || {
black_box(r.wrap(&fk, &mut vault::rng::os()));
});
let st = r.wrap(&fk, &mut vault::rng::os());
bench("unwrap", 0, || {
black_box(id.unwrap(std::slice::from_ref(&st)).unwrap());
});
let pt = pattern(4 << 20, 3);
bench("object 4MiB", pt.len(), || {
let mut ct = Vec::new();
fury::encrypt_to(&mut ct, &r, &pt, &mut vault::rng::os());
black_box(fury::decrypt_bounded(&ct[..], &id, pt.len()).unwrap());
});
let key = [0u8; 32];
let big = vec![0u8; 64 << 20];
bench("stream 64MiB", big.len(), || {
let mut w = stream::StreamWriter::new(&key, Vec::with_capacity(big.len() + (1 << 16)));
w.write_all(&big).unwrap();
let ct = w.finish().unwrap();
let mut out = Vec::with_capacity(big.len());
stream::StreamReader::new(&key, &ct[..]).read_to_end(&mut out).unwrap();
black_box(out);
});
let body = vault::keyfile::new_body(b"bench", b"", &mut vault::rng::os());
bench("unlock", 0, || {
black_box(vault::keyfile::unlock_body(&body, b"bench").unwrap());
});
}
+7 -5
View File
@@ -73,11 +73,14 @@ out=$(cd -- "$out" && pwd -P)
case $out in case $out in
/ | "$(cd ~ && pwd -P)" | "$root") echo "refusing OUT=$out: use a dedicated directory" >&2; exit 2 ;; / | "$(cd ~ && pwd -P)" | "$root") echo "refusing OUT=$out: use a dedicated directory" >&2; exit 2 ;;
esac esac
rm -f -- "$out"/vault-* "$out"/SHA256SUMS "$out"/*.log rm -f -- "$out"/vault-* "$out"/SHA256SUMS
export CARGO_TARGET_DIR_BASE=$root/target/build-sh export CARGO_TARGET_DIR_BASE=$root/target/build-sh
export RUSTFLAGS="${RUSTFLAGS:-} --remap-path-prefix=$root=vault --remap-path-prefix=${CARGO_HOME:-$HOME/.cargo}=cargo" export RUSTFLAGS="${RUSTFLAGS:-} --remap-path-prefix=$root=vault --remap-path-prefix=${CARGO_HOME:-$HOME/.cargo}=cargo"
export OUT_DIR=$out JOBS_N=$jobs HOST_OS=$(uname -s) export OUT_DIR=$out JOBS_N=$jobs HOST_OS=$(uname -s) root
logs=$root/target/build-sh-logs
mkdir -p -- "$logs"
export logs
build_one() { build_one() {
local want=$1 row os triple method name ext="" tdir bin local want=$1 row os triple method name ext="" tdir bin
@@ -115,15 +118,14 @@ build_one() {
# zig's musl getrandom() never reaches the kernel on riscv64 (it spins); # zig's musl getrandom() never reaches the kernel on riscv64 (it spins);
# make the getrandom crate issue the system call itself. # make the getrandom crate issue the system call itself.
[[ $triple == riscv64*-musl ]] && flags+=' --cfg getrandom_backend="linux_raw"' [[ $triple == riscv64*-musl ]] && flags+=' --cfg getrandom_backend="linux_raw"'
if ! RUSTFLAGS=$flags CARGO_TARGET_DIR=$tdir "${cmd[@]}" --release --locked --target "$triple" >"$OUT_DIR/$name.log" 2>&1; then if ! RUSTFLAGS=$flags CARGO_TARGET_DIR=$tdir "${cmd[@]}" --release --locked --target "$triple" >"$logs/$name.log" 2>&1; then
echo "FAILED $want ($triple)" >&2 echo "FAILED $want ($triple)" >&2
tail -20 "$OUT_DIR/$name.log" >&2 tail -20 "$logs/$name.log" >&2
return 1 return 1
fi fi
bin=$tdir/$triple/release/vault$ext bin=$tdir/$triple/release/vault$ext
[[ $os == wasip1 ]] && bin=$tdir/$triple/release/vault.wasm [[ $os == wasip1 ]] && bin=$tdir/$triple/release/vault.wasm
cp -- "$bin" "$OUT_DIR/$name" cp -- "$bin" "$OUT_DIR/$name"
rm -f -- "$OUT_DIR/$name.log"
if [[ $triple == *-musl* ]] && command -v file >/dev/null; then if [[ $triple == *-musl* ]] && command -v file >/dev/null; then
file -L "$OUT_DIR/$name" | grep -q 'static' || { echo "FAILED $want: not statically linked" >&2; return 1; } file -L "$OUT_DIR/$name" | grep -q 'static' || { echo "FAILED $want: not statically linked" >&2; return 1; }
fi fi
+95
View File
@@ -0,0 +1,95 @@
# Migrating from the Go implementation
vault 0.6 is a rewrite in Rust of the Go vault (0.5 and earlier). It is an
implementation migration only: key files, vaults, the lock and the unlock
agent are the same formats, byte for byte where the format is
deterministic (see `COMPATIBILITY.md`). There is nothing to convert. Go and
Rust builds can be used on the same vault, even at the same time: they
honour each other's locks, and either can use an agent the other started.
To go back, install 0.5: everything 0.6 writes, 0.5 reads.
## Intentional behaviour changes
Safety and correctness:
- **`get` never replaces anything.** Output is published with an atomic
no-replace rename or link, so a file, a symlink (even a dangling one) or a
file created while `get` runs is left alone. Go checked first and then
renamed, which a concurrent creator could race.
- **`get -r` does not follow symlinks inside the output directory.** Each
directory below the chosen one is opened without following links; a link
or non-directory in the way stops the run. (Windows: checked just before
use, see `SECURITY.md`.)
- **Extracted files are synced** to disk before they appear under their
name, and their directory afterwards.
- **A source file that changes while being added is not stored.** If it
grows, shrinks, is modified or replaced during `add`/`migrate`, the
command fails and nothing is stored. Go stored whatever it read (for a
file that grew, a truncated copy).
- **`migrate` deletes an original only if it is still the file stored**
(same inode, size and time), after the record is durable.
- **Deletes are durable in order.** Removing a file syncs the removal of its
record before deleting its objects, so a crash cannot leave a record
pointing at deleted objects. A failed sync after a record is already in
place no longer deletes its objects.
- **Lock errors are reported.** Go let a reader continue without the lock on
any error; Rust does so only on a read-only filesystem.
- **Stricter records.** A record must have every field, with exactly the
field names Go writes, and nothing after the JSON. Go also accepted
differently-cased names, missing fields (as zero) and trailing bytes;
Go never writes such records, so no existing vault is affected.
- **Key files over 64 KiB** are rejected without reading them whole.
Agent (`unlock`):
- The agent **refuses the key if it cannot protect it**: locking its memory,
disabling core dumps and (Linux) becoming undumpable must all succeed. Go
ignored failures there.
- Its environment is stripped of `VAULT_PASSPHRASE` and
`VAULT_DECOY_PASSPHRASE`; it runs in `/` with standard error closed.
- Expiry no longer waits for the request in progress beyond its deadline,
and a key is never handed out after expiry.
- `unlock` waits at most ten seconds for the agent to start and kills it on
failure; the agent removes its socket only if it is still its own.
- **Windows: `unlock` is not available** (Go's agent there had no memory
protection). Every command asks for the passphrase.
Network (`add URL`, `mirror`):
- Explicit limits, all adjustable: connect 30 s, response headers 60 s,
whole request 2 h (`add`) or 5 min (`mirror`, per request), download size
16 GiB (`add`) or 4 GiB in total (`mirror`), 10 redirects, 10 000 mirror
pages, 16 MiB per HTML or CSS page. Bytes are counted after decompression.
New flags `--max-bytes`, `--timeout`, `--max-pages`. Go had no limits.
- Redirects go to http or https only; a `mirror` redirect to another host is
skipped (Go followed it).
- `mirror` resolves links against the final address after redirects and the
page's `<base href>`; it parses HTML instead of matching a pattern, so it
rewrites only `href`, `src`, `action` and `poster` on real tags, plus
`url(...)` in CSS, `<style>` and `style=`, keeping every other byte and
the original quoting. Text in scripts and comments is no longer touched.
- `mirror` never lets two different URLs in one crawl overwrite each other's
file; the second is skipped. It exits with status **3** when it stored
some resources but skipped others (Go always exited 0).
- Proxies follow `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` as in Go; local
addresses are never proxied.
Command line:
- New option `--workers N` (1–256) on every command.
- `--key`/`--vault` no longer need a home directory when given explicitly.
- Command-line arguments must be valid UTF-8 (Go accepted any bytes, but
vault names had to be UTF-8 already).
- `ls`/`list` piped into a reader that stops early end quietly with
status 141, as Go's did by `SIGPIPE`.
Platforms: see `PLATFORMS.md`. Some targets Go built have no stable Rust
equivalent; they are listed there and need a decision before release.
## Unchanged on purpose
Formats, limits (1024-byte names, 1 TiB files, 4 MiB objects, 4096-byte
records, 16 MiB headers, 1 MiB stanzas), Argon2id parameters, the hybrid
suite, filler, decoy behaviour, messages and exit statuses (apart from the
above), the shape of `ls` and `list` output, and the agent protocol.
+6 -3
View File
@@ -7,7 +7,9 @@ sandbox path and file owner are shown as /home/sam and "sam sam", key lines
are highlighted in amber for the reader, and "moment" frames show a prompt are highlighted in amber for the reader, and "moment" frames show a prompt
with a cursor, exactly as vault prints it before you type. with a cursor, exactly as vault prints it before you type.
Needs: go, python3, and Chrome or Chromium (set CHROME=/path if not found). Needs: cargo (with the pinned toolchain), cargo-zigbuild and zig for the
build screenshot, python3, and Chrome or Chromium (set CHROME=/path if not
found).
Output: docs/guide/img/*.png Output: docs/guide/img/*.png
""" """
@@ -75,7 +77,8 @@ class Sandbox:
self.bin = os.path.join(self.root, "bin") self.bin = os.path.join(self.root, "bin")
os.makedirs(self.home) os.makedirs(self.home)
os.makedirs(self.bin) os.makedirs(self.bin)
subprocess.run(["go", "build", "-o", os.path.join(self.bin, "vault"), "."], cwd=REPO, check=True) subprocess.run(["cargo", "build", "--release", "--locked", "--quiet"], cwd=REPO, check=True)
shutil.copy2(os.path.join(REPO, "target", "release", "vault"), os.path.join(self.bin, "vault"))
def env(self, extra=None): def env(self, extra=None):
e = { e = {
@@ -406,7 +409,7 @@ def main():
# building, into a temporary OUT so an existing dist/ is left alone # building, into a temporary OUT so an existing dist/ is left alone
rel = tempfile.mkdtemp(prefix="vault-release-") rel = tempfile.mkdtemp(prefix="vault-release-")
targets = "linux/amd64 linux/arm darwin/arm64 windows/amd64" targets = "linux/amd64 linux/arm64 linux/armv7 windows/amd64"
t = Term(box, title="sam@laptop: ~/src/vault") t = Term(box, title="sam@laptop: ~/src/vault")
out = subprocess.run(["./build.sh"], cwd=REPO, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, out = subprocess.run(["./build.sh"], cwd=REPO, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
text=True, env=dict(os.environ, TARGETS=targets, OUT=rel)).stdout text=True, env=dict(os.environ, TARGETS=targets, OUT=rel)).stdout
+36 -17
View File
@@ -222,14 +222,17 @@ Linux on ARM64 & Pi 4/5 (64-bit OS), Graviton, Ampere & vault-linux-arm64 \\
Raspberry Pi, 32-bit & Pi 2, 3, 4 on a 32-bit OS & vault-linux-armv7 \\ Raspberry Pi, 32-bit & Pi 2, 3, 4 on a 32-bit OS & vault-linux-armv7 \\
Raspberry Pi Zero / 1 & the tiny older ones & vault-linux-armv6 \\ Raspberry Pi Zero / 1 & the tiny older ones & vault-linux-armv6 \\
Android & in Termux & vault-android-arm64 \\ Android & in Termux & vault-android-arm64 \\
FreeBSD, OpenBSD, NetBSD & amd64, arm64, 386, arm\ldots & vault-freebsd-amd64 etc. \\ FreeBSD, NetBSD & amd64 (FreeBSD also 386) & vault-freebsd-amd64 etc. \\
Something exotic & RISC-V, POWER, s390x, MIPS, LoongArch & see \cmd{dist/} \\ Something exotic & RISC-V, POWER, s390x, LoongArch, illumos, Solaris & see \cmd{dist/} \\
\end{vtable} \end{vtable}
\begin{squirrel} \begin{squirrel}
The full release covers 43 platform builds, including AIX, Solaris, illumos, The release covers some twenty platforms, from Windows on ARM to IBM
Plan~9 and even WebAssembly. Will you ever need \file{vault-plan9-arm}? mainframes, and even WebAssembly. A few that the older Go builds reached,
Almost certainly not. Is it pleasing that it exists? Extremely. such as Plan~9, OpenBSD and MIPS, have no stable Rust toolchain yet;
\file{docs/PLATFORMS.md} keeps the list. Will you ever need
\file{vault-linux-s390x}? Almost certainly not. Is it pleasing that it
exists? Extremely.
\end{squirrel} \end{squirrel}
\begin{headsup} \begin{headsup}
@@ -299,24 +302,29 @@ Run \cmd{vault help}. You should see the full command list:
\section{Building it yourself} \section{Building it yourself}
Prefer to build from source? You need Go 1.26.6 or newer. Then: Prefer to build from source? You need Rust, through \cmd{rustup}, which
fetches the exact compiler version the project pins. Then:
\begin{shell} \begin{shell}
$ go build . # just your platform, produces ./vault $ cargo build --release --locked # just your platform: target/release/vault
$ ./build.sh # every platform, into ./dist, with SHA256SUMS $ ./build.sh # every platform it can, into ./dist, with SHA256SUMS
\end{shell} \end{shell}
\cmd{build.sh} cross-compiles every target Go supports without C libraries. \cmd{build.sh} cross-compiles Linux and Windows with \cmd{cargo-zigbuild}
and \cmd{zig}, macOS on a Mac, and the BSDs, illumos and Solaris with
\cmd{cross} and Docker; it says which targets your machine can't build.
32-bit ARM Linux gets two builds, armv6 and armv7. Want just a few? Set 32-bit ARM Linux gets two builds, armv6 and armv7. Want just a few? Set
\env{TARGETS}. Want them somewhere else? Set \env{OUT}. \env{TARGETS}. Want them somewhere else? Set \env{OUT}.
\shot{build}{A partial build, followed by checking every binary against SHA256SUMS} \shot{build}{A partial build, followed by checking every binary against SHA256SUMS}
\begin{whyso} \begin{whyso}
The binaries are \emph{static} and \emph{reproducible}: no C code, no system The Linux binaries are \emph{static}: they need no system libraries at all.
libraries, file paths stripped, build IDs zeroed. Two people building the same Every build uses the pinned compiler and the locked versions of every
commit get byte-identical files, so you can check a release against your own dependency, with build paths stripped, so the result doesn't depend on whose
build. machine or which folder built it. Built with the same compiler for the same
target, the same commit should give the same file, so you can check a
release against your own build.
\end{whyso} \end{whyso}
\begin{pocket} \begin{pocket}
@@ -324,7 +332,7 @@ build.
\item One file per platform. Rename it \cmd{vault} and put it on your \env{PATH}. \item One file per platform. Rename it \cmd{vault} and put it on your \env{PATH}.
\item Always check the SHA-256 against \file{SHA256SUMS}. \item Always check the SHA-256 against \file{SHA256SUMS}.
\item macOS: clear the quarantine flag after checking. \item macOS: clear the quarantine flag after checking.
\item \cmd{./build.sh} builds all 43 platform binaries from source. \item \cmd{./build.sh} builds every platform your machine can, from source.
\end{pocketlist} \end{pocketlist}
\end{pocket} \end{pocket}
@@ -1071,8 +1079,9 @@ who's slow at typing never blocks you.
\begin{whyso} \begin{whyso}
\textbf{What if vault crashes while holding the lock?} On Linux, macOS, the \textbf{What if vault crashes while holding the lock?} On Linux, macOS, the
BSDs and Windows, the operating system drops the lock the moment the process BSDs and Windows, the operating system drops the lock the moment the process
dies. Nothing to clean up. On Plan~9 and WebAssembly, which have no file dies. Nothing to clean up. Older Go builds of vault use the very same locks,
locks, vault uses a \file{.lock} file instead. If one's ever left behind, so mixing versions is safe. On WebAssembly, which has no file locks, vault
uses a \file{.lock} file instead. If one's ever left behind,
vault tells you after a minute exactly which file to delete. vault tells you after a minute exactly which file to delete.
\end{whyso} \end{whyso}
@@ -1326,7 +1335,17 @@ could not read passphrase (set VAULT\_PASSPHRASE\ldots) & No terminal to ask on,
agent did not start: \ldots & \cmd{unlock} couldn't start its helper. The reason follows; commands still work, asking for the passphrase. \\ agent did not start: \ldots & \cmd{unlock} couldn't start its helper. The reason follows; commands still work, asking for the passphrase. \\
nothing was unlocked & \cmd{vault lock} found nothing to lock. Not an error. \\ nothing was unlocked & \cmd{vault lock} found nothing to lock. Not an error. \\
waiting for another vault process to finish & Not an error. Another command is writing; this one continues when it's done. \\ waiting for another vault process to finish & Not an error. Another command is writing; this one continues when it's done. \\
vault still locked after a minute\ldots & Plan~9 and WebAssembly only. If nothing else is running, delete the file it names. \\ vault still locked after a minute\ldots & WebAssembly only. If nothing else is running, delete the file it names. \\
cannot lock the vault: \ldots & The lock file couldn't be opened or locked; the reason follows. Check the vault folder's permissions. \\
\ldots\ changed while being added & The file was being written while vault read it. Nothing was stored; try again when it's finished. \\
\ldots\ changed after it was stored; not deleting it & \cmd{migrate} stored the file, but it changed before it could be deleted. The original is kept; the vault has the version it read. \\
\ldots\ already exists & \cmd{get} never overwrites, not even a symlink. Pick another name with \cmd{-o}. \\
\ldots\ refusing to extract through it & \cmd{get -r} found a symlink (or a file) where a folder should be inside the output folder, and stopped rather than follow it. \\
cannot lock agent memory \ldots, cannot make the agent undumpable \ldots & \cmd{unlock}'s helper couldn't protect the key in memory, so it refused it. Commands still work, asking for the passphrase. \\
vault unlock is not supported on this platform\ldots & Windows: every command asks for the passphrase. \\
download exceeds \ldots\ bytes & A download passed its size limit. Raise it with \cmd{--max-bytes} if you meant it. \\
redirected to another host: \ldots & \cmd{mirror} only follows redirects within the site it is copying. \\
mirror incomplete: stored \ldots, skipped \ldots & Some pages failed; the reasons were printed as they happened. Exit status 3. \\
skipping unreadable record \ldots & A catalogue entry is damaged. Everything else still works; restore \file{records/} from backup to recover that file. \\ skipping unreadable record \ldots & A catalogue entry is damaged. Everything else still works; restore \file{records/} from backup to recover that file. \\
object does not match its catalogue record & The encrypted contents were swapped or damaged. Nothing was written. Restore \file{objects/} from backup. \\ object does not match its catalogue record & The encrypted contents were swapped or damaged. Nothing was written. Restore \file{objects/} from backup. \\
object size does not match its catalogue record & Same as above: damage or tampering, caught before any output appeared. \\ object size does not match its catalogue record & Same as above: damage or tampering, caught before any output appeared. \\
+21 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT-ADD 1 "September 2026" "vault" "User Commands" .TH VAULT-ADD 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault-add \- encrypt files, directories or URLs into the vault vault-add \- encrypt files, directories or URLs into the vault
.SH SYNOPSIS .SH SYNOPSIS
@@ -7,6 +7,10 @@ vault-add \- encrypt files, directories or URLs into the vault
.IR PATH ] .IR PATH ]
.RB [ \-\-vault .RB [ \-\-vault
.IR DIR ] .IR DIR ]
.RB [ \-\-max\-bytes
.IR SIZE ]
.RB [ \-\-timeout
.IR DURATION ]
.IR FILE | DIR | URL .IR FILE | DIR | URL
.RI [ FOLDER ] .RI [ FOLDER ]
.SH DESCRIPTION .SH DESCRIPTION
@@ -17,7 +21,14 @@ and the key are skipped. A stored name that already exists is replaced.
.PP .PP
An \fBhttp://\fR or \fBhttps://\fR \fIURL\fR is downloaded with a Chrome user An \fBhttp://\fR or \fBhttps://\fR \fIURL\fR is downloaded with a Chrome user
agent and stored under the last element of its final (post-redirect) path, or agent and stored under the last element of its final (post-redirect) path, or
the host name if there is none. Only a 200 response is accepted. the host name if there is none. Only a 200 response is accepted. Certificates
are always verified; at most 10 redirects are followed, to http or https
only. The download is staged in a private temporary file (it must be
complete before it can be stored), which is removed afterwards whatever
happens; it is plaintext on disk meanwhile.
.PP
A file that changes while it is being added (it grows, shrinks, or is
replaced) is not stored, and the command fails.
.PP .PP
\fIFOLDER\fR places the result inside a folder in the vault. File contents \fIFOLDER\fR places the result inside a folder in the vault. File contents
are cut into 4 MiB chunks, the last bucket-padded. are cut into 4 MiB chunks, the last bucket-padded.
@@ -28,6 +39,14 @@ Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP .TP
.BI \-\-vault " DIR" .BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store. Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.TP
.BI \-\-max\-bytes " SIZE"
Largest download accepted, counted after decompression, such as 500M or 50G.
Default 16G; at most 1T.
.TP
.BI \-\-timeout " DURATION"
Deadline for the whole download, such as 30m. Default 2h. Connecting must
take under 30 seconds and the response headers must arrive within a minute.
.SH ENVIRONMENT .SH ENVIRONMENT
.TP .TP
.B VAULT_PASSPHRASE .B VAULT_PASSPHRASE
+9 -4
View File
@@ -1,4 +1,4 @@
.TH VAULT-GET 1 "September 2026" "vault" "User Commands" .TH VAULT-GET 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault-get \- decrypt a stored file vault-get \- decrypt a stored file
.SH SYNOPSIS .SH SYNOPSIS
@@ -13,13 +13,18 @@ vault-get \- decrypt a stored file
.I NAME .I NAME
.SH DESCRIPTION .SH DESCRIPTION
Decrypts \fINAME\fR. Nothing appears at the output path until every object has Decrypts \fINAME\fR. Nothing appears at the output path until every object has
authenticated. The stored mode and modification time are restored. It refuses authenticated, padding and end of file included; until then the output is a
to overwrite an existing file. Takes a shared lock. private temporary file in the same directory, removed on failure. The stored
mode and modification time are restored. It never replaces anything already
at the output path, including a symbolic link (even a dangling one) or a
file created while it runs. Takes a shared lock.
.PP .PP
With \fB\-r\fR, \fINAME\fR is a folder, and every file under it is With \fB\-r\fR, \fINAME\fR is a folder, and every file under it is
decrypted into the output directory, keeping the folder structure. Each file decrypted into the output directory, keeping the folder structure. Each file
is checked as above; a clash stops the run, leaving the files already is checked as above; a clash stops the run, leaving the files already
written. written. Directories are created as needed, mode 0700; a symbolic link (or
anything but a directory) where a directory belongs inside the output
directory stops the run rather than being followed.
.SH OPTIONS .SH OPTIONS
.TP .TP
.BI \-o " OUTPUT" .BI \-o " OUTPUT"
+5 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT-MIGRATE 1 "September 2026" "vault" "User Commands" .TH VAULT-MIGRATE 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault-migrate \- move files into the vault, deleting the originals vault-migrate \- move files into the vault, deleting the originals
.SH SYNOPSIS .SH SYNOPSIS
@@ -13,7 +13,10 @@ vault-migrate \- move files into the vault, deleting the originals
.SH DESCRIPTION .SH DESCRIPTION
Works like Works like
.BR vault\-add (1) .BR vault\-add (1)
but deletes each original once it is stored, then any directories left empty. but deletes each original once it is stored (and its record is on disk),
then any directories left empty. An original is deleted only if it is still
the same file, unchanged, that was stored; otherwise it is kept and the
command fails.
Directories still holding skipped files stay. A URL cannot be migrated. Directories still holding skipped files stay. A URL cannot be migrated.
.PP .PP
Before storing anything it lists the vault names that would be overwritten and Before storing anything it lists the vault names that would be overwritten and
+60 -17
View File
@@ -1,4 +1,4 @@
.TH VAULT-MIRROR 1 "September 2026" "vault" "User Commands" .TH VAULT-MIRROR 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault-mirror \- mirror a website into an encrypted vault vault-mirror \- mirror a website into an encrypted vault
.SH SYNOPSIS .SH SYNOPSIS
@@ -7,6 +7,12 @@ vault-mirror \- mirror a website into an encrypted vault
.IR PATH ] .IR PATH ]
.RB [ \-\-vault .RB [ \-\-vault
.IR DIR ] .IR DIR ]
.RB [ \-\-max\-pages
.IR N ]
.RB [ \-\-max\-bytes
.IR SIZE ]
.RB [ \-\-timeout
.IR DURATION ]
.I URL .I URL
.RI [ FOLDER ] .RI [ FOLDER ]
.SH DESCRIPTION .SH DESCRIPTION
@@ -20,21 +26,30 @@ and stores every page and asset it finds in the vault, beneath
if no folder is given). if no folder is given).
.PP .PP
The crawl is single threaded: one request at a time, in breadth-first order. The crawl is single threaded: one request at a time, in breadth-first order.
It follows only links on the same host as It follows only links on the same host (and port) as
.IR URL , .IR URL ,
and requests are sent with a Chrome user agent string. and only redirects to that host; a redirect elsewhere skips the page.
Requests are sent with a Chrome user agent string, and certificates are
always verified.
.PP .PP
Links in HTML Links in HTML start tags
.RB ( href , .RB ( href ,
.BR src , .BR src ,
.BR action .BR action
and and
.B poster .B poster
attributes) and in CSS attributes) and CSS
.RB ( url(...) ) .RB ( url(...)
that point into the mirror are rewritten to relative paths, so the copy in style sheets,
browses correctly once fetched out of the vault. Links to other hosts, and .B <style>
fragments, are left as they were. elements and
.B style
attributes) that point into the mirror are rewritten to relative paths, so
the copy browses correctly once fetched out of the vault. Relative links are
resolved against the page's final address and its
.B <base href>
if it has one. Links to other hosts, script and comment text, and every
other byte of the page are left as they were. Fragments are kept.
.PP .PP
Pages are named after their URL path. A path that is empty or ends in Pages are named after their URL path. A path that is empty or ends in
.B / .B /
@@ -42,19 +57,39 @@ becomes
.BR index.html ; .BR index.html ;
a query string is appended as a query string is appended as
.BI _ query .BI _ query
(URL-escaped). Anything already stored under the same name is replaced. (URL-escaped). A name already stored by an earlier run is replaced, but if
two different URLs in one crawl would get the same name, the second is
skipped rather than overwriting the first.
.PP .PP
A page that fails to download, or answers with anything but A page that fails to download, answers with anything but
.BR "200 OK" , .BR "200 OK" ,
is reported on standard error and skipped; the crawl carries on. or exceeds a limit is reported on standard error as
The vault is locked exclusively for the whole run. .BI "skipping " URL ": " reason
and skipped; the crawl carries on. HTML and CSS are rewritten in memory;
other files are staged in a private temporary file (plaintext on disk
meanwhile) that is always removed. The vault is locked exclusively for the
whole run.
.SH OPTIONS .SH OPTIONS
.TP .TP
.BI \-\-max\-pages " N"
Pages and assets to fetch at most. Default 10000.
.TP
.BI \-\-max\-bytes " SIZE"
Total bytes to download at most, counted after decompression. Default 4G.
.TP
.BI \-\-timeout " DURATION"
Deadline for each request. Default 5m.
.TP
.BI \-\-key " PATH" .BI \-\-key " PATH"
Key file. Default: $VAULT_KEY, else ~/.vault/key. Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP .TP
.BI \-\-vault " DIR" .BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store. Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.PP
A single HTML or CSS file larger than 16 MiB is skipped.
.SH EXIT STATUS
0 when everything found was stored, 3 when some of it was skipped, 1 on
an error that stops the crawl.
.SH ENVIRONMENT .SH ENVIRONMENT
.TP .TP
.B VAULT_PASSPHRASE .B VAULT_PASSPHRASE
@@ -75,11 +110,19 @@ vault mirror https://example.com/ archive/2026
vault get archive/2026/example.com/index.html vault get archive/2026/example.com/index.html
.RE .RE
.SH LIMITATIONS .SH LIMITATIONS
Links are found by pattern matching, not by parsing HTML: URLs assembled by URLs assembled by scripts, and
scripts, and
.B srcset .B srcset
lists, are neither followed nor rewritten. There is no depth limit, rate lists, are neither followed nor rewritten. In attribute values only numeric
limit or robots.txt handling; only mirror sites you may copy. character references and
.BR &amp; ,
.BR &lt; ,
.BR &gt; ,
.BR &quot; ,
.B &apos;
and
.B &nbsp;
are decoded. There is no rate limit or robots.txt handling; only mirror
sites you may copy.
.SH SEE ALSO .SH SEE ALSO
.BR vault (1) .BR vault (1)
.SH AUTHOR .SH AUTHOR
+8
View File
@@ -32,6 +32,14 @@ memory is locked against swapping and excluded from core dumps; on Linux the
helper is also not debuggable. It answers only on a Unix socket inside helper is also not debuggable. It answers only on a Unix socket inside
\fI$XDG_RUNTIME_DIR/vault-UID\fR (or the temporary directory), a directory \fI$XDG_RUNTIME_DIR/vault-UID\fR (or the temporary directory), a directory
that must belong to this user with mode 0700. Rebooting ends it too. that must belong to this user with mode 0700. Rebooting ends it too.
If the helper cannot lock its memory, disable core dumps or (on Linux) make
itself undebuggable, it refuses the key and
.B unlock
fails, saying why. The helper is the same program, started with the
passphrase variables removed from its environment. Helpers started by
earlier (Go) versions of vault are compatible and can be used and locked.
.PP
Not available on Windows, where every command asks for the passphrase.
.PP .PP
While the key is unlocked, any program running as this user can use it, and While the key is unlocked, any program running as this user can use it, and
it is whichever key the passphrase opened, real or decoy. Lock before it is whichever key the passphrase opened, real or decoy. Lock before
+23 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT 1 "September 2026" "vault" "User Commands" .TH VAULT 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault \- post-quantum encrypted file vault vault \- post-quantum encrypted file vault
.SH SYNOPSIS .SH SYNOPSIS
@@ -77,6 +77,11 @@ Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP .TP
.BI \-\-vault " DIR" .BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store. Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.TP
.BI \-\-workers " N"
Threads for encryption, decryption and file I/O, 1 to 256. Default: the
number of CPUs, at most 8. Up to eight 4 MiB objects are in flight at once
whatever the count.
.SH ENVIRONMENT .SH ENVIRONMENT
.TP .TP
.B VAULT_KEY .B VAULT_KEY
@@ -92,6 +97,19 @@ Passphrase, instead of prompting on the terminal or using a key remembered by
.B VAULT_DECOY_PASSPHRASE .B VAULT_DECOY_PASSPHRASE
Decoy passphrase used by Decoy passphrase used by
.BR "vault genkey" . .BR "vault genkey" .
.TP
.BR HTTPS_PROXY ", " HTTP_PROXY ", " NO_PROXY
Proxies for
.B add
.I URL
and
.BR mirror ,
as other programs use them; local addresses are never proxied.
.TP
.BR XDG_RUNTIME_DIR ", " TMPDIR
Where
.BR vault\-unlock (1)
keeps its socket, and where downloads are staged.
.SH LOCKING .SH LOCKING
Commands that change the vault hold an exclusive lock on Commands that change the vault hold an exclusive lock on
.IR DIR/.lock ; .IR DIR/.lock ;
@@ -106,7 +124,10 @@ and
fills each vault with unreadable filler, so hidden files look like filler to fills each vault with unreadable filler, so hidden files look like filler to
anyone holding only the decoy passphrase. anyone holding only the decoy passphrase.
.SH EXIT STATUS .SH EXIT STATUS
0 on success; 1 on error (diagnostics on standard error); 2 when run with no command. 0 on success; 1 on error (diagnostics on standard error); 2 when run with no
command; 3 when
.BR vault\-mirror (1)
stored some resources but had to skip others.
.SH EXAMPLES .SH EXAMPLES
.nf .nf
vault genkey vault genkey
@@ -0,0 +1,57 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
package pqcrypt
import (
"bytes"
"io"
"testing"
)
func BenchmarkOracleDerive(b *testing.B) {
seed := pattern(32, 1)
for b.Loop() {
id, _ := NewPQIdentityFromSeed(0x0001, seed)
id.Close()
}
}
func BenchmarkOracleWrap(b *testing.B) {
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
r := id.Recipient()
fk := pattern(32, 2)
for b.Loop() {
r.Wrap(fk)
}
}
func BenchmarkOracleUnwrap(b *testing.B) {
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
st, _ := id.Recipient().Wrap(pattern(32, 2))
for b.Loop() {
if _, err := id.Unwrap(st); err != nil {
b.Fatal(err)
}
}
}
// Whole-file encryption and decryption of 4 MiB: one vault object.
func BenchmarkOracleObject4MiB(b *testing.B) {
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
pt := pattern(4<<20, 3)
var ct bytes.Buffer
b.SetBytes(int64(len(pt)))
for b.Loop() {
ct.Reset()
w, _ := Encrypt(&ct, id.Recipient())
w.Write(pt)
w.Close()
r, err := Decrypt(bytes.NewReader(ct.Bytes()), id)
if err != nil {
b.Fatal(err)
}
io.Copy(io.Discard, r)
}
}
@@ -0,0 +1,27 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
package stream
import (
"bytes"
"io"
"testing"
)
// STREAM seal and open of 64 MiB, one goroutine (the sequential path).
func BenchmarkOracleStream64MiB(b *testing.B) {
key := make([]byte, 32)
pt := make([]byte, 64<<20)
var ct bytes.Buffer
b.SetBytes(int64(len(pt)))
for b.Loop() {
ct.Reset()
w, _ := NewEncryptWriter(key, &ct)
w.Write(pt)
w.Close()
r, _ := NewDecryptReader(key, bytes.NewReader(ct.Bytes()))
io.Copy(io.Discard, r)
}
}
+23
View File
@@ -0,0 +1,23 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
package main
import (
"path/filepath"
"testing"
)
// Unlocking a key file: Argon2id for both slots.
func BenchmarkOracleUnlock(b *testing.B) {
p := filepath.Join(b.TempDir(), "key")
if err := writeKeyFile(p, "bench", ""); err != nil {
b.Fatal(err)
}
for b.Loop() {
if _, err := unlockKeyFile(p, "bench"); err != nil {
b.Fatal(err)
}
}
}
+96
View File
@@ -0,0 +1,96 @@
#!/usr/bin/env bash
# Compare the Go oracle and the Rust build on this machine, end to end.
#
# scripts/bench.sh [WORKDIR] (default: target/bench, on the repo's disk)
# RUNS=5 BIG_MB=2048 SMALL=5000 scripts/bench.sh
#
# For each implementation, in a fresh key and vault on the same filesystem:
# add one BIG_MB file, get it back, add SMALL small files (1-16 KiB) as a
# folder, list the catalogue (filler plus SMALL+1 records), and get one small
# file. Each step is timed RUNS times (median, min, max), with peak RSS and
# peak thread count. Every step includes unlocking the key (two Argon2id
# runs). Both implementations fsync every object and record, so `add`
# timings include durable writes. Page caches are warm: dropping them needs
# root, so cold-cache numbers are not measured here.
#
# Needs: go (to build the oracle), cargo, /usr/bin/time (GNU).
set -euo pipefail
cd "$(dirname "$0")/.."
root=$PWD
work=${1:-$root/target/bench}
runs=${RUNS:-3}
big_mb=${BIG_MB:-1024}
small=${SMALL:-2000}
mkdir -p -- "$work"
work=$(cd -- "$work" && pwd -P)
[[ $work == "$root" || $work == / ]] && { echo "refusing WORKDIR=$work" >&2; exit 2; }
cargo build --release --locked --quiet
(cd oracle/go && CGO_ENABLED=0 go build -trimpath -o "$work/vault-go" .)
cp target/release/vault "$work/vault-rust"
data=$work/data
if [[ ! -f $data/big || $(stat -c %s "$data/big") != $((big_mb << 20)) ]]; then
rm -rf -- "$data"
mkdir -p -- "$data/small"
head -c $((big_mb << 20)) /dev/urandom >"$data/big"
for i in $(seq -w 1 "$small"); do
head -c $((1024 + RANDOM % 15360)) /dev/urandom >"$data/small/f$i"
done
fi
export VAULT_PASSPHRASE=bench-passphrase
# measure LABEL CMD...: one timed run, printing "seconds rss_kib threads".
measure() {
local out=$work/time.$$ peak=0 pid t
/usr/bin/time -f '%e %M' -o "$out" "$@" >/dev/null 2>&1 &
pid=$!
while kill -0 $pid 2>/dev/null; do
for t in /proc/[0-9]*/status; do
if grep -q "^PPid:[[:space:]]*$pid$" "$t" 2>/dev/null; then
n=$(awk '/^Threads:/{print $2}' "$t" 2>/dev/null || echo 0)
((n > peak)) && peak=$n
fi
done
sleep 0.05
done
wait $pid
echo "$(cat "$out") $peak"
rm -f -- "$out"
}
summarise() { # lines of "secs rss threads" -> median min max, max rss, max threads
sort -n | awk '{t[NR]=$1; if($2>r)r=$2; if($3>th)th=$3} END {printf "%8.2f s (min %.2f, max %.2f) rss %6.0f MiB threads %d\n", t[int((NR+1)/2)], t[1], t[NR], r/1024, th}'
}
for impl in go rust; do
bin=$work/vault-$impl
echo "== $impl ($(stat -c %s "$bin") bytes)"
for step in add-big get-big add-small list get-small; do
results=""
for run in $(seq 1 "$runs"); do
v=$work/run-$impl
rm -rf -- "$v"
mkdir -p -- "$v"
export VAULT_KEY=$v/key VAULT_DIR=$v/store
"$bin" genkey 2>/dev/null
"$bin" init 2>/dev/null
case $step in
get-big | list | get-small) "$bin" add "$data/big" 2>/dev/null; "$bin" add "$data/small" 2>/dev/null ;;
esac
case $step in
add-big) r=$(measure "$bin" add "$data/big") ;;
get-big) r=$(measure "$bin" get big -o "$v/big.out") ;;
add-small) r=$(measure "$bin" add "$data/small") ;;
list) r=$(measure "$bin" list) ;;
get-small) r=$(measure "$bin" get "small/f$(printf "%0${#small}d" 1)" -o "$v/small.out") ;;
esac
results+="$r"$'\n'
done
printf ' %-10s ' "$step"
printf '%s' "$results" | grep . | summarise
done
rm -rf -- "$work/run-$impl"
done