30 Commits
Author SHA1 Message Date
Andy Dixon b06094b2aa Tests: skip Go-oracle CLI interop on Windows
The frozen Go version cannot sync directories there.
2026-10-13 14:47:30 +01:00
Andy Dixon c9725f18d4 Tests on Windows: no console for CLI children, .exe for the Go oracle; CI test timeout
A hidden console on Windows runners let a confirmation prompt wait
forever, and the oracle built without .exe could not be run.
2026-10-13 14:23:43 +01:00
Andy Dixon f62e74a3e9 0.6.1 2026-10-13 12:55:23 +01:00
Andy Dixon 1811ff4d33 Tests: find agents with pgrep, since macOS has no /proc 2026-10-13 12:50:55 +01:00
Andy Dixon 3347efe418 CI: keep test vectors byte-exact on Windows, short agent paths on macOS
Windows checkouts converted LF vectors to CRLF; SystemTime there has
100 ns resolution. macOS's long $TMPDIR pushed the agent socket past
SUN_LEN.
2026-10-13 12:42:23 +01:00
Andy DixonandClaude e8f21f68a6 Guide: scale the taller help screenshot to fit its page
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 19:15:00 +01:00
Andy DixonandClaude 660e9cd5bf Guide: split the error table so it fits the page; PLATFORMS: no Homebrew yet
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 19:12:00 +01:00
Andy Dixon 404aa91573 Merge branch 'rust-port': screenshots and macOS cross-build support 2026-10-08 18:28:04 +01:00
Andy DixonandClaude 8aee1f0f04 Screenshots from the Rust build; system-roots feature for macOS cross-builds
docs/guide/img is re-shot from the 0.6 binary. TLS roots come from the
operating system by default; build.sh cross-builds macOS on Linux without
that feature (no Apple SDK), trusting the bundled Mozilla roots. Lock
tests wait for the reader to report it is blocked rather than assuming
how long unlocking takes on a busy machine.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 18:27:32 +01:00
Andy Dixon 0ebf6ce97c Merge pull request #1 from andydixon/rust-port
Rewrite vault in Rust (format-compatible with 0.5)
2026-10-08 17:38:32 +01:00
Andy DixonandClaude 06cf7355dc Optimised dev builds; idle-agent test measures only the agent
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 14:31:58 +01:00
Andy DixonandClaude 31dbdc7009 Benchmark results: Go 0.5 against Rust 0.6
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 14:15:30 +01:00
Andy DixonandClaude 0877cc7447 Parallel Argon2id lanes (unlock 180 ms -> 129 ms, Go 122 ms); AEAD throughput probe
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 13:13:25 +01:00
Andy DixonandClaude 438180b6fb Update fuzz lockfile
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 13:03:31 +01:00
Andy DixonandClaude 25496ddb97 Release matrix: Linux, Windows, macOS and FreeBSD
Drop android (same binary as linux/arm64), big-endian ppc64 (cannot be
verified: even hello-world crashes under qemu-ppc64 with the cross
toolchain), NetBSD (cross 0.2.5's sysroot lacks libexecinfo), illumos,
Solaris and WASI, as agreed. docs/PLATFORMS.md records each decision.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 12:40:30 +01:00
Andy DixonandClaude 7506c1dbef Clippy fixes
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 12:24:37 +01:00
Andy DixonandClaude 91d49aeecf Fix review findings: PAX sizes, agent stdin, PEM parity, sticky writer
- absorb uses the PAX-corrected entry size (large Go/bsdtar archives were
  stored truncated) and refuses GNU sparse entries
- the agent reads the key from fd 0 straight into locked memory, and never
  unlinks a live agent's socket
- the key file PEM parser is a port of Go's pem.Decode
- the STREAM writer fails permanently after a write error; plaintext is
  cleared on reader error paths; AEAD, HMAC and hash states are zeroized
- add/migrate refuse paths inside the vault; publish reports filesystems
  without an atomic no-replace primitive clearly and treats a leftover
  temporary name as a warning; Windows rejects NTFS-special names
- the leak test runs in its own process

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 12:23:27 +01:00
Andy DixonandClaude 25f50ed73b Network review fixes: confine mirror paths, bound rewrites, charge partial downloads
Mirror refuses local paths with ., .. or empty segments (decoded %2e%2e
could climb out of the mirror folder) and checks names stay under it;
dedupes found links and skips a page whose rewritten links would pass
four times the rewrite limit; charges bytes received before a failure to
the budget. add URL falls back to the host name for . and .. names.
Staged downloads are unlinked at creation on Unix and stored from the
open handle. Rewritten links are percent-escaped (./ before a colon), a
<base href> always points at the page itself, userinfo and proxy values
stay out of messages, ureq I/O errors are lower-cased, CGI detection
needs a non-empty REQUEST_METHOD and ::ffff:127.0.0.1 is loopback.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 12:16:50 +01:00
Andy DixonandClaude 53626a7de7 Crash and leak tests; platform matrix; ppc64 via cross; CI runs builds under qemu
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 11:56:41 +01:00
Andy DixonandClaude afdfbc5ebd Oracle README: exact provenance check and the full list of added test files
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 11:51:14 +01:00
Andy DixonandClaude 53968f8822 Docs for the Rust version; benchmarks; build.sh fixes
README, CLAUDE.md, man pages, the user guide and its screenshot script
now describe the Rust build and the intentional behaviour changes, which
docs/MIGRATION.md lists. Go and Rust micro-benchmarks for the same
operations, and scripts/bench.sh for end-to-end comparison. build.sh
exports its tree root to the per-target jobs and keeps logs out of the
output directory; riscv64 musl builds use getrandom's raw syscall
backend, since zig's musl getrandom never reaches the kernel there.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 11:50:55 +01:00
Andy DixonandClaude cddde5ca52 Cross builds, packaging split, CI, fuzzing, supply-chain policy
build.sh builds the Rust targets with the Go-era names, refuses an OUT
that is /, $HOME or the tree, checks static linking and names the Go
targets Rust cannot build. packaging/package.sh builds and inspects
packages without signing or publishing; repo.sh reuses it. Seven fuzz
targets (one found a URL-parser panic, fixed here with a regression
test), cargo-deny policy, CI for lint, tests on Linux/macOS/Windows,
cross builds with qemu runs, supply chain and fuzzing. Compatibility
specification and security notes.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 11:44:23 +01:00
Andy DixonandClaude a98931779a CLI, agent and absorb; stop tracking target/
Every Go command with the same flags, prompts and messages; the agent is
wire-compatible with Go's; tests drive the real Rust and Go binaries.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 10:36:20 +01:00
Andy DixonandClaude aedc605a82 Network import: bounded add URL and mirror
An explicit ureq client with platform TLS verification, connect/header/
request deadlines, manual bounded redirects, decoded-byte budgets and
Go-style proxy selection. A Go-compatible URL module checked against
oracle vectors, and a byte-preserving HTML tokenizer for link rewriting
(href/src/action/poster, <style> and style= url(), <base href>). Mirror
keeps to one host across redirects, bounds pages and bytes, refuses
local-name collisions and reports partial results.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 10:34:28 +01:00
Andy DixonandClaude 24ebede9bb Placeholder net module
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 10:24:34 +01:00
Andy DixonandClaude b97c690ef7 Fix ciphertext buffer preallocation precedence; clippy fixes
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 10:22:06 +01:00
Andy DixonandClaude a6610449f2 Store, validated types, bounded pipeline, no-clobber extraction
The store reads and writes the Go on-disk format (it opens the oracle's
fixture vault). Object I/O runs on one bounded worker pool with eight
reusable cleared buffers; sequential and parallel runs produce identical
bytes under seeded test randomness. Extraction publishes with
RENAME_NOREPLACE/link and only after full authentication; failed or
after-publish writes keep crash-safe ordering.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 10:21:44 +01:00
Andy DixonandClaude c324f6a306 Rust crypto core: FrodoKEM port, hybrid suite, Fury, STREAM, key files
FrodoKEM-640-SHAKE is ported from CIRCL v1.6.4 (the RustCrypto crate
implements the incompatible ISO variant) and reproduces the reference
NIST KAT digest. ML-KEM-768 comes from RustCrypto ml-kem. Derivation,
encapsulation, implicit rejection, KEK, header MAC, header parsing,
STREAM and Argon2id key files all match vectors from the Go oracle, Go's
files decrypt in Rust, and Go decrypts Rust's.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 10:12:53 +01:00
Andy DixonandClaude 69f4b90777 Freeze the Go implementation as a test-only oracle, with vectors
oracle/go is the Go code at 4c1507c, moved unchanged, plus test-only
generators that record what it computes. tests/vectors holds the
deterministic crypto/store vectors and synthetic fixtures they produce.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-08 09:55:17 +01:00
Andy Dixon 4c1507c463 Add unlock --idle, and packaging/repo.sh for repo.dixon.cx
vault unlock --idle counts the duration from the last command that used
the key rather than from the unlock: the agent resets its timer each
time it hands the key out. The agent loop no longer needs its own
goroutine.

packaging/repo.sh builds the dixon-vault .deb, .rpm and Arch packages
from a release tag (a clean git archive) with nfpm for amd64, arm64,
armv7, 386 and riscv64, then publishes them to the apt, dnf and pacman
repositories at repo.dixon.cx: apt-ftparchive with a signed InRelease,
createrepo_c (in a Fedora container) with signed RPMs and repomd.xml,
and repo-add (in an Arch container) with signed packages and database.
Indexes cover every package in the repository, under a lock, so other
projects publishing with their copy of the script are kept. The package
is dixon-vault because vault is HashiCorp Vault in Arch's [extra] and
HashiCorp's own repositories; it conflicts with vault.

README, guide and CLAUDE.md point at the repository.
2026-10-03 09:44:49 +01:00
153 changed files with 24521 additions and 182 deletions
+2
View File
@@ -0,0 +1,2 @@
# Test vectors are compared byte for byte; never convert line endings.
tests/vectors/** -text
+112
View File
@@ -0,0 +1,112 @@
name: ci
on:
push:
pull_request:
permissions:
contents: read
env:
CARGO_TERM_COLOR: always
# Oracle-backed interoperability tests must run, not skip, in CI.
VAULT_REQUIRE_ORACLE: "1"
jobs:
lint:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- run: rustup show active-toolchain || rustup toolchain install
- run: cargo fmt --check
- run: cargo clippy --locked --all-targets --all-features -- -D warnings
test:
strategy:
fail-fast: false
matrix:
os: [ubuntu-24.04, ubuntu-24.04-arm, macos-15, macos-15-intel, windows-2025]
runs-on: ${{ matrix.os }}
timeout-minutes: 30
steps:
- uses: actions/checkout@v5
# The Go toolchain builds the test-only oracle; vault itself needs no Go.
- uses: actions/setup-go@v6
with:
go-version: "1.26.x"
cache-dependency-path: oracle/go/go.sum
- run: rustup show active-toolchain || rustup toolchain install
- name: test
shell: bash
run: |
if [[ $RUNNER_OS == Windows ]]; then unset VAULT_REQUIRE_ORACLE; fi
cargo test --locked --all-targets --all-features
- run: cargo test --locked --doc
- run: cargo build --locked --release
- name: oracle self-test
if: runner.os == 'Linux'
working-directory: oracle/go
run: go test ./...
cross:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- uses: actions/setup-go@v6
with:
go-version: "1.26.x"
cache-dependency-path: oracle/go/go.sum
- run: rustup show active-toolchain || rustup toolchain install
- name: zig and cargo-zigbuild
run: |
curl -sSLo zig.tar.xz https://ziglang.org/download/0.16.0/zig-x86_64-linux-0.16.0.tar.xz
echo "70e49664a74374b48b51e6f3fdfbf437f6395d42509050588bd49abe52ba3d00 zig.tar.xz" | sha256sum -c
mkdir -p "$HOME/zig" && tar -xJf zig.tar.xz -C "$HOME/zig" --strip-components=1
echo "$HOME/zig" >>"$GITHUB_PATH"
cargo install cargo-zigbuild --version 0.23.4 --locked
cargo install cross --version 0.2.5 --locked
- name: targets
run: |
rustup target add x86_64-unknown-linux-musl aarch64-unknown-linux-musl i686-unknown-linux-musl \
arm-unknown-linux-musleabihf armv7-unknown-linux-musleabihf riscv64gc-unknown-linux-musl \
loongarch64-unknown-linux-musl powerpc64le-unknown-linux-musl \
s390x-unknown-linux-gnu x86_64-pc-windows-gnu i686-pc-windows-gnu aarch64-pc-windows-gnullvm \
x86_64-unknown-freebsd i686-unknown-freebsd
- name: build
run: |
# Everything but macOS, which the test job builds on a Mac.
TARGETS="linux/amd64 linux/arm64 linux/386 linux/armv6 linux/armv7 linux/riscv64 linux/loong64 \
linux/ppc64le linux/s390x windows/amd64 windows/386 windows/arm64 freebsd/amd64 freebsd/386" ./build.sh
- name: run builds under qemu, cross-checked with the Go oracle
run: |
sudo apt-get update -q && sudo apt-get install -yq qemu-user libc6-s390x-cross
(cd oracle/go && go build -o ../../target/vault-go .)
for t in "arm64 qemu-aarch64" "armv7 qemu-arm" "riscv64 qemu-riscv64" "ppc64le qemu-ppc64le" \
"s390x qemu-s390x -L /usr/s390x-linux-gnu"; do
EMU="${t#* }" scripts/smoke.sh "dist/vault-linux-${t%% *}" target/vault-go
done
- uses: actions/upload-artifact@v4
with:
name: dist
path: dist/
supply-chain:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- run: rustup show active-toolchain || rustup toolchain install
- run: cargo install cargo-deny --version 0.20.2 --locked
- run: cargo deny --locked check
fuzz:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- run: rustup toolchain install nightly-2026-10-01 --profile minimal
- run: cargo +nightly-2026-10-01 install cargo-fuzz --version 0.13.2 --locked
- name: a short run of every target
working-directory: fuzz
run: |
for t in $(cargo +nightly-2026-10-01 fuzz list); do
cargo +nightly-2026-10-01 fuzz run "$t" -- -max_total_time=60
done
+5
View File
@@ -6,3 +6,8 @@
/docs/guide/version.tex /docs/guide/version.tex
/docs/guide/vault-guide.pdf /docs/guide/vault-guide.pdf
/docs/guide/*.log /docs/guide/*.log
/target
fuzz/target
fuzz/corpus
fuzz/artifacts
fuzz/coverage
+65 -39
View File
@@ -4,67 +4,93 @@ This file provides guidance to Claude Code when working with this repository.
## Overview ## Overview
vault is a post-quantum encrypted file vault: one `package main` at the repo vault is a post-quantum encrypted file vault: one Rust package (`Cargo.toml`
root that builds a single `vault` binary. Module `github.com/andydixon/vault`. at the root, toolchain pinned in `rust-toolchain.toml`) that builds a single
`vault` binary. The library (`src/lib.rs`) exists so tests and benchmarks can
reach the same code; `src/main.rs` only calls `cli::main`.
- `main.go` - CLI: `genkey`, `init`, `add` (file, directory - `src/cli.rs` - commands and Go-`flag`-compatible parsing: `genkey`,
or URL), `migrate` (add, then delete originals), `mirror`, `absorb` (tar `init`, `add` (file, directory or URL), `migrate`, `mirror`, `absorb`,
on stdin, `absorb.go`), `ls` (one folder level), `list` (recursive), `ls`/`list`, `get` (`-r`), `mv`/`cp` (`transfers` resolves names like
`get` (`-r` for a folder), `mv`/`cp` (files or folders; `transfers` mv(1)), `rm` (`-r`, asks unless `-f`), `unlock`/`lock`, hidden `__agent`.
resolves names like mv(1), copies share objects), `rm` (`-r` for a - `src/agent.rs` - the unlock agent: `unlock` re-executes the binary as
folder, asks unless `-f`), `unlock`/`lock`. `__agent` (passphrase variables stripped, secret over a pipe); it hardens
- `agent*.go` - `vault unlock` re-executes itself as a hidden `__agent` itself (`platform::memory`) before reading the key, serves it on a Unix
process holding the suite ID and seed (pipe in, mlocked, undumpable) and socket in a 0700 per-user dir until a deadline (fixed or `--idle`).
serving them on a Unix socket in a 0700 per-user dir until a hard Wire-compatible with the Go agent. Not available on Windows.
deadline. `keyIdentity` asks it first unless `$VAULT_PASSPHRASE` is set. - `src/keyfile.rs` - key file: two Argon2id/XChaCha20-Poly1305 slots, decoy
- `keyfile.go` - key file with two passphrase slots (Argon2id + or random bytes in the second, random order, both always tried.
XChaCha20-Poly1305 sealing a suite ID and 32-byte master seed). The second - `src/crypto/` - `frodo.rs` (FrodoKEM-640-SHAKE ported from CIRCL v1.6.4),
slot is either a decoy identity or random bytes; slot order is random. `kem.rs` (hybrid suite 0x0001 with RustCrypto ml-kem, `Identity`),
- `store.go` - on-disk vault: `records/` holds fixed-size encrypted catalogue `header.rs` (Fury header format), `fury.rs` (file encryption), `stream.rs`
records, `objects/` holds file bodies cut into 4 MiB chunks (last one (STREAM, including random access).
bucket-padded) whose IDs are HMAC'd from a per-record seed. Both are - `src/store/` - on-disk vault: `records/`, `objects/` sharded by ID prefix,
sharded into 256 `xx/` subdirectories by ID prefix. Records a key cannot catalogue, add/remove/copy/extract, filler; `types.rs` holds the validated
open are skipped, not treated as errors. types (`VaultName`, `Id`, `ObjectSeed`, `Record`) and record JSON.
- `lock*.go` - vault lock on `DIR/.lock`: fcntl on unix, LockFileEx on - `src/pipeline.rs` - the one bounded worker pool and the eight reusable
Windows, an exclusive lock file elsewhere. Writers lock exclusively, object buffers; coordinators run on the caller's thread.
readers shared, taken after the passphrase prompt. - `src/platform/` - all `unsafe` and OS specifics: `fs.rs` (no-clobber
- `docs/guide` - LaTeX user guide (`build-pdf.sh` via Tectonic; `shots.py` publish, `Dir` confined extraction, epoch times, dir sync), `lock.rs`
re-shoots screenshots from the real binary). Keep it in step with the CLI. (fcntl / LockFileEx / lock file), `memory.rs` (mlock, undumpable),
- `build.sh` - cross-compiles every `go tool dist list` target except `term.rs` (prompts).
iOS and android/386, amd64, arm (cgo-only). - `src/net/` - `add URL` and `mirror` with limits; `url.rs` (Go `net/url`
- `internal/pqcrypt` - file encryption (header + STREAM ChaCha20-Poly1305), semantics), `html.rs` (tokenizer and link rewriting).
suite registry, hybrid FrodoKEM-640 + ML-KEM-768 default suite. - `src/absorb.rs` - tar (optionally gzipped) from stdin, streamed in.
- `internal/format`, `internal/stream`, `internal/term` - header encoding, - `src/secret.rs`, `src/rng.rs`, `src/util.rs`, `src/error.rs` - secret
chunked payload encryption, terminal passphrase prompts. buffers, randomness (seeded only under `cfg(test)`), Go-compatible
parsing/printing helpers, the error type.
- `oracle/go` - the frozen Go implementation (test-only reference, see
`oracle/README.md`); `oracle/gen-vectors.sh` writes `tests/vectors`.
- `tests/` - CLI and interoperability tests against the real binaries;
`fuzz/` - cargo-fuzz targets; `benches/micro.rs` - micro-benchmarks.
- `docs/` - man pages, `COMPATIBILITY.md`, `MIGRATION.md`, `PLATFORMS.md`,
`BENCHMARKS.md`, and the LaTeX guide (`docs/guide`, `build-pdf.sh` via
Tectonic; `shots.py` re-shoots screenshots from the real binary). Keep
them in step with the CLI.
- `build.sh` - cross-builds every supported target (see `PLATFORMS.md`).
- `packaging/package.sh` builds and inspects dixon-vault packages locally;
`packaging/repo.sh [TAG]` signs and publishes them to repo.dixon.cx.
## Commands ## Commands
```sh ```sh
go test ./... cargo test --locked # needs go on PATH for interop tests
go build . cargo clippy --locked --all-targets --all-features -- -D warnings
./build.sh # all platforms, into ./dist cargo fmt --check
cargo build --release --locked
./build.sh # all buildable targets, into ./dist
TARGETS="linux/amd64" ./build.sh # a subset TARGETS="linux/amd64" ./build.sh # a subset
cargo deny --locked check # advisories and licences
(cd fuzz && cargo +nightly fuzz run header)
``` ```
End-to-end check: End-to-end check:
```sh ```sh
export VAULT_PASSPHRASE=test VAULT_KEY=$PWD/k VAULT_DIR=$PWD/s export VAULT_PASSPHRASE=test VAULT_KEY=$PWD/k VAULT_DIR=$PWD/s
./vault genkey && ./vault init && ./vault add README.md docs && ./vault list target/release/vault genkey && target/release/vault init && target/release/vault add README.md docs && target/release/vault list
``` ```
## Invariants ## Invariants
- The on-disk formats are frozen and shared with the Go version; check any
change against `docs/COMPATIBILITY.md` and the oracle vectors.
- Never write a seed or private key unencrypted (the agent gets it by pipe). - Never write a seed or private key unencrypted (the agent gets it by pipe).
- Keep suite IDs, stanza tags, HKDF info strings and `fileKeySize` (32) stable. - Keep suite IDs, stanza tags, HKDF info strings and the 32-byte file key stable.
- Every ciphertext's and shard directory's mtime is pinned to the epoch; - Every ciphertext's and shard directory's mtime is pinned to the epoch;
records stay one fixed size; objects stay at most 4 MiB and bucket-padded. records stay one fixed size; objects stay at most 4 MiB and bucket-padded.
These hide metadata from the storage host. These hide metadata from the storage host.
- An object's plaintext starts with its own ID, binding it to its record - An object's plaintext starts with its own ID, binding it to its record
and its position in the file. and its position in the file.
- `get` publishes output only after every object has authenticated. - `get` publishes output only after every object has authenticated, with a
no-replace primitive; never fall back to check-then-rename.
- Decoy deniability: both key slots stay the same size and are both always - Decoy deniability: both key slots stay the same size and are both always
tried; `init` always writes filler; foreign records are skipped silently. tried; `init` always writes filler; foreign records are skipped silently.
- Every command that changes the vault holds the exclusive lock. - Every command that changes the vault holds the exclusive lock (POSIX
- Keep dependencies pure Go and CGO_ENABLED=0 builds working. fcntl locks, compatible with Go's).
- Crash-safe order: objects before records, new records before removing old
ones, record removal (synced) before object removal.
- Secrets live in `SecretArray`/`SecretVec`/`LockedBuf`: no Clone, Debug or
serialisation. Production randomness is the OS only.
- `unsafe` only in `src/platform`, each block with a SAFETY comment.
- Diagnostics go to standard error, lower-case, no trailing full stops. - Diagnostics go to standard error, lower-case, no trailing full stops.
Generated
+1320
View File
File diff suppressed because it is too large Load Diff
+63
View File
@@ -0,0 +1,63 @@
[package]
name = "vault"
version = "0.6.1"
edition = "2024"
rust-version = "1.99"
license = "GPL-3.0-or-later"
description = "Post-quantum encrypted file vault with a decoy passphrase"
homepage = "https://dixon.cx/vault"
repository = "https://github.com/andydixon/vault"
publish = false
[dependencies]
argon2 = { version = "0.6", default-features = false, features = ["alloc", "zeroize", "parallel"] }
chacha20poly1305 = { version = "0.11", default-features = false, features = ["alloc", "zeroize"] }
hkdf = "0.13"
hmac = { version = "0.13", features = ["zeroize"] }
sha2 = { version = "0.11", default-features = false, features = ["zeroize"] }
sha3 = { version = "0.11", default-features = false, features = ["zeroize"] }
ml-kem = { version = "0.3.2", features = ["zeroize", "hazmat"] }
zeroize = { version = "1.9", features = ["derive", "serde"] }
getrandom = "0.4"
base64 = "0.23"
serde = { version = "1", features = ["derive"] }
serde_json = "1"
jiff = "0.2"
tar = { version = "0.4.46", default-features = false }
flate2 = { version = "1.1", default-features = false, features = ["rust_backend"] }
ureq = { version = "3.4", default-features = false, features = ["rustls", "gzip"] }
[features]
default = ["system-roots"]
# Verify TLS certificates against the operating system's trust store. Off,
# the bundled Mozilla roots are used (for cross-built macOS binaries).
system-roots = ["ureq/platform-verifier"]
[dev-dependencies]
aes = "0.9"
[target.'cfg(unix)'.dependencies]
rustix = { version = "1.1.5", features = ["fs", "process", "mm", "termios", "event", "net", "stdio", "param"] }
[target.'cfg(windows)'.dependencies]
windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Storage_FileSystem", "Win32_System_Console", "Win32_System_IO", "Win32_System_Memory", "Win32_Security"] }
# Tests run the real KEMs and Argon2id; unoptimised they are far too slow.
[profile.dev]
opt-level = 1
[profile.dev.package."*"]
opt-level = 3
[profile.release]
codegen-units = 1
lto = true
strip = true
[[bench]]
name = "micro"
harness = false
[[bench]]
name = "aead_probe"
harness = false
+48 -14
View File
@@ -11,18 +11,48 @@ whatever the file's size. Object names derive from a secret seed in the
record, so nothing on disk links a file's pieces. Records and objects are record, so nothing on disk links a file's pieces. Records and objects are
spread over 256 subfolders (`records/3f/3fa9…`) to keep directories small. spread over 256 subfolders (`records/3f/3fa9…`) to keep directories small.
## Install
On Debian, Ubuntu, Fedora, RHEL, openSUSE and Arch, install the
`dixon-vault` package from the signed repository at repo.dixon.cx; see
https://dixon.cx/repo for the two or three commands. (The package isn't
called `vault` because that name is HashiCorp Vault's.) Every platform also
has a single binary at https://download.dixon.cx/vault/.
Releases are packaged and published with `packaging/repo.sh [TAG]`.
## Build ## Build
vault is written in Rust (the toolchain is pinned in `rust-toolchain.toml`;
`rustup` installs it on first use). Releases up to 0.5 were Go; the Go code
lives on, frozen, in `oracle/go` as the test reference. Nothing at run time
needs Go.
```sh ```sh
go build . # produces ./vault cargo build --release --locked # produces target/release/vault
go test ./... cargo test --locked # unit, CLI and Go-interoperability tests
./build.sh # every OS and CPU architecture, into ./dist ./build.sh # every target this host can build, into ./dist
TARGETS="linux/amd64 windows/amd64" ./build.sh
``` ```
`build.sh` cross-compiles static binaries for every target in `build.sh` cross-compiles with `cargo-zigbuild` and `zig` (Linux, statically
`go tool dist list` except iOS and 32-bit or x86 Android, which need cgo. It builds 32-bit ARM Linux for linked with musl, and Windows), on a Mac for macOS, and with `cross` and
both armv6 and armv7, and writes `dist/SHA256SUMS`. Set `TARGETS` to build a Docker for FreeBSD. The release covers Linux, Windows, macOS and FreeBSD. It names binaries as before
subset, for example `TARGETS="linux/amd64 darwin/arm64" ./build.sh`. (`vault-linux-arm64`, `vault-windows-amd64.exe`, …), builds 32-bit ARM Linux
for both armv6 and armv7, and writes `dist/SHA256SUMS`. `docs/PLATFORMS.md`
lists every target, how it is built and tested, and the Go targets Rust
cannot build.
The interoperability tests build the Go oracle when `go` is installed and
are skipped (and say so) otherwise; set `VAULT_REQUIRE_ORACLE=1` to make a
missing oracle an error, as CI does. `oracle/gen-vectors.sh` regenerates the
Go test vectors. Packages are built and inspected locally, unsigned, with
`packaging/package.sh`, and published with `packaging/repo.sh [TAG]`.
Further reading: `SECURITY.md` (what is and is not protected, memory and
filesystem guarantees), `docs/COMPATIBILITY.md` (the formats),
`docs/MIGRATION.md` (what changed from the Go version) and
`docs/BENCHMARKS.md`.
## User guide ## User guide
@@ -65,18 +95,20 @@ vault rm old/report.pdf
vault rm -r old # a whole folder; lists it and asks first vault rm -r old # a whole folder; lists it and asks first
vault rm -rf old # the same, without asking vault rm -rf old # the same, without asking
vault unlock # remember the key for 5 minutes (--for 30m) vault unlock # remember the key for 5 minutes (--for 30m)
vault unlock --idle # ... or until 5 minutes go by unused
vault lock # forget it now vault lock # forget it now
``` ```
Commands that change the vault take an exclusive lock on `DIR/.lock`, and Commands that change the vault take an exclusive lock on `DIR/.lock`, and
commands that read it take a shared one. A second process waits, and says commands that read it take a shared one. A second process waits, and says
so, until the first finishes. The operating system releases the lock if a so, until the first finishes. The operating system releases the lock if a
process dies. Plan 9 and WebAssembly have no file locks, so there the lock process dies; the locks are the same ones the Go version used, so old and
file itself is the lock. After a crash on those platforms you remove it by new versions exclude each other.
hand; the error message names it.
Adding a name that already exists replaces it. `get` refuses to overwrite an Adding a name that already exists replaces it. `get` never replaces an
existing file and writes nothing until the whole object has authenticated. existing file (or symlink) and writes nothing until the whole file has
authenticated. `add URL` and `mirror` have size and time limits; see
`vault(1)`.
`--key PATH` and `--vault DIR` work with every command. They default to `--key PATH` and `--vault DIR` work with every command. They default to
`$VAULT_KEY` and `$VAULT_DIR`, then to `~/.vault/key` and `~/.vault/store`. `$VAULT_KEY` and `$VAULT_DIR`, then to `~/.vault/key` and `~/.vault/store`.
@@ -85,13 +117,15 @@ scripts.
`vault unlock` asks once, then starts a helper process, like `ssh-agent`, `vault unlock` asks once, then starts a helper process, like `ssh-agent`,
that holds the unlocked key in memory until a hard deadline (5 minutes by that holds the unlocked key in memory until a hard deadline (5 minutes by
default) or `vault lock`. Until then commands skip the prompt, unless default; with `--idle`, 5 minutes since a command last used it) or
`vault lock`. Until then commands skip the prompt, unless
`$VAULT_PASSPHRASE` is set, which always wins. The key reaches the helper `$VAULT_PASSPHRASE` is set, which always wins. The key reaches the helper
over a pipe and never touches the disk; the helper's memory is locked over a pipe and never touches the disk; the helper's memory is locked
against swapping, it cannot be core-dumped (or, on Linux, debugged), and it against swapping, it cannot be core-dumped (or, on Linux, debugged), and it
answers only on a socket in a directory private to your user. While it is answers only on a socket in a directory private to your user. While it is
unlocked, anything running as you can use the vault, decoy or not, so lock unlocked, anything running as you can use the vault, decoy or not, so lock
before handing the machine over. before handing the machine over. If the helper cannot protect its memory it
refuses the key. `unlock` is not available on Windows.
## Decoy passphrase ## Decoy passphrase
+157
View File
@@ -0,0 +1,157 @@
# Security notes
What vault protects, how, and where the protection stops. See
`docs/COMPATIBILITY.md` for the exact formats.
## What is protected
- **Contents, names, sizes, modes and times** of stored files, against
anyone who reads the vault directory without the passphrase: everything is
inside Fury files sealed to a hybrid FrodoKEM-640 + ML-KEM-768 key, so the
file key stays secret unless both KEMs are broken.
- **Structure**: records are one fixed size; objects are at most 4 MiB and
padded to power-of-two buckets; object names derive from a secret per-file
seed, so nothing on disk links a file's objects; every ciphertext and shard
directory has its timestamps pinned to the epoch; all 256 shards exist from
the start; `init` adds filler sealed to a discarded key.
- **Integrity**: every byte that is acted on is authenticated (header MAC,
STREAM tags with a final-chunk flag, no trailing data); an object starts
with its own ID, so objects cannot be swapped, reordered or replayed under
another record; `get` publishes output only after the whole file, padding
and end of file included, has authenticated.
- **Deniability**: the key file always has two equal slots, one possibly
random; both are always tried; decoy and real files are indistinguishable
from filler to the other passphrase.
## What is not protected
- The number and total size of ciphertexts, and when the directory changes,
are visible to whoever stores it.
- A passphrase is only as strong as it is: Argon2id (3 passes, 64 MiB, 4
lanes) slows guessing, it does not prevent it.
- Anyone who can run code as you while the vault is unlocked (or watch you
type the passphrase) can read everything.
- Nothing here hides that you use vault.
## Secrets in memory
Passphrases, seeds, private keys, key-encryption keys and plaintext buffers
are held in owned types (`SecretArray`, `SecretVec`, `LockedBuf`) that are
not `Clone`, `Copy`, `Debug` or serialisable, have fixed allocations that
never reallocate, and are cleared with `zeroize` when dropped — on success,
on error and while unwinding from a panic (release builds unwind).
What cannot be cleared, so is not claimed:
- copies the compiler leaves in registers or on the stack;
- copies inside third-party key objects: `chacha20poly1305` cipher states
(the key schedule), `ml-kem`'s `Seed` passed by value (the decapsulation
key itself is zeroized on drop with the crate's `zeroize` feature),
`hmac`/`hkdf` internal states, `argon2` internals other than its working
memory (which vault allocates and clears itself);
- the passphrase in `$VAULT_PASSPHRASE` (it is in the process environment
for the process lifetime) and in the terminal driver's buffers;
- anything after `abort`, `kill -9`, a power cut, or once swapped out;
- file data in the OS page cache, and plaintext temporary files on disk
(below).
None of this is erasure from physical memory.
## The agent
`vault unlock` re-executes the binary as `vault __agent` with
`VAULT_PASSPHRASE` and `VAULT_DECOY_PASSPHRASE` removed from its environment,
no secret in its arguments, its working directory `/` and standard error
closed. It hands over the 34-byte suite ID and seed through a pipe. Before
reading it (straight from the pipe into the locked page, with no
intermediate buffer), the agent:
1. sets the core-dump size limit to zero;
2. on Linux, sets `PR_SET_DUMPABLE` to 0 (no core dumps, no same-user
`ptrace`, `/proc/PID/mem` or `/proc/PID/environ`) and checks it took;
3. maps a private page, `mlock`s it and, on Linux, marks it
`MADV_DONTDUMP`, all checked.
Any failure is reported to `unlock` and the agent exits without the key;
there is no fallback to unprotected memory. On other Unix systems (macOS,
the BSDs, illumos) steps 1 and 3 apply; there is no portable equivalent of
`PR_SET_DUMPABLE`, so another process of the same user may be able to read
the agent's memory with a debugger.
The socket lives in a directory that must be owned by you with mode exactly
0700 (refused otherwise); the socket is 0600, checked before readiness. Each
request has a two-second deadline; the expiry deadline is enforced with a
timed `poll`, so it does not depend on another client arriving, and a key
is never handed out after expiry. On exit the agent clears the page and
removes the socket only if it is still its own. `unlock` waits at most ten
seconds for the agent to report and kills it otherwise.
Windows: the agent is not supported (`unlock` says so); every command asks
for the passphrase.
## Filesystem safety
- Output is published with an atomic no-replace primitive
(`renameat2(RENAME_NOREPLACE)`, `renameatx_np(RENAME_EXCL)`, `link`+`unlink`,
or `MoveFileExW` without replace), so an existing file, symlink (dangling
or not) or one created concurrently is never replaced; only vault's own
temporary file is removed. There is no check-then-rename fallback.
- The temporary output file is mode 0600 until the content has fully
authenticated; only then are the recorded mode and times applied.
- `get -r` opens each directory below the chosen destination with
`O_NOFOLLOW|O_DIRECTORY` relative to its parent, so a symlink planted
inside the destination cannot redirect files elsewhere. On Windows, which
has no `openat`, each component is checked not to be a link or junction
just before use; a concurrent swap between check and use is not excluded.
- Writes follow crash-safe order with explicit `fsync` of files and
directories (Unix): objects before their record; a new record before the
removal of the one it replaces; a record's removal before its objects'.
A failure after a record is already in place never deletes its objects.
Windows has no directory fsync through the APIs used; NTFS journals
metadata.
- `migrate` checks an original is still the file it stored (device, inode,
size, time; on Windows only size and time) and then deletes it by name; a
replacement in the instant between the check and the deletion is not
excluded. Paths inside the vault directory are refused outright.
- A filesystem with neither `RENAME_NOREPLACE` nor hard links (some FUSE
mounts, FAT) cannot publish without risking a replace, so `get` fails
there with a message saying so rather than fall back.
- The vault lock uses the same POSIX `fcntl` locks as the Go version, so
mixed Go and Rust processes exclude each other. Only a read-only
filesystem lets a reader proceed without the lock.
## Plaintext on disk
- `vault add URL` and `vault mirror` stream non-HTML downloads to a 0600
temporary file in `$TMPDIR` before storing them, because the size must be
known first. On Unix the file is unlinked as soon as it is created, so it
has no name and disappears even if vault is killed; on other systems it
is a named file removed on every normal path, including errors. HTML and
CSS are rewritten in memory. `vault absorb` stores tar
entries straight from the stream with no temporary file.
- `vault get` writes plaintext where you ask; `vault migrate` deletes the
originals with `unlink`, which is not secure deletion.
## Network
Downloads verify TLS certificates (rustls with the operating system's trust
store; macOS binaries cross-built on Linux use the bundled Mozilla roots
instead, see `docs/PLATFORMS.md`; verification is never disabled), follow at most 10 redirects (http and https only; for
`mirror`, only to the same host), and are bounded by connect, response and
whole-request deadlines and a byte budget counted after decompression. See
`vault(1)` for the defaults and flags.
## Cryptographic implementations
FrodoKEM-640-SHAKE is a port of CIRCL v1.6.4's implementation (constant-time
in the same places: no secret-dependent branches or indices; the
re-encryption check selects the rejection key with a mask). It reproduces
CIRCL's NIST KAT digest. ML-KEM-768 (`ml-kem`), ChaCha20-Poly1305 and
XChaCha20-Poly1305 (`chacha20poly1305`), Argon2id (`argon2`), HKDF, HMAC,
SHA-2 and SHA-3 are RustCrypto crates; TLS is rustls with `ring`. A Rust
rewrite is not stronger cryptography: the algorithms and parameters are
unchanged.
## Reporting
Please report vulnerabilities privately to andy@dixon.cx.
+25
View File
@@ -0,0 +1,25 @@
use chacha20poly1305::aead::{AeadInOut, KeyInit};
use chacha20poly1305::{ChaCha20Poly1305, Key, Nonce};
use std::time::Instant;
fn main() {
let c = ChaCha20Poly1305::new(&Key::from([1u8; 32]));
let mut buf = vec![0u8; 64 * 1024];
let n = 1024; // 64 MiB
let t = Instant::now();
for i in 0..n {
let mut nb = [0u8; 12];
nb[3..11].copy_from_slice(&(i as u64).to_be_bytes());
std::hint::black_box(c.encrypt_inout_detached(&Nonce::from(nb), &[], buf.as_mut_slice().into()).unwrap());
}
let s = t.elapsed().as_secs_f64();
println!("raw seal: {:.0} MB/s", (n * 65536) as f64 / s / 1e6);
let key = [1u8; 32];
let data = vec![0u8; 64 << 20];
let t = Instant::now();
let mut out = Vec::with_capacity(data.len() + (1 << 20));
vault::crypto::stream::seal_all(&key, &data, &mut out);
println!("seal_all: {:.0} MB/s", data.len() as f64 / t.elapsed().as_secs_f64() / 1e6);
let t = Instant::now();
let n = vault::crypto::stream::open_all(&key, &mut out).unwrap();
println!("open_all: {:.0} MB/s ({n})", data.len() as f64 / t.elapsed().as_secs_f64() / 1e6);
}
+79
View File
@@ -0,0 +1,79 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! The Rust side of scripts/bench.sh: the same operations as the Go
//! oracle's Benchmark* functions, timed as the median of repeated runs.
//! Run with `cargo bench --bench micro`.
use std::hint::black_box;
use std::io::{Read, Write};
use std::time::{Duration, Instant};
use vault::crypto::{fury, kem, stream};
fn pattern(n: usize, salt: u8) -> Vec<u8> {
(0..n).map(|i| (i.wrapping_mul(7) as u8) ^ salt).collect()
}
/// Runs `f` until about two seconds have passed (at least 5 times) and
/// prints the median and the spread.
fn bench(name: &str, bytes: usize, mut f: impl FnMut()) {
f(); // warm up
let mut times = Vec::new();
let start = Instant::now();
while times.len() < 5 || (start.elapsed() < Duration::from_secs(2) && times.len() < 10_000) {
let t = Instant::now();
f();
times.push(t.elapsed());
}
times.sort();
let med = times[times.len() / 2];
let (lo, hi) = (times[times.len() / 10], times[times.len() * 9 / 10]);
let rate = if bytes > 0 {
format!(" {:.0} MB/s", bytes as f64 / med.as_secs_f64() / 1e6)
} else {
String::new()
};
println!(
"{name:<24} median {:>10.3} ms (p10 {:.3}, p90 {:.3}, n={}){rate}",
med.as_secs_f64() * 1e3,
lo.as_secs_f64() * 1e3,
hi.as_secs_f64() * 1e3,
times.len()
);
}
fn main() {
let seed = pattern(32, 1);
bench("derive", 0, || {
black_box(kem::Identity::from_suite_seed(kem::SUITE_ID, &seed).unwrap());
});
let id = kem::Identity::from_suite_seed(kem::SUITE_ID, &seed).unwrap();
let r = id.recipient();
let fk: [u8; 32] = pattern(32, 2).try_into().unwrap();
bench("wrap", 0, || {
black_box(r.wrap(&fk, &mut vault::rng::os()));
});
let st = r.wrap(&fk, &mut vault::rng::os());
bench("unwrap", 0, || {
black_box(id.unwrap(std::slice::from_ref(&st)).unwrap());
});
let pt = pattern(4 << 20, 3);
bench("object 4MiB", pt.len(), || {
let mut ct = Vec::new();
fury::encrypt_to(&mut ct, &r, &pt, &mut vault::rng::os());
black_box(fury::decrypt_bounded(&ct[..], &id, pt.len()).unwrap());
});
let key = [0u8; 32];
let big = vec![0u8; 64 << 20];
bench("stream 64MiB", big.len(), || {
let mut w = stream::StreamWriter::new(&key, Vec::with_capacity(big.len() + (1 << 16)));
w.write_all(&big).unwrap();
let ct = w.finish().unwrap();
let mut out = Vec::with_capacity(big.len());
stream::StreamReader::new(&key, &ct[..]).read_to_end(&mut out).unwrap();
black_box(out);
});
let body = vault::keyfile::new_body(b"bench", b"", &mut vault::rng::os());
bench("unlock", 0, || {
black_box(vault::keyfile::unlock_body(&body, b"bench").unwrap());
});
}
+128 -31
View File
@@ -1,48 +1,145 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Build vault for every OS and CPU architecture Go can target without cgo. # Build vault for the supported operating systems and CPU architectures.
# #
# ./build.sh # everything, into ./dist # ./build.sh # every target this host can build, into ./dist
# TARGETS="linux/amd64 darwin/arm64" ./build.sh # TARGETS="linux/amd64 darwin/arm64" ./build.sh
# OUT=/tmp/release ./build.sh # OUT=/tmp/release JOBS=4 ./build.sh
# #
# Binaries are static, stripped and reproducible (CGO_ENABLED=0, -trimpath). # Binaries keep the Go-era names, vault-OS-ARCH[.exe], and are stripped
# iOS and most Android targets are skipped (they need cgo); see below. # release builds from the pinned toolchain (rust-toolchain.toml) and the
# 32-bit ARM Linux is built twice: armv6 (Raspberry Pi Zero/1) and armv7. # locked dependencies (Cargo.lock), with build paths remapped so the output
# SHA256SUMS is written next to the binaries. # does not depend on where the tree lives. Linux binaries are static (musl)
# except s390x, which links glibc dynamically; static linking is
# checked. SHA256SUMS is written next to the binaries.
#
# How each target is built, and what the host needs:
# zig cargo-zigbuild and zig on PATH (Linux, Windows GNU ABI)
# mac a Mac with Xcode, or zig (macOS; zig builds trust the
# bundled Mozilla roots, not the
# keychain)
# cross `cross` and Docker (FreeBSD)
# A requested target the host cannot build is reported, the others are still
# built, and the script exits non-zero. See docs/PLATFORMS.md for the Go
# targets with no Rust equivalent; asking for one is an error.
#
# JOBS builds several targets at once, each in its own target directory
# under target/build-sh (default 1: one at a time, cargo using every CPU).
set -euo pipefail set -euo pipefail
cd "$(dirname "$0")" cd "$(dirname "$0")"
root=$PWD
table() {
# os/arch Rust target method
cat <<-'EOF'
linux/amd64 x86_64-unknown-linux-musl zig
linux/arm64 aarch64-unknown-linux-musl zig
linux/386 i686-unknown-linux-musl zig
linux/armv6 arm-unknown-linux-musleabihf zig
linux/armv7 armv7-unknown-linux-musleabihf zig
linux/riscv64 riscv64gc-unknown-linux-musl zig
linux/loong64 loongarch64-unknown-linux-musl zig
linux/ppc64le powerpc64le-unknown-linux-musl zig
linux/s390x s390x-unknown-linux-gnu zig
windows/amd64 x86_64-pc-windows-gnu zig
windows/386 i686-pc-windows-gnu zig
windows/arm64 aarch64-pc-windows-gnullvm zig
darwin/amd64 x86_64-apple-darwin mac
darwin/arm64 aarch64-apple-darwin mac
freebsd/amd64 x86_64-unknown-freebsd cross
freebsd/386 i686-unknown-freebsd cross
EOF
}
# Go-era targets no longer released (the release covers Linux, Windows,
# macOS and FreeBSD). See docs/PLATFORMS.md for why each was dropped.
unsupported="aix/ppc64 android/arm64 dragonfly/amd64 freebsd/arm freebsd/arm64 freebsd/riscv64
illumos/amd64 js/wasm linux/mips linux/mipsle linux/mips64 linux/mips64le linux/ppc64
netbsd/386 netbsd/amd64 netbsd/arm netbsd/arm64 openbsd/386 openbsd/amd64 openbsd/arm
openbsd/arm64 openbsd/ppc64 openbsd/riscv64 plan9/386 plan9/amd64 plan9/arm solaris/amd64 wasip1/wasm"
out=${OUT:-dist} out=${OUT:-dist}
# iOS and 32-bit/x86 Android need cgo and a vendor SDK. android/arm64 builds jobs=${JOBS:-1}
# a static binary that runs in Termux. targets=${TARGETS:-$(table | awk '{print $1}')}
targets=${TARGETS:-$(go tool dist list | grep -vE '^ios/|^android/(386|amd64|arm)$')}
jobs=${JOBS:-$(getconf _NPROCESSORS_ONLN 2>/dev/null || echo 4)}
rm -rf "$out" # Never clean anything but our own output, and never in a directory that
mkdir -p "$out" # matters: OUT must be a dedicated directory, not /, $HOME or the tree.
mkdir -p -- "$out"
out=$(cd -- "$out" && pwd -P)
case $out in
/ | "$(cd ~ && pwd -P)" | "$root") echo "refusing OUT=$out: use a dedicated directory" >&2; exit 2 ;;
esac
rm -f -- "$out"/vault-* "$out"/SHA256SUMS
export CARGO_TARGET_DIR_BASE=$root/target/build-sh
export RUSTFLAGS="${RUSTFLAGS:-} --remap-path-prefix=$root=vault --remap-path-prefix=${CARGO_HOME:-$HOME/.cargo}=cargo"
export OUT_DIR=$out JOBS_N=$jobs HOST_OS=$(uname -s) root
logs=$root/target/build-sh-logs
mkdir -p -- "$logs"
export logs
build_one() { build_one() {
local os=${1%/*} arch=${1#*/} variants=("") name ext="" local want=$1 row os triple method name ext="" tdir bin
[[ $os == windows ]] && ext=.exe row=$(table | awk -v t="$want" '$1 == t')
[[ $arch == wasm ]] && ext=.wasm if [[ -z $row ]]; then
[[ $os/$arch == linux/arm ]] && variants=(6 7) if grep -qw -- "$want" <<<"$unsupported"; then
for v in "${variants[@]}"; do echo "UNSUPPORTED $want: not released; see docs/PLATFORMS.md" >&2
name=vault-$os-$arch${v:+v$v}$ext else
if ! GOOS=$os GOARCH=$arch GOARM=${v:-} CGO_ENABLED=0 \ echo "UNKNOWN $want" >&2
go build -trimpath -buildvcs=false -ldflags='-s -w -buildid=' -o "$OUT_DIR/$name" . 2>"$OUT_DIR/$name.log"; then fi
echo "FAILED $name" >&2
cat "$OUT_DIR/$name.log" >&2
return 1 return 1
fi fi
rm "$OUT_DIR/$name.log" read -r _ triple method <<<"$row"
echo "built $name" os=${want%/*}
done [[ $os == windows ]] && ext=.exe
name=vault-${want/\//-}$ext
tdir=$root/target
[[ $JOBS_N != 1 ]] && tdir=$CARGO_TARGET_DIR_BASE/$triple
case $method in
zig)
command -v cargo-zigbuild >/dev/null && command -v zig >/dev/null ||
{ echo "SKIPPED $want: needs cargo-zigbuild and zig" >&2; return 1; }
cmd=(cargo zigbuild) ;;
mac)
if [[ $HOST_OS == Darwin ]]; then
cmd=(cargo build)
elif command -v cargo-zigbuild >/dev/null && command -v zig >/dev/null; then
# Without Apple's SDK there is no Security framework to link, so
# the cross-built binary trusts the bundled Mozilla roots rather
# than the macOS keychain.
cmd=(cargo zigbuild --no-default-features)
else
echo "SKIPPED $want: needs a Mac, or cargo-zigbuild and zig" >&2; return 1
fi ;;
cross)
command -v cross >/dev/null && command -v docker >/dev/null ||
{ echo "SKIPPED $want: needs cross and docker" >&2; return 1; }
cmd=(cross build) ;;
esac
local flags=$RUSTFLAGS
# zig's musl getrandom() never reaches the kernel on riscv64 (it spins);
# make the getrandom crate issue the system call itself.
[[ $triple == riscv64*-musl ]] && flags+=' --cfg getrandom_backend="linux_raw"'
if ! RUSTFLAGS=$flags CARGO_TARGET_DIR=$tdir "${cmd[@]}" --release --locked --target "$triple" >"$logs/$name.log" 2>&1; then
echo "FAILED $want ($triple)" >&2
tail -20 "$logs/$name.log" >&2
return 1
fi
bin=$tdir/$triple/release/vault$ext
cp -- "$bin" "$OUT_DIR/$name"
if [[ $triple == *-musl* ]] && command -v file >/dev/null; then
file -L "$OUT_DIR/$name" | grep -q 'static' || { echo "FAILED $want: not statically linked" >&2; return 1; }
fi
echo "built $name ($triple)"
} }
export -f build_one export -f build_one table
export OUT_DIR=$out export unsupported
printf '%s\n' $targets | xargs -P "$jobs" -I{} bash -c 'build_one {}' status=0
printf '%s\n' $targets | xargs -P "$jobs" -I{} bash -c 'build_one {}' || status=1
cd "$out" cd -- "$out"
if ls vault-* >/dev/null 2>&1; then
if command -v sha256sum >/dev/null; then sha256sum vault-* >SHA256SUMS; else shasum -a 256 vault-* >SHA256SUMS; fi if command -v sha256sum >/dev/null; then sha256sum vault-* >SHA256SUMS; else shasum -a 256 vault-* >SHA256SUMS; fi
echo "$(ls vault-* | wc -l | tr -d ' ') binaries in $out" fi
echo "$(ls vault-* 2>/dev/null | wc -l | tr -d ' ') binaries in $out"
exit $status
+33
View File
@@ -0,0 +1,33 @@
# cargo-deny policy: run `cargo deny --locked check`.
[graph]
all-features = true
[advisories]
# Every advisory is an error unless listed here with a reason.
ignore = []
[licenses]
# Licences compatible with distributing vault under GPL-3.0-or-later.
allow = [
"Apache-2.0",
"Apache-2.0 WITH LLVM-exception",
"BSD-3-Clause",
"CDLA-Permissive-2.0",
"GPL-3.0-or-later",
"ISC",
"MIT",
"Unicode-3.0",
"Unlicense",
"Zlib",
]
confidence-threshold = 0.9
[bans]
multiple-versions = "warn"
wildcards = "deny"
[sources]
unknown-registry = "deny"
unknown-git = "deny"
allow-registry = ["https://github.com/rust-lang/crates.io-index"]
+85
View File
@@ -0,0 +1,85 @@
# Benchmarks: Go 0.5 against Rust 0.6
Measured on 2026-10-08 on one machine: Intel Core i7-7700 (4 cores, 8
threads), 31 GiB RAM, ext4 on a two-disk RAID 1 (`/dev/md2`), Linux 7.0.
Both binaries were built from this tree: the Go oracle (`oracle/go`, Go
1.27.1, `CGO_ENABLED=0`) and the Rust release build (Rust 1.99.0, LTO).
**Limitation:** the machine was shared. Other users' Docker builds kept the
load average between 4 and 12 throughout, so absolute times are noisy.
Go and Rust were run alternately under the same conditions, and every
figure is a median with its spread, but small differences mean nothing.
Page caches were warm (dropping them needs root), so cold-cache numbers
were not measured.
## Primitives
`cargo bench --bench micro` and
`(cd oracle/go && go test -run XXX -bench Oracle -benchtime 2s ./ ./internal/pqcrypt ./internal/stream)`,
run alternately twice (load 4.4 to 5.4). Medians in milliseconds; the Rust
harness's p10 to p90 range is in brackets.
| Operation | Go | Rust |
| --- | --- | --- |
| Derive hybrid identity from seed | 5.6 | 3.0 (2.9 to 4.6) |
| Wrap file key (both encapsulations) | 7.4 to 8.0 | 3.1 (3.0 to 4.8) |
| Unwrap file key (both decapsulations) | 7.6 to 8.3 | 3.1 (3.0 to 3.6) |
| Encrypt and decrypt one 4 MiB object | 19.9 to 20.1 | 19.0 to 19.5 |
| STREAM seal and open, 64 MiB, streaming API | 89 to 92 | 187 to 188 |
| Unlock key file (Argon2id for both slots) | 121 to 125 | 129 (125 to 147) |
- The FrodoKEM port is about twice as fast as CIRCL's Go code, mostly from
multiplying A row by row instead of walking it by column.
- ChaCha20-Poly1305 is slower: RustCrypto reaches about 1.2 GB/s on this CPU
(`cargo bench --bench aead_probe`), Go's assembly about twice that. The
64 MiB row uses the streaming reader and writer, which copy through
64 KiB buffers. The vault itself uses the in-memory path (`seal_all`
0.85 GB/s, `open_all` 1.4 GB/s), and per 4 MiB object the two are
level because the KEM work is cheaper.
- Unlock takes 180 ms with Argon2id lanes computed in turn. Computing the
four lanes in parallel (argon2's `parallel` feature) brings it to 129 ms,
close to Go. The parameters are unchanged.
## End to end
`RUNS=3 BIG_MB=1024 SMALL=500 scripts/bench.sh` (load 8 to 13). Every step
includes unlocking the key. The 1 GiB file and the 500 small files (1 to
16 KiB each) are each added to a fresh vault. The reads use one vault
holding both, plus filler. Times are medians of 3 (min to max).
| Step | Go | Rust |
| --- | --- | --- |
| add 1 GiB file | 41.7 s (32.0 to 50.3) | 29.6 s (28.5 to 31.0) |
| add 500 small files | 123.9 s (113.3 to 156.2) | 79.0 s (77.3 to 133.3) |
| get 1 GiB file | 9.8 s (8.8 to 10.0) | 16.2 s (15.5 to 16.7) |
| list (about 530 records) | 4.95 s (4.89 to 5.20) | 0.74 s (0.71 to 0.77) |
| get one small file | 5.24 s (4.88 to 6.00) | 0.74 s (0.70 to 0.81) |
| Peak RSS | 137 to 140 MiB | 132 MiB |
| Peak threads | 18 | 17 |
| Binary size | 11.5 MB | 3.5 MB (static musl: 3.4 MB) |
- **Catalogue reads** (`list`, and every command that looks a name up) are
about 6 times faster. Records are decrypted in parallel on the worker pool
(bounded at twice the worker count in flight), and each FrodoKEM
decapsulation is cheaper.
- **Adds** are faster. Both implementations fsync every object, its shard
directory and every record. Rust overlaps up to eight objects' encryption
and syncs on the pool.
- **`get` of a large file is slower, on purpose.** Rust fsyncs the output
file before giving it its name, so a crash just after `get` cannot leave
the name pointing at an empty or partial file. Go did not. On an idle
rerun (load about 8, output on the same disk), Go took 3.6 s and Rust
15.5 s. Of Rust's time, 13.3 s was the single `fsync` of 1 GiB of
freshly written data (measured with `strace -T`). The other 2 s or so is
unlock, decryption and writing. This is the cost of the
durability barrier on this RAID 1 array, and it scales with the disk's
write speed.
- Peak memory is dominated by Argon2id: 64 MiB for each of the two slots,
computed at the same time in both implementations. Rust's object pipeline
adds at most eight 4 MiB plaintext buffers and their ciphertexts. Peak
threads: Rust has 8 workers, 8 Argon2 threads while unlocking, and the
main thread.
Reproduce with `scripts/bench.sh` (it builds both binaries, creates
synthetic data in `target/bench`, and prints the tables' numbers) and the
two micro-benchmark commands above.
+187
View File
@@ -0,0 +1,187 @@
# Compatibility specification
This is the on-disk and on-wire contract between the Go implementation
(frozen in `oracle/go`, commit `4c1507c`) and the Rust implementation. It is
derived from the Go sources and pinned by vectors the Go oracle generated
(`tests/vectors`, regenerated with `oracle/gen-vectors.sh`). The Rust port is
an implementation migration, not a format migration: nothing here changes,
no data is rekeyed or converted, and no new suite becomes the default.
Each section names the tests that hold the Rust code to it. "Go" below always
means the oracle.
## 1. Key file
| Item | Value |
| --- | --- |
| Encoding | one PEM block, type `VAULT KEY`, standard base64 with padding, 64 columns, as Go's `pem.Encode` |
| Body | exactly 198 bytes: an 18-byte header and two 90-byte slots |
| Header | `VAULTKEY1`, Argon2id time `3` (u32 BE), memory `65536` KiB (u32 BE), lanes `4` (u8) |
| Slot | 16-byte salt, 24-byte XChaCha20-Poly1305 nonce, 34 bytes sealed (u16 BE suite ID, 32-byte master seed), 16-byte tag |
| Slot key | Argon2id v1.3 (passphrase bytes as given, no normalisation, salt, t=3, m=64 MiB, p=4, 32 bytes) |
| Associated data | the 18-byte header |
| Second slot | a decoy identity's slot, or 90 random bytes; which slot is real is random |
| Unlock | Argon2id runs for both slots every time (Rust runs them concurrently); the first slot that opens wins; "incorrect passphrase" if neither |
Accepted framing: optional text before the block (Go's `pem.Decode`), only
whitespace after it, header lines inside the block ignored. Files over 64 KiB
are refused outright (Go would read them, then reject them).
Tests: `keyfile::tests::*` (Argon2id vectors, Go-written key files with and
without decoy open to the Go-recorded seeds, byte-identical PEM re-encoding,
malformed framing, NFC/NFD passphrases differ).
## 2. Hybrid suite `0x0001`, tag `frodo640shake+mlkem768`
- Master seed: 32 bytes. Expansion:
`HKDF-SHA-256(ikm = seed, salt = none, info = "fury/v1 seed-expand frodo640shake+mlkem768")`,
read as 48 bytes of FrodoKEM seed then 64 bytes of ML-KEM seed.
- FrodoKEM-640-SHAKE exactly as CIRCL v1.6.4 `kem/frodo/frodo640shake`
(the round-3 FrodoKEM, not the ISO/eFrodoKEM variants): key seed
`s(16) || seedSE(16) || z(16)`; public key 9616 bytes (`seedA || pack(B)`);
private key 19888 bytes (`s || pk || S as u16 LE || H(pk)`); ciphertext 9720
bytes; 16-byte shared secret; 16-byte encapsulation seed `mu`; implicit
rejection returns `SHAKE128(ct || s)`.
- ML-KEM-768 (FIPS 203) with seed `d || z`; encapsulation key 1184 bytes,
expanded decapsulation key 2400 bytes, ciphertext 1088 bytes, 32-byte
shared secret, implicit rejection.
- Public key: FrodoKEM public key `||` ML-KEM encapsulation key (10800 bytes).
Private key (for comparison only): FrodoKEM private `||` ML-KEM expanded.
- Wrap: fresh, independent randomness for both encapsulations every time.
`KEK = HKDF-SHA-256(ikm = ss1 || ss2, salt = none, info = "fury/v1 kek frodo640shake+mlkem768" || ct1 || ct2)`;
the 32-byte file key is sealed with ChaCha20-Poly1305 under the KEK and an
all-zero 12-byte nonce. The zero nonce is safe only because each KEK is new
and used once; the Rust code keeps that sealing function private to the
suite.
- Stanza: type `frodo640shake+mlkem768`, two arguments
`base64(ct1)`, `base64(ct2)`, body = 48-byte sealed file key.
- Unwrap: a stanza of another type is "not mine"; wrong argument count,
bad base64 or wrong ciphertext lengths are fatal for the file; an AEAD
failure (wrong key) is "not mine". The first stanza that opens wins.
Rust implementation: FrodoKEM is a line-by-line port of CIRCL
(`src/crypto/frodo.rs`) because no maintained Rust crate implements CIRCL's
variant (RustCrypto `frodo-kem` 0.1 implements the salted ISO version with
9752-byte ciphertexts). ML-KEM is RustCrypto `ml-kem` 0.3.2.
Tests: `crypto::frodo::tests::nist_kat_digest` (reproduces the SHA-256
CIRCL checks against the FrodoKEM reference KAT file),
`crypto::kem::tests::{frodo_matches_circl, mlkem_matches_circl,
hybrid_derivation_matches_go, wrap_matches_go_and_unwraps,
fresh_wraps_differ_and_foreign_keys_do_not_match}`.
## 3. Fury header
```text
fury-encryption.org/v1\n
-> TYPE ARG ARG\n per stanza; TYPE and ARGs are 1+ bytes of 0x21-0x7e
BODY, 64 base64 columns per line, ending with a line shorter than 64
--- MAC\n MAC = base64 of 32 bytes
```
- Base64: standard alphabet, no padding, non-zero trailing bits rejected,
CR or LF anywhere in an encoded value rejected.
- A body whose encoding fills its last line (the 48-byte wrapped key encodes
to exactly 64 characters) is followed by an empty line; serialisation is
canonical and Rust reproduces Go's bytes exactly.
- Limits: 1 MiB per stanza (all its lines), 16 MiB per header; a line must
end in LF; three bytes of look-ahead decide between a stanza and the
footer; the payload starts immediately after the footer's LF, with any
read-ahead returned first.
- MAC: `HMAC-SHA-256(key = HKDF-SHA-256(file key, salt none, info "header"), message = header up to and including "---")`,
compared in constant time.
- Payload: 16-byte nonce, then STREAM keyed by
`HKDF-SHA-256(file key, salt = nonce, info "payload")`.
Tests: `crypto::fury::tests::{header_vectors_match_go,
header_parser_agrees_with_go, decrypts_go_files}` (the parser's accept or
reject verdict, and the payload bytes left after the header, match Go on
every case in `crypto.json`), `crypto::header::tests::*`,
`tests/interop_crypto.rs` (Go decrypts files Rust wrote, and rejects
truncated, extended and corrupted ones).
## 4. STREAM
- 64 KiB plaintext chunks; ChaCha20-Poly1305; 16-byte tag per chunk.
- Nonce: 12 bytes, bytes 3..11 the big-endian chunk index, last byte `1`
on the final chunk only, `0` otherwise.
- Only the final chunk may be short; it may be empty only when it is the
only chunk (empty plaintext). A plaintext that is an exact multiple of
64 KiB ends with a full-length final chunk.
- Rejected: no final chunk (truncation), an empty final chunk after
others, bytes after the final chunk, any chunk failing authentication,
a payload length no STREAM encoding can have.
- The index never wraps; Rust refuses to go past 2^64 − 1 chunks (Go's
counter is 88 bits; neither limit is reachable in practice).
- Random access (`StreamReaderAt`) authenticates the final chunk when
constructed, then any chunk on demand.
Tests: `crypto::stream::tests::*` (Go vectors for sizes around every
boundary, any write/read step sizes, malformations rejected identically by
the streaming, in-memory and random-access readers, concurrency of the
random-access reader, counter overflow).
## 5. Store
- `DIR/.vault` contains exactly `VAULT 2\n`.
- `DIR/records/xx/ID` and `DIR/objects/xx/ID`: `ID` is 16 bytes as
lowercase hex, `xx` its first byte; all 256 shards of both exist after
`init`. Entries whose names are not exactly that form, or are in the
wrong shard, are ignored.
- Every ciphertext, every shard and both top directories have access and
modification times pinned to the Unix epoch; removals re-pin the shard.
- Record: a Fury file whose plaintext is exactly 4096 bytes: a u32 BE JSON
length (≤ 4092), the JSON, random padding. JSON is Go's `encoding/json`
output for `{"name","seed","size","mode","modified","added"}`: HTML
characters and U+2028/2029 escaped, times RFC 3339 with up to nine
fractional digits, trailing zeros removed, `Z`. Rust writes it
byte-identically.
- Validation on read: name valid (below), seed 64 lowercase hex characters,
size 0..=1 TiB, both times valid RFC 3339; unknown fields refused.
- Names: non-empty UTF-8, ≤ 1024 bytes, already clean (Go `path.Clean`),
relative, not `.`/`..`/`../…`, no backslash, no control characters.
- Object IDs: `HMAC-SHA-256(key = seed, msg = index as big-endian i64)[:16]`.
- Objects: plaintext = own ID (16 bytes), then up to `4 MiB − 16` file
bytes, then random padding to the next power of two from 64 KiB to 4 MiB.
An empty file still has one object. These sizes are plaintext; the
ciphertext adds the header (~14.6 KB) and 16 bytes per 64 KiB.
- Reading an object checks: it opens, its first 16 bytes are its ID, its
length is exactly the padded size for its part, and the whole STREAM
(final chunk, end of file) authenticates.
- Records the key cannot open (no stanza matches) are skipped silently;
ones that open but are malformed are reported and skipped.
- Duplicates (an interrupted replace) are ordered newest `added` first; the
next write of that name removes them all.
- Filler: `init` writes 8–32 entries (or more, to reach `--filler`) of
log-uniform sizes from 1 KiB to `clamp(target/8, 4 MiB, 1 GiB)`, sealed to
a throwaway key, through the same code as `add`.
Tests: `store::types::tests::*` (names, object IDs, padded sizes, object
counts, record JSON byte-identical to Go), `store::tests::reads_go_vault`
(the oracle's fixture vault: real files, decoy file, foreign record),
`store::tests::*`, `tests/cli.rs` lifecycle tests in both directions.
## 6. Locks
`DIR/.lock`, created mode 0600. Unix: POSIX `fcntl` record locks over the
whole file (`F_SETLK` then `F_SETLKW`), shared for readers, exclusive for
writers, released by the kernel when the process dies. Windows:
`LockFileEx` on byte 0, length 1. Other systems: an `O_EXCL` lock file.
Tests: `tests/cli.rs::lock_*` (Rust waits for Go, Go waits for Rust, both
proceed after the holder is killed).
## 7. Agent
Socket `$XDG_RUNTIME_DIR` (else `$TMPDIR`, else `/tmp`) `/vault-UID/agent-`
followed by the first 8 bytes of SHA-256 of the key file's absolute path
(Go `filepath.Abs`, including its use of `$PWD`) in hex. The directory must
be the user's own with mode exactly 0700. One request per connection: `g`
→ 34 bytes (u16 BE suite ID, seed); `q` → the agent wipes the key and
closes. Two-second deadline per request. Tests:
`tests/cli.rs::agent_*` (Rust client with Go agent and the reverse).
## 8. What is deliberately not identical
See `docs/MIGRATION.md` for the full list of intentional behaviour changes.
None of them alters a byte that is written to the vault or the key file
beyond what the Go format already allows (fresh randomness, timestamps).
+103
View File
@@ -0,0 +1,103 @@
# Migrating from the Go implementation
vault 0.6 is a rewrite in Rust of the Go vault (0.5 and earlier). It is an
implementation migration only: key files, vaults, the lock and the unlock
agent are the same formats, byte for byte where the format is
deterministic (see `COMPATIBILITY.md`). There is nothing to convert. Go and
Rust builds can be used on the same vault, even at the same time: they
honour each other's locks, and either can use an agent the other started.
To go back, install 0.5: everything 0.6 writes, 0.5 reads.
## Intentional behaviour changes
Safety and correctness:
- **`get` never replaces anything.** Output is published with an atomic
no-replace rename or link, so a file, a symlink (even a dangling one) or a
file created while `get` runs is left alone. Go checked first and then
renamed, which a concurrent creator could race.
- **`get -r` does not follow symlinks inside the output directory.** Each
directory below the chosen one is opened without following links; a link
or non-directory in the way stops the run. (Windows: checked just before
use, see `SECURITY.md`.)
- **Extracted files are synced** to disk before they appear under their
name, and their directory afterwards.
- **`get` needs an atomic no-replace primitive.** On a filesystem with
neither `RENAME_NOREPLACE` nor hard links (some FUSE mounts, FAT) it fails
with a message saying so, where Go would rename.
- **`add`/`migrate` refuse paths inside the vault directory** (Go refused
only the directory itself; `migrate store/records` would have deleted the
vault's records).
- **`absorb` refuses GNU sparse files** (Go expanded them), and honours PAX
size records as Go does.
- **A source file that changes while being added is not stored.** If it
grows, shrinks, is modified or replaced during `add`/`migrate`, the
command fails and nothing is stored. Go stored whatever it read (for a
file that grew, a truncated copy).
- **`migrate` deletes an original only if it is still the file stored**
(same inode, size and time), after the record is durable.
- **Deletes are durable in order.** Removing a file syncs the removal of its
record before deleting its objects, so a crash cannot leave a record
pointing at deleted objects. A failed sync after a record is already in
place no longer deletes its objects.
- **Lock errors are reported.** Go let a reader continue without the lock on
any error; Rust does so only on a read-only filesystem.
- **Stricter records.** A record must have every field, with exactly the
field names Go writes, and nothing after the JSON. Go also accepted
differently-cased names, missing fields (as zero) and trailing bytes;
Go never writes such records, so no existing vault is affected.
- **Key files over 64 KiB** are rejected without reading them whole.
Agent (`unlock`):
- The agent **refuses the key if it cannot protect it**: locking its memory,
disabling core dumps and (Linux) becoming undumpable must all succeed. Go
ignored failures there.
- Its environment is stripped of `VAULT_PASSPHRASE` and
`VAULT_DECOY_PASSPHRASE`; it runs in `/` with standard error closed.
- Expiry no longer waits for the request in progress beyond its deadline,
and a key is never handed out after expiry.
- `unlock` waits at most ten seconds for the agent to start and kills it on
failure; the agent removes its socket only if it is still its own.
- **Windows: `unlock` is not available** (Go's agent there had no memory
protection). Every command asks for the passphrase.
Network (`add URL`, `mirror`):
- Explicit limits, all adjustable: connect 30 s, response headers 60 s,
whole request 2 h (`add`) or 5 min (`mirror`, per request), download size
16 GiB (`add`) or 4 GiB in total (`mirror`), 10 redirects, 10 000 mirror
pages, 16 MiB per HTML or CSS page. Bytes are counted after decompression.
New flags `--max-bytes`, `--timeout`, `--max-pages`. Go had no limits.
- Redirects go to http or https only; a `mirror` redirect to another host is
skipped (Go followed it).
- `mirror` resolves links against the final address after redirects and the
page's `<base href>`; it parses HTML instead of matching a pattern, so it
rewrites only `href`, `src`, `action` and `poster` on real tags, plus
`url(...)` in CSS, `<style>` and `style=`, keeping every other byte and
the original quoting. Text in scripts and comments is no longer touched.
- `mirror` never lets two different URLs in one crawl overwrite each other's
file; the second is skipped. It exits with status **3** when it stored
some resources but skipped others (Go always exited 0).
- Proxies follow `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` as in Go; local
addresses are never proxied.
Command line:
- New option `--workers N` (1–256) on every command.
- `--key`/`--vault` no longer need a home directory when given explicitly.
- Command-line arguments must be valid UTF-8 (Go accepted any bytes, but
vault names had to be UTF-8 already).
- `ls`/`list` piped into a reader that stops early end quietly with
status 141, as Go's did by `SIGPIPE`.
Platforms: releases now cover Linux, Windows, macOS and FreeBSD (16
binaries, down from 43); `PLATFORMS.md` lists what was dropped and why.
## Unchanged on purpose
Formats, limits (1024-byte names, 1 TiB files, 4 MiB objects, 4096-byte
records, 16 MiB headers, 1 MiB stanzas), Argon2id parameters, the hybrid
suite, filler, decoy behaviour, messages and exit statuses (apart from the
above), the shape of `ls` and `list` output, and the agent protocol.
+73
View File
@@ -0,0 +1,73 @@
# Platforms
The Go version was built for every target of `go tool dist list` that needs
no cgo: 43 binaries. The Rust release is narrower by decision (Linux,
Windows, macOS, FreeBSD): 16 binaries. This is the target-by-target record:
how each is built, and how far it has actually been verified. "Runtime" means
a binary was executed; "build" means it only compiled and linked.
Tooling used for verification on the development machine (x86-64 Linux):
the pinned Rust 1.99.0, zig 0.16.0 with cargo-zigbuild 0.23.4, qemu-user
10.2.1 (Ubuntu package, unpacked locally, no system changes), glibc cross
sysroots from Ubuntu's `libc6-*-cross` packages, and the Go oracle built for
x86-64. `scripts/smoke.sh` is the runtime check: create a key and vault, store
files across object boundaries, read them back, then have the Go oracle read
what that binary wrote and the binary read what the oracle wrote.
## Released
The release covers Linux, Windows, macOS and FreeBSD (a decision made during
the port: the Go release's more exotic targets are no longer built).
| Go name | Rust target | Linking | Built by | Verified |
| --- | --- | --- | --- | --- |
| linux/amd64 | x86_64-unknown-linux-musl | static | zig | runtime: full test suite (glibc host build) and smoke test of the static binary |
| linux/arm64 | aarch64-unknown-linux-musl | static | zig | runtime: smoke test under qemu-aarch64, cross-checked with the Go oracle |
| linux/386 | i686-unknown-linux-musl | static | zig | runtime: smoke test under qemu-i386, cross-checked |
| linux/armv6 | arm-unknown-linux-musleabihf | static | zig | runtime: smoke test under qemu-arm, cross-checked |
| linux/armv7 | armv7-unknown-linux-musleabihf | static | zig | runtime: smoke test under qemu-arm, cross-checked |
| linux/riscv64 | riscv64gc-unknown-linux-musl | static | zig | runtime: smoke test and crypto unit tests under qemu-riscv64, cross-checked (note 1) |
| linux/loong64 | loongarch64-unknown-linux-musl | static | zig | runtime: smoke test under qemu-loongarch64, cross-checked |
| linux/ppc64le | powerpc64le-unknown-linux-musl | static | zig | runtime: smoke test under qemu-ppc64le, cross-checked |
| linux/s390x | s390x-unknown-linux-gnu | glibc, dynamic | zig | runtime: smoke test under qemu-s390x (big-endian) with an Ubuntu glibc sysroot, cross-checked |
| windows/amd64 | x86_64-pc-windows-gnu | static CRT | zig | build; tests cross-compile cleanly; runtime in CI only (windows-2025 job, not yet run) |
| windows/386 | i686-pc-windows-gnu | static CRT | zig | build only |
| windows/arm64 | aarch64-pc-windows-gnullvm | static CRT | zig | build only |
| darwin/amd64 | x86_64-apple-darwin | system | a Mac, or zig (note 2) | build only from Linux; tested in CI on macos-15-intel |
| darwin/arm64 | aarch64-apple-darwin | system | a Mac, or zig (note 2) | build only from Linux; tested in CI on macos-15 |
| freebsd/amd64 | x86_64-unknown-freebsd | system | cross 0.2.5 (Docker) | build only (no emulator) |
| freebsd/386 | i686-unknown-freebsd | system | cross 0.2.5 (Docker) | build only |
Feature notes: the `unlock` agent works on Linux, macOS and FreeBSD;
Windows asks for the passphrase every time. Full agent hardening
(`PR_SET_DUMPABLE`) is Linux-only; elsewhere the agent locks its memory and
disables core dumps. See `SECURITY.md`.
1. zig's bundled musl `getrandom()` never reaches the kernel on riscv64 (the
process spins). `build.sh` and `packaging/package.sh` build that target
with `--cfg getrandom_backend="linux_raw"`, which makes the `getrandom`
crate issue the system call itself; with that flag the binary passes.
2. Without Apple's SDK there is no Security framework to link, so `build.sh`
on Linux cross-builds macOS with `--no-default-features`: those binaries
verify TLS for `add URL` and `mirror` against the bundled Mozilla roots
instead of the macOS keychain (so a corporate CA added to the keychain is
not trusted). Built on a Mac (`cargo build --release`), vault uses the
keychain.
## No longer released
| Go name | Why |
| --- | --- |
| android/arm64 | was a copy of the static Linux arm64 binary; that binary still runs in Termux, but `add URL`/`mirror` need `/etc/resolv.conf` |
| linux/ppc64 (big-endian) | zig links it for the ELFv2 loader that BE distributions lack; built with `cross` instead, even a hello-world crashes at start-up under qemu-ppc64, so it cannot be verified |
| linux/mips, mipsle, mips64, mips64le | Rust tier 3 (no stable std) since 1.72 |
| netbsd/amd64 | builds only with a newer sysroot than cross 0.2.5 provides (std needs `libexecinfo`); netbsd/386, arm, arm64 are tier 3 |
| openbsd/* | Rust tier 3; OpenBSD packages Rust itself, so a native build there would work |
| freebsd/arm, arm64, riscv64 | Rust tier 3 |
| dragonfly/amd64, aix/ppc64 | Rust tier 3 |
| illumos/amd64, solaris/amd64 | outside the chosen matrix (illumos builds with cross; Solaris has no cross 0.2.5 image) |
| wasip1/wasm | outside the chosen matrix (it builds; no agent, no prompts) |
| js/wasm | no filesystem in the browser target |
| plan9/* | no Rust target |
Asking `build.sh` for any of these reports it and fails.
Binary file not shown.

Before

Width:  |  Height:  |  Size: 168 KiB

After

Width:  |  Height:  |  Size: 169 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 165 KiB

After

Width:  |  Height:  |  Size: 176 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 140 KiB

After

Width:  |  Height:  |  Size: 142 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 131 KiB

After

Width:  |  Height:  |  Size: 134 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 74 KiB

After

Width:  |  Height:  |  Size: 74 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 525 KiB

After

Width:  |  Height:  |  Size: 604 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 89 KiB

After

Width:  |  Height:  |  Size: 89 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 166 KiB

After

Width:  |  Height:  |  Size: 172 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 96 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 146 KiB

After

Width:  |  Height:  |  Size: 149 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 139 KiB

After

Width:  |  Height:  |  Size: 140 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 191 KiB

After

Width:  |  Height:  |  Size: 196 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 87 KiB

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 180 KiB

After

Width:  |  Height:  |  Size: 195 KiB

+6 -3
View File
@@ -7,7 +7,9 @@ sandbox path and file owner are shown as /home/sam and "sam sam", key lines
are highlighted in amber for the reader, and "moment" frames show a prompt are highlighted in amber for the reader, and "moment" frames show a prompt
with a cursor, exactly as vault prints it before you type. with a cursor, exactly as vault prints it before you type.
Needs: go, python3, and Chrome or Chromium (set CHROME=/path if not found). Needs: cargo (with the pinned toolchain), cargo-zigbuild and zig for the
build screenshot, python3, and Chrome or Chromium (set CHROME=/path if not
found).
Output: docs/guide/img/*.png Output: docs/guide/img/*.png
""" """
@@ -75,7 +77,8 @@ class Sandbox:
self.bin = os.path.join(self.root, "bin") self.bin = os.path.join(self.root, "bin")
os.makedirs(self.home) os.makedirs(self.home)
os.makedirs(self.bin) os.makedirs(self.bin)
subprocess.run(["go", "build", "-o", os.path.join(self.bin, "vault"), "."], cwd=REPO, check=True) subprocess.run(["cargo", "build", "--release", "--locked", "--quiet"], cwd=REPO, check=True)
shutil.copy2(os.path.join(REPO, "target", "release", "vault"), os.path.join(self.bin, "vault"))
def env(self, extra=None): def env(self, extra=None):
e = { e = {
@@ -406,7 +409,7 @@ def main():
# building, into a temporary OUT so an existing dist/ is left alone # building, into a temporary OUT so an existing dist/ is left alone
rel = tempfile.mkdtemp(prefix="vault-release-") rel = tempfile.mkdtemp(prefix="vault-release-")
targets = "linux/amd64 linux/arm darwin/arm64 windows/amd64" targets = "linux/amd64 linux/arm64 linux/armv7 windows/amd64"
t = Term(box, title="sam@laptop: ~/src/vault") t = Term(box, title="sam@laptop: ~/src/vault")
out = subprocess.run(["./build.sh"], cwd=REPO, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, out = subprocess.run(["./build.sh"], cwd=REPO, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
text=True, env=dict(os.environ, TARGETS=targets, OUT=rel)).stdout text=True, env=dict(os.environ, TARGETS=targets, OUT=rel)).stdout
+70 -25
View File
@@ -186,6 +186,26 @@ Download the one file for your system, check its SHA-256, rename it to
Or build every platform yourself with \cmd{./build.sh}. Or build every platform yourself with \cmd{./build.sh}.
\end{tldr} \end{tldr}
\section{The easy way on Linux: my package repository}
On Debian, Ubuntu, Fedora, RHEL, openSUSE or Arch, add my signed repository
once and install the \cmd{dixon-vault} package with your usual package
manager. You get the manual pages too, and new releases arrive with your
normal system updates. On Debian or Ubuntu:
\begin{shell}
$ sudo install -d -m 0755 /etc/apt/keyrings
$ curl -fsSL https://repo.dixon.cx/dixon.gpg | sudo tee /etc/apt/keyrings/dixon.gpg >/dev/null
$ echo "deb [signed-by=/etc/apt/keyrings/dixon.gpg] https://repo.dixon.cx/apt stable main" |
sudo tee /etc/apt/sources.list.d/dixon.list
$ sudo apt update && sudo apt install dixon-vault
\end{shell}
The commands for dnf, zypper and pacman are at \file{dixon.cx/repo}. The
package is called \cmd{dixon-vault} because plain \cmd{vault} is HashiCorp
Vault's name; the command it installs is still \cmd{vault}. Everyone else,
read on.
\section{Pick your file} \section{Pick your file}
vault is a single self-contained program. No installer, no runtime, no vault is a single self-contained program. No installer, no runtime, no
@@ -201,15 +221,17 @@ Linux PC or server & almost everything & vault-linux-amd64 \\
Linux on ARM64 & Pi 4/5 (64-bit OS), Graviton, Ampere & vault-linux-arm64 \\ Linux on ARM64 & Pi 4/5 (64-bit OS), Graviton, Ampere & vault-linux-arm64 \\
Raspberry Pi, 32-bit & Pi 2, 3, 4 on a 32-bit OS & vault-linux-armv7 \\ Raspberry Pi, 32-bit & Pi 2, 3, 4 on a 32-bit OS & vault-linux-armv7 \\
Raspberry Pi Zero / 1 & the tiny older ones & vault-linux-armv6 \\ Raspberry Pi Zero / 1 & the tiny older ones & vault-linux-armv6 \\
Android & in Termux & vault-android-arm64 \\ Android & in Termux: the Linux ARM64 file & vault-linux-arm64 \\
FreeBSD, OpenBSD, NetBSD & amd64, arm64, 386, arm\ldots & vault-freebsd-amd64 etc. \\ FreeBSD & amd64 or 386 & vault-freebsd-amd64 etc. \\
Something exotic & RISC-V, POWER, s390x, MIPS, LoongArch & see \cmd{dist/} \\ Something exotic & RISC-V, POWER (little-endian), s390x, LoongArch & see \cmd{dist/} \\
\end{vtable} \end{vtable}
\begin{squirrel} \begin{squirrel}
The full release covers 43 platform builds, including AIX, Solaris, illumos, The release covers sixteen builds for Linux, Windows, macOS and FreeBSD,
Plan~9 and even WebAssembly. Will you ever need \file{vault-plan9-arm}? from Windows on ARM to IBM mainframes. The older Go builds reached further,
Almost certainly not. Is it pleasing that it exists? Extremely. to Plan~9 and WebAssembly; \file{docs/PLATFORMS.md} says why those went. Will you ever need
\file{vault-linux-s390x}? Almost certainly not. Is it pleasing that it
exists? Extremely.
\end{squirrel} \end{squirrel}
\begin{headsup} \begin{headsup}
@@ -267,36 +289,40 @@ so Gatekeeper quarantines it. Once you've checked the SHA-256, clear the flag:
\subsection{Android (Termux)} \subsection{Android (Termux)}
\begin{shell} \begin{shell}
$ curl -LO https://.../vault-android-arm64 $ curl -LO https://.../vault-linux-arm64
$ chmod +x vault-android-arm64 && mv vault-android-arm64 $PREFIX/bin/vault $ chmod +x vault-linux-arm64 && mv vault-linux-arm64 $PREFIX/bin/vault
\end{shell} \end{shell}
\section{Did it work?} \section{Did it work?}
Run \cmd{vault help}. You should see the full command list: Run \cmd{vault help}. You should see the full command list:
\shot{help}{\cmd{vault help}: everything vault can do, on one screen} \shot[0.95]{help}{\cmd{vault help}: everything vault can do, on one screen}
\section{Building it yourself} \section{Building it yourself}
Prefer to build from source? You need Go 1.26.6 or newer. Then: Prefer to build from source? You need Rust, through \cmd{rustup}, which
fetches the exact compiler version the project pins. Then:
\begin{shell} \begin{shell}
$ go build . # just your platform, produces ./vault $ cargo build --release --locked # just your platform: target/release/vault
$ ./build.sh # every platform, into ./dist, with SHA256SUMS $ ./build.sh # every platform it can, into ./dist, with SHA256SUMS
\end{shell} \end{shell}
\cmd{build.sh} cross-compiles every target Go supports without C libraries. \cmd{build.sh} cross-compiles Linux and Windows with \cmd{cargo-zigbuild}
and \cmd{zig}, macOS on a Mac, and FreeBSD with \cmd{cross} and Docker; it says which targets your machine can't build.
32-bit ARM Linux gets two builds, armv6 and armv7. Want just a few? Set 32-bit ARM Linux gets two builds, armv6 and armv7. Want just a few? Set
\env{TARGETS}. Want them somewhere else? Set \env{OUT}. \env{TARGETS}. Want them somewhere else? Set \env{OUT}.
\shot{build}{A partial build, followed by checking every binary against SHA256SUMS} \shot{build}{A partial build, followed by checking every binary against SHA256SUMS}
\begin{whyso} \begin{whyso}
The binaries are \emph{static} and \emph{reproducible}: no C code, no system The Linux binaries are \emph{static}: they need no system libraries at all.
libraries, file paths stripped, build IDs zeroed. Two people building the same Every build uses the pinned compiler and the locked versions of every
commit get byte-identical files, so you can check a release against your own dependency, with build paths stripped, so the result doesn't depend on whose
build. machine or which folder built it. Built with the same compiler for the same
target, the same commit should give the same file, so you can check a
release against your own build.
\end{whyso} \end{whyso}
\begin{pocket} \begin{pocket}
@@ -304,7 +330,7 @@ build.
\item One file per platform. Rename it \cmd{vault} and put it on your \env{PATH}. \item One file per platform. Rename it \cmd{vault} and put it on your \env{PATH}.
\item Always check the SHA-256 against \file{SHA256SUMS}. \item Always check the SHA-256 against \file{SHA256SUMS}.
\item macOS: clear the quarantine flag after checking. \item macOS: clear the quarantine flag after checking.
\item \cmd{./build.sh} builds all 43 platform binaries from source. \item \cmd{./build.sh} builds every platform your machine can, from source.
\end{pocketlist} \end{pocketlist}
\end{pocket} \end{pocket}
@@ -714,13 +740,16 @@ vault.
\begin{shell} \begin{shell}
$ vault unlock # ask once, remember the key for 5 minutes $ vault unlock # ask once, remember the key for 5 minutes
$ vault unlock --for 30m # or for as long as you say $ vault unlock --for 30m # or for as long as you say
$ vault unlock --idle # 5 minutes since you last used it
$ vault lock # forget it right now $ vault lock # forget it right now
\end{shell} \end{shell}
After \cmd{unlock}, every command skips the passphrase prompt until the time After \cmd{unlock}, every command skips the passphrase prompt until the time
is up. It's a hard deadline, counted from when you unlocked, not from when you is up. Normally that's a hard deadline, counted from when you unlocked, so a
last used it, so a laptop left open still locks itself. Run \cmd{unlock} laptop left open still locks itself; run \cmd{unlock} again for a fresh
again to start a fresh countdown. countdown. If you'd rather it stayed unlocked while you're busy, \cmd{--idle}
counts from the last command that used the key instead, so it locks only
after that long without use.
\shot{unlock}{Unlock once, work without prompts, lock when you're done} \shot{unlock}{Unlock once, work without prompts, lock when you're done}
@@ -1048,8 +1077,8 @@ who's slow at typing never blocks you.
\begin{whyso} \begin{whyso}
\textbf{What if vault crashes while holding the lock?} On Linux, macOS, the \textbf{What if vault crashes while holding the lock?} On Linux, macOS, the
BSDs and Windows, the operating system drops the lock the moment the process BSDs and Windows, the operating system drops the lock the moment the process
dies. Nothing to clean up. On Plan~9 and WebAssembly, which have no file dies. Nothing to clean up. Older Go builds of vault use the very same locks,
locks, vault uses a \file{.lock} file instead. If one's ever left behind, so mixing versions is safe. If one's ever left behind,
vault tells you after a minute exactly which file to delete. vault tells you after a minute exactly which file to delete.
\end{whyso} \end{whyso}
@@ -1303,7 +1332,6 @@ could not read passphrase (set VAULT\_PASSPHRASE\ldots) & No terminal to ask on,
agent did not start: \ldots & \cmd{unlock} couldn't start its helper. The reason follows; commands still work, asking for the passphrase. \\ agent did not start: \ldots & \cmd{unlock} couldn't start its helper. The reason follows; commands still work, asking for the passphrase. \\
nothing was unlocked & \cmd{vault lock} found nothing to lock. Not an error. \\ nothing was unlocked & \cmd{vault lock} found nothing to lock. Not an error. \\
waiting for another vault process to finish & Not an error. Another command is writing; this one continues when it's done. \\ waiting for another vault process to finish & Not an error. Another command is writing; this one continues when it's done. \\
vault still locked after a minute\ldots & Plan~9 and WebAssembly only. If nothing else is running, delete the file it names. \\
skipping unreadable record \ldots & A catalogue entry is damaged. Everything else still works; restore \file{records/} from backup to recover that file. \\ skipping unreadable record \ldots & A catalogue entry is damaged. Everything else still works; restore \file{records/} from backup to recover that file. \\
object does not match its catalogue record & The encrypted contents were swapped or damaged. Nothing was written. Restore \file{objects/} from backup. \\ object does not match its catalogue record & The encrypted contents were swapped or damaged. Nothing was written. Restore \file{objects/} from backup. \\
object size does not match its catalogue record & Same as above: damage or tampering, caught before any output appeared. \\ object size does not match its catalogue record & Same as above: damage or tampering, caught before any output appeared. \\
@@ -1311,6 +1339,22 @@ cannot find home directory; set VAULT\_KEY & Unusual environments only. Set \env
unknown command / wrong number of arguments & A typo in the command itself. The usage text follows. \\ unknown command / wrong number of arguments & A typo in the command itself. The usage text follows. \\
\end{vtable} \end{vtable}
\subsection{Files, the lock and downloads}
\begin{vtable}{>{\ttfamily\footnotesize}p{58mm}Y}
\thead{MESSAGE} & \thead{WHAT IT MEANS, AND THE FIX} \\ \midrule
cannot lock the vault: \ldots & The lock file couldn't be opened or locked; the reason follows. Check the vault folder's permissions. \\
\ldots\ changed while being added & The file was being written while vault read it. Nothing was stored; try again when it's finished. \\
\ldots\ changed after it was stored; not deleting it & \cmd{migrate} stored the file, but it changed before it could be deleted. The original is kept; the vault has the version it read. \\
\ldots\ already exists & \cmd{get} never overwrites, not even a symlink. Pick another name with \cmd{-o}. \\
\ldots\ refusing to extract through it & \cmd{get -r} found a symlink (or a file) where a folder should be inside the output folder, and stopped rather than follow it. \\
cannot lock agent memory \ldots, cannot make the agent undumpable \ldots & \cmd{unlock}'s helper couldn't protect the key in memory, so it refused it. Commands still work, asking for the passphrase. \\
vault unlock is not supported on this platform\ldots & Windows: every command asks for the passphrase. \\
download exceeds \ldots\ bytes & A download passed its size limit. Raise it with \cmd{--max-bytes} if you meant it. \\
redirected to another host: \ldots & \cmd{mirror} only follows redirects within the site it is copying. \\
mirror incomplete: stored \ldots, skipped \ldots & Some pages failed; the reasons were printed as they happened. Exit status 3. \\
\end{vtable}
\section{``My files have vanished!''} \section{``My files have vanished!''}
Deep breath. Nine times out of ten, one of these: Deep breath. Nine times out of ten, one of these:
@@ -1362,7 +1406,7 @@ passphrase. Nothing in this chapter helps with that. Back up your key.
\begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault mv OLD NEW}\\[2pt]\small Rename or move a file or folder. Instant.\end{tcolorbox} \begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault mv OLD NEW}\\[2pt]\small Rename or move a file or folder. Instant.\end{tcolorbox}
\begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault cp [-r] OLD NEW}\\[2pt]\small Copy a file, or a folder with -r. Instant, no extra space.\end{tcolorbox} \begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault cp [-r] OLD NEW}\\[2pt]\small Copy a file, or a folder with -r. Instant, no extra space.\end{tcolorbox}
\begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault rm [-r] [-f] NAME}\\[2pt]\small Delete a file, or a folder with -r. Frees the space. No undo.\end{tcolorbox} \begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault rm [-r] [-f] NAME}\\[2pt]\small Delete a file, or a folder with -r. Frees the space. No undo.\end{tcolorbox}
\begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault unlock [--for 5m]}\\[2pt]\small Remember the key, so commands stop asking. Hard deadline.\end{tcolorbox} \begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault unlock [--for 5m] [--idle]}\\[2pt]\small Remember the key, so commands stop asking. Fixed deadline, or since last use.\end{tcolorbox}
\begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault lock}\\[2pt]\small Forget the remembered key now.\end{tcolorbox} \begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault lock}\\[2pt]\small Forget the remembered key now.\end{tcolorbox}
\begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault help}\\[2pt]\small Show the command summary.\end{tcolorbox} \begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries vault help}\\[2pt]\small Show the command summary.\end{tcolorbox}
\begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries man vault}\\[2pt]\small A manual page for vault and for every command.\end{tcolorbox} \begin{tcolorbox}{\ttfamily\color{vtealdeep}\bfseries man vault}\\[2pt]\small A manual page for vault and for every command.\end{tcolorbox}
@@ -1380,6 +1424,7 @@ passphrase. Nothing in this chapter helps with that. Back up your key.
--yes & \textrm{---} & ask, for \cmd{migrate} \\ --yes & \textrm{---} & ask, for \cmd{migrate} \\
-r & \textrm{---} & \cmd{rm}, \cmd{get}, \cmd{cp}: one file; \cmd{ls}: one level \\ -r & \textrm{---} & \cmd{rm}, \cmd{get}, \cmd{cp}: one file; \cmd{ls}: one level \\
--for DURATION & \textrm{---} & \cmd{5m}, for \cmd{unlock} \\ --for DURATION & \textrm{---} & \cmd{5m}, for \cmd{unlock} \\
--idle & \textrm{---} & \cmd{unlock}: count from the unlock, not the last use \\
-f & \textrm{---} & \cmd{rm}: ask before \cmd{-r}, fail if missing \\ -f & \textrm{---} & \cmd{rm}: ask before \cmd{-r}, fail if missing \\
\end{vtable} \end{vtable}
+21 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT-ADD 1 "September 2026" "vault" "User Commands" .TH VAULT-ADD 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault-add \- encrypt files, directories or URLs into the vault vault-add \- encrypt files, directories or URLs into the vault
.SH SYNOPSIS .SH SYNOPSIS
@@ -7,6 +7,10 @@ vault-add \- encrypt files, directories or URLs into the vault
.IR PATH ] .IR PATH ]
.RB [ \-\-vault .RB [ \-\-vault
.IR DIR ] .IR DIR ]
.RB [ \-\-max\-bytes
.IR SIZE ]
.RB [ \-\-timeout
.IR DURATION ]
.IR FILE | DIR | URL .IR FILE | DIR | URL
.RI [ FOLDER ] .RI [ FOLDER ]
.SH DESCRIPTION .SH DESCRIPTION
@@ -17,7 +21,14 @@ and the key are skipped. A stored name that already exists is replaced.
.PP .PP
An \fBhttp://\fR or \fBhttps://\fR \fIURL\fR is downloaded with a Chrome user An \fBhttp://\fR or \fBhttps://\fR \fIURL\fR is downloaded with a Chrome user
agent and stored under the last element of its final (post-redirect) path, or agent and stored under the last element of its final (post-redirect) path, or
the host name if there is none. Only a 200 response is accepted. the host name if there is none. Only a 200 response is accepted. Certificates
are always verified; at most 10 redirects are followed, to http or https
only. The download is staged in a private temporary file (it must be
complete before it can be stored), which is removed afterwards whatever
happens; it is plaintext on disk meanwhile.
.PP
A file that changes while it is being added (it grows, shrinks, or is
replaced) is not stored, and the command fails.
.PP .PP
\fIFOLDER\fR places the result inside a folder in the vault. File contents \fIFOLDER\fR places the result inside a folder in the vault. File contents
are cut into 4 MiB chunks, the last bucket-padded. are cut into 4 MiB chunks, the last bucket-padded.
@@ -28,6 +39,14 @@ Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP .TP
.BI \-\-vault " DIR" .BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store. Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.TP
.BI \-\-max\-bytes " SIZE"
Largest download accepted, counted after decompression, such as 500M or 50G.
Default 16G; at most 1T.
.TP
.BI \-\-timeout " DURATION"
Deadline for the whole download, such as 30m. Default 2h. Connecting must
take under 30 seconds and the response headers must arrive within a minute.
.SH ENVIRONMENT .SH ENVIRONMENT
.TP .TP
.B VAULT_PASSPHRASE .B VAULT_PASSPHRASE
+9 -4
View File
@@ -1,4 +1,4 @@
.TH VAULT-GET 1 "September 2026" "vault" "User Commands" .TH VAULT-GET 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault-get \- decrypt a stored file vault-get \- decrypt a stored file
.SH SYNOPSIS .SH SYNOPSIS
@@ -13,13 +13,18 @@ vault-get \- decrypt a stored file
.I NAME .I NAME
.SH DESCRIPTION .SH DESCRIPTION
Decrypts \fINAME\fR. Nothing appears at the output path until every object has Decrypts \fINAME\fR. Nothing appears at the output path until every object has
authenticated. The stored mode and modification time are restored. It refuses authenticated, padding and end of file included; until then the output is a
to overwrite an existing file. Takes a shared lock. private temporary file in the same directory, removed on failure. The stored
mode and modification time are restored. It never replaces anything already
at the output path, including a symbolic link (even a dangling one) or a
file created while it runs. Takes a shared lock.
.PP .PP
With \fB\-r\fR, \fINAME\fR is a folder, and every file under it is With \fB\-r\fR, \fINAME\fR is a folder, and every file under it is
decrypted into the output directory, keeping the folder structure. Each file decrypted into the output directory, keeping the folder structure. Each file
is checked as above; a clash stops the run, leaving the files already is checked as above; a clash stops the run, leaving the files already
written. written. Directories are created as needed, mode 0700; a symbolic link (or
anything but a directory) where a directory belongs inside the output
directory stops the run rather than being followed.
.SH OPTIONS .SH OPTIONS
.TP .TP
.BI \-o " OUTPUT" .BI \-o " OUTPUT"
+5 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT-MIGRATE 1 "September 2026" "vault" "User Commands" .TH VAULT-MIGRATE 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault-migrate \- move files into the vault, deleting the originals vault-migrate \- move files into the vault, deleting the originals
.SH SYNOPSIS .SH SYNOPSIS
@@ -13,7 +13,10 @@ vault-migrate \- move files into the vault, deleting the originals
.SH DESCRIPTION .SH DESCRIPTION
Works like Works like
.BR vault\-add (1) .BR vault\-add (1)
but deletes each original once it is stored, then any directories left empty. but deletes each original once it is stored (and its record is on disk),
then any directories left empty. An original is deleted only if it is still
the same file, unchanged, that was stored; otherwise it is kept and the
command fails.
Directories still holding skipped files stay. A URL cannot be migrated. Directories still holding skipped files stay. A URL cannot be migrated.
.PP .PP
Before storing anything it lists the vault names that would be overwritten and Before storing anything it lists the vault names that would be overwritten and
+60 -17
View File
@@ -1,4 +1,4 @@
.TH VAULT-MIRROR 1 "September 2026" "vault" "User Commands" .TH VAULT-MIRROR 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault-mirror \- mirror a website into an encrypted vault vault-mirror \- mirror a website into an encrypted vault
.SH SYNOPSIS .SH SYNOPSIS
@@ -7,6 +7,12 @@ vault-mirror \- mirror a website into an encrypted vault
.IR PATH ] .IR PATH ]
.RB [ \-\-vault .RB [ \-\-vault
.IR DIR ] .IR DIR ]
.RB [ \-\-max\-pages
.IR N ]
.RB [ \-\-max\-bytes
.IR SIZE ]
.RB [ \-\-timeout
.IR DURATION ]
.I URL .I URL
.RI [ FOLDER ] .RI [ FOLDER ]
.SH DESCRIPTION .SH DESCRIPTION
@@ -20,21 +26,30 @@ and stores every page and asset it finds in the vault, beneath
if no folder is given). if no folder is given).
.PP .PP
The crawl is single threaded: one request at a time, in breadth-first order. The crawl is single threaded: one request at a time, in breadth-first order.
It follows only links on the same host as It follows only links on the same host (and port) as
.IR URL , .IR URL ,
and requests are sent with a Chrome user agent string. and only redirects to that host; a redirect elsewhere skips the page.
Requests are sent with a Chrome user agent string, and certificates are
always verified.
.PP .PP
Links in HTML Links in HTML start tags
.RB ( href , .RB ( href ,
.BR src , .BR src ,
.BR action .BR action
and and
.B poster .B poster
attributes) and in CSS attributes) and CSS
.RB ( url(...) ) .RB ( url(...)
that point into the mirror are rewritten to relative paths, so the copy in style sheets,
browses correctly once fetched out of the vault. Links to other hosts, and .B <style>
fragments, are left as they were. elements and
.B style
attributes) that point into the mirror are rewritten to relative paths, so
the copy browses correctly once fetched out of the vault. Relative links are
resolved against the page's final address and its
.B <base href>
if it has one. Links to other hosts, script and comment text, and every
other byte of the page are left as they were. Fragments are kept.
.PP .PP
Pages are named after their URL path. A path that is empty or ends in Pages are named after their URL path. A path that is empty or ends in
.B / .B /
@@ -42,19 +57,39 @@ becomes
.BR index.html ; .BR index.html ;
a query string is appended as a query string is appended as
.BI _ query .BI _ query
(URL-escaped). Anything already stored under the same name is replaced. (URL-escaped). A name already stored by an earlier run is replaced, but if
two different URLs in one crawl would get the same name, the second is
skipped rather than overwriting the first.
.PP .PP
A page that fails to download, or answers with anything but A page that fails to download, answers with anything but
.BR "200 OK" , .BR "200 OK" ,
is reported on standard error and skipped; the crawl carries on. or exceeds a limit is reported on standard error as
The vault is locked exclusively for the whole run. .BI "skipping " URL ": " reason
and skipped; the crawl carries on. HTML and CSS are rewritten in memory;
other files are staged in a private temporary file (plaintext on disk
meanwhile) that is always removed. The vault is locked exclusively for the
whole run.
.SH OPTIONS .SH OPTIONS
.TP .TP
.BI \-\-max\-pages " N"
Pages and assets to fetch at most. Default 10000.
.TP
.BI \-\-max\-bytes " SIZE"
Total bytes to download at most, counted after decompression. Default 4G.
.TP
.BI \-\-timeout " DURATION"
Deadline for each request. Default 5m.
.TP
.BI \-\-key " PATH" .BI \-\-key " PATH"
Key file. Default: $VAULT_KEY, else ~/.vault/key. Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP .TP
.BI \-\-vault " DIR" .BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store. Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.PP
A single HTML or CSS file larger than 16 MiB is skipped.
.SH EXIT STATUS
0 when everything found was stored, 3 when some of it was skipped, 1 on
an error that stops the crawl.
.SH ENVIRONMENT .SH ENVIRONMENT
.TP .TP
.B VAULT_PASSPHRASE .B VAULT_PASSPHRASE
@@ -75,11 +110,19 @@ vault mirror https://example.com/ archive/2026
vault get archive/2026/example.com/index.html vault get archive/2026/example.com/index.html
.RE .RE
.SH LIMITATIONS .SH LIMITATIONS
Links are found by pattern matching, not by parsing HTML: URLs assembled by URLs assembled by scripts, and
scripts, and
.B srcset .B srcset
lists, are neither followed nor rewritten. There is no depth limit, rate lists, are neither followed nor rewritten. In attribute values only numeric
limit or robots.txt handling; only mirror sites you may copy. character references and
.BR &amp; ,
.BR &lt; ,
.BR &gt; ,
.BR &quot; ,
.B &apos;
and
.B &nbsp;
are decoded. There is no rate limit or robots.txt handling; only mirror
sites you may copy.
.SH SEE ALSO .SH SEE ALSO
.BR vault (1) .BR vault (1)
.SH AUTHOR .SH AUTHOR
+16 -2
View File
@@ -5,6 +5,7 @@ vault-unlock, vault-lock \- remember the unlocked key for a while, or forget it
.B vault unlock .B vault unlock
.RB [ \-\-for .RB [ \-\-for
.IR DURATION ] .IR DURATION ]
.RB [ \-\-idle ]
.RB [ \-\-key .RB [ \-\-key
.IR PATH ] .IR PATH ]
.br .br
@@ -17,8 +18,9 @@ asks for the passphrase, checks it, and starts a helper process, much like
.BR ssh\-agent (1), .BR ssh\-agent (1),
that holds the unlocked key in memory until \fIDURATION\fR has passed that holds the unlocked key in memory until \fIDURATION\fR has passed
(default 5m). Until then every command using the same key file skips the (default 5m). Until then every command using the same key file skips the
passphrase prompt. The deadline is fixed when you unlock; using the key does passphrase prompt. By default the deadline is fixed when you unlock and
not extend it. Running using the key does not extend it; with \fB\-\-idle\fR it is \fIDURATION\fR
after the last command that used the key. Running
.B unlock .B unlock
again replaces the helper with a fresh one. again replaces the helper with a fresh one.
.PP .PP
@@ -30,6 +32,14 @@ memory is locked against swapping and excluded from core dumps; on Linux the
helper is also not debuggable. It answers only on a Unix socket inside helper is also not debuggable. It answers only on a Unix socket inside
\fI$XDG_RUNTIME_DIR/vault-UID\fR (or the temporary directory), a directory \fI$XDG_RUNTIME_DIR/vault-UID\fR (or the temporary directory), a directory
that must belong to this user with mode 0700. Rebooting ends it too. that must belong to this user with mode 0700. Rebooting ends it too.
If the helper cannot lock its memory, disable core dumps or (on Linux) make
itself undebuggable, it refuses the key and
.B unlock
fails, saying why. The helper is the same program, started with the
passphrase variables removed from its environment. Helpers started by
earlier (Go) versions of vault are compatible and can be used and locked.
.PP
Not available on Windows, where every command asks for the passphrase.
.PP .PP
While the key is unlocked, any program running as this user can use it, and While the key is unlocked, any program running as this user can use it, and
it is whichever key the passphrase opened, real or decoy. Lock before it is whichever key the passphrase opened, real or decoy. Lock before
@@ -38,6 +48,10 @@ handing the machine to anyone.
.TP .TP
.BI \-\-for " DURATION" .BI \-\-for " DURATION"
How long to remember the key, such as 90s, 5m or 2h. Default 5m. How long to remember the key, such as 90s, 5m or 2h. Default 5m.
.TP
.B \-\-idle
Count \fIDURATION\fR from the last time a command used the key, rather than
from the unlock, so the key stays while you keep working.
.PP .PP
See also \fB\-\-key\fR in See also \fB\-\-key\fR in
.BR vault (1). .BR vault (1).
+23 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT 1 "September 2026" "vault" "User Commands" .TH VAULT 1 "October 2026" "vault" "User Commands"
.SH NAME .SH NAME
vault \- post-quantum encrypted file vault vault \- post-quantum encrypted file vault
.SH SYNOPSIS .SH SYNOPSIS
@@ -77,6 +77,11 @@ Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP .TP
.BI \-\-vault " DIR" .BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store. Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.TP
.BI \-\-workers " N"
Threads for encryption, decryption and file I/O, 1 to 256. Default: the
number of CPUs, at most 8. Up to eight 4 MiB objects are in flight at once
whatever the count.
.SH ENVIRONMENT .SH ENVIRONMENT
.TP .TP
.B VAULT_KEY .B VAULT_KEY
@@ -92,6 +97,19 @@ Passphrase, instead of prompting on the terminal or using a key remembered by
.B VAULT_DECOY_PASSPHRASE .B VAULT_DECOY_PASSPHRASE
Decoy passphrase used by Decoy passphrase used by
.BR "vault genkey" . .BR "vault genkey" .
.TP
.BR HTTPS_PROXY ", " HTTP_PROXY ", " NO_PROXY
Proxies for
.B add
.I URL
and
.BR mirror ,
as other programs use them; local addresses are never proxied.
.TP
.BR XDG_RUNTIME_DIR ", " TMPDIR
Where
.BR vault\-unlock (1)
keeps its socket, and where downloads are staged.
.SH LOCKING .SH LOCKING
Commands that change the vault hold an exclusive lock on Commands that change the vault hold an exclusive lock on
.IR DIR/.lock ; .IR DIR/.lock ;
@@ -106,7 +124,10 @@ and
fills each vault with unreadable filler, so hidden files look like filler to fills each vault with unreadable filler, so hidden files look like filler to
anyone holding only the decoy passphrase. anyone holding only the decoy passphrase.
.SH EXIT STATUS .SH EXIT STATUS
0 on success; 1 on error (diagnostics on standard error); 2 when run with no command. 0 on success; 1 on error (diagnostics on standard error); 2 when run with no
command; 3 when
.BR vault\-mirror (1)
stored some resources but had to skip others.
.SH EXAMPLES .SH EXAMPLES
.nf .nf
vault genkey vault genkey
+1287
View File
File diff suppressed because it is too large Load Diff
+68
View File
@@ -0,0 +1,68 @@
[package]
name = "vault-fuzz"
version = "0.0.0"
publish = false
edition = "2024"
[package.metadata]
cargo-fuzz = true
[dependencies]
libfuzzer-sys = "0.4"
tar = { version = "0.4.46", default-features = false }
vault = { path = ".." }
# Not part of the main package's workspace.
[workspace]
[profile.release]
debug = 1
[[bin]]
name = "header"
path = "fuzz_targets/header.rs"
test = false
doc = false
bench = false
[[bin]]
name = "stream"
path = "fuzz_targets/stream.rs"
test = false
doc = false
bench = false
[[bin]]
name = "keyfile"
path = "fuzz_targets/keyfile.rs"
test = false
doc = false
bench = false
[[bin]]
name = "record"
path = "fuzz_targets/record.rs"
test = false
doc = false
bench = false
[[bin]]
name = "names"
path = "fuzz_targets/names.rs"
test = false
doc = false
bench = false
[[bin]]
name = "html"
path = "fuzz_targets/html.rs"
test = false
doc = false
bench = false
[[bin]]
name = "tar"
path = "fuzz_targets/tar.rs"
test = false
doc = false
bench = false
+15
View File
@@ -0,0 +1,15 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! The Fury header parser never panics, and accepts only canonical headers:
//! whatever it accepts re-serialises to exactly the bytes it consumed.
#![no_main]
use std::io::Read;
use vault::crypto::header;
libfuzzer_sys::fuzz_target!(|data: &[u8]| {
if let Ok((h, mut rest)) = header::parse(data) {
let mut tail = Vec::new();
rest.read_to_end(&mut tail).unwrap();
let consumed = &data[..data.len() - tail.len()];
assert_eq!(h.marshal(), consumed, "non-canonical header accepted");
}
});
+15
View File
@@ -0,0 +1,15 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! The mirror's HTML and CSS rewriters never panic, and rewriting nothing
//! returns the input byte for byte.
#![no_main]
use vault::net::html;
libfuzzer_sys::fuzz_target!(|data: &[u8]| {
let _ = html::tokenize(data);
assert_eq!(html::rewrite_html(data, &mut |_| None, &mut |_| None), data);
assert_eq!(html::rewrite_css(data, false, &mut |_| None), data);
let _ = html::rewrite_html(data, &mut |_| None, &mut |u| Some(format!("x/{}", u.len())));
if let Ok(s) = std::str::from_utf8(data) {
let _ = vault::net::Url::parse(s).map(|u| u.join("../a?b#c"));
}
});
+11
View File
@@ -0,0 +1,11 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! PEM decoding never panics, and decoding what it encoded is the identity.
#![no_main]
use vault::keyfile::{decode_pem, encode_pem, parse_body};
libfuzzer_sys::fuzz_target!(|data: &[u8]| {
if let Some(body) = decode_pem(data) {
let _ = parse_body(&body);
assert_eq!(decode_pem(encode_pem(&body).as_bytes()), Some(body));
}
});
+18
View File
@@ -0,0 +1,18 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! Path validation: a valid vault name is clean, relative and confined; the
//! tar name check only ever yields valid-shaped relative names.
#![no_main]
use vault::store::VaultName;
libfuzzer_sys::fuzz_target!(|data: &[u8]| {
let Ok(s) = std::str::from_utf8(data) else { return };
if let Some(n) = VaultName::new(s) {
let n = n.as_str();
assert_eq!(vault::util::clean(n), n);
assert!(!n.starts_with('/') && !n.contains('\\') && !n.split('/').any(|c| c == ".." || c == "." || c.is_empty()));
assert!(n.len() <= 1024);
}
if let Some(n) = vault::absorb::safe_name(s) {
assert!(!n.is_empty() && !n.starts_with('/') && !n.split('/').any(|c| c == ".."));
}
});
+15
View File
@@ -0,0 +1,15 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! Record JSON parsing never panics; an accepted record re-encodes (Go's
//! JSON) to something that parses back to the same values.
#![no_main]
use vault::store::{Id, Record};
libfuzzer_sys::fuzz_target!(|data: &[u8]| {
if let Ok(r) = Record::from_json(data, Id([0; 16])) {
let Some(j) = r.to_json() else { return };
let r2 = Record::from_json(&j, Id([0; 16])).expect("own encoding parses");
assert_eq!(r.name, r2.name);
assert_eq!((r.size, r.mode, r.modified, r.added), (r2.size, r2.mode, r2.modified, r2.added));
assert!(r.seed.same_as(&r2.seed));
}
});
+46
View File
@@ -0,0 +1,46 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! The streaming, in-memory and random-access STREAM readers agree on every
//! input: the same plaintext or all reject. Inputs are mostly forgeries;
//! the first byte selects a mode that starts from a valid stream instead.
#![no_main]
use std::io::{Read, Write};
use vault::crypto::stream::{StreamReader, StreamReaderAt, StreamWriter, open_all};
const KEY: [u8; 32] = [7; 32];
libfuzzer_sys::fuzz_target!(|data: &[u8]| {
let Some((&mode, rest)) = data.split_first() else { return };
let input = if mode & 1 == 1 {
// A valid stream of the given plaintext, then a mutation.
let mut w = StreamWriter::new(&KEY, Vec::new());
let n = rest.len().min(200_000);
w.write_all(&rest[..n]).unwrap();
let mut c = w.finish().unwrap();
if mode & 2 == 2 && !c.is_empty() {
let i = (mode as usize * 7919) % c.len();
c[i] ^= mode | 1;
}
if mode & 4 == 4 {
c.truncate(c.len().saturating_sub(mode as usize % 40));
}
c
} else {
rest.to_vec()
};
let mut a = Vec::new();
let streaming = StreamReader::new(&KEY, &input[..]).read_to_end(&mut a).map(|_| a);
let mut b = input.clone();
let whole = open_all(&KEY, &mut b).map(|n| b[..n].to_vec());
assert_eq!(streaming.is_ok(), whole.is_ok());
if let (Ok(x), Ok(y)) = (&streaming, &whole) {
assert_eq!(x, y);
}
if let Ok(r) = StreamReaderAt::new(&KEY, &input[..], input.len() as u64) {
let mut out = vec![0u8; r.size() as usize];
let got = r.read_at(&mut out, 0).map(|n| out[..n].to_vec());
assert_eq!(got.is_ok(), whole.is_ok());
if let (Ok(x), Ok(y)) = (got, whole) {
assert_eq!(x, y);
}
}
});
+19
View File
@@ -0,0 +1,19 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! Walking arbitrary tar input as absorb does never panics, and every name
//! it would store is confined.
#![no_main]
use std::io::Read;
libfuzzer_sys::fuzz_target!(|data: &[u8]| {
let mut a = tar::Archive::new(data);
let Ok(entries) = a.entries() else { return };
for e in entries.take(64) {
let Ok(mut e) = e else { return };
let name = e.path_bytes().into_owned();
if let Some(n) = std::str::from_utf8(&name).ok().and_then(vault::absorb::safe_name) {
assert!(!n.split('/').any(|c| c == ".."));
}
let mut sink = Vec::new();
let _ = e.by_ref().take(1 << 20).read_to_end(&mut sink);
}
});
+46
View File
@@ -0,0 +1,46 @@
# Go oracle (test-only)
`oracle/go` is a frozen copy of the Go implementation of vault, taken from
commit `4c1507c` ("Add unlock --idle, and packaging/repo.sh for
repo.dixon.cx"), the last Go release on `master`. It is kept as the
reference that the Rust implementation is checked against. It is **not**
part of the shipped product: release builds, packages and the `vault`
binary contain no Go code and need no Go toolchain.
Provenance:
- All 32 Go sources and `go.mod`/`go.sum` under `oracle/go` are
byte-identical to the same paths at the repository root at `4c1507c`.
- The only additions are test files whose names contain `oracle`
(`*oracle_vectors_test.go`, `oracle_decrypt_test.go`,
`*oracle_bench_test.go`). They call the frozen code and record what it
computes; they skip themselves unless `VAULT_ORACLE_OUT` or
`VAULT_ORACLE_DECRYPT` is set (benchmarks run only with `-bench`), and add
nothing to the oracle binary.
- Dependencies are pinned by `go.mod`/`go.sum`: CIRCL v1.6.4,
golang.org/x/crypto v0.53.0, x/sys v0.46.0, x/term v0.44.0. The module
needs Go 1.26.6 or newer.
Check the copy is still faithful:
```sh
for f in $(git ls-tree -r --name-only 4c1507c | grep -E '\.go$|^go\.(mod|sum)$'); do
cmp <(git show 4c1507c:$f) oracle/go/$f || echo "DIFFERS: $f"
done
```
Use:
```sh
oracle/gen-vectors.sh # rewrite tests/vectors from the oracle
(cd oracle/go && go build -o ../../target/vault-go .) # build the oracle CLI
(cd oracle/go && go test ./...) # its own test-suite
```
`cargo test` builds the oracle binary itself for the interoperability tests
when `go` is on `PATH` (or uses `$VAULT_GO_ORACLE`), and reports them as
skipped otherwise.
All passphrases, seeds and plaintexts in the vectors are synthetic and
public (`oracle-real passphrase`, `oracle-decoy passphrase`, fixed byte
patterns). Never commit real keys or real vault contents.
+14
View File
@@ -0,0 +1,14 @@
#!/usr/bin/env bash
# Regenerate the Go-oracle vectors and fixtures in tests/vectors.
#
# Everything is computed by the frozen Go implementation in oracle/go (see
# oracle/README.md); nothing comes from the Rust code under test. Key files
# and ciphertexts use fresh randomness, so regenerating changes them; the
# deterministic vectors (*.json fields derived from fixed seeds) do not.
set -euo pipefail
cd "$(dirname "$0")/.."
out=$PWD/tests/vectors
mkdir -p "$out"
(cd oracle/go && VAULT_ORACLE_OUT=$out go test -count=1 -run '^TestOracleVectors$' ./ ./internal/pqcrypt ./internal/stream)
(cd oracle/go && VAULT_ORACLE_OUT=$out go test -count=1 -run '^TestOracleNetVectors$' ./)
echo "vectors written to $out"
View File
+29 -27
View File
@@ -26,7 +26,8 @@ import (
// seed over a pipe; it never touches the disk. The agent serves them on a // seed over a pipe; it never touches the disk. The agent serves them on a
// Unix socket in a directory only this user can enter, keeps them in locked, // Unix socket in a directory only this user can enter, keeps them in locked,
// undumpable memory where the platform allows, and wipes them and exits at // undumpable memory where the platform allows, and wipes them and exits at
// a fixed deadline, or on `vault lock`. Using the agent never extends it. // a deadline, or on `vault lock`. The deadline is fixed, or with --idle it
// moves on each time a command uses the key.
const ( const (
agentCmd = "__agent" // hidden: the agent process itself agentCmd = "__agent" // hidden: the agent process itself
@@ -104,8 +105,8 @@ func agentStop(keyPath string) bool {
} }
// agentStart replaces any agent for keyPath with one holding id until ttl // agentStart replaces any agent for keyPath with one holding id until ttl
// has passed. // has passed, since it started or, with idle, since the key was last used.
func agentStart(keyPath string, id *pqcrypt.PQIdentity, ttl time.Duration) error { func agentStart(keyPath string, id *pqcrypt.PQIdentity, ttl time.Duration, idle bool) error {
sock, err := agentSocket(keyPath) sock, err := agentSocket(keyPath)
if err != nil { if err != nil {
return err return err
@@ -115,7 +116,11 @@ func agentStart(keyPath string, id *pqcrypt.PQIdentity, ttl time.Duration) error
if err != nil { if err != nil {
return err return err
} }
cmd := exec.Command(self, agentCmd, sock, ttl.String()) mode := "fixed"
if idle {
mode = "idle"
}
cmd := exec.Command(self, agentCmd, sock, ttl.String(), mode)
stdin, err := cmd.StdinPipe() stdin, err := cmd.StdinPipe()
if err != nil { if err != nil {
return err return err
@@ -150,12 +155,13 @@ func agentStart(keyPath string, id *pqcrypt.PQIdentity, ttl time.Duration) error
// runAgent is the agent process: it reads the key from in, says "ok" or // runAgent is the agent process: it reads the key from in, says "ok" or
// why not on out and closes it, then serves the key on sock until ttl has // why not on out and closes it, then serves the key on sock until ttl has
// passed or it is told to quit. // passed, since it started or with mode "idle" since the key was last
func runAgent(sock, ttlArg string, in io.Reader, out io.WriteCloser) { // handed out, or until it is told to quit.
func runAgent(sock, ttlArg, mode string, in io.Reader, out io.WriteCloser) {
report := func(msg string) { io.WriteString(out, msg); out.Close() } report := func(msg string) { io.WriteString(out, msg); out.Close() }
ttl, err := time.ParseDuration(ttlArg) ttl, err := time.ParseDuration(ttlArg)
if err != nil || ttl <= 0 { if err != nil || ttl <= 0 || mode != "fixed" && mode != "idle" {
report("bad duration") report("bad duration or mode")
return return
} }
secret := make([]byte, agentReply) secret := make([]byte, agentReply)
@@ -175,44 +181,40 @@ func runAgent(sock, ttlArg string, in io.Reader, out io.WriteCloser) {
} }
os.Chmod(sock, 0o600) os.Chmod(sock, 0o600)
report("ok") report("ok")
done := make(chan struct{}) expiry := time.AfterFunc(ttl, func() { l.Close() })
time.AfterFunc(ttl, func() { l.Close() })
go func() {
defer close(done)
for { for {
c, err := l.Accept() c, err := l.Accept()
if err != nil { if err != nil {
return break
}
op := serveAgent(c, secret)
c.Close()
if op == agentQuit {
break
}
if op == agentGet && mode == "idle" {
expiry.Reset(ttl)
}
} }
if !serveAgent(c, secret) {
l.Close() l.Close()
c.Close()
return
}
c.Close()
}
}()
<-done
clear(secret) clear(secret)
os.Remove(sock) os.Remove(sock)
} }
// serveAgent answers one request, reporting whether to keep going. // serveAgent answers one request and returns its operation.
func serveAgent(c net.Conn, secret []byte) bool { func serveAgent(c net.Conn, secret []byte) byte {
c.SetDeadline(time.Now().Add(agentTimeout)) c.SetDeadline(time.Now().Add(agentTimeout))
op := make([]byte, 1) op := make([]byte, 1)
if _, err := io.ReadFull(c, op); err != nil { if _, err := io.ReadFull(c, op); err != nil {
return true return 0
} }
switch op[0] { switch op[0] {
case agentGet: case agentGet:
c.Write(secret) c.Write(secret)
return true
case agentQuit: case agentQuit:
clear(secret) clear(secret)
return false
} }
return true return op[0]
} }
// keyIdentity unlocks the key at keyPath: from the agent if one is running // keyIdentity unlocks the key at keyPath: from the agent if one is running
View File
View File
@@ -0,0 +1,57 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
package pqcrypt
import (
"bytes"
"io"
"testing"
)
func BenchmarkOracleDerive(b *testing.B) {
seed := pattern(32, 1)
for b.Loop() {
id, _ := NewPQIdentityFromSeed(0x0001, seed)
id.Close()
}
}
func BenchmarkOracleWrap(b *testing.B) {
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
r := id.Recipient()
fk := pattern(32, 2)
for b.Loop() {
r.Wrap(fk)
}
}
func BenchmarkOracleUnwrap(b *testing.B) {
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
st, _ := id.Recipient().Wrap(pattern(32, 2))
for b.Loop() {
if _, err := id.Unwrap(st); err != nil {
b.Fatal(err)
}
}
}
// Whole-file encryption and decryption of 4 MiB: one vault object.
func BenchmarkOracleObject4MiB(b *testing.B) {
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
pt := pattern(4<<20, 3)
var ct bytes.Buffer
b.SetBytes(int64(len(pt)))
for b.Loop() {
ct.Reset()
w, _ := Encrypt(&ct, id.Recipient())
w.Write(pt)
w.Close()
r, err := Decrypt(bytes.NewReader(ct.Bytes()), id)
if err != nil {
b.Fatal(err)
}
io.Copy(io.Discard, r)
}
}
@@ -0,0 +1,54 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only: decrypts files another implementation wrote, for the Rust
// port's interoperability tests. Skips unless VAULT_ORACLE_DECRYPT is set.
package pqcrypt
import (
"bytes"
"io"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
)
// TestOracleDecrypt opens every DIR/pattern-SIZE.fury with the identity of
// master seed pattern(32, 1) and checks it holds pattern(SIZE, 0x5a), and
// that every DIR/bad-*.fury is rejected.
func TestOracleDecrypt(t *testing.T) {
dir := os.Getenv("VAULT_ORACLE_DECRYPT")
if dir == "" {
t.Skip("set VAULT_ORACLE_DECRYPT")
}
id, err := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
if err != nil {
t.Fatal(err)
}
files, _ := filepath.Glob(filepath.Join(dir, "*.fury"))
if len(files) == 0 {
t.Fatal("no files")
}
for _, f := range files {
data, _ := os.ReadFile(f)
r, err := Decrypt(bytes.NewReader(data), id)
var got []byte
if err == nil {
got, err = io.ReadAll(r)
}
base := filepath.Base(f)
if strings.HasPrefix(base, "bad-") {
if err == nil {
t.Errorf("%s: accepted", base)
}
continue
}
size, _ := strconv.Atoi(strings.TrimSuffix(strings.TrimPrefix(base, "pattern-"), ".fury"))
if err != nil || !bytes.Equal(got, pattern(size, 0x5a)) {
t.Errorf("%s: %v", base, err)
}
t.Logf("%s ok", base)
}
}
@@ -0,0 +1,298 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only vector generator for the Rust port. It adds nothing to the
// production code: it only calls the frozen implementation and records what
// it computes. Run through oracle/gen-vectors.sh.
package pqcrypt
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"io"
"os"
"path/filepath"
"testing"
"github.com/andydixon/vault/internal/format"
"github.com/cloudflare/circl/kem"
"github.com/cloudflare/circl/kem/frodo/frodo640shake"
"github.com/cloudflare/circl/kem/mlkem/mlkem768"
"golang.org/x/crypto/hkdf"
)
type kemVector struct {
Seed string `json:"seed"`
PubSHA256 string `json:"pub_sha256"`
PrivSHA256 string `json:"priv_sha256"`
EncapSeed string `json:"encap_seed"`
CtSHA256 string `json:"ct_sha256"`
SharedSecret string `json:"ss"`
RejectFlipIdx int `json:"reject_flip_index"`
RejectSS string `json:"reject_ss"`
}
type hybridVector struct {
MasterSeed string `json:"master_seed"`
FrodoSeed string `json:"frodo_seed"`
MLKEMSeed string `json:"mlkem_seed"`
PubSHA256 string `json:"pub_sha256"`
PrivSHA256 string `json:"priv_sha256"`
PubPrefix string `json:"pub_prefix"`
}
type wrapVector struct {
MasterSeed string `json:"master_seed"`
FrodoEncap string `json:"frodo_encap_seed"`
MLKEMEncap string `json:"mlkem_encap_seed"`
FileKey string `json:"file_key"`
KEK string `json:"kek"`
Header string `json:"header_hex"` // full serialised header, MAC line included
MAC string `json:"mac"`
Nonce string `json:"nonce"`
PayloadKey string `json:"payload_key"`
StanzaArg0 string `json:"stanza_arg0_sha256"`
StanzaBody string `json:"stanza_body"`
}
type headerCase struct {
Input string `json:"input_hex"`
Accept bool `json:"accept"`
Rest string `json:"rest_hex,omitempty"` // payload bytes after the header
}
func sum(b []byte) string { s := sha256.Sum256(b); return hex.EncodeToString(s[:]) }
func pattern(n int, salt byte) []byte {
b := make([]byte, n)
for i := range b {
b[i] = byte(i*7) ^ salt
}
return b
}
func kemVectors(t *testing.T, s kem.Scheme, n int) []kemVector {
var out []kemVector
for i := range n {
seed := pattern(s.SeedSize(), byte(i*31))
pk, sk := s.DeriveKeyPair(seed)
pkb, _ := pk.MarshalBinary()
skb, _ := sk.MarshalBinary()
es := pattern(s.EncapsulationSeedSize(), byte(i*17+5))
ct, ss, err := s.EncapsulateDeterministically(pk, es)
if err != nil {
t.Fatal(err)
}
ss2, err := s.Decapsulate(sk, ct)
if err != nil || !bytes.Equal(ss, ss2) {
t.Fatal("decapsulation mismatch")
}
flip := (i * 997) % len(ct)
bad := append([]byte(nil), ct...)
bad[flip] ^= 0x01
rss, err := s.Decapsulate(sk, bad)
if err != nil {
t.Fatal(err)
}
out = append(out, kemVector{
Seed: hex.EncodeToString(seed), PubSHA256: sum(pkb), PrivSHA256: sum(skb),
EncapSeed: hex.EncodeToString(es), CtSHA256: sum(ct), SharedSecret: hex.EncodeToString(ss),
RejectFlipIdx: flip, RejectSS: hex.EncodeToString(rss),
})
}
return out
}
func TestOracleVectors(t *testing.T) {
dir := os.Getenv("VAULT_ORACLE_OUT")
if dir == "" {
t.Skip("set VAULT_ORACLE_OUT to write vectors")
}
suite := frodoMLKEM768{}
frodo, ml := frodo640shake.Scheme(), mlkem768.Scheme()
var hybrids []hybridVector
for i := range 6 {
master := pattern(32, byte(i*53+1))
if i == 5 {
master = bytes.Repeat([]byte{0xff}, 32)
}
r := hkdf.New(sha256.New, master, nil, []byte("fury/v1 seed-expand "+suite.tag()))
fs := make([]byte, frodo.SeedSize())
ms := make([]byte, ml.SeedSize())
io.ReadFull(r, fs)
io.ReadFull(r, ms)
pub, priv, err := suite.deriveKeypair(master)
if err != nil {
t.Fatal(err)
}
hybrids = append(hybrids, hybridVector{
MasterSeed: hex.EncodeToString(master), FrodoSeed: hex.EncodeToString(fs), MLKEMSeed: hex.EncodeToString(ms),
PubSHA256: sum(pub), PrivSHA256: sum(priv), PubPrefix: hex.EncodeToString(pub[:64]),
})
}
// A wrap with controlled encapsulation randomness, built from the real
// deriveKEK, aeadEncrypt, header marshalling, headerMAC and streamKey.
var wraps []wrapVector
for i := range 3 {
master := pattern(32, byte(i*53+1))
id, err := NewPQIdentityFromSeed(suite.id(), master)
if err != nil {
t.Fatal(err)
}
pub := id.Recipient().Bytes()
fpk, _ := frodo.UnmarshalBinaryPublicKey(pub[:frodo.PublicKeySize()])
mpk, _ := ml.UnmarshalBinaryPublicKey(pub[frodo.PublicKeySize():])
fe, me := pattern(16, byte(i+100)), pattern(32, byte(i+150))
ct1, ss1, _ := frodo.EncapsulateDeterministically(fpk, fe)
ct2, ss2, _ := ml.EncapsulateDeterministically(mpk, me)
kek := suite.deriveKEK(ss1, ss2, ct1, ct2)
fileKey := pattern(32, byte(i+200))
body, _ := aeadEncrypt(kek, fileKey)
st := &format.Stanza{Type: suite.tag(), Args: []string{format.EncodeToString(ct1), format.EncodeToString(ct2)}, Body: body}
hdr := &format.Header{Recipients: []*format.Stanza{st}}
mac, err := headerMAC(fileKey, hdr)
if err != nil {
t.Fatal(err)
}
hdr.MAC = mac
var hb bytes.Buffer
hdr.Marshal(&hb)
// The real identity must open what was built.
got, err := id.Unwrap([]*Stanza{(*Stanza)(st)})
if err != nil || !bytes.Equal(got, fileKey) {
t.Fatal("identity did not unwrap the vector")
}
nonce := pattern(16, byte(i+250))
wraps = append(wraps, wrapVector{
MasterSeed: hex.EncodeToString(master), FrodoEncap: hex.EncodeToString(fe), MLKEMEncap: hex.EncodeToString(me),
FileKey: hex.EncodeToString(fileKey), KEK: hex.EncodeToString(kek), Header: hex.EncodeToString(hb.Bytes()),
MAC: hex.EncodeToString(mac), Nonce: hex.EncodeToString(nonce), PayloadKey: hex.EncodeToString(streamKey(fileKey, nonce)),
StanzaArg0: sum([]byte(st.Args[0])), StanzaBody: hex.EncodeToString(body),
})
}
// Whole files from the real Encrypt, with fresh randomness, for the
// first hybrid identity. Plaintexts are pattern(size, 0x5a).
id, _ := NewPQIdentityFromSeed(suite.id(), pattern(32, 1))
fdir := filepath.Join(dir, "fury")
os.MkdirAll(fdir, 0o755)
for _, size := range []int{0, 1, 65535, 65536, 65537, 131072, 200000} {
var buf bytes.Buffer
w, err := Encrypt(&buf, id.Recipient())
if err != nil {
t.Fatal(err)
}
w.Write(pattern(size, 0x5a))
if err := w.Close(); err != nil {
t.Fatal(err)
}
os.WriteFile(filepath.Join(fdir, "pattern-"+itoa(size)+".fury"), buf.Bytes(), 0o644)
}
// Header parser verdicts on valid and malformed headers.
var cases []headerCase
valid, _ := hex.DecodeString(wraps[0].Header)
add := func(b []byte) {
h, payload, err := format.Parse(bytes.NewReader(b))
c := headerCase{Input: hex.EncodeToString(b), Accept: err == nil}
if err == nil {
rest, _ := io.ReadAll(payload)
c.Rest = hex.EncodeToString(rest)
var again bytes.Buffer
h.Marshal(&again)
}
cases = append(cases, c)
}
for _, m := range headerMutations(valid) {
add(m)
}
write(t, filepath.Join(dir, "crypto.json"), map[string]any{
"provenance": "oracle/go at 4c1507c, internal/pqcrypt/oracle_vectors_test.go",
"frodo640shake": kemVectors(t, frodo, 8),
"mlkem768": kemVectors(t, ml, 8),
"hybrid": hybrids,
"wrap": wraps,
"header_cases": cases,
})
}
func itoa(n int) string { return string(appendInt(nil, n)) }
func appendInt(b []byte, n int) []byte {
if n >= 10 {
b = appendInt(b, n/10)
}
return append(b, byte('0'+n%10))
}
// headerMutations returns valid headers plus a range of malformed ones.
func headerMutations(valid []byte) [][]byte {
s := string(valid)
introEnd := bytes.IndexByte(valid, '\n') + 1
footer := bytes.LastIndex(valid, []byte("\n---")) + 1
lines := bytes.SplitAfter(valid, []byte("\n"))
join := func(ls [][]byte) []byte { return bytes.Join(ls, nil) }
var out [][]byte
push := func(b []byte) { out = append(out, append([]byte(nil), b...)) }
push(valid)
push(append(append([]byte(nil), valid...), []byte("payload\x00\x01")...))
push([]byte(""))
push([]byte("fury-encryption.org/v1\n--- " + s[len(s)-44:])) // no stanzas
push([]byte("fury-encryption.org/v2\n" + s[introEnd:])) // wrong intro
push([]byte("fury-encryption.org/v1" + s[introEnd:])) // intro without newline
push(valid[:footer]) // no footer
push(valid[:len(valid)-1]) // footer without newline
push(append(append([]byte(nil), valid[:len(valid)-1]...), '=', '\n')) // padded MAC
push([]byte(s[:footer] + "--- " + s[footer+4:])) // double space
push([]byte(s[:footer] + "---x " + s[footer+4:])) // bad footer prefix
push([]byte(s[:footer] + "--- " + s[footer+4:len(s)-2] + "\n")) // short MAC
push([]byte(s[:footer] + "--- " + s[footer+4:len(s)-1] + " x\n"))
// Stanza line edits.
stanza := lines[1]
for _, edit := range []string{"-> \n", "->\n", "-> x\n", "-> x\x7f\n", "-> x é\n", "-x a\n", "-> x y\n", "-> x\r\n"} {
ls := append([][]byte(nil), lines...)
ls[1] = []byte(edit)
push(join(ls))
_ = stanza
}
// Body edits: the body is the two lines after the stanza line.
body := 2
for _, edit := range []string{"\n", "AAAA\n", "AAAA=\n", "A\n", "AB\n", "AAB\n", "AA==\n", " AAAA\n", "AAAA\r\n",
string(bytes.Repeat([]byte("A"), 64)) + "\n", string(bytes.Repeat([]byte("A"), 68)) + "\n", string(bytes.Repeat([]byte("A"), 65)) + "\n"} {
ls := append([][]byte(nil), lines...)
ls[body] = []byte(edit)
push(join(ls))
}
{
// Drop the empty line that ends a 48-byte body: the stanza then runs
// into the footer.
ls := append([][]byte(nil), lines...)
ls = append(ls[:3], ls[4:]...)
push(join(ls))
}
{
// A second, foreign stanza ahead of the real one.
ls := append([][]byte(nil), lines[:1]...)
ls = append(ls, []byte("-> other-suite arg\n"), []byte("AAAA\n"))
ls = append(ls, lines[1:]...)
push(join(ls))
}
// The 1 MiB stanza and 16 MiB header limits are exercised by the Rust
// tests directly; vectors that size would bloat the repository.
return out
}
func write(t *testing.T, path string, v any) {
b, err := json.MarshalIndent(v, "", " ")
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(path, append(b, '\n'), 0o644); err != nil {
t.Fatal(err)
}
}
@@ -0,0 +1,27 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
package stream
import (
"bytes"
"io"
"testing"
)
// STREAM seal and open of 64 MiB, one goroutine (the sequential path).
func BenchmarkOracleStream64MiB(b *testing.B) {
key := make([]byte, 32)
pt := make([]byte, 64<<20)
var ct bytes.Buffer
b.SetBytes(int64(len(pt)))
for b.Loop() {
ct.Reset()
w, _ := NewEncryptWriter(key, &ct)
w.Write(pt)
w.Close()
r, _ := NewDecryptReader(key, bytes.NewReader(ct.Bytes()))
io.Copy(io.Discard, r)
}
}
@@ -0,0 +1,65 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only vector generator for the Rust port; see oracle/gen-vectors.sh.
package stream
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"io"
"os"
"path/filepath"
"testing"
)
func TestOracleVectors(t *testing.T) {
dir := os.Getenv("VAULT_ORACLE_OUT")
if dir == "" {
t.Skip("set VAULT_ORACLE_OUT to write vectors")
}
key := make([]byte, 32)
for i := range key {
key[i] = byte(i + 1)
}
type vec struct {
Size int `json:"size"`
CtLen int `json:"ct_len"`
CtSHA256 string `json:"ct_sha256"`
CtHex string `json:"ct_hex,omitempty"`
}
var vs []vec
for _, size := range []int{0, 1, 15, 16, 17, ChunkSize - 1, ChunkSize, ChunkSize + 1, 2 * ChunkSize, 2*ChunkSize + 1, 3*ChunkSize - 16, 300000} {
pt := make([]byte, size)
for i := range pt {
pt[i] = byte(i*13 + 7)
}
var buf bytes.Buffer
w, _ := NewEncryptWriter(key, &buf)
w.Write(pt)
w.Close()
// The reader-side encryptor must agree.
r, _ := NewEncryptReader(key, bytes.NewReader(pt))
alt, _ := io.ReadAll(r)
if !bytes.Equal(alt, buf.Bytes()) {
t.Fatal("EncryptReader disagrees with EncryptWriter")
}
s := sha256.Sum256(buf.Bytes())
v := vec{Size: size, CtLen: buf.Len(), CtSHA256: hex.EncodeToString(s[:])}
if size <= 17 {
v.CtHex = hex.EncodeToString(buf.Bytes())
}
vs = append(vs, v)
}
b, _ := json.MarshalIndent(map[string]any{
"provenance": "oracle/go at 4c1507c, internal/stream/oracle_vectors_test.go",
"key": hex.EncodeToString(key),
"plaintext": "byte(i*13+7)",
"vectors": vs,
}, "", " ")
if err := os.WriteFile(filepath.Join(dir, "stream.json"), append(b, '\n'), 0o644); err != nil {
t.Fatal(err)
}
}
View File
View File
+10 -4
View File
@@ -57,7 +57,8 @@ const usage = `Usage:
vault rm -r FOLDER [-f] delete a folder and everything under it; vault rm -r FOLDER [-f] delete a folder and everything under it;
asks first unless -f (or -rf) is given asks first unless -f (or -rf) is given
vault unlock [--for DURATION] remember the passphrase for DURATION vault unlock [--for DURATION] remember the passphrase for DURATION
(default 5m), so commands skip asking [--idle] (default 5m), so commands skip asking;
with --idle, DURATION since last used
vault lock forget it now vault lock forget it now
Options (any command): Options (any command):
@@ -91,8 +92,8 @@ func main() {
os.Exit(2) os.Exit(2)
} }
cmd, args := os.Args[1], os.Args[2:] cmd, args := os.Args[1], os.Args[2:]
if cmd == agentCmd && len(args) == 2 { if cmd == agentCmd && len(args) == 3 {
runAgent(args[0], args[1], os.Stdin, os.Stdout) runAgent(args[0], args[1], args[2], os.Stdin, os.Stdout)
return return
} }
fset := flag.NewFlagSet("vault "+cmd, flag.ContinueOnError) fset := flag.NewFlagSet("vault "+cmd, flag.ContinueOnError)
@@ -103,6 +104,7 @@ func main() {
filler := fset.String("filler", "0", "") filler := fset.String("filler", "0", "")
yes := fset.Bool("yes", false, "") yes := fset.Bool("yes", false, "")
ttl := fset.Duration("for", 5*time.Minute, "") ttl := fset.Duration("for", 5*time.Minute, "")
idle := fset.Bool("idle", false, "")
// rm's -r and -f; Go's flag package has no combined short flags, so // rm's -r and -f; Go's flag package has no combined short flags, so
// -rf and -fr are spelled out. // -rf and -fr are spelled out.
recurse, force := fset.Bool("r", false, ""), fset.Bool("f", false, "") recurse, force := fset.Bool("r", false, ""), fset.Bool("f", false, "")
@@ -295,10 +297,14 @@ func main() {
check(err) check(err)
id, err := unlockKeyFile(*keyPath, pass) id, err := unlockKeyFile(*keyPath, pass)
check(err) check(err)
check(agentStart(*keyPath, id, *ttl)) check(agentStart(*keyPath, id, *ttl, *idle))
id.Close() id.Close()
if *idle {
fmt.Fprintf(os.Stderr, "unlocked until unused for %s; vault lock forgets the key sooner\n", *ttl)
} else {
fmt.Fprintf(os.Stderr, "unlocked until %s; vault lock forgets the key sooner\n", fmt.Fprintf(os.Stderr, "unlocked until %s; vault lock forgets the key sooner\n",
time.Now().Add(*ttl).Format("15:04:05")) time.Now().Add(*ttl).Format("15:04:05"))
}
case "lock": case "lock":
want(0, 0) want(0, 0)
if agentStop(*keyPath) { if agentStop(*keyPath) {
View File
+100
View File
@@ -0,0 +1,100 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only vector generator for the Rust port's URL handling and mirror
// link mapping; see oracle/gen-vectors.sh.
package main
import (
"encoding/json"
"net/http"
"net/url"
"os"
"path/filepath"
"testing"
)
func TestOracleNetVectors(t *testing.T) {
dir := os.Getenv("VAULT_ORACLE_OUT")
if dir == "" {
t.Skip("set VAULT_ORACLE_OUT to write vectors")
}
type parsed struct {
Input string `json:"input"`
Err bool `json:"err"`
String string `json:"string,omitempty"`
Scheme string `json:"scheme,omitempty"`
Host string `json:"host,omitempty"`
Hostname string `json:"hostname,omitempty"`
Path string `json:"path,omitempty"`
RawQuery string `json:"raw_query,omitempty"`
Fragment string `json:"fragment,omitempty"`
Local string `json:"local_path,omitempty"`
Key string `json:"key,omitempty"`
}
describe := func(in string, u *url.URL, err error) parsed {
if err != nil {
return parsed{Input: in, Err: true}
}
return parsed{Input: in, String: u.String(), Scheme: u.Scheme, Host: u.Host, Hostname: u.Hostname(), Path: u.Path,
RawQuery: u.RawQuery, Fragment: u.Fragment, Local: localPath(u), Key: key(u)}
}
refs := []string{
"", "a", "a/b", "./a", "../a", "../../../../a", "/a/./b/../c", "/", "//other.example/x", "//OTHER.example:8080",
"?q=1", "?", "#frag", "a#f", "a?x=1&y=2#f", "/sp ace", "/%41%2fb", "/%zz", "/caf%C3%A9", "/café", "http://H.Example:80/A",
"https://x/a/b/../../..", "HTTP://x", "mailto:a@b", "javascript:alert(1)", "ftp://x/y", "x:y", "a:b/c", "/a?b?c", "a/..",
"a/.", ".", "..", "/a//b", "https://user:pw@x/p", "http://[::1]:8080/v6", "http://x:port/", "http://x:/p", "/a%20b?c%20d#e%20f",
"http://x/\x01", "/a+b", "/q?a+b=c d", "#", "?#", "/p#a%23b", "//x/?", "http://x?only",
}
bases := []string{"http://h.example/dir/page.html?q=1#top", "https://h.example", "http://h.example:8080/a/b/", "http://h.example/a/b"}
var parses []parsed
for _, r := range refs {
u, err := url.Parse(r)
parses = append(parses, describe(r, u, err))
}
type resolved struct {
Base string `json:"base"`
Ref string `json:"ref"`
Result parsed `json:"result"`
}
var resolves []resolved
for _, b := range bases {
bu, err := url.Parse(b)
if err != nil {
t.Fatal(err)
}
for _, r := range refs {
u, err := bu.Parse(r)
resolves = append(resolves, resolved{Base: b, Ref: r, Result: describe(r, u, err)})
}
}
rels := [][2]string{{"index.html", "sub/p.html"}, {"sub/p.html", "index.html"}, {"sub/p.html", "img/a.png"}, {"a/b/c.html", "a/b/d.html"},
{"a/b/c.html", "a/x/y/z.css"}, {"x.html", "x.html"}, {"a/b/", "a/c"}, {"s.css", "img/a.png"}, {"a/b/c", "a"}}
var relOut [][3]string
for _, p := range rels {
relOut = append(relOut, [3]string{p[0], p[1], relative(p[0], p[1])})
}
escapes := map[string]string{}
for _, s := range []string{"a=1&b=2", "x y", "é/?#", "~-_.!*'()", "a+b"} {
escapes[s] = url.QueryEscape(s)
}
times := map[string]any{}
for _, s := range []string{"Mon, 02 Jan 2006 15:04:05 GMT", "Monday, 02-Jan-06 15:04:05 GMT", "Mon Jan 2 15:04:05 2006", "Tue, 15 Nov 1994 08:12:31 GMT", "bogus", "Mon, 02 Jan 2006 15:04:05 UTC", "Mon Jan 12 15:04:05 2006"} {
if tm, err := http.ParseTime(s); err == nil {
times[s] = tm.Unix()
} else {
times[s] = nil
}
}
b, _ := json.MarshalIndent(map[string]any{
"provenance": "oracle/go at 4c1507c, net_oracle_vectors_test.go",
"parse": parses,
"resolve": resolves,
"relative": relOut,
"query_escape": escapes,
"http_time": times,
}, "", " ")
if err := os.WriteFile(filepath.Join(dir, "net.json"), append(b, '\n'), 0o644); err != nil {
t.Fatal(err)
}
}
+23
View File
@@ -0,0 +1,23 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
package main
import (
"path/filepath"
"testing"
)
// Unlocking a key file: Argon2id for both slots.
func BenchmarkOracleUnlock(b *testing.B) {
p := filepath.Join(b.TempDir(), "key")
if err := writeKeyFile(p, "bench", ""); err != nil {
b.Fatal(err)
}
for b.Loop() {
if _, err := unlockKeyFile(p, "bench"); err != nil {
b.Fatal(err)
}
}
}
+208
View File
@@ -0,0 +1,208 @@
// SPDX-License-Identifier: GPL-3.0-or-later
// Test-only vector and fixture generator for the Rust port. It calls the
// frozen implementation and records what it computes; it changes nothing in
// the vault binary. Run through oracle/gen-vectors.sh.
package main
import (
"encoding/hex"
"encoding/json"
"fmt"
"os"
"path"
"path/filepath"
"testing"
"time"
"github.com/andydixon/vault/internal/pqcrypt"
"golang.org/x/crypto/argon2"
)
// Synthetic, public test passphrases. Never use them for anything real.
const (
oracleReal = "oracle-real passphrase"
oracleDecoy = "oracle-decoy passphrase"
)
func TestOracleVectors(t *testing.T) {
dir := os.Getenv("VAULT_ORACLE_OUT")
if dir == "" {
t.Skip("set VAULT_ORACLE_OUT to write vectors")
}
out := map[string]any{"provenance": "oracle/go at 4c1507c, oracle_vectors_test.go"}
// Key files.
keys := map[string]any{}
for _, k := range []struct{ name, real, decoy string }{
{"key-with-decoy.pem", oracleReal, oracleDecoy},
{"key-no-decoy.pem", oracleReal, ""},
} {
p := filepath.Join(dir, k.name)
os.Remove(p)
if err := writeKeyFile(p, k.real, k.decoy); err != nil {
t.Fatal(err)
}
entry := map[string]any{}
for label, pass := range map[string]string{"real": k.real, "decoy": k.decoy} {
if pass == "" {
continue
}
id, err := unlockKeyFile(p, pass)
if err != nil {
t.Fatal(err)
}
entry[label] = map[string]any{"passphrase": pass, "suite": id.SuiteID(), "seed": hex.EncodeToString(id.Seed())}
}
keys[k.name] = entry
}
out["keys"] = keys
var argon []map[string]string
for i, pass := range []string{"", "a", "correct horse", "pässwörd ü", "x\x00y"} {
salt := make([]byte, 16)
for j := range salt {
salt[j] = byte(i*16 + j)
}
k := argon2.IDKey([]byte(pass), salt, argonTime, argonMemory, argonThreads, 32)
argon = append(argon, map[string]string{"passphrase_hex": hex.EncodeToString([]byte(pass)), "salt": hex.EncodeToString(salt), "key": hex.EncodeToString(k)})
}
out["argon2id"] = argon
out["key_header_hex"] = hex.EncodeToString(keyHeader())
// Object IDs and sizes.
seed := "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f"
var ids []string
for i := range int64(4) {
ids = append(ids, objectID(seed, i))
}
out["object_ids"] = map[string]any{"seed": seed, "ids": ids, "big_index": objectID(seed, 1<<40), "big_index_value": int64(1 << 40)}
padded := map[string]int64{}
counts := map[string]int64{}
for _, n := range []int64{0, 1, 16, 65535, 65536, 65537, 1 << 20, chunkSize - 1, chunkSize, chunkData - 1, chunkData, chunkData + 1, 2 * chunkData, fileLimit} {
padded[fmt.Sprint(n)] = paddedSize(n)
counts[fmt.Sprint(n)] = objectCount(n)
}
out["padded_size"] = padded
out["object_count"] = counts
names := map[string]bool{}
for _, n := range []string{"", "a", "a/b", "/a", "a/", "a//b", "./a", "a/./b", "../a", "a/..", "..", ".", "a/../b", "a\\b", "a\x00b",
"a\nb", "a\x7fb", "a\u0085b", "ü/ñ", "\xff", "a b", " a", "...", "a/...", "..a", "a/.b", string(make([]byte, 1024)), longName(1024), longName(1025), "a​b", "
"} {
names[hex.EncodeToString([]byte(n))] = validName(n)
}
out["valid_name_hex"] = names
sizes := map[string]any{}
for _, s := range []string{"0", "512", "2k", "500M", "2G", "1TB", "", "G", "-1", "2X", "1.5G", "2T", "+5", "1KB", "1MK", "1kb", "B", "1T", "1099511627776", "1099511627777", " 1", "0x10", "1024G", "1025G", "-0", "1bb"} {
n, err := parseSize(s)
if err != nil {
sizes[s] = nil
} else {
sizes[s] = n
}
}
out["parse_size"] = sizes
human := map[string]string{}
for _, n := range []int64{0, 1, 1023, 1024, 1025, 1536, 1101, 10239, 10240, 1048575, 1048576, 1 << 30, 1<<40 - 1, 1 << 40, 1<<62 + 12345, 1<<63 - 1, 1075, 1126, 1177, 1638, 1689} {
human[fmt.Sprint(n)] = humanSize(n)
}
out["human_size"] = human
clean := map[string]string{}
for _, p := range []string{"", ".", "..", "/", "//", "a/b/../c", "/../a", "a/./b/", "../../a", "a//b", "./", "/a/b/..", "a/b/../../..", "abc/def/ghi/../.."} {
clean[p] = path.Clean(p)
}
out["path_clean"] = clean
durs := map[string]any{}
for _, d := range []string{"5m", "1h", "90s", "1.5h", "300ms", "1h30m", "0", "-5m", "", "5", "1d", "2h45m30.5s", "1us", "1µs", "1ns", ".5s", "5.s", "+5m", "9223372036854775807ns", "9223372036854775808ns"} {
v, err := time.ParseDuration(d)
if err != nil {
durs[d] = nil
} else {
durs[d] = map[string]any{"ns": int64(v), "string": v.String()}
}
}
out["durations"] = durs
// Record JSON, exactly as written.
var recs []map[string]any
for _, r := range []record{
{Name: "plain.txt", Seed: seed, Size: 5, Mode: 0o644, Modified: time.Unix(1700000000, 123456789).UTC(), Added: time.Unix(1700000001, 0).UTC()},
{Name: "<a&b>/

'\"\\x", Seed: seed, Size: 0, Mode: 0o755, Modified: time.Unix(0, 0).UTC(), Added: time.Unix(1, 100).UTC()},
{Name: "ü/日本語", Seed: seed, Size: 1 << 40, Mode: 0o600, Modified: time.Unix(-1, 500000000).UTC(), Added: time.Unix(253402300799, 999999999).UTC()},
{Name: "tab\there", Seed: seed, Size: 1, Mode: 0, Modified: time.Unix(1234567890, 120000000).UTC(), Added: time.Unix(1234567890, 1000).UTC()},
} {
b, err := json.Marshal(&r)
if err != nil {
t.Fatal(err)
}
recs = append(recs, map[string]any{"json": string(b), "name": r.Name, "modified_unix_nano": r.Modified.UnixNano(), "added_unix_nano": r.Added.UnixNano()})
}
out["records"] = recs
writeMiniVault(t, filepath.Join(dir, "mini-vault"), filepath.Join(dir, "key-with-decoy.pem"))
b, _ := json.MarshalIndent(out, "", " ")
if err := os.WriteFile(filepath.Join(dir, "store.json"), append(b, '\n'), 0o644); err != nil {
t.Fatal(err)
}
}
func longName(n int) string {
b := make([]byte, n)
for i := range b {
b[i] = 'a' + byte(i%26)
}
return string(b)
}
// writeMiniVault builds a small vault through the real store code, without
// init's filler (which would make the fixture megabytes): three real files,
// one decoy file and one record sealed to a throwaway key.
func writeMiniVault(t *testing.T, root, keyPath string) {
os.RemoveAll(root)
for _, d := range []string{recordsDir, objectsDir} {
for i := range 256 {
if err := os.MkdirAll(filepath.Join(root, d, fmt.Sprintf("%02x", i)), 0o700); err != nil {
t.Fatal(err)
}
}
}
os.WriteFile(filepath.Join(root, markerName), []byte(markerContent), 0o600)
src := filepath.Join(t.TempDir(), "src")
put := func(s *store, name string, body []byte, mode os.FileMode, mtime time.Time) {
os.WriteFile(src, body, 0o600)
os.Chmod(src, mode)
os.Chtimes(src, mtime, mtime)
cat, err := s.catalogue()
if err != nil {
t.Fatal(err)
}
if _, err := s.add(cat, src, name); err != nil {
t.Fatal(err)
}
}
real, _ := unlockKeyFile(keyPath, oracleReal)
decoy, _ := unlockKeyFile(keyPath, oracleDecoy)
throwaway, _ := pqcrypt.GeneratePQIdentity()
rs := &store{root: root, id: real}
boundary := make([]byte, minPadding-idSize)
for i := range boundary {
boundary[i] = byte(i * 7)
}
put(rs, "hello.txt", []byte("hello"), 0o644, time.Unix(1700000000, 123456789))
put(rs, "dir/empty", nil, 0o600, time.Unix(1600000000, 0))
put(rs, "dir/ünïcode.bin", boundary, 0o755, time.Unix(1500000000, 999999999))
put(&store{root: root, id: decoy}, "shopping.txt", []byte("milk\n"), 0o600, time.Unix(1700000000, 0))
put(&store{root: root, id: throwaway}, "foreign", []byte("not yours"), 0o600, time.Unix(1700000000, 0))
for _, d := range []string{recordsDir, objectsDir} {
shards, _ := filepath.Glob(filepath.Join(root, d, "??"))
for _, p := range append(shards, filepath.Join(root, d)) {
os.Chtimes(p, epoch, epoch)
}
}
}
View File
+37 -1
View File
@@ -21,6 +21,8 @@ import (
"strings" "strings"
"testing" "testing"
"time" "time"
"github.com/andydixon/vault/internal/pqcrypt"
) )
func TestKeyFile(t *testing.T) { func TestKeyFile(t *testing.T) {
@@ -668,7 +670,7 @@ func TestAgent(t *testing.T) {
secret = append(secret, id.Seed()...) secret = append(secret, id.Seed()...)
pr, pw := io.Pipe() pr, pw := io.Pipe()
stopped := make(chan struct{}) stopped := make(chan struct{})
go func() { runAgent(sock, "1m", bytes.NewReader(secret), pw); close(stopped) }() go func() { runAgent(sock, "1m", "fixed", bytes.NewReader(secret), pw); close(stopped) }()
if msg, _ := io.ReadAll(pr); string(msg) != "ok" { if msg, _ := io.ReadAll(pr); string(msg) != "ok" {
t.Fatalf("agent said %q", msg) t.Fatalf("agent said %q", msg)
} }
@@ -690,3 +692,37 @@ func TestAgent(t *testing.T) {
t.Fatal("agent still running") t.Fatal("agent still running")
} }
} }
func TestAgentIdle(t *testing.T) {
dir := t.TempDir()
t.Setenv("XDG_RUNTIME_DIR", dir)
keyPath := filepath.Join(dir, "key")
sock, err := agentSocket(keyPath)
if err != nil {
t.Fatal(err)
}
id, err := pqcrypt.GeneratePQIdentity()
if err != nil {
t.Fatal(err)
}
secret := binary.BigEndian.AppendUint16(nil, id.SuiteID())
secret = append(secret, id.Seed()...)
pr, pw := io.Pipe()
stopped := make(chan struct{})
go func() { runAgent(sock, "400ms", "idle", bytes.NewReader(secret), pw); close(stopped) }()
if msg, _ := io.ReadAll(pr); string(msg) != "ok" {
t.Fatalf("agent said %q", msg)
}
// Used every 150ms, it outlives its 400ms by well over twice.
for range 7 {
time.Sleep(150 * time.Millisecond)
if _, err := agentIdentity(keyPath); err != nil {
t.Fatal("idle agent expired while in use: ", err)
}
}
select {
case <-stopped:
case <-time.After(2 * time.Second):
t.Fatal("idle agent outlived its idle time")
}
}
+140
View File
@@ -0,0 +1,140 @@
#!/usr/bin/env bash
# Build the dixon-vault .deb, .rpm and Arch packages locally and show what
# is in them. Nothing is signed or published: packaging/repo.sh does that,
# reusing this script's build step.
#
# packaging/package.sh # from the working tree, into dist/packages
# packaging/package.sh v0.6 # from a tag (a clean export of it)
# OUT=/tmp/pkgs ARCHS="amd64 arm64" packaging/package.sh
#
# Needs: the pinned Rust toolchain with cargo-zigbuild and zig (the static
# Linux binaries are cross-built), and go, only to run the pinned nfpm
# packager. dpkg-deb, rpm and zstd are used for inspection when present.
# None of these is needed to run vault itself.
#
# ARCHS package architectures (default below)
# RPM_SIGNING_KEY armoured private key file to sign the .rpm with
# (repo.sh sets it; unset here means unsigned)
set -euo pipefail
# ---- project ---------------------------------------------------------------
NAME=vault # the command
# The package: plain "vault" is HashiCorp Vault in Arch's [extra] and in
# HashiCorp's own apt and dnf repositories, which would win upgrades. Both
# install /usr/bin/vault, so they conflict.
PACKAGE=dixon-vault
CONFLICTS=vault
SUMMARY="Post-quantum encrypted file vault with a decoy passphrase"
HOMEPAGE=https://dixon.cx/vault
LICENSE=GPL-3.0-or-later
MANPAGES=docs/*.1
DOCS="README.md LICENSE SECURITY.md"
# Package architecture -> Rust target (all static musl binaries).
triple() {
case $1 in
amd64) echo x86_64-unknown-linux-musl ;;
arm64) echo aarch64-unknown-linux-musl ;;
arm) echo armv7-unknown-linux-musleabihf ;;
386) echo i686-unknown-linux-musl ;;
riscv64) echo riscv64gc-unknown-linux-musl ;;
*) return 1 ;;
esac
}
# ----------------------------------------------------------------------------
ARCHS=${ARCHS:-"amd64 arm64 arm 386 riscv64"}
NFPM="go run github.com/goreleaser/nfpm/v2/cmd/nfpm@v2.41.1"
say() { echo "package: $*" >&2; }
die() { say "$*"; exit 1; }
# build_packages SRC OUTDIR VERSION: binaries, then one .deb, .rpm and
# .pkg.tar.zst per architecture, into OUTDIR.
build_packages() {
local src=$1 outdir=$2 version=$3 arch nfarch t work
for tool in cargo cargo-zigbuild zig go; do
command -v $tool >/dev/null || die "needs $tool"
done
work=$(mktemp -d)
mkdir -p "$outdir"
for arch in $ARCHS; do
t=$(triple "$arch") || die "no Rust target for $arch"
nfarch=$arch
[[ $arch == arm ]] && nfarch=arm7
say "building $NAME for $arch ($t)"
flags="--remap-path-prefix=$src=vault"
# zig's musl getrandom() never reaches the kernel on riscv64; have
# the getrandom crate issue the system call itself.
[[ $t == riscv64*-musl ]] && flags+=' --cfg getrandom_backend="linux_raw"'
(cd "$src" && RUSTFLAGS=$flags cargo zigbuild --quiet --release --locked --target "$t") || die "build for $arch failed"
install -D -m 0755 "$src/target/$t/release/$NAME" "$work/bin-$arch/$NAME"
{
cat <<-EOF
name: $PACKAGE
arch: $nfarch
platform: linux
version: $version
release: 1
maintainer: Andy Dixon <andy@dixon.cx>
description: $SUMMARY
homepage: $HOMEPAGE
license: $LICENSE
conflicts: [${CONFLICTS:-}]
contents:
- src: $work/bin-$arch/$NAME
dst: /usr/bin/$NAME
file_info: {mode: 0755}
EOF
for m in $(cd "$src" && ls $MANPAGES); do
printf ' - src: %s\n dst: /usr/share/man/man1/%s\n file_info: {mode: 0644}\n' \
"$src/$m" "$(basename "$m")"
done
for d in $DOCS; do
printf ' - src: %s\n dst: /usr/share/doc/%s/%s\n file_info: {mode: 0644}\n' \
"$src/$d" "$PACKAGE" "$d"
done
if [[ -n ${RPM_SIGNING_KEY:-} ]]; then
printf 'rpm:\n signature:\n key_file: %s\n' "$RPM_SIGNING_KEY"
fi
} >"$work/nfpm-$arch.yaml"
for fmt in deb rpm archlinux; do
$NFPM pkg -f "$work/nfpm-$arch.yaml" -p $fmt -t "$outdir/" >/dev/null
done
done
rm -rf -- "$work"
}
# inspect_packages DIR: list each package's metadata and files.
inspect_packages() {
local p
for p in "$1"/*; do
echo "== $(basename "$p")"
case $p in
*.deb) if command -v dpkg-deb >/dev/null; then dpkg-deb -f "$p" Package Version Architecture Conflicts; dpkg-deb -c "$p"; fi ;;
*.rpm) if command -v rpm >/dev/null; then rpm -qpi "$p" 2>/dev/null | head -12; rpm -qplv "$p"; fi ;;
*.pkg.tar.zst) if command -v zstd >/dev/null; then zstd -dcq "$p" | tar -tvf -; fi ;;
esac
done
}
# When run (not sourced by repo.sh): build from the tree or a tag, inspect.
if [[ ${BASH_SOURCE[0]} == "$0" ]]; then
cd "$(git rev-parse --show-toplevel)"
out=${OUT:-dist/packages}
mkdir -p -- "$out"
out=$(cd -- "$out" && pwd -P)
rm -f -- "$out"/*.deb "$out"/*.rpm "$out"/*.pkg.tar.zst
if [[ $# -gt 0 ]]; then
git rev-parse -q --verify "refs/tags/$1" >/dev/null || die "no tag $1"
src=$(mktemp -d)
trap 'rm -rf -- "$src"' EXIT
git archive "$1" | tar -x -C "$src"
version=${1#v}
else
src=$PWD
version=$(cargo metadata --no-deps --format-version 1 | python3 -c 'import sys,json; print(json.load(sys.stdin)["packages"][0]["version"])')
fi
build_packages "$src" "$out" "$version"
inspect_packages "$out"
say "$(ls "$out" | wc -l) packages in $out (unsigned, not published)"
fi
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env bash
# Build Linux packages (.deb, .rpm, Arch .pkg.tar.zst) for a release tag and
# publish them to the signed apt, dnf and pacman repositories served at
# https://repo.dixon.cx.
#
# packaging/repo.sh # the newest tag
# packaging/repo.sh 0.5 # a given tag
#
# Builds from a clean export of the tag, so the working tree doesn't matter.
# Every index is rebuilt over all the packages in the repository, so other
# projects publishing there with their copy of this script are kept.
#
# Needs: the pinned Rust toolchain with cargo-zigbuild and zig, go (to run
# nfpm), docker, gpg, apt-ftparchive (apt-utils), flock.
# REPO_DIR where the repository lives (default below)
# REPO_GNUPGHOME GnuPG home holding the signing key (default below)
# ARCHS architectures to package (see packaging/package.sh)
#
# To build and inspect packages without signing or publishing, run
# packaging/package.sh instead.
#
# The same script, with its own project block, is in ../mapsize/packaging.
set -euo pipefail
# The project (package name, contents, architectures) and the build live in
# packaging/package.sh, which can also be run on its own to build and inspect
# packages without signing or publishing anything.
. "$(dirname "$0")/package.sh"
REPO_DIR=${REPO_DIR:-/home/andy/domains/repo.dixon.cx/public_html}
REPO_GNUPGHOME=${REPO_GNUPGHOME:-$HOME/.config/repo.dixon.cx/gnupg}
SUITE=stable
COMPONENT=main
ARCH_REPO=dixon # the pacman repository name: [dixon]
say() { echo "repo: $*" >&2; }
die() { say "$*"; exit 1; }
for tool in cargo cargo-zigbuild zig go docker gpg apt-ftparchive flock; do
command -v $tool >/dev/null || die "needs $tool"
done
[[ -d $REPO_DIR ]] || die "$REPO_DIR does not exist"
export GNUPGHOME=$REPO_GNUPGHOME
KEY=$(gpg --list-secret-keys --with-colons 2>/dev/null | awk -F: '/^fpr/{print $10; exit}')
[[ -n $KEY ]] || die "no signing key in $GNUPGHOME"
cd "$(git rev-parse --show-toplevel)"
TAG=${1:-$(git describe --tags --abbrev=0)}
git rev-parse -q --verify "refs/tags/$TAG" >/dev/null || die "no tag $TAG"
VERSION=${TAG#v}
export VERSION
WORK=$(mktemp -d)
trap 'rm -rf "$WORK"' EXIT
mkdir -p "$WORK/src"
git archive "$TAG" | tar -x -C "$WORK/src"
gpg --batch --armor --export-secret-keys "$KEY" >"$WORK/signing.asc"
# ---- build ----------------------------------------------------------------
say "building $PACKAGE $VERSION for $ARCHS"
RPM_SIGNING_KEY=$WORK/signing.asc build_packages "$WORK/src" "$WORK/out" "$VERSION"
# ---- publish ---------------------------------------------------------------
# One publisher at a time: the indexes cover every project's packages.
exec 9>"$REPO_DIR/../.publish.lock"
flock 9
say "publishing to $REPO_DIR"
gpg --batch --armor --export "$KEY" >"$REPO_DIR/dixon.asc"
gpg --batch --export "$KEY" >"$REPO_DIR/dixon.gpg"
sign() { gpg --batch --yes --local-user "$KEY" "$@"; }
# apt: pool/main/<letter>/<name>/, indexed by apt-ftparchive.
pool=$REPO_DIR/apt/pool/$COMPONENT/${PACKAGE:0:1}/$PACKAGE
mkdir -p "$pool"
cp "$WORK"/out/*.deb "$pool/"
(
cd "$REPO_DIR/apt"
# Built beside the live suite and swapped in, so clients never see half.
dists=dists/.$SUITE.new
rm -rf "$dists"
debarchs=$(find pool -name '*.deb' | sed -E 's/.*_([^_]+)\.deb$/\1/' | sort -u | tr '\n' ' ')
for a in $debarchs; do
d=$dists/$COMPONENT/binary-$a
mkdir -p "$d"
apt-ftparchive --arch "$a" packages pool >"$d/Packages"
gzip -9kn "$d/Packages"
done
apt-ftparchive \
-o APT::FTPArchive::Release::Origin=repo.dixon.cx \
-o APT::FTPArchive::Release::Label=repo.dixon.cx \
-o APT::FTPArchive::Release::Suite=$SUITE \
-o APT::FTPArchive::Release::Codename=$SUITE \
-o APT::FTPArchive::Release::Components=$COMPONENT \
-o "APT::FTPArchive::Release::Architectures=${debarchs% }" \
release "$dists" >"$WORK/Release"
mv "$WORK/Release" "$dists/Release"
sign --clearsign -o "$dists/InRelease" "$dists/Release"
sign --armor --detach-sign -o "$dists/Release.gpg" "$dists/Release"
rm -rf "dists/.$SUITE.old"
[[ -d dists/$SUITE ]] && mv "dists/$SUITE" "dists/.$SUITE.old"
mv "$dists" "dists/$SUITE"
rm -rf "dists/.$SUITE.old"
)
# dnf: every architecture in one directory; dnf picks its own.
mkdir -p "$REPO_DIR/rpm"
cp "$WORK"/out/*.rpm "$REPO_DIR/rpm/"
docker image inspect repo-tools:fedora >/dev/null 2>&1 ||
docker build -q -t repo-tools:fedora - >/dev/null <<-'EOF'
FROM fedora:42
RUN dnf -y install createrepo_c && dnf clean all
EOF
docker run --rm --user "$(id -u):$(id -g)" -v "$REPO_DIR/rpm:/repo" repo-tools:fedora \
createrepo_c --quiet --update --general-compress-type=gz /repo
sign --armor --detach-sign -o "$REPO_DIR/rpm/repodata/repomd.xml.asc" "$REPO_DIR/rpm/repodata/repomd.xml"
cat >"$REPO_DIR/rpm/dixon.repo" <<EOF
[dixon]
name=repo.dixon.cx
baseurl=https://repo.dixon.cx/rpm
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://repo.dixon.cx/dixon.asc
EOF
# pacman: one directory per architecture, each package signed, then
# repo-add (from an Arch container) and a signed database.
added=
for pkg in "$WORK"/out/*.pkg.tar.zst; do
a=$(basename "$pkg" .pkg.tar.zst)
a=${a##*-}
mkdir -p "$REPO_DIR/arch/$a"
cp "$pkg" "$REPO_DIR/arch/$a/"
sign --detach-sign -o "$REPO_DIR/arch/$a/$(basename "$pkg").sig" "$pkg"
added+=" $a"
done
for a in $(tr ' ' '\n' <<<"$added" | sort -u); do
dir=$REPO_DIR/arch/$a
# A fresh database of every package here; -p keeps the newest version.
rm -f "$dir/$ARCH_REPO".{db,files}{,.tar.gz}{,.sig}
docker run --rm --user "$(id -u):$(id -g)" -e HOME=/tmp -v "$dir:/repo" -w /repo archlinux:latest \
sh -c "repo-add --quiet --prevent-downgrade $ARCH_REPO.db.tar.gz *.pkg.tar.zst" 2>&1 |
{ grep -v "newer version\|^$" >&2 || true; }
for db in $ARCH_REPO.db $ARCH_REPO.files; do
sign --detach-sign -o "$dir/$db.tar.gz.sig" "$dir/$db.tar.gz"
ln -sfn "$db.tar.gz.sig" "$dir/$db.sig"
done
rm -f "$dir"/*.old "$dir"/*.old.sig
done
say "published $PACKAGE $VERSION: $(cd "$WORK/out" && ls | wc -l) packages"
+4
View File
@@ -0,0 +1,4 @@
[toolchain]
channel = "1.99.0"
components = ["rustfmt", "clippy"]
profile = "minimal"
+1
View File
@@ -0,0 +1 @@
max_width = 140
+113
View File
@@ -0,0 +1,113 @@
#!/usr/bin/env bash
# Compare the Go oracle and the Rust build on this machine, end to end.
#
# scripts/bench.sh [WORKDIR] (default: target/bench, on the repo's disk)
# RUNS=5 BIG_MB=2048 SMALL=5000 scripts/bench.sh
#
# For each implementation, in a fresh key and vault on the same filesystem:
# add one BIG_MB file, and add SMALL small files (1-16 KiB) as a folder, each
# into a fresh vault; then, in one vault holding both, get the big file,
# list the catalogue (filler plus SMALL+1 records), and get one small file. Each step is timed RUNS times (median, min, max), with peak RSS and
# peak thread count. Every step includes unlocking the key (two Argon2id
# runs). Both implementations fsync every object and record, so `add`
# timings include durable writes. Page caches are warm: dropping them needs
# root, so cold-cache numbers are not measured here.
#
# Needs: go (to build the oracle), cargo, /usr/bin/time (GNU).
set -euo pipefail
cd "$(dirname "$0")/.."
root=$PWD
work=${1:-$root/target/bench}
runs=${RUNS:-3}
big_mb=${BIG_MB:-1024}
small=${SMALL:-500}
mkdir -p -- "$work"
work=$(cd -- "$work" && pwd -P)
[[ $work == "$root" || $work == / ]] && { echo "refusing WORKDIR=$work" >&2; exit 2; }
cargo build --release --locked --quiet
(cd oracle/go && CGO_ENABLED=0 go build -trimpath -o "$work/vault-go" .)
cp target/release/vault "$work/vault-rust"
data=$work/data
if [[ ! -f $data/big || $(stat -c %s "$data/big") != $((big_mb << 20)) ]]; then
rm -rf -- "$data"
mkdir -p -- "$data/small"
head -c $((big_mb << 20)) /dev/urandom >"$data/big"
for i in $(seq -w 1 "$small"); do
head -c $((1024 + RANDOM % 15360)) /dev/urandom >"$data/small/f$i"
done
fi
export VAULT_PASSPHRASE=bench-passphrase
# measure LABEL CMD...: one timed run, printing "seconds rss_kib threads".
measure() {
local out=$work/time.$$ peak=0 pid t
/usr/bin/time -f '%e %M' -o "$out" "$@" >/dev/null 2>&1 &
pid=$!
while kill -0 $pid 2>/dev/null; do
for t in /proc/[0-9]*/status; do
if grep -q "^PPid:[[:space:]]*$pid$" "$t" 2>/dev/null; then
n=$(awk '/^Threads:/{print $2}' "$t" 2>/dev/null || echo 0)
((n > peak)) && peak=$n
fi
done
sleep 0.05
done
wait $pid
echo "$(cat "$out") $peak"
rm -f -- "$out"
}
summarise() { # lines of "secs rss threads" -> median min max, max rss, max threads
sort -n | awk '{t[NR]=$1; if($2>r)r=$2; if($3>th)th=$3} END {printf "%8.2f s (min %.2f, max %.2f) rss %6.0f MiB threads %d\n", t[int((NR+1)/2)], t[1], t[NR], r/1024, th}'
}
fresh() { # fresh key and vault at $1
rm -rf -- "$1"
mkdir -p -- "$1"
export VAULT_KEY=$1/key VAULT_DIR=$1/store
"$bin" genkey 2>/dev/null
"$bin" init 2>/dev/null
}
for impl in go rust; do
bin=$work/vault-$impl
echo "== $impl ($(stat -c %s "$bin") bytes)"
# Writes: a fresh vault for every run.
for step in add-big add-small; do
results=""
for run in $(seq 1 "$runs"); do
fresh "$work/run-$impl"
case $step in
add-big) r=$(measure "$bin" add "$data/big") ;;
add-small) r=$(measure "$bin" add "$data/small") ;;
esac
results+="$r"$'\n'
done
printf ' %-10s ' "$step"
printf '%s' "$results" | grep . | summarise
done
# Reads: one populated vault, read repeatedly.
v=$work/run-$impl
fresh "$v"
"$bin" add "$data/big" 2>/dev/null
"$bin" add "$data/small" 2>/dev/null
for step in get-big list get-small; do
results=""
for run in $(seq 1 "$runs"); do
rm -f -- "$v/big.out" "$v/small.out"
case $step in
get-big) r=$(measure "$bin" get big -o "$v/big.out") ;;
list) r=$(measure "$bin" list) ;;
get-small) r=$(measure "$bin" get "small/f$(printf "%0${#small}d" 1)" -o "$v/small.out") ;;
esac
results+="$r"$'\n'
done
printf ' %-10s ' "$step"
printf '%s' "$results" | grep . | summarise
done
rm -rf -- "$work/run-$impl"
done
+43
View File
@@ -0,0 +1,43 @@
#!/usr/bin/env bash
# Runtime smoke test of one vault binary, optionally under an emulator:
# create a key and vault, store files across object boundaries, read them
# back, then (if a reference binary is given) check the reference can read
# what this binary wrote and the other way round.
#
# scripts/smoke.sh BINARY [REFERENCE_BINARY]
# EMU="qemu-aarch64" scripts/smoke.sh dist/vault-linux-arm64 target/release/vault
#
# Uses a private temporary directory, removed afterwards. The passphrase is
# a public test value.
set -euo pipefail
bin=$(realpath "$1")
ref=${2:+$(realpath "$2")}
emu=${EMU:-}
work=$(mktemp -d "${TMPDIR:-/tmp}/vault-smoke.XXXXXX")
trap 'rm -rf -- "$work"' EXIT
cd "$work"
export VAULT_PASSPHRASE=smoke-test-passphrase VAULT_KEY=$work/key VAULT_DIR=$work/store
export XDG_RUNTIME_DIR=$work/run
mkdir -m 700 run
v() { $emu "$bin" "$@"; }
v genkey 2>/dev/null
v init 2>/dev/null
head -c $((4 * 1024 * 1024 + 123)) /dev/urandom >big
printf 'small\n' >small
: >empty
v add big 2>/dev/null
v add small dir 2>/dev/null
v add empty dir 2>/dev/null
[[ $(v list | wc -l) == 3 ]] || { echo "FAIL list"; exit 1; }
v get big -o big.out 2>/dev/null && cmp big big.out
v get dir/small -o small.out 2>/dev/null && cmp small small.out
if [[ -n $ref ]]; then
"$ref" get big -o big.ref 2>/dev/null && cmp big big.ref
printf 'from the reference\n' >r.txt
"$ref" add r.txt 2>/dev/null
v get r.txt -o r.out 2>/dev/null && cmp r.txt r.out
fi
v rm -rf dir 2>/dev/null
[[ $(v list | wc -l) == $([[ -n $ref ]] && echo 2 || echo 1) ]] || { echo "FAIL rm"; exit 1; }
echo "ok $(basename "$bin")${emu:+ (under $emu)}"
+325
View File
@@ -0,0 +1,325 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! `vault absorb`: add every regular file of a tar archive (optionally
//! gzipped, possibly in several gzip members) read from a stream, each
//! streamed straight into the vault so no plaintext touches the disk.
//! Other entry types, and names that would escape the destination folder,
//! are reported and skipped, as in the Go version.
use crate::error::{Error, Kind, Result};
use crate::store::{Catalogue, Store, VaultName};
use crate::util::{Instant, clean, join};
use std::io::{BufRead, Read};
pub fn absorb(
store: &Store,
cat: &mut Catalogue,
mut input: impl BufRead,
dest: &str,
report: &mut dyn FnMut(&VaultName, bool),
) -> Result<()> {
let gz = input
.fill_buf()
.map_err(|e| Error::new(format!("reading archive: {}", crate::util::io_reason(&e))))?
.starts_with(&[0x1f, 0x8b]);
let r: Box<dyn Read> = if gz {
Box::new(flate2::bufread::MultiGzDecoder::new(input))
} else {
Box::new(input)
};
let mut archive = tar::Archive::new(r);
let entries = archive
.entries()
.map_err(|e| Error::new(format!("reading archive: {}", crate::util::io_reason(&e))))?;
let mut n = 0u64;
for entry in entries {
let mut entry = entry.map_err(|e| Error::new(format!("reading archive: {}", crate::util::io_reason(&e))))?;
let raw = entry.path_bytes().into_owned();
let shown = String::from_utf8_lossy(&raw).into_owned();
let kind = entry.header().entry_type();
if !kind.is_file() {
if !kind.is_dir() {
eprintln!("skipping {shown}: not a regular file");
}
continue;
}
let Ok(text) = std::str::from_utf8(&raw) else {
crate::bail!("invalid vault path {shown:?}");
};
let Some(rel) = safe_name(text) else {
eprintln!("skipping {shown}: unsafe name");
continue;
};
let name = join(&[dest, &rel]);
let name = VaultName::new(&name).ok_or_else(|| Error::new(format!("invalid vault path {name:?}")))?;
// The entry's size, with a PAX "size" override applied, as Go's
// hdr.Size; the fixed header field alone would truncate files that
// Go's archive/tar writes for sizes past 8 GiB.
let size = entry.size();
if is_gnu_sparse(&mut entry) {
eprintln!("skipping {shown}: sparse files are not supported");
continue;
}
let mode = entry.header().mode().unwrap_or(0) & 0o777;
let mtime = mtime(&mut entry)?;
match store.add_reader(cat, &mut entry, size, mode, mtime, &name) {
Ok(a) => report(&name, a.replaced),
Err(e) if e.kind == Kind::SourceShort => crate::bail!("{shown}: archive ends early"),
Err(e) => return Err(e),
}
n += 1;
}
if n == 0 {
crate::bail!("no files in archive");
}
Ok(())
}
/// Go's check: the name, cleaned as if rooted, must equal the name with a
/// leading "./" or "/" removed and cleaned, and contain no backslash.
pub fn safe_name(name: &str) -> Option<String> {
let rooted = clean(&format!("/{name}"));
let n = rooted.strip_prefix('/').unwrap_or(&rooted).to_string();
let plain = name.strip_prefix("./").unwrap_or(name);
let plain = plain.strip_prefix('/').unwrap_or(plain);
if name.contains('\\') || n.is_empty() || n != clean(plain) {
return None;
}
Some(n)
}
/// A PAX-format GNU sparse file, whose data is an encoded map, not the
/// file's bytes.
fn is_gnu_sparse<R: Read>(entry: &mut tar::Entry<R>) -> bool {
if entry.header().entry_type() == tar::EntryType::GNUSparse {
return true;
}
match entry.pax_extensions() {
Ok(Some(exts)) => exts.flatten().any(|e| e.key_bytes().starts_with(b"GNU.sparse.")),
_ => false,
}
}
/// The entry's modification time: the PAX `mtime` (with its fraction) if
/// present, else the header's whole seconds.
fn mtime<R: Read>(entry: &mut tar::Entry<R>) -> Result<Instant> {
if let Ok(Some(exts)) = entry.pax_extensions() {
for ext in exts.flatten() {
if ext.key_bytes() == b"mtime"
&& let Some(t) = std::str::from_utf8(ext.value_bytes()).ok().and_then(parse_pax_time)
{
return Ok(t);
}
}
}
let secs = entry
.header()
.mtime()
.map_err(|e| Error::new(format!("reading archive: {}", crate::util::io_reason(&e))))?;
Ok(Instant {
secs: secs.min(i64::MAX as u64) as i64,
nanos: 0,
})
}
/// A PAX time: optional sign, decimal seconds, optional fraction.
fn parse_pax_time(s: &str) -> Option<Instant> {
let (neg, s) = match s.strip_prefix('-') {
Some(r) => (true, r),
None => (false, s),
};
let (whole, frac) = s.split_once('.').unwrap_or((s, ""));
if whole.is_empty() || !whole.bytes().all(|c| c.is_ascii_digit()) || !frac.bytes().all(|c| c.is_ascii_digit()) {
return None;
}
let secs: i64 = whole.parse().ok()?;
let mut nanos: u32 = format!("{:0<9}", &frac[..frac.len().min(9)]).parse().ok()?;
let mut secs = if neg { -secs } else { secs };
if neg && nanos > 0 {
secs -= 1;
nanos = 1_000_000_000 - nanos;
}
Some(Instant { secs, nanos })
}
#[cfg(test)]
mod tests {
use super::*;
use crate::crypto::kem::{Identity, SUITE_ID};
use crate::pipeline::Pool;
use crate::testutil::TempDir;
use std::sync::Arc;
fn store(tag: &str) -> (TempDir, Store) {
let d = TempDir::new(tag);
let root = d.0.join("v");
Store::init(&root, 0, Arc::new(Pool::new(2))).unwrap();
let s = Store::open(
&root,
Identity::from_suite_seed(SUITE_ID, &[1; 32]).unwrap(),
Arc::new(Pool::new(2)),
)
.unwrap();
(d, s)
}
fn header(name: &str, kind: tar::EntryType, size: u64, mode: u32, mtime: u64) -> tar::Header {
let mut h = tar::Header::new_gnu();
h.set_entry_type(kind);
h.set_size(size);
h.set_mode(mode);
h.set_mtime(mtime);
// Raw bytes, so unsafe names reach the reader unaltered.
let n = name.as_bytes();
h.as_old_mut().name[..n.len()].copy_from_slice(n);
h.set_cksum();
h
}
#[test]
fn absorbs_regular_files_only() {
let (d, s) = store("absorb");
let mut tw = tar::Builder::new(Vec::new());
for (name, kind, body) in [
("./d/", tar::EntryType::Directory, &b""[..]),
("./d/a.txt", tar::EntryType::Regular, b"hello"),
("../escape", tar::EntryType::Regular, b"nope!"),
("/abs", tar::EntryType::Regular, b"abs"),
("a\\b", tar::EntryType::Regular, b"bs"),
("./d/link", tar::EntryType::Symlink, b""),
] {
let h = header(name, kind, body.len() as u64, 0o600, 1577934245);
tw.append(&h, body).unwrap();
}
let tarball = tw.into_inner().unwrap();
let mut gz = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default());
std::io::Write::write_all(&mut gz, &tarball).unwrap();
let gz = gz.finish().unwrap();
let mut cat = s.catalogue().unwrap();
let mut added = Vec::new();
absorb(&s, &mut cat, &gz[..], "in", &mut |n, _| added.push(n.to_string())).unwrap();
assert_eq!(added, ["in/d/a.txt", "in/abs"]);
let r = cat.current("in/d/a.txt").unwrap();
assert_eq!(
(r.mode, r.modified),
(
0o600,
Instant {
secs: 1577934245,
nanos: 0
}
)
);
let dir = crate::platform::fs::Dir::open(&d.0).unwrap();
s.extract(r, &dir, std::ffi::OsStr::new("out")).unwrap();
assert_eq!(std::fs::read(d.0.join("out")).unwrap(), b"hello");
// Plain tar works too, and an archive with nothing to add fails.
absorb(&s, &mut cat, &tarball[..], "", &mut |_, _| {}).unwrap();
let mut empty = tar::Builder::new(Vec::new());
empty
.append(&header("dir/", tar::EntryType::Directory, 0, 0o755, 0), &b""[..])
.unwrap();
let e = absorb(&s, &mut cat, &empty.into_inner().unwrap()[..], "", &mut |_, _| {}).unwrap_err();
assert_eq!(e.msg, "no files in archive");
}
#[test]
fn truncated_and_malicious_archives() {
let (_d, s) = store("absorb-bad");
let mut tw = tar::Builder::new(Vec::new());
tw.append(&header("big", tar::EntryType::Regular, 100_000, 0o600, 0), &vec![7u8; 100_000][..])
.unwrap();
let t = tw.into_inner().unwrap();
let before = s.catalogue().unwrap().len();
let mut cat = s.catalogue().unwrap();
let e = absorb(&s, &mut cat, &t[..60_000], "", &mut |_, _| {}).unwrap_err();
assert!(e.msg.contains("archive ends early") || e.msg.starts_with("reading archive"), "{e}");
assert_eq!(s.catalogue().unwrap().len(), before, "a truncated entry stored something");
for junk in [
&b"not a tar archive at all, but long enough to fill a header block"[..],
&[0x1f, 0x8b, 0, 0][..],
&[0u8; 1024][..],
] {
assert!(absorb(&s, &mut cat, junk, "", &mut |_, _| {}).is_err());
}
// A corrupted gzip stream fails rather than storing a partial file.
let mut gz = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default());
std::io::Write::write_all(&mut gz, &t).unwrap();
let mut g = gz.finish().unwrap();
let mid = g.len() / 2;
g[mid] ^= 0xff;
assert!(absorb(&s, &mut cat, &g[..], "", &mut |_, _| {}).is_err());
assert_eq!(s.catalogue().unwrap().len(), before);
}
/// A PAX "size" record overrides the header's size field, as Go writes
/// for files over 8 GiB; the whole body must be stored.
#[test]
fn pax_size_override() {
let (d, s) = store("absorb-pax");
let mut t = Vec::new();
let rec = |k: &str, v: &str| {
let body = format!(" {k}={v}\n");
let mut n = body.len() + 1;
while format!("{n}{body}").len() != n {
n += 1;
}
format!("{n}{body}")
};
let pax = rec("size", "11");
let mut h = header("PaxHeaders/big", tar::EntryType::XHeader, pax.len() as u64, 0o644, 0);
h.set_cksum();
t.extend_from_slice(h.as_bytes());
let mut p = pax.into_bytes();
p.resize(p.len().div_ceil(512) * 512, 0);
t.extend_from_slice(&p);
let h = header("big.bin", tar::EntryType::Regular, 0, 0o644, 0);
t.extend_from_slice(h.as_bytes());
let mut body = b"hello world".to_vec();
body.resize(512, 0);
t.extend_from_slice(&body);
t.extend_from_slice(&[0u8; 1024]);
let mut cat = s.catalogue().unwrap();
absorb(&s, &mut cat, &t[..], "", &mut |_, _| {}).unwrap();
let r = cat.current("big.bin").unwrap();
assert_eq!(r.size, 11);
let dir = crate::platform::fs::Dir::open(&d.0).unwrap();
s.extract(r, &dir, std::ffi::OsStr::new("o")).unwrap();
assert_eq!(std::fs::read(d.0.join("o")).unwrap(), b"hello world");
}
#[test]
fn names_and_times() {
// As in Go, only names that would climb out are refused; others are
// cleaned.
for (n, want) in [
("./a/b", Some("a/b")),
("a/../b", Some("b")),
("../x", None),
("a/../../x", None),
("/x", Some("x")),
("a//b", Some("a/b")),
("a\\b", None),
("", None),
(".", None),
("a/", Some("a")),
] {
assert_eq!(safe_name(n).as_deref(), want, "{n:?}");
}
assert_eq!(
parse_pax_time("1577934245.5"),
Some(Instant {
secs: 1577934245,
nanos: 500_000_000
})
);
assert_eq!(
parse_pax_time("-1.25"),
Some(Instant {
secs: -2,
nanos: 750_000_000
})
);
assert_eq!(parse_pax_time("x"), None);
}
}
+420
View File
@@ -0,0 +1,420 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! The key agent, wire-compatible with the Go version: `vault unlock`
//! starts a copy of this binary as a hidden `__agent` process, hands it the
//! suite ID and master seed over a pipe (never argv, never disk), and the
//! agent serves them on a Unix socket in a per-user 0700 directory until a
//! deadline, fixed or (with `--idle`) moved on each use, or `vault lock`.
//!
//! Protocol: the client connects and sends one byte, `g` (get: the agent
//! replies with the 2-byte big-endian suite ID and 32-byte seed) or `q`
//! (quit: the agent wipes the key and closes). Every request has a
//! two-second deadline. The socket is
//! `$XDG_RUNTIME_DIR` (else the temp dir) `/vault-UID/agent-` + the first 8
//! bytes of SHA-256(absolute key path) in hex, exactly as Go computes it,
//! so a Go client can use a Rust agent and the other way round.
//!
//! Before it accepts the secret, the agent disables core dumps, makes
//! itself undumpable (Linux) and locks the page that will hold the secret;
//! if any of that fails it reports why and exits without the key.
use crate::crypto::kem::Identity;
use crate::error::{Error, Result};
use std::path::{Path, PathBuf};
use std::time::Duration;
pub const AGENT_CMD: &str = "__agent";
#[cfg_attr(not(unix), allow(dead_code))]
const GET: u8 = b'g';
#[cfg_attr(not(unix), allow(dead_code))]
const QUIT: u8 = b'q';
#[cfg_attr(not(unix), allow(dead_code))]
const REPLY: usize = 2 + crate::crypto::kem::SEED_SIZE;
#[cfg_attr(not(unix), allow(dead_code))]
const TIMEOUT: Duration = Duration::from_secs(2);
/// How long `unlock` waits for a starting agent to say it is ready.
#[cfg_attr(not(unix), allow(dead_code))]
const START_TIMEOUT: Duration = Duration::from_secs(10);
#[cfg_attr(not(unix), allow(dead_code))]
const SECRET_ENVS: [&str; 2] = ["VAULT_PASSPHRASE", "VAULT_DECOY_PASSPHRASE"];
/// Go's `filepath.Abs`: lexical, with `os.Getwd`'s preference for `$PWD`.
pub fn go_abs(p: &Path) -> std::io::Result<PathBuf> {
#[cfg(unix)]
{
use std::os::unix::ffi::OsStrExt;
let joined = if p.is_absolute() { p.to_path_buf() } else { go_getwd()?.join(p) };
let s = std::str::from_utf8(joined.as_os_str().as_bytes()).map_err(|_| std::io::Error::other("key path is not valid UTF-8"))?;
Ok(PathBuf::from(crate::util::clean(s)))
}
#[cfg(not(unix))]
{
std::path::absolute(p)
}
}
#[cfg(unix)]
fn go_getwd() -> std::io::Result<PathBuf> {
use std::os::unix::fs::MetadataExt;
let dot = std::fs::metadata(".")?;
if let Some(pwd) = std::env::var_os("PWD").map(PathBuf::from)
&& pwd.is_absolute()
&& let Ok(m) = std::fs::metadata(&pwd)
&& m.dev() == dot.dev()
&& m.ino() == dot.ino()
{
return Ok(pwd);
}
std::env::current_dir()
}
/// The agent socket for the key file at `key`, creating the private
/// per-user directory if needed and refusing one that is not private.
pub fn socket_path(key: &Path) -> Result<PathBuf> {
use sha2::Digest;
let abs = go_abs(key).map_err(|e| Error::io("abs", key, &e))?;
let base = std::env::var_os("XDG_RUNTIME_DIR")
.filter(|v| !v.is_empty())
.map(PathBuf::from)
.unwrap_or_else(go_temp_dir);
let dir = base.join(format!("vault-{}", uid()));
match crate::platform::fs::create_private_dir(&dir) {
Ok(()) => {}
Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => {}
Err(e) => return Err(Error::io("mkdir", &dir, &e)),
}
private_dir(&dir)?;
let sum = sha2::Sha256::digest(abs.as_os_str().as_encoded_bytes());
Ok(dir.join(format!("agent-{}", crate::store::hex(&sum[..8]))))
}
/// Go's `os.TempDir`: `$TMPDIR` or /tmp on Unix.
fn go_temp_dir() -> PathBuf {
#[cfg(unix)]
{
std::env::var_os("TMPDIR")
.filter(|v| !v.is_empty())
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from("/tmp"))
}
#[cfg(not(unix))]
{
std::env::temp_dir()
}
}
fn uid() -> u32 {
#[cfg(unix)]
{
rustix::process::getuid().as_raw()
}
#[cfg(not(unix))]
{
// Go's os.Getuid returns -1 on Windows.
u32::MAX
}
}
/// A real directory, owned by this user, with mode exactly 0700.
#[cfg(unix)]
fn private_dir(dir: &Path) -> Result<()> {
use std::os::unix::fs::{MetadataExt, PermissionsExt};
let m = std::fs::symlink_metadata(dir).map_err(|e| Error::io("lstat", dir, &e))?;
if !m.is_dir() || m.uid() != uid() || m.permissions().mode() & 0o777 != 0o700 {
crate::bail!("{} is not a private directory of this user", dir.display());
}
Ok(())
}
#[cfg(not(unix))]
fn private_dir(_: &Path) -> Result<()> {
Ok(())
}
#[cfg(unix)]
mod imp {
use super::*;
use crate::platform::memory::{LockedBuf, harden_process};
use std::io::{Read, Write};
use std::os::unix::net::{UnixListener, UnixStream};
fn dial(key: &Path, op: u8) -> Result<UnixStream> {
let sock = socket_path(key)?;
let c = UnixStream::connect(&sock).map_err(|e| Error::io("dial", &sock, &e))?;
c.set_read_timeout(Some(TIMEOUT))?;
c.set_write_timeout(Some(TIMEOUT))?;
(&c).write_all(&[op])?;
Ok(c)
}
/// The identity a running agent holds for `key`.
pub fn identity(key: &Path) -> Result<Identity> {
let mut c = dial(key, GET)?;
let mut buf = crate::secret::SecretArray::<REPLY>::zero();
c.read_exact(buf.expose_mut())?;
let b = buf.expose();
Identity::from_suite_seed(u16::from_be_bytes([b[0], b[1]]), &b[2..]).map_err(Error::from)
}
/// Tells the agent for `key` to forget it; whether one was running.
pub fn stop(key: &Path) -> bool {
let Ok(mut c) = dial(key, QUIT) else { return false };
// Returns once the agent has wiped the key and closed.
let _ = std::io::copy(&mut c, &mut std::io::sink());
true
}
/// The command that starts an agent: this binary, with passphrase
/// variables removed from its environment and nothing secret in argv.
pub fn command(sock: &Path, ttl: Duration, idle: bool) -> Result<std::process::Command> {
let exe =
std::env::current_exe().map_err(|e| Error::new(format!("cannot find this executable: {}", crate::util::io_reason(&e))))?;
let mut c = std::process::Command::new(exe);
c.arg(AGENT_CMD)
.arg(sock)
.arg(crate::util::format_duration(ttl))
.arg(if idle { "idle" } else { "fixed" });
for v in SECRET_ENVS {
c.env_remove(v);
}
c.current_dir("/")
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::null());
Ok(c)
}
/// Replaces any agent for `key` with one holding `id` for `ttl`.
pub fn start(key: &Path, id: &Identity, ttl: Duration, idle: bool) -> Result<()> {
let sock = socket_path(key)?;
stop(key);
let mut child = command(&sock, ttl, idle)?
.spawn()
.map_err(|e| Error::new(format!("cannot start the agent: {}", crate::util::io_reason(&e))))?;
let mut secret = crate::secret::SecretArray::<REPLY>::zero();
secret.expose_mut()[..2].copy_from_slice(&id.suite_id().to_be_bytes());
secret.expose_mut()[2..].copy_from_slice(id.seed());
let mut stdin = child.stdin.take().expect("piped");
let wrote = stdin.write_all(secret.expose());
drop(secret);
drop(stdin);
let mut stdout = child.stdout.take().expect("piped");
let (tx, rx) = std::sync::mpsc::channel();
std::thread::spawn(move || {
let mut msg = Vec::new();
let _ = stdout.by_ref().take(4096).read_to_end(&mut msg);
let _ = tx.send(msg);
});
// The agent closes its standard output once listening, after saying
// "ok" or why not.
let msg = rx.recv_timeout(START_TIMEOUT);
let fail = |child: &mut std::process::Child, why: String| {
let _ = child.kill();
let _ = child.wait();
Err(Error::new(why))
};
if let Err(e) = wrote {
return fail(
&mut child,
format!("cannot hand the key to the agent: {}", crate::util::io_reason(&e)),
);
}
match msg {
Ok(m) if m == b"ok" => Ok(()),
Ok(m) => fail(&mut child, format!("agent did not start: {}", String::from_utf8_lossy(&m).trim())),
Err(_) => fail(&mut child, "agent did not start in time".into()),
}
}
fn read_fd0(buf: &mut [u8]) -> std::io::Result<()> {
let mut done = 0;
while done < buf.len() {
match rustix::io::read(rustix::stdio::stdin(), &mut buf[done..]) {
Ok(0) => return Err(std::io::ErrorKind::UnexpectedEof.into()),
Ok(n) => done += n,
Err(rustix::io::Errno::INTR) => {}
Err(e) => return Err(e.into()),
}
}
Ok(())
}
/// The agent process. Returns when it expires or is told to quit.
pub fn run(sock: &Path, ttl: &str, mode: &str) {
let mut out = std::io::stdout();
let report = |out: &mut std::io::Stdout, msg: &str| {
let _ = out.write_all(msg.as_bytes());
let _ = out.flush();
// Closing standard output (by pointing it at /dev/null) tells
// `unlock` the outcome is final.
if let Ok(null) = rustix::fs::open("/dev/null", rustix::fs::OFlags::WRONLY, rustix::fs::Mode::empty()) {
let _ = rustix::stdio::dup2_stdout(&null);
}
};
let ttl = match crate::util::parse_duration(ttl) {
Ok(n) if n > 0 && (mode == "fixed" || mode == "idle") => Duration::from_nanos(n as u64),
_ => return report(&mut out, "bad duration or mode"),
};
if let Err(e) = harden_process() {
return report(&mut out, &e.to_string());
}
let mut secret = match LockedBuf::new(REPLY) {
Ok(b) => b,
Err(e) => return report(&mut out, &e.to_string()),
};
// Read straight from fd 0 into the locked page: std's buffered
// stdin would leave a copy in ordinary heap memory.
if read_fd0(secret.as_mut_slice()).is_err() {
return report(&mut out, "no key received");
}
{
let s = secret.as_slice();
if let Err(e) = Identity::from_suite_seed(u16::from_be_bytes([s[0], s[1]]), &s[2..]) {
return report(&mut out, &e);
}
}
// A socket left by an agent that died stops bind. Remove it only if
// nothing answers on it, so a concurrent unlock's live agent is not
// cut off from `vault lock`.
if UnixStream::connect(sock).is_ok() {
return report(&mut out, "another agent is already running for this key");
}
let _ = std::fs::remove_file(sock);
let l = match UnixListener::bind(sock) {
Ok(l) => l,
Err(e) => return report(&mut out, &crate::util::io_msg("listen unix", sock, &e)),
};
let ours = match std::fs::symlink_metadata(sock) {
Ok(m) => m,
Err(e) => return report(&mut out, &crate::util::io_msg("stat", sock, &e)),
};
let cleanup = || {
// Remove the socket only if it is still the one this agent made.
use std::os::unix::fs::MetadataExt;
if let Ok(m) = std::fs::symlink_metadata(sock)
&& m.ino() == ours.ino()
&& m.dev() == ours.dev()
{
let _ = std::fs::remove_file(sock);
}
};
{
use std::os::unix::fs::PermissionsExt;
if let Err(e) = std::fs::set_permissions(sock, std::fs::Permissions::from_mode(0o600)).and_then(|_| l.set_nonblocking(true)) {
cleanup();
return report(&mut out, &crate::util::io_msg("chmod", sock, &e));
}
}
report(&mut out, "ok");
serve(&l, &mut secret, ttl, mode == "idle");
secret.clear();
drop(secret);
cleanup();
}
/// Serves until the deadline. The listener is polled with a timeout, so
/// expiry never waits for another client; each request is bounded by
/// two seconds and by the time left.
fn serve(l: &UnixListener, secret: &mut LockedBuf, ttl: Duration, idle: bool) {
use rustix::event::{PollFd, PollFlags, Timespec, poll};
let mut deadline = std::time::Instant::now() + ttl;
loop {
let left = deadline.saturating_duration_since(std::time::Instant::now());
if left.is_zero() {
return;
}
let t = Timespec {
tv_sec: left.as_secs() as i64,
tv_nsec: left.subsec_nanos() as _,
};
let mut fds = [PollFd::new(l, PollFlags::IN)];
match poll(&mut fds, Some(&t)) {
Ok(0) => continue,
Ok(_) => {}
Err(rustix::io::Errno::INTR) => continue,
Err(_) => return,
}
let Ok((c, _)) = l.accept() else { continue };
let left = deadline.saturating_duration_since(std::time::Instant::now());
if left.is_zero() {
return;
}
let limit = left.min(TIMEOUT);
if c.set_nonblocking(false)
.and_then(|_| c.set_read_timeout(Some(limit)))
.and_then(|_| c.set_write_timeout(Some(limit)))
.is_err()
{
continue;
}
let mut op = [0u8; 1];
if (&c).read_exact(&mut op).is_err() {
continue;
}
match op[0] {
GET => {
if std::time::Instant::now() < deadline {
let _ = (&c).write_all(secret.as_slice());
if idle {
deadline = std::time::Instant::now() + ttl;
}
}
}
QUIT => {
secret.clear();
drop(c);
return;
}
_ => {}
}
}
}
}
#[cfg(unix)]
pub use imp::{command, identity, run, start, stop};
#[cfg(not(unix))]
mod imp {
use super::*;
const UNSUPPORTED: &str = "vault unlock is not supported on this platform; use the passphrase each time";
pub fn identity(_: &Path) -> Result<Identity> {
Err(Error::new(UNSUPPORTED))
}
pub fn stop(_: &Path) -> bool {
false
}
pub fn start(_: &Path, _: &Identity, _: Duration, _: bool) -> Result<()> {
Err(Error::new(UNSUPPORTED))
}
pub fn run(_: &Path, _: &str, _: &str) {}
}
#[cfg(not(unix))]
pub use imp::{identity, run, start, stop};
#[cfg(all(test, unix))]
mod tests {
use super::*;
#[test]
fn command_strips_secrets_from_the_environment() {
let c = command(Path::new("/x/sock"), Duration::from_secs(300), true).unwrap();
let removed: Vec<_> = c
.get_envs()
.filter(|(_, v)| v.is_none())
.map(|(k, _)| k.to_string_lossy().into_owned())
.collect();
for v in SECRET_ENVS {
assert!(removed.iter().any(|r| r == v), "{v} not removed");
}
let args: Vec<_> = c.get_args().map(|a| a.to_string_lossy().into_owned()).collect();
assert_eq!(args, [AGENT_CMD, "/x/sock", "5m0s", "idle"]);
}
#[test]
fn abs_is_lexical() {
assert_eq!(go_abs(Path::new("/a/b/../c/./d")).unwrap(), PathBuf::from("/a/c/d"));
}
}
+973
View File
@@ -0,0 +1,973 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! The command line, matching the Go version's commands, flags (Go `flag`
//! package syntax, flags before, between or after arguments), prompts,
//! messages and exit statuses.
use crate::agent;
use crate::error::{Error, Result};
use crate::keyfile;
use crate::pipeline::{MAX_WORKERS, Pool};
use crate::platform::{fs as pfs, lock, term};
use crate::secret::SecretVec;
use crate::store::{Catalogue, FILE_LIMIT, SourceStamp, Store, VaultName};
use crate::util::{self, human_size};
use std::ffi::OsString;
use std::io::Write;
use std::path::{Path, PathBuf};
use std::sync::Arc;
use std::time::Duration;
pub const USAGE: &str = "Usage:
vault genkey create a new passphrase-protected key
vault init [--filler SIZE] create a new, empty vault, with at least
SIZE of filler (e.g. 500M, 2G)
vault add FILE|DIR|URL [FOLDER] encrypt a file, a directory (recursively)
or a downloaded URL into the vault,
optionally under a folder inside the vault
vault migrate FILE|DIR [FOLDER] like add, then delete each original once
[--yes] it is stored; asks before overwriting
anything already in the vault, unless
--yes is given
vault mirror URL [FOLDER] crawl a website (same host only, one request
at a time) into FOLDER/HOST, rewriting its
links to relative paths
vault absorb [FOLDER] read a tar archive (optionally gzipped) from
standard input and add every file in it,
optionally under a folder inside the vault
vault ls [FOLDER] list one folder: its files, and its
sub-folders (ending in /) with their
total size; -r lists everything under it
vault list [FOLDER] list every stored file under FOLDER,
with full names
vault get NAME [-o OUTPUT] decrypt a stored file (default: its base name)
vault get -r FOLDER [-o DIR] decrypt a folder and everything under it
into DIR (default: the folder's base name)
vault mv OLD NEW rename or move a file or folder; NEW
ending in / or naming an existing folder
moves OLD into it
vault cp [-r] OLD NEW copy a file, or with -r a folder, like mv;
copies share their encrypted data
vault rm NAME delete a stored file
vault rm -r FOLDER [-f] delete a folder and everything under it;
asks first unless -f (or -rf) is given
vault unlock [--for DURATION] remember the passphrase for DURATION
[--idle] (default 5m), so commands skip asking;
with --idle, DURATION since last used
vault lock forget it now
Options (any command):
--key PATH key file (default $VAULT_KEY, else ~/.vault/key)
--vault DIR vault directory (default $VAULT_DIR, else ~/.vault/store)
--workers N encryption and I/O threads, 1 to 256 (default: CPUs, at
most 8)
Options for add URL and mirror:
--max-bytes SIZE largest download (add) or total (mirror); defaults
16G and 4G, at most 1T
--timeout DURATION deadline for each request; defaults 2h and 5m
--max-pages N pages and assets mirror fetches (default 10000)
The passphrase is read from the terminal, or from $VAULT_PASSPHRASE. After
vault unlock, commands use the remembered key instead, unless
$VAULT_PASSPHRASE is set.
Decoy passphrase: genkey offers a second, optional passphrase
($VAULT_DECOY_PASSPHRASE when scripted). It unlocks a separate key, so
every command run with it sees and changes only the decoy files, and the
real files stay invisible. The key file does not reveal whether a decoy
exists, and init fills each vault with unreadable filler, so hidden
files look like filler to anyone holding only the decoy passphrase.";
const PASSPHRASE_ENV: &str = "VAULT_PASSPHRASE";
const DECOY_ENV: &str = "VAULT_DECOY_PASSPHRASE";
/// Exit status when `mirror` stored some resources but skipped others.
pub const EXIT_PARTIAL: i32 = 3;
/// Prints to standard error, ignoring failure (as Go's Fprintf does).
macro_rules! say {
($($t:tt)*) => {{ let _ = writeln!(std::io::stderr(), $($t)*); }};
}
/// Prints to standard output; a closed pipe ends the program quietly, as
/// Go's SIGPIPE on standard output does.
fn out(line: &str) {
let mut o = std::io::stdout().lock();
if let Err(e) = o.write_all(line.as_bytes()).and_then(|_| o.flush())
&& e.kind() == std::io::ErrorKind::BrokenPipe
{
std::process::exit(128 + 13);
}
}
#[derive(Default)]
struct Flags {
key: Option<String>,
vault: Option<String>,
output: Option<String>,
filler: Option<String>,
yes: bool,
ttl: Option<i64>,
idle: bool,
r: bool,
f: bool,
rf: bool,
fr: bool,
workers: Option<String>,
max_bytes: Option<String>,
timeout: Option<i64>,
max_pages: Option<String>,
}
enum FlagKind {
Str,
Bool,
Dur,
}
fn flag_kind(name: &str) -> Option<FlagKind> {
Some(match name {
"key" | "vault" | "o" | "filler" | "workers" | "max-bytes" | "max-pages" => FlagKind::Str,
"yes" | "idle" | "r" | "f" | "rf" | "fr" => FlagKind::Bool,
"for" | "timeout" => FlagKind::Dur,
_ => return None,
})
}
fn go_parse_bool(v: &str) -> Option<bool> {
match v {
"1" | "t" | "T" | "TRUE" | "true" | "True" => Some(true),
"0" | "f" | "F" | "FALSE" | "false" | "False" => Some(false),
_ => None,
}
}
impl Flags {
fn set(&mut self, name: &str, value: &str) -> std::result::Result<(), String> {
match flag_kind(name).expect("known flag") {
FlagKind::Bool => {
let b = go_parse_bool(value).ok_or_else(|| format!("invalid boolean value {value:?} for -{name}: parse error"))?;
*match name {
"yes" => &mut self.yes,
"idle" => &mut self.idle,
"r" => &mut self.r,
"f" => &mut self.f,
"rf" => &mut self.rf,
_ => &mut self.fr,
} = b;
}
FlagKind::Dur => {
let d = util::parse_duration(value).map_err(|_| format!("invalid value {value:?} for flag -{name}: parse error"))?;
*if name == "for" { &mut self.ttl } else { &mut self.timeout } = Some(d);
}
FlagKind::Str => {
let v = Some(value.to_string());
match name {
"key" => self.key = v,
"vault" => self.vault = v,
"o" => self.output = v,
"filler" => self.filler = v,
"workers" => self.workers = v,
"max-bytes" => self.max_bytes = v,
_ => self.max_pages = v,
}
}
}
Ok(())
}
}
/// Go's `flag.FlagSet.Parse`: consumes flags from the front of `args`,
/// returning the remaining arguments.
fn parse_flags(fl: &mut Flags, mut args: &[String]) -> std::result::Result<Vec<String>, String> {
loop {
let Some(s) = args.first() else { return Ok(Vec::new()) };
let b = s.as_bytes();
if b.len() < 2 || b[0] != b'-' {
return Ok(args.to_vec());
}
let mut minuses = 1;
if b[1] == b'-' {
minuses = 2;
if b.len() == 2 {
return Ok(args[1..].to_vec());
}
}
let mut name = &s[minuses..];
if name.is_empty() || name.starts_with('-') || name.starts_with('=') {
return Err(format!("bad flag syntax: {s}"));
}
args = &args[1..];
let mut value = None;
if let Some(i) = name[1..].find('=') {
value = Some(&name[i + 2..]);
name = &name[..i + 1];
}
let Some(kind) = flag_kind(name) else {
if name == "help" || name == "h" {
return Err("flag: help requested".into());
}
return Err(format!("flag provided but not defined: -{name}"));
};
match (kind, value) {
(FlagKind::Bool, None) => fl.set(name, "true")?,
(_, Some(v)) => fl.set(name, v)?,
(_, None) => {
let Some(v) = args.first() else {
return Err(format!("flag needs an argument: -{name}"));
};
fl.set(name, v)?;
args = &args[1..];
}
}
}
}
/// Flags may come before, between or after positional arguments.
fn parse_interleaved(fl: &mut Flags, args: &[String]) -> std::result::Result<Vec<String>, String> {
let mut pos = Vec::new();
let mut rest = args.to_vec();
loop {
let left = parse_flags(fl, &rest)?;
let Some(first) = left.first() else { return Ok(pos) };
pos.push(first.clone());
rest = left[1..].to_vec();
}
}
fn env_bytes(name: &str) -> Option<Vec<u8>> {
let v = std::env::var_os(name)?;
#[cfg(unix)]
{
Some(std::os::unix::ffi::OsStringExt::into_vec(v))
}
#[cfg(not(unix))]
{
Some(v.to_string_lossy().into_owned().into_bytes())
}
}
fn read_passphrase(prompt: &str) -> Result<SecretVec> {
match env_bytes(PASSPHRASE_ENV) {
Some(v) if v.is_empty() => Err(Error::new(format!("{PASSPHRASE_ENV} is set but empty"))),
Some(v) => Ok(SecretVec::from_vec(v)),
None => term::read_secret(prompt).map_err(|e| {
Error::new(format!(
"could not read passphrase (set {PASSPHRASE_ENV} for non-interactive use): {}",
util::io_reason(&e)
))
}),
}
}
fn new_passphrase(what: &str) -> Result<SecretVec> {
if let Some(v) = env_bytes(PASSPHRASE_ENV).filter(|v| !v.is_empty()) {
return Ok(SecretVec::from_vec(v));
}
let p = read_passphrase(&format!("Enter {what}:"))?;
if p.is_empty() {
crate::bail!("{what} must not be empty");
}
let again = read_passphrase(&format!("Confirm {what}:"))?;
if !(p == *again.expose()) {
crate::bail!("{what}s do not match");
}
Ok(p)
}
/// An optional second passphrase; empty means none.
fn decoy_passphrase() -> Result<SecretVec> {
if let Some(v) = env_bytes(DECOY_ENV) {
return Ok(SecretVec::from_vec(v));
}
if std::env::var_os(PASSPHRASE_ENV).is_some() {
return Ok(SecretVec::from_vec(Vec::new())); // scripted, no decoy
}
let read = |p: &str| term::read_secret(p).map_err(|e| Error::new(util::io_reason(&e)));
let p = read("Enter decoy passphrase (empty for none):")?;
if p.is_empty() {
return Ok(p);
}
let again = read("Confirm decoy passphrase:")?;
if !(p == *again.expose()) {
crate::bail!("decoy passphrases do not match");
}
Ok(p)
}
fn home() -> Option<PathBuf> {
let v = if cfg!(windows) {
std::env::var_os("USERPROFILE")
} else {
std::env::var_os("HOME")
};
v.filter(|h| !h.is_empty()).map(PathBuf::from)
}
fn default_path(flag: &Option<String>, env: &str, name: &str) -> Result<PathBuf> {
if let Some(f) = flag {
return Ok(PathBuf::from(f));
}
if let Some(v) = std::env::var_os(env).filter(|v| !v.is_empty()) {
return Ok(PathBuf::from(v));
}
home()
.map(|h| h.join(".vault").join(name))
.ok_or_else(|| Error::new(format!("cannot find home directory; set {env}")))
}
/// The key: from a running agent unless a passphrase is in the
/// environment, else from the key file with the passphrase.
fn key_identity(key: &Path) -> Result<crate::crypto::kem::Identity> {
if std::env::var_os(PASSPHRASE_ENV).is_none()
&& let Ok(id) = agent::identity(key)
{
return Ok(id);
}
let pass = read_passphrase("Enter passphrase:")?;
keyfile::unlock(key, pass.expose()).map_err(|e| Error::new(e.to_string()))
}
struct Ctx {
flags: Flags,
pool: Option<Arc<Pool>>,
}
impl Ctx {
fn key(&self) -> Result<PathBuf> {
default_path(&self.flags.key, "VAULT_KEY", "key")
}
fn vault_dir(&self) -> Result<PathBuf> {
default_path(&self.flags.vault, "VAULT_DIR", "store")
}
fn pool(&mut self) -> Result<Arc<Pool>> {
if self.pool.is_none() {
let n = match &self.flags.workers {
None => Pool::default_workers(),
Some(w) => match w.parse::<usize>() {
Ok(n) if (1..=MAX_WORKERS).contains(&n) => n,
_ => crate::bail!("--workers needs a number from 1 to {MAX_WORKERS}"),
},
};
self.pool = Some(Arc::new(Pool::new(n)));
}
Ok(Arc::clone(self.pool.as_ref().expect("set")))
}
/// Unlocks the key, opens the vault, then takes the vault lock:
/// exclusive for writers, shared for readers.
fn open(&mut self, exclusive: bool) -> Result<(Store, lock::Lock)> {
let key = self.key()?;
let dir = self.vault_dir()?;
let id = key_identity(&key)?;
let s = Store::open(&dir, id, self.pool()?)?;
let l = lock::lock_vault(&dir, exclusive)?;
Ok((s, l))
}
fn net_limits(&self, mirror: bool) -> Result<crate::net::Limits> {
let mut l = if mirror {
crate::net::Limits::mirror_defaults()
} else {
crate::net::Limits::add_defaults()
};
if let Some(b) = &self.flags.max_bytes {
l.max_bytes = util::parse_size(b, FILE_LIMIT)?;
}
if let Some(t) = self.flags.timeout {
if t <= 0 {
crate::bail!("--timeout needs a positive duration, such as 30s or 1h");
}
l.request = Duration::from_nanos(t as u64);
}
if let Some(p) = &self.flags.max_pages {
l.max_pages = p
.parse()
.ok()
.filter(|&n| n > 0)
.ok_or_else(|| Error::new("--max-pages needs a positive number"))?;
}
Ok(l)
}
}
fn report(verb: &str) -> impl FnMut(&VaultName, bool) + '_ {
move |n, replaced| say!("{} {n}", if replaced { "replaced" } else { verb })
}
/// Runs the command line; returns the exit status.
pub fn main() -> i32 {
let args: Vec<OsString> = std::env::args_os().collect();
if args.len() < 2 {
say!("{USAGE}");
return 2;
}
if args[1] == agent::AGENT_CMD && args.len() == 5 {
agent::run(Path::new(&args[2]), &args[3].to_string_lossy(), &args[4].to_string_lossy());
return 0;
}
let mut strs = Vec::with_capacity(args.len());
for a in &args[1..] {
match a.to_str() {
Some(s) => strs.push(s.to_string()),
None => {
say!("vault: arguments must be valid UTF-8: {}", a.to_string_lossy());
return 1;
}
}
}
match run(&strs[0], &strs[1..]) {
Ok(code) => code,
Err(e) => {
say!("vault: {e}");
1
}
}
}
fn run(cmd: &str, args: &[String]) -> Result<i32> {
let mut flags = Flags::default();
let pos = parse_interleaved(&mut flags, args).map_err(|e| Error::new(format!("{e}\n\n{USAGE}")))?;
let want = |min: usize, max: usize| -> Result<()> {
if pos.len() < min || pos.len() > max {
crate::bail!("wrong number of arguments for {cmd}\n\n{USAGE}");
}
Ok(())
};
let mut ctx = Ctx { flags, pool: None };
match cmd {
"genkey" => {
want(0, 0)?;
let key = ctx.key()?;
if std::fs::metadata(&key).is_ok() {
crate::bail!("{} already exists", key.display());
}
let pass = new_passphrase("passphrase")?;
let decoy = decoy_passphrase()?;
if pass == *decoy.expose() {
crate::bail!("decoy passphrase must differ from the passphrase");
}
keyfile::write(&key, pass.expose(), decoy.expose()).map_err(|e| Error::new(e.to_string()))?;
// One left from an old key at this path would serve it.
agent::stop(&key);
say!("key written to {}", key.display());
}
"init" => {
want(0, 0)?;
let size = util::parse_size(ctx.flags.filler.as_deref().unwrap_or("0"), FILE_LIMIT)?;
let dir = ctx.vault_dir()?;
Store::init(&dir, size, ctx.pool()?)?;
say!("vault created at {}", dir.display());
}
"add" | "migrate" => {
want(1, 2)?;
let dest = pos.get(1).map(|d| d.trim_matches('/').to_string()).unwrap_or_default();
let migrate = cmd == "migrate";
if crate::net::is_url(&pos[0]) {
if migrate {
crate::bail!("cannot migrate a URL; use add");
}
let limits = ctx.net_limits(false)?;
let (s, _lock) = ctx.open(true)?;
let mut cat = s.catalogue()?;
crate::net::add_url(&s, &mut cat, &pos[0], &dest, &limits, &mut report("added"))?;
return Ok(0);
}
let (s, _lock) = ctx.open(true)?;
let mut cat = s.catalogue()?;
// Never ingest, or delete, the vault, anything inside it, or its key.
let vault_dir = ctx.vault_dir()?;
if let (Ok(v), Ok(src)) = (std::fs::canonicalize(&vault_dir), std::fs::canonicalize(&pos[0]))
&& src != v
&& src.starts_with(&v)
{
crate::bail!("{} is inside the vault", pos[0]);
}
let skip: Vec<FileId> = [vault_dir, ctx.key()?].iter().filter_map(|p| FileId::of(p, true)).collect();
let yes = ctx.flags.yes;
add_path(&s, &mut cat, Path::new(&pos[0]), &dest, migrate, &skip, &mut |names| {
yes || confirm_overwrite(names)
})?;
}
"mirror" => {
want(1, 2)?;
if !crate::net::is_url(&pos[0]) {
crate::bail!("mirror needs an http:// or https:// URL");
}
let dest = pos.get(1).map(|d| d.trim_matches('/').to_string()).unwrap_or_default();
let limits = ctx.net_limits(true)?;
let (s, _lock) = ctx.open(true)?;
let mut cat = s.catalogue()?;
match crate::net::mirror(&s, &mut cat, &pos[0], &dest, &limits, &mut report("added"))? {
crate::net::MirrorOutcome::Complete { .. } => {}
crate::net::MirrorOutcome::Partial { stored, skipped } => {
say!("vault: mirror incomplete: stored {stored}, skipped {skipped}");
return Ok(EXIT_PARTIAL);
}
}
}
"absorb" => {
want(0, 1)?;
if stdin_is_terminal() {
crate::bail!("absorb reads a tar archive from standard input; pipe one in");
}
let dest = pos.first().map(|d| d.trim_matches('/').to_string()).unwrap_or_default();
// Wait for the archive to start arriving before asking for the
// passphrase, so a program feeding the pipe (gpg, ssh) has
// finished with the terminal and cannot bury the prompt.
let mut input = std::io::BufReader::with_capacity(1 << 16, std::io::stdin().lock());
if std::io::BufRead::fill_buf(&mut input).map_or(true, |b| b.is_empty()) {
crate::bail!("nothing to absorb on standard input");
}
let (s, _lock) = ctx.open(true)?;
let mut cat = s.catalogue()?;
crate::absorb::absorb(&s, &mut cat, input, &dest, &mut report("added"))?;
}
"list" | "ls" => {
want(0, 1)?;
let prefix = pos.first().map(|d| d.trim_matches('/').to_string()).unwrap_or_default();
let (s, _lock) = ctx.open(false)?;
let cat = s.catalogue()?;
if cmd == "ls" && !ctx.flags.r {
for line in ls(&cat, &prefix)? {
out(&line);
}
} else {
let names: Vec<String> = if prefix.is_empty() {
cat.names().map(String::from).collect()
} else {
cat.under(&prefix)
};
for n in names {
let r = cat.current(&n).expect("listed");
out(&format!("{:>10} {} {n}\n", human_size(r.size), r.modified.local_minute()));
}
}
}
"get" => {
want(1, 1)?;
let out_path = ctx.flags.output.clone().unwrap_or_else(|| util::base(pos[0].trim_end_matches('/')));
let (s, _lock) = ctx.open(false)?;
let cat = s.catalogue()?;
if !ctx.flags.r {
let r = cat
.current(&pos[0])
.ok_or_else(|| Error::new(format!("{}: not in the vault", pos[0])))?;
let p = PathBuf::from(&out_path);
let Some(fname) = p.file_name() else {
crate::bail!("{out_path} already exists");
};
let parent = p.parent().unwrap_or(Path::new(""));
let dir = pfs::Dir::open(parent).map_err(|e| Error::io("open", parent, &e))?;
s.extract(r, &dir, fname)?;
return Ok(0);
}
let folder = pos[0].trim_matches('/');
let names = cat.under(folder);
if folder.is_empty() || names.is_empty() || cat.contains(folder) {
crate::bail!("{}: not a folder in the vault", pos[0]);
}
let root = PathBuf::from(&out_path);
pfs::create_private_dir_all(&root).map_err(|e| Error::io("mkdir", &root, &e))?;
let top = pfs::Dir::open(&root).map_err(|e| Error::io("open", &root, &e))?;
for n in names {
let rel = &n[folder.len() + 1..];
let parts: Vec<&str> = rel.split('/').collect();
let mut dir = None::<pfs::Dir>;
for p in &parts[..parts.len() - 1] {
let next = dir
.as_ref()
.unwrap_or(&top)
.subdir(p.as_ref())
.map_err(|e| Error::new(util::io_reason(&e)))?;
dir = Some(next);
}
let d = dir.as_ref().unwrap_or(&top);
s.extract(cat.current(&n).expect("listed"), d, parts[parts.len() - 1].as_ref())?;
say!("extracted {}", root.join(rel.replace('/', std::path::MAIN_SEPARATOR_STR)).display());
}
}
"mv" | "rename" | "cp" => {
want(2, 2)?;
let (s, _lock) = ctx.open(true)?;
let mut cat = s.catalogue()?;
let pairs = transfers(&cat, &pos[0], &pos[1], cmd != "cp" || ctx.flags.r)?;
let verb = if cmd == "cp" { "copied" } else { "moved" };
s.copy_all(&mut cat, &pairs, cmd != "cp", &mut |from, to| say!("{verb} {from} -> {to}"))?;
}
"rm" | "delete" => {
want(1, 1)?;
let f = &ctx.flags;
let (recursive, forced) = (f.r || f.rf || f.fr, f.f || f.rf || f.fr);
let mut name = pos[0].clone();
if recursive {
name = name.trim_matches('/').to_string();
if name.is_empty() {
crate::bail!("refusing to remove the whole vault");
}
}
let (s, _lock) = ctx.open(true)?;
let mut cat = s.catalogue()?;
let names = if recursive {
cat.under(&name)
} else if cat.contains(&name) {
vec![name.clone()]
} else {
Vec::new()
};
if names.is_empty() {
if forced {
return Ok(0);
}
crate::bail!("{name}: not in the vault");
}
let size: u64 = names.iter().map(|n| cat.current(n).expect("listed").size).sum();
if recursive
&& !forced
&& !confirm(
"these files will be deleted, with no undo:",
&names,
&format!("delete {} file(s), {}? [y/N]", names.len(), human_size(size)),
)
{
crate::bail!("rm cancelled; nothing changed (-f skips the question)");
}
s.remove_all(&mut cat, &names, &mut |n| say!("removed {n}"))?;
if recursive {
say!("removed {} file(s), {}", names.len(), human_size(size));
}
}
"unlock" => {
want(0, 0)?;
let ttl = ctx.flags.ttl.unwrap_or(5 * 60 * 1_000_000_000);
if ttl <= 0 {
crate::bail!("--for needs a positive duration, such as 5m or 1h");
}
let ttl = Duration::from_nanos(ttl as u64);
let key = ctx.key()?;
let pass = read_passphrase("Enter passphrase:")?;
let id = keyfile::unlock(&key, pass.expose()).map_err(|e| Error::new(e.to_string()))?;
drop(pass);
agent::start(&key, &id, ttl, ctx.flags.idle)?;
id.close();
if ctx.flags.idle {
say!(
"unlocked until unused for {}; vault lock forgets the key sooner",
util::format_duration(ttl)
);
} else {
let until = util::Instant::from_system(std::time::SystemTime::now() + ttl);
say!("unlocked until {}; vault lock forgets the key sooner", until.local_clock());
}
}
"lock" => {
want(0, 0)?;
if agent::stop(&ctx.key()?) {
say!("locked");
} else {
say!("nothing was unlocked");
}
}
"help" | "-h" | "--help" => out(&format!("{USAGE}\n")),
_ => crate::bail!("unknown command {cmd:?}\n\n{USAGE}"),
}
Ok(0)
}
fn stdin_is_terminal() -> bool {
use std::io::IsTerminal;
std::io::stdin().is_terminal()
}
/// Works out, like mv(1) and cp(1), what `src` and `dst` mean: each file
/// to copy or move, with its new name. `src` is a file, or with
/// `recursive` a folder; `dst` ending in / or naming an existing folder
/// means into it. No new name may already exist.
pub fn transfers(cat: &Catalogue, src: &str, dst: &str, recursive: bool) -> Result<Vec<(String, VaultName)>> {
let mut into = dst.is_empty() || dst.ends_with('/');
let (src, dst) = (src.trim_matches('/'), dst.trim_matches('/'));
if src.is_empty() {
crate::bail!("name a file or folder to copy or move");
}
if dst.is_empty() || (!cat.contains(dst) && !cat.under(dst).is_empty()) {
into = true;
}
let dst = if into {
util::join(&[dst, &util::base(src)])
} else {
dst.to_string()
};
let raw: Vec<(String, String)> = if cat.contains(src) {
vec![(src.to_string(), dst)]
} else {
let names = cat.under(src);
if names.is_empty() {
crate::bail!("{src}: not in the vault");
}
if !recursive {
crate::bail!("{src} is a folder; use -r");
}
if dst == src || dst.starts_with(&format!("{src}/")) {
crate::bail!("cannot put {src} inside itself");
}
names
.into_iter()
.map(|n| (n.clone(), format!("{dst}{}", &n[src.len()..])))
.collect()
};
let mut pairs = Vec::with_capacity(raw.len());
for (from, to) in raw {
let Some(name) = VaultName::new(&to) else {
crate::bail!("invalid vault path {to:?}")
};
if cat.contains(&to) {
crate::bail!("{to} already exists");
}
pairs.push((from, name));
}
Ok(pairs)
}
/// One level of the vault, like ls(1): the files directly in `folder`
/// and each sub-folder once, with a trailing slash, the total size and
/// the newest date beneath it. Folders first. A file's own name lists it.
pub fn ls(cat: &Catalogue, folder: &str) -> Result<Vec<String>> {
let mut entries: std::collections::BTreeMap<String, (u64, Option<util::Instant>)> = Default::default();
for (name, r) in cat.iter() {
let rel = if folder.is_empty() {
name.to_string()
} else if name == folder {
util::base(name)
} else if let Some(rest) = name.strip_prefix(folder).and_then(|r| r.strip_prefix('/')) {
rest.to_string()
} else {
continue;
};
let rel = match rel.find('/') {
Some(i) => rel[..i + 1].to_string(),
None => rel,
};
let e = entries.entry(rel).or_default();
e.0 += r.size;
if e.1.is_none_or(|m| r.modified > m) {
e.1 = Some(r.modified);
}
}
if entries.is_empty() && !folder.is_empty() {
crate::bail!("{folder}: not in the vault");
}
let mut names: Vec<_> = entries.into_iter().collect();
names.sort_by(|a, b| b.0.ends_with('/').cmp(&a.0.ends_with('/')).then(a.0.cmp(&b.0)));
Ok(names
.into_iter()
.map(|(n, (size, m))| {
format!(
"{:>10} {} {n}\n",
human_size(size),
m.unwrap_or(util::Instant { secs: 0, nanos: 0 }).local_minute()
)
})
.collect())
}
fn confirm_overwrite(names: &[String]) -> bool {
confirm(
"these files are already in the vault and would be overwritten:",
names,
&format!("overwrite {} file(s)? [y/N]", names.len()),
)
}
/// Shows `intro` and the first 20 names, then asks on the terminal. No
/// terminal means no.
fn confirm(intro: &str, names: &[String], question: &str) -> bool {
say!("{intro}");
for (i, n) in names.iter().enumerate() {
if i == 20 {
say!(" … and {} more", names.len() - i);
break;
}
say!(" {n}");
}
match term::confirm(question) {
Ok(b) => b,
Err(e) => {
say!("cannot ask for confirmation: {}", util::io_reason(&e));
false
}
}
}
/// Identifies a file for Go's `os.SameFile`.
#[derive(PartialEq, Eq)]
struct FileId(u64, u64, PathBuf);
impl FileId {
fn of(p: &Path, follow: bool) -> Option<FileId> {
let m = if follow {
std::fs::metadata(p)
} else {
std::fs::symlink_metadata(p)
}
.ok()?;
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
let _ = &m;
Some(FileId(m.dev(), m.ino(), PathBuf::new()))
}
#[cfg(not(unix))]
{
let _ = m;
Some(FileId(0, 0, std::fs::canonicalize(p).ok()?))
}
}
}
fn utf8_rel(p: &Path) -> Result<String> {
let mut parts = Vec::new();
for c in p.components() {
let s = c
.as_os_str()
.to_str()
.ok_or_else(|| Error::new(format!("invalid vault path {:?}", p.to_string_lossy())))?;
parts.push(s.to_string());
}
Ok(parts.join("/"))
}
/// Adds a file, or every regular file under a directory, beneath `dest`,
/// keeping the structure. A directory named "." or ".." adds its contents
/// directly. With `migrate`, nothing is stored unless `overwrite` approves
/// the names it would replace, each original is deleted once stored and
/// checked unchanged, then emptied directories go.
fn add_path(
s: &Store,
cat: &mut Catalogue,
src: &Path,
dest: &str,
migrate: bool,
skip: &[FileId],
overwrite: &mut dyn FnMut(&[String]) -> bool,
) -> Result<()> {
let info = std::fs::metadata(src).map_err(|e| Error::io("stat", src, &e))?;
let skipped = |p: &Path, follow: bool| FileId::of(p, follow).is_some_and(|id| skip.contains(&id));
if skipped(src, true) {
crate::bail!("{} is the vault or its key", src.display());
}
let mut items: Vec<(PathBuf, String)> = Vec::new();
let mut dirs = Vec::new();
if !info.is_dir() {
let base = src.file_name().map(|b| utf8_rel(Path::new(b))).transpose()?.unwrap_or_default();
items.push((src.to_path_buf(), util::join(&[dest, &base])));
} else {
let base = match src.components().next_back() {
Some(std::path::Component::Normal(b)) => utf8_rel(Path::new(b))?,
_ => String::new(),
};
walk(src, src, &mut |p, rel, ft| {
if skipped(p, false) {
say!("skipping {}: the vault or its key", p.display());
return Ok(false);
}
if ft.is_dir() {
dirs.push(p.to_path_buf());
return Ok(true);
}
if !ft.is_file() {
say!("skipping {}: not a regular file", p.display());
return Ok(false);
}
items.push((p.to_path_buf(), util::join(&[dest, &base, &utf8_rel(rel)?])));
Ok(false)
})?;
}
if migrate {
let existing: Vec<String> = items.iter().filter(|(_, n)| cat.contains(n)).map(|(_, n)| n.clone()).collect();
if !existing.is_empty() && !overwrite(&existing) {
crate::bail!("migrate cancelled; nothing changed");
}
}
for (p, n) in &items {
let name = VaultName::new(n).ok_or_else(|| Error::new(format!("invalid vault path {n:?}")))?;
let (added, stamp) = s.add_file(cat, p, &name)?;
let mut verb = if added.replaced { "replaced" } else { "added" };
if migrate {
remove_source(p, stamp)?;
verb = "migrated";
}
say!("{verb} {n}");
}
if migrate {
// Children come after their parents in walk order, so go backwards.
// Directories still holding skipped files stay.
for d in dirs.iter().rev() {
let _ = std::fs::remove_dir(d);
}
}
Ok(())
}
/// Deletes a migrated original, but only if it is still the very file
/// that was stored (same inode, size and modification time).
fn remove_source(p: &Path, stamp: SourceStamp) -> Result<()> {
match std::fs::metadata(p) {
Ok(m) if SourceStamp::of(&m) == stamp => std::fs::remove_file(p).map_err(|e| Error::io("remove", p, &e)),
_ => crate::bail!("{} changed after it was stored; not deleting it", p.display()),
}
}
/// Go's `filepath.WalkDir`: lexical order, the root first, symlinks not
/// followed. `visit` returns whether to descend into a directory.
fn walk(root: &Path, p: &Path, visit: &mut dyn FnMut(&Path, &Path, &std::fs::FileType) -> Result<bool>) -> Result<()> {
let ft = std::fs::symlink_metadata(p).map_err(|e| Error::io("lstat", p, &e))?.file_type();
let rel = p.strip_prefix(root).unwrap_or(Path::new(""));
if !visit(p, rel, &ft)? || !ft.is_dir() {
return Ok(());
}
let mut entries: Vec<_> = std::fs::read_dir(p)
.map_err(|e| Error::io("open", p, &e))?
.map(|e| e.map(|e| e.file_name()))
.collect::<std::io::Result<_>>()
.map_err(|e| Error::io("readdir", p, &e))?;
entries.sort();
for e in entries {
walk(root, &p.join(e), visit)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
fn s(v: &[&str]) -> Vec<String> {
v.iter().map(|x| x.to_string()).collect()
}
#[test]
fn go_flag_syntax() {
let mut f = Flags::default();
let pos = parse_interleaved(&mut f, &s(&["a", "-o", "out", "--yes", "b", "-for=1h", "-r=false"])).unwrap();
assert_eq!(pos, ["a", "b"]);
assert_eq!(f.output.as_deref(), Some("out"));
assert!(f.yes && !f.r);
assert_eq!(f.ttl, Some(3_600_000_000_000));
let mut f = Flags::default();
assert_eq!(parse_interleaved(&mut f, &s(&["--", "-x", "y"])).unwrap(), ["-x", "y"]);
assert_eq!(parse_interleaved(&mut f, &s(&["-", "y"])).unwrap(), ["-", "y"]);
for (args, err) in [
(vec!["-x"], "flag provided but not defined: -x"),
(vec!["-o"], "flag needs an argument: -o"),
(vec!["---o"], "bad flag syntax: ---o"),
(vec!["-=x"], "bad flag syntax: -=x"),
(vec!["-for", "soon"], "invalid value \"soon\" for flag -for: parse error"),
(vec!["-yes=maybe"], "invalid boolean value \"maybe\" for -yes: parse error"),
(vec!["-h"], "flag: help requested"),
] {
assert_eq!(parse_interleaved(&mut Flags::default(), &s(&args)).err().as_deref(), Some(err));
}
}
}
+497
View File
@@ -0,0 +1,497 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! FrodoKEM-640-SHAKE exactly as CIRCL v1.6.4 implements it
//! (`kem/frodo/frodo640shake`): the round-3 FrodoKEM with a 48-byte key
//! seed, 16-byte encapsulation seed, 16-byte shared secret and 9720-byte
//! ciphertext. The RustCrypto `frodo-kem` crate implements the later ISO
//! variant (salted, 9752-byte ciphertexts, a different seed layout), which
//! cannot open existing vaults, so this module is a line-by-line port of
//! CIRCL's code, checked against vectors from the Go oracle and against the
//! NIST KAT digest CIRCL itself tests.
//!
//! All arithmetic is mod 2^16 with masking to 15 bits where CIRCL masks;
//! loops never branch on secret data, and the decapsulation re-encryption
//! check selects the rejection secret with a constant-time mask.
use sha3::Shake128;
use sha3::digest::{ExtendableOutput, Update, XofReader};
use zeroize::Zeroize;
const N: usize = 640;
const NBAR: usize = 8;
const LOG_Q: u32 = 15;
const Q_MASK: u16 = (1 << LOG_Q) - 1;
const SEED_A: usize = 16;
const PK_HASH: usize = 16;
const EXTRACTED_BITS: u32 = 2;
const MESSAGE: usize = 16;
const BP_PACKED: usize = (LOG_Q as usize * N * NBAR) / 8;
pub const SHARED_KEY_SIZE: usize = 16;
pub const KEY_SEED_SIZE: usize = SHARED_KEY_SIZE + SHARED_KEY_SIZE + 16;
pub const ENCAPSULATION_SEED_SIZE: usize = 16;
pub const CIPHERTEXT_SIZE: usize = 9720;
pub const PUBLIC_KEY_SIZE: usize = 9616;
pub const PRIVATE_KEY_SIZE: usize = 19888;
const CDF: [u16; 13] = [
4643, 13363, 20579, 25843, 29227, 31145, 32103, 32525, 32689, 32745, 32762, 32766, 32767,
];
/// A packed public key (`seedA || pack(B)`), as CIRCL's `MarshalBinary`.
pub struct PublicKey {
seed_a: [u8; SEED_A],
b: Vec<u16>, // N x NBAR
}
/// A private key. Its secret parts are cleared on drop.
pub struct PrivateKey {
s_reject: [u8; SHARED_KEY_SIZE],
pk: PublicKey,
s: Vec<u16>, // transpose(S), NBAR x N
hpk: [u8; PK_HASH],
}
impl Drop for PrivateKey {
fn drop(&mut self) {
self.s_reject.zeroize();
self.s.zeroize();
}
}
fn shake(parts: &[&[u8]], out: &mut [u8]) {
let mut h = Shake128::default();
for p in parts {
h.update(p);
}
h.finalize_xof().read(out);
}
fn le_u16s(bytes: &[u8], out: &mut [u16]) {
for (o, c) in out.iter_mut().zip(bytes.as_chunks::<2>().0) {
*o = u16::from_le_bytes([c[0], c[1]]);
}
}
/// Inverse-transform sampling from the FrodoKEM error distribution.
fn sample(v: &mut [u16]) {
for x in v.iter_mut() {
let sign = *x & 1;
let unif = *x >> 1;
let mut g: u16 = 0;
for c in &CDF[..CDF.len() - 1] {
g = g.wrapping_add(c.wrapping_sub(unif) >> 15);
}
*x = (sign.wrapping_neg() ^ g).wrapping_add(sign);
}
}
/// Row i of A is SHAKE128(le16(i) || seedA), as little-endian u16s.
fn expand_a(seed_a: &[u8; SEED_A]) -> Vec<u16> {
let mut a = vec![0u16; N * N];
let mut input = [0u8; 2 + SEED_A];
input[2..].copy_from_slice(seed_a);
let mut row = [0u8; N * 2];
for i in 0..N {
input[0] = i as u8;
input[1] = (i >> 8) as u8;
shake(&[&input], &mut row);
le_u16s(&row, &mut a[i * N..(i + 1) * N]);
}
a
}
/// out (N x NBAR) += A*S + E, with S stored transposed (NBAR x N).
fn mul_add_as_plus_e(out: &mut [u16], a: &[u16], s: &[u16], e: &[u16]) {
for i in 0..N {
let row = &a[i * N..(i + 1) * N];
for k in 0..NBAR {
let sk = &s[k * N..(k + 1) * N];
let mut sum = e[i * NBAR + k];
for j in 0..N {
sum = sum.wrapping_add(row[j].wrapping_mul(sk[j]));
}
out[i * NBAR + k] = out[i * NBAR + k].wrapping_add(sum);
}
}
}
/// out (NBAR x N) += S*A + E. CIRCL walks A by column; summing row by row
/// gives the same result mod 2^16 with far better locality.
fn mul_add_sa_plus_e(out: &mut [u16], s: &[u16], a: &[u16], e: &[u16]) {
let mut acc = e[..NBAR * N].to_vec();
for j in 0..N {
let row = &a[j * N..(j + 1) * N];
for k in 0..NBAR {
let skj = s[k * N + j];
let dst = &mut acc[k * N..(k + 1) * N];
for i in 0..N {
dst[i] = dst[i].wrapping_add(row[i].wrapping_mul(skj));
}
}
}
for (o, v) in out.iter_mut().zip(&acc) {
*o = o.wrapping_add(*v);
}
acc.zeroize();
}
/// out (NBAR x NBAR) = S*B + E, masked.
fn mul_add_sb_plus_e(out: &mut [u16], s: &[u16], b: &[u16], e: &[u16]) {
for k in 0..NBAR {
for i in 0..NBAR {
let mut v = e[k * NBAR + i];
for j in 0..N {
v = v.wrapping_add(s[k * N + j].wrapping_mul(b[j * NBAR + i]));
}
out[k * NBAR + i] = v & Q_MASK;
}
}
}
/// out (NBAR x NBAR) = B'*S, with S transposed, masked.
fn mul_bs(out: &mut [u16], b: &[u16], s: &[u16]) {
for i in 0..NBAR {
for j in 0..NBAR {
let mut v: u16 = 0;
for k in 0..N {
v = v.wrapping_add(b[i * N + k].wrapping_mul(s[j * N + k]));
}
out[i * NBAR + j] = v & Q_MASK;
}
}
}
fn pack(out: &mut [u8], v: &[u16]) {
for (o, x) in out.as_chunks_mut::<15>().0.iter_mut().zip(v.as_chunks::<8>().0) {
let m: [u16; 8] = core::array::from_fn(|i| x[i] & Q_MASK);
o[0] = (m[0] >> 7) as u8;
o[1] = (((m[0] & 0x7f) as u8) << 1) | (m[1] >> 14) as u8;
o[2] = (m[1] >> 6) as u8;
o[3] = (((m[1] & 0x3f) as u8) << 2) | (m[2] >> 13) as u8;
o[4] = (m[2] >> 5) as u8;
o[5] = (((m[2] & 0x1f) as u8) << 3) | (m[3] >> 12) as u8;
o[6] = (m[3] >> 4) as u8;
o[7] = (((m[3] & 0x0f) as u8) << 4) | (m[4] >> 11) as u8;
o[8] = (m[4] >> 3) as u8;
o[9] = (((m[4] & 0x07) as u8) << 5) | (m[5] >> 10) as u8;
o[10] = (m[5] >> 2) as u8;
o[11] = (((m[5] & 0x03) as u8) << 6) | (m[6] >> 9) as u8;
o[12] = (m[6] >> 1) as u8;
o[13] = (((m[6] & 0x01) as u8) << 7) | (m[7] >> 8) as u8;
o[14] = m[7] as u8;
}
}
fn unpack(out: &mut [u16], b: &[u8]) {
for (o, i) in out.as_chunks_mut::<8>().0.iter_mut().zip(b.as_chunks::<15>().0) {
let w = |x: u8| x as u16;
o[0] = (w(i[0]) << 7) | (w(i[1] & 0xfe) >> 1);
o[1] = (w(i[1] & 0x01) << 14) | (w(i[2]) << 6) | (w(i[3] & 0xfc) >> 2);
o[2] = (w(i[3] & 0x03) << 13) | (w(i[4]) << 5) | (w(i[5] & 0xf8) >> 3);
o[3] = (w(i[5] & 0x07) << 12) | (w(i[6]) << 4) | (w(i[7] & 0xf0) >> 4);
o[4] = (w(i[7] & 0x0f) << 11) | (w(i[8]) << 3) | (w(i[9] & 0xe0) >> 5);
o[5] = (w(i[9] & 0x1f) << 10) | (w(i[10]) << 2) | (w(i[11] & 0xc0) >> 6);
o[6] = (w(i[11] & 0x3f) << 9) | (w(i[12]) << 1) | (w(i[13] & 0x80) >> 7);
o[7] = (w(i[13] & 0x7f) << 8) | w(i[14]);
}
}
fn encode_message(out: &mut [u16], msg: &[u8; MESSAGE]) {
let mask = (1u16 << EXTRACTED_BITS) - 1;
let mut pos = 0;
for c in msg.as_chunks::<2>().0 {
let mut v = u16::from_le_bytes([c[0], c[1]]);
for _ in 0..(16 / EXTRACTED_BITS) {
out[pos] = (v & mask) << (LOG_Q - EXTRACTED_BITS);
pos += 1;
v >>= EXTRACTED_BITS;
}
}
}
fn decode_message(out: &mut [u8; MESSAGE], m: &[u16]) {
let mask = (1u16 << EXTRACTED_BITS) - 1;
let mut pos = 0;
for o in out.iter_mut() {
for j in 0..(8 / EXTRACTED_BITS) {
let mut t = (m[pos] & Q_MASK) + (1 << (LOG_Q - EXTRACTED_BITS - 1));
t >>= LOG_Q - EXTRACTED_BITS;
t &= mask;
*o |= (t as u8) << (j * EXTRACTED_BITS);
pos += 1;
}
}
}
/// 0 if equal, 1 otherwise, without branching on the contents.
fn ct_differs(a: &[u16], b: &[u16]) -> u8 {
let mut v: u16 = 0;
for (x, y) in a.iter().zip(b) {
v |= x ^ y;
}
((v | v.wrapping_neg()) >> 15) as u8
}
impl PublicKey {
pub fn to_bytes(&self) -> Vec<u8> {
let mut out = vec![0u8; PUBLIC_KEY_SIZE];
out[..SEED_A].copy_from_slice(&self.seed_a);
pack(&mut out[SEED_A..], &self.b);
out
}
/// Any 9616 bytes form a key, as in CIRCL; only the length is checked.
pub fn from_bytes(buf: &[u8]) -> Option<Self> {
if buf.len() != PUBLIC_KEY_SIZE {
return None;
}
let mut seed_a = [0u8; SEED_A];
seed_a.copy_from_slice(&buf[..SEED_A]);
let mut b = vec![0u16; N * NBAR];
unpack(&mut b, &buf[SEED_A..]);
Some(PublicKey { seed_a, b })
}
fn hash(&self) -> [u8; PK_HASH] {
let mut h = [0u8; PK_HASH];
shake(&[&self.to_bytes()], &mut h);
h
}
/// Encapsulates with the 16-byte message `mu` (CIRCL's
/// `EncapsulateTo` seed), returning the ciphertext and shared secret.
/// Callers must pass fresh randomness; reusing `mu` repeats the secret.
pub fn encapsulate(&self, mu: &[u8; ENCAPSULATION_SEED_SIZE]) -> (Vec<u8>, [u8; SHARED_KEY_SIZE]) {
let hpk = self.hash();
let mut g2 = [0u8; 2 * SHARED_KEY_SIZE];
shake(&[&hpk, mu], &mut g2);
let (sp, ep, epp, mut noise) = noise(&g2[..SHARED_KEY_SIZE]);
let a = expand_a(&self.seed_a);
let mut bp = vec![0u16; NBAR * N];
mul_add_sa_plus_e(&mut bp, &noise[sp.clone()], &a, &noise[ep]);
let mut v = [0u16; NBAR * NBAR];
mul_add_sb_plus_e(&mut v, &noise[sp], &self.b, &noise[epp]);
let mut c = [0u16; NBAR * NBAR];
encode_message(&mut c, mu);
for (c, v) in c.iter_mut().zip(&v) {
*c = c.wrapping_add(*v) & Q_MASK;
}
let mut ct = vec![0u8; CIPHERTEXT_SIZE];
pack(&mut ct[..BP_PACKED], &bp);
pack(&mut ct[BP_PACKED..], &c);
let mut ss = [0u8; SHARED_KEY_SIZE];
shake(&[&ct, &g2[SHARED_KEY_SIZE..]], &mut ss);
noise.zeroize();
g2.zeroize();
v.zeroize();
(ct, ss)
}
}
type Span = core::ops::Range<usize>;
/// SHAKE128(0x96 || seedSE) expanded and sampled into S', E', E''.
fn noise(seed_se: &[u8]) -> (Span, Span, Span, Vec<u16>) {
let total = 2 * N * NBAR + NBAR * NBAR;
let mut bytes = vec![0u8; 2 * total];
shake(&[&[0x96], seed_se], &mut bytes);
let mut v = vec![0u16; total];
le_u16s(&bytes, &mut v);
bytes.zeroize();
sample(&mut v);
(0..N * NBAR, N * NBAR..2 * N * NBAR, 2 * N * NBAR..total, v)
}
impl PrivateKey {
/// CIRCL's `newKeyFromSeed`: seed is s || seedSE || z.
pub fn derive(seed: &[u8; KEY_SEED_SIZE]) -> Self {
let mut seed_a = [0u8; SEED_A];
shake(&[&seed[2 * SHARED_KEY_SIZE..]], &mut seed_a);
let mut bytes = vec![0u8; 2 * 2 * N * NBAR];
shake(&[&[0x5f], &seed[SHARED_KEY_SIZE..2 * SHARED_KEY_SIZE]], &mut bytes);
let mut s = vec![0u16; N * NBAR];
let mut e = vec![0u16; N * NBAR];
le_u16s(&bytes[..2 * N * NBAR], &mut s);
le_u16s(&bytes[2 * N * NBAR..], &mut e);
bytes.zeroize();
sample(&mut s);
sample(&mut e);
let a = expand_a(&seed_a);
let mut b = vec![0u16; N * NBAR];
mul_add_as_plus_e(&mut b, &a, &s, &e);
e.zeroize();
let pk = PublicKey { seed_a, b };
let hpk = pk.hash();
let mut s_reject = [0u8; SHARED_KEY_SIZE];
s_reject.copy_from_slice(&seed[..SHARED_KEY_SIZE]);
PrivateKey { s_reject, pk, s, hpk }
}
pub fn public(&self) -> &PublicKey {
&self.pk
}
/// CIRCL's packed form: s || pk || S (little-endian) || H(pk).
pub fn to_bytes(&self) -> zeroize::Zeroizing<Vec<u8>> {
let mut out = zeroize::Zeroizing::new(Vec::with_capacity(PRIVATE_KEY_SIZE));
out.extend_from_slice(&self.s_reject);
out.extend_from_slice(&self.pk.to_bytes());
for x in &self.s {
out.extend_from_slice(&x.to_le_bytes());
}
out.extend_from_slice(&self.hpk);
out
}
/// Decapsulates with implicit rejection: a ciphertext that does not
/// re-encrypt identically yields SHAKE(ct || s) instead of an error.
pub fn decapsulate(&self, ct: &[u8]) -> Option<[u8; SHARED_KEY_SIZE]> {
if ct.len() != CIPHERTEXT_SIZE {
return None;
}
let mut bp = vec![0u16; NBAR * N];
let mut c = [0u16; NBAR * NBAR];
unpack(&mut bp, &ct[..BP_PACKED]);
unpack(&mut c, &ct[BP_PACKED..]);
let mut w = [0u16; NBAR * NBAR];
mul_bs(&mut w, &bp, &self.s);
for (w, c) in w.iter_mut().zip(&c) {
*w = c.wrapping_sub(*w) & Q_MASK;
}
let mut mu = [0u8; MESSAGE];
decode_message(&mut mu, &w);
let mut g2 = [0u8; 2 * SHARED_KEY_SIZE];
shake(&[&self.hpk, &mu], &mut g2);
let (sp, ep, epp, mut noise) = noise(&g2[..SHARED_KEY_SIZE]);
let a = expand_a(&self.pk.seed_a);
let mut bbp = vec![0u16; NBAR * N];
mul_add_sa_plus_e(&mut bbp, &noise[sp.clone()], &a, &noise[ep]);
for x in bbp.iter_mut() {
*x &= Q_MASK;
}
mul_add_sb_plus_e(&mut w, &noise[sp], &self.pk.b, &noise[epp]);
let mut cc = [0u16; NBAR * NBAR];
encode_message(&mut cc, &mu);
for (cc, w) in cc.iter_mut().zip(&w) {
*cc = cc.wrapping_add(*w) & Q_MASK;
}
let differs = ct_differs(&bp, &bbp) | ct_differs(&c, &cc);
let mask = differs.wrapping_neg(); // 0xff to reject, 0 to accept
let mut k = [0u8; SHARED_KEY_SIZE];
for (i, k) in k.iter_mut().enumerate() {
*k = (g2[SHARED_KEY_SIZE + i] & !mask) | (self.s_reject[i] & mask);
}
let mut ss = [0u8; SHARED_KEY_SIZE];
shake(&[ct, &k], &mut ss);
for z in [&mut mu[..], &mut g2[..], &mut k[..]] {
z.zeroize();
}
noise.zeroize();
w.zeroize();
cc.zeroize();
bbp.zeroize();
Some(ss)
}
}
#[cfg(test)]
mod tests {
use super::*;
use aes::cipher::{BlockCipherEncrypt, KeyInit};
use sha2::Digest;
/// NIST's AES-256 CTR_DRBG from the KAT generator's randombytes.c.
struct Drbg {
key: [u8; 32],
v: [u8; 16],
}
impl Drbg {
fn new(entropy: &[u8; 48]) -> Self {
let mut d = Drbg { key: [0; 32], v: [0; 16] };
d.update(Some(entropy));
d
}
fn block(&mut self) -> [u8; 16] {
for b in self.v.iter_mut().rev() {
*b = b.wrapping_add(1);
if *b != 0 {
break;
}
}
let c = aes::Aes256::new(&self.key.into());
let mut blk = self.v.into();
c.encrypt_block(&mut blk);
blk.into()
}
fn update(&mut self, data: Option<&[u8; 48]>) {
let mut t = [0u8; 48];
for i in 0..3 {
t[16 * i..16 * i + 16].copy_from_slice(&self.block());
}
if let Some(d) = data {
for (t, d) in t.iter_mut().zip(d) {
*t ^= d;
}
}
self.key.copy_from_slice(&t[..32]);
self.v.copy_from_slice(&t[32..]);
}
fn fill(&mut self, out: &mut [u8]) {
for c in out.chunks_mut(16) {
let b = self.block();
c.copy_from_slice(&b[..c.len()]);
}
self.update(None);
}
}
fn upper(b: &[u8]) -> String {
b.iter().map(|x| format!("{x:02X}")).collect()
}
/// CIRCL's TestPQCgenKATKem: the SHA-256 of the KAT response file it
/// regenerates, which matches the FrodoKEM reference implementation's
/// PQCkemKAT_19888_shake.rsp.
#[test]
fn nist_kat_digest() {
let mut seed = [0u8; 48];
for (i, s) in seed.iter_mut().enumerate() {
*s = i as u8;
}
let mut g = Drbg::new(&seed);
let mut f = sha2::Sha256::new();
Digest::update(&mut f, b"# FrodoKEM-640-SHAKE\n\n");
for i in 0..100 {
g.fill(&mut seed);
Digest::update(&mut f, format!("count = {i}\nseed = {}\n", upper(&seed)).as_bytes());
let mut g2 = Drbg::new(&seed);
let mut kseed = [0u8; KEY_SEED_SIZE];
g2.fill(&mut kseed);
let sk = PrivateKey::derive(&kseed);
let pk = sk.public().to_bytes();
let mut eseed = [0u8; ENCAPSULATION_SEED_SIZE];
g2.fill(&mut eseed);
let (ct, ss) = PublicKey::from_bytes(&pk).unwrap().encapsulate(&eseed);
assert_eq!(sk.decapsulate(&ct).unwrap(), ss);
Digest::update(
&mut f,
format!(
"pk = {}\nsk = {}\nct = {}\nss = {}\n\n",
upper(&pk),
upper(&sk.to_bytes()),
upper(&ct),
upper(&ss)
)
.as_bytes(),
);
}
let digest: String = f.finalize().iter().map(|x| format!("{x:02x}")).collect();
assert_eq!(digest, "604a10cfc871dfaed9cb5b057c644ab03b16852cea7f39bc7f9831513b5b1cfa");
}
}

Some files were not shown because too many files have changed in this diff Show More