Rewrite vault in Rust (format-compatible with 0.5) #1

2026-10-08T17:57:52+02:00

andydixon commented

2026-10-08 17:57:52 +02:00

(Migrated from github.com)

Rewrites vault in Rust as an implementation migration. The formats don't change: key files, vaults, the .lock and the unlock agent protocol stay byte-compatible with 0.5, so nothing needs converting, and Go 0.5 and Rust 0.6 can share a vault at the same time.

Compatibility evidence

The Go code is frozen at 4c1507c in oracle/go as a test-only reference. Its 32 sources are byte-identical; the only additions are test-only generators and benchmarks. Vectors it generated live in tests/vectors.
Byte-for-byte matches against those vectors: FrodoKEM-640-SHAKE (a line-by-line port of CIRCL 1.6.4, which also reproduces its NIST KAT digest), ML-KEM-768 (RustCrypto ml-kem), hybrid derivation, KEK, header MAC and serialisation, header parser verdicts, STREAM, Argon2id key files, object IDs and record JSON.
Bidirectional tests against the real binaries: Go encrypts / Rust decrypts and the reverse; the full CLI lifecycle (with decoy) both ways; Go↔Rust lock contention, including a killed holder; agent wire compatibility both ways.
Cross-built Linux binaries (arm64, 386, armv6/7, riscv64, loong64, ppc64le, big-endian s390x) run under qemu and are cross-checked against the Go oracle.

See docs/COMPATIBILITY.md.

Changes

docs/MIGRATION.md lists every intentional behaviour change. The main ones:

get: never replaces an existing file, even a symlink or one created while it runs (atomic no-replace publish); get -r doesn't follow symlinks out of the output folder; output is synced before it appears.
Crash safety: deletion order is durable; a file that changes while being added is not stored; migrate deletes an original only if it is still the file stored; paths inside the vault are refused.
Agent: hardened (mlock, MADV_DONTDUMP, PR_SET_DUMPABLE, checked), and it refuses the key if protection fails; it doesn't wait for a client to expire.
add URL / mirror: time and size limits; same-host redirects only; an HTML parser instead of regexes; confined local paths (fixes a %2e%2e traversal Go also has); exit status 3 on a partial mirror.
Throughput: one bounded worker pool with at most eight object buffers; parallel catalogue reads.

Release matrix

Linux (9 architectures), Windows, macOS and FreeBSD: 16 binaries, down from 43, as agreed. docs/PLATFORMS.md records each dropped target and why. unlock is not available on Windows.

Verification

Run locally, all passing: cargo fmt --check, clippy -D warnings, 92 unit + 19 CLI + 1 interop + 1 doc test (VAULT_REQUIRE_ORACLE=1), release build, cargo deny check, the Go oracle's own tests.
Fuzzing: 7 targets; one crash found in the URL parser and fixed.
Miri: clean on the secret buffers, worker pool and parsers.
Reviews: three independent reviews (crypto, filesystem/agent, network); their findings are fixed in this branch.
Not run: Windows and macOS tests run only in the new CI workflow, which this PR triggers for the first time.

Performance

Measured on a shared, busy machine; details in docs/BENCHMARKS.md.

list: 0.74 s vs 5 s.
Adding 1 GiB: 30 s vs 42 s.
Unlock: 129 ms vs 122 ms.
Binary: 3.5 MB vs 11.5 MB.
Getting 1 GiB is slower (15.5 s vs 3.6 s): 13.3 s of it is the new pre-publish sync of the output file.

Rewrites vault in Rust as an implementation migration. The formats don't change: key files, vaults, the `.lock` and the `unlock` agent protocol stay byte-compatible with 0.5, so nothing needs converting, and Go 0.5 and Rust 0.6 can share a vault at the same time. ## Compatibility evidence - The Go code is frozen at 4c1507c in `oracle/go` as a test-only reference. Its 32 sources are byte-identical; the only additions are test-only generators and benchmarks. Vectors it generated live in `tests/vectors`. - Byte-for-byte matches against those vectors: FrodoKEM-640-SHAKE (a line-by-line port of CIRCL 1.6.4, which also reproduces its NIST KAT digest), ML-KEM-768 (RustCrypto `ml-kem`), hybrid derivation, KEK, header MAC and serialisation, header parser verdicts, STREAM, Argon2id key files, object IDs and record JSON. - Bidirectional tests against the real binaries: Go encrypts / Rust decrypts and the reverse; the full CLI lifecycle (with decoy) both ways; Go↔Rust lock contention, including a killed holder; agent wire compatibility both ways. - Cross-built Linux binaries (arm64, 386, armv6/7, riscv64, loong64, ppc64le, big-endian s390x) run under qemu and are cross-checked against the Go oracle. See `docs/COMPATIBILITY.md`. ## Changes `docs/MIGRATION.md` lists every intentional behaviour change. The main ones: - **`get`:** never replaces an existing file, even a symlink or one created while it runs (atomic no-replace publish); `get -r` doesn't follow symlinks out of the output folder; output is synced before it appears. - **Crash safety:** deletion order is durable; a file that changes while being added is not stored; `migrate` deletes an original only if it is still the file stored; paths inside the vault are refused. - **Agent:** hardened (mlock, `MADV_DONTDUMP`, `PR_SET_DUMPABLE`, checked), and it refuses the key if protection fails; it doesn't wait for a client to expire. - **`add URL` / `mirror`:** time and size limits; same-host redirects only; an HTML parser instead of regexes; confined local paths (fixes a `%2e%2e` traversal Go also has); exit status 3 on a partial mirror. - **Throughput:** one bounded worker pool with at most eight object buffers; parallel catalogue reads. ## Release matrix Linux (9 architectures), Windows, macOS and FreeBSD: 16 binaries, down from 43, as agreed. `docs/PLATFORMS.md` records each dropped target and why. `unlock` is not available on Windows. ## Verification - **Run locally, all passing:** `cargo fmt --check`, clippy `-D warnings`, 92 unit + 19 CLI + 1 interop + 1 doc test (`VAULT_REQUIRE_ORACLE=1`), release build, `cargo deny check`, the Go oracle's own tests. - **Fuzzing:** 7 targets; one crash found in the URL parser and fixed. - **Miri:** clean on the secret buffers, worker pool and parsers. - **Reviews:** three independent reviews (crypto, filesystem/agent, network); their findings are fixed in this branch. - **Not run:** Windows and macOS tests run only in the new CI workflow, which this PR triggers for the first time. ## Performance Measured on a shared, busy machine; details in `docs/BENCHMARKS.md`. - `list`: 0.74 s vs 5 s. - Adding 1 GiB: 30 s vs 42 s. - Unlock: 129 ms vs 122 ms. - Binary: 3.5 MB vs 11.5 MB. - Getting 1 GiB is slower (15.5 s vs 3.6 s): 13.3 s of it is the new pre-publish sync of the output file.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: andy/vault#1