Docs for the Rust version; benchmarks; build.sh fixes
README, CLAUDE.md, man pages, the user guide and its screenshot script now describe the Rust build and the intentional behaviour changes, which docs/MIGRATION.md lists. Go and Rust micro-benchmarks for the same operations, and scripts/bench.sh for end-to-end comparison. build.sh exports its tree root to the per-target jobs and keeps logs out of the output directory; riscv64 musl builds use getrandom's raw syscall backend, since zig's musl getrandom never reaches the kernel there. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -4,71 +4,93 @@ This file provides guidance to Claude Code when working with this repository.
|
|||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
vault is a post-quantum encrypted file vault: one `package main` at the repo
|
vault is a post-quantum encrypted file vault: one Rust package (`Cargo.toml`
|
||||||
root that builds a single `vault` binary. Module `github.com/andydixon/vault`.
|
at the root, toolchain pinned in `rust-toolchain.toml`) that builds a single
|
||||||
|
`vault` binary. The library (`src/lib.rs`) exists so tests and benchmarks can
|
||||||
|
reach the same code; `src/main.rs` only calls `cli::main`.
|
||||||
|
|
||||||
- `main.go` - CLI: `genkey`, `init`, `add` (file, directory
|
- `src/cli.rs` - commands and Go-`flag`-compatible parsing: `genkey`,
|
||||||
or URL), `migrate` (add, then delete originals), `mirror`, `absorb` (tar
|
`init`, `add` (file, directory or URL), `migrate`, `mirror`, `absorb`,
|
||||||
on stdin, `absorb.go`), `ls` (one folder level), `list` (recursive),
|
`ls`/`list`, `get` (`-r`), `mv`/`cp` (`transfers` resolves names like
|
||||||
`get` (`-r` for a folder), `mv`/`cp` (files or folders; `transfers`
|
mv(1)), `rm` (`-r`, asks unless `-f`), `unlock`/`lock`, hidden `__agent`.
|
||||||
resolves names like mv(1), copies share objects), `rm` (`-r` for a
|
- `src/agent.rs` - the unlock agent: `unlock` re-executes the binary as
|
||||||
folder, asks unless `-f`), `unlock`/`lock`.
|
`__agent` (passphrase variables stripped, secret over a pipe); it hardens
|
||||||
- `agent*.go` - `vault unlock` re-executes itself as a hidden `__agent`
|
itself (`platform::memory`) before reading the key, serves it on a Unix
|
||||||
process holding the suite ID and seed (pipe in, mlocked, undumpable) and
|
socket in a 0700 per-user dir until a deadline (fixed or `--idle`).
|
||||||
serving them on a Unix socket in a 0700 per-user dir until a deadline
|
Wire-compatible with the Go agent. Not available on Windows.
|
||||||
(fixed, or with `--idle` reset on each use). `keyIdentity` asks it first
|
- `src/keyfile.rs` - key file: two Argon2id/XChaCha20-Poly1305 slots, decoy
|
||||||
unless `$VAULT_PASSPHRASE` is set.
|
or random bytes in the second, random order, both always tried.
|
||||||
- `keyfile.go` - key file with two passphrase slots (Argon2id +
|
- `src/crypto/` - `frodo.rs` (FrodoKEM-640-SHAKE ported from CIRCL v1.6.4),
|
||||||
XChaCha20-Poly1305 sealing a suite ID and 32-byte master seed). The second
|
`kem.rs` (hybrid suite 0x0001 with RustCrypto ml-kem, `Identity`),
|
||||||
slot is either a decoy identity or random bytes; slot order is random.
|
`header.rs` (Fury header format), `fury.rs` (file encryption), `stream.rs`
|
||||||
- `store.go` - on-disk vault: `records/` holds fixed-size encrypted catalogue
|
(STREAM, including random access).
|
||||||
records, `objects/` holds file bodies cut into 4 MiB chunks (last one
|
- `src/store/` - on-disk vault: `records/`, `objects/` sharded by ID prefix,
|
||||||
bucket-padded) whose IDs are HMAC'd from a per-record seed. Both are
|
catalogue, add/remove/copy/extract, filler; `types.rs` holds the validated
|
||||||
sharded into 256 `xx/` subdirectories by ID prefix. Records a key cannot
|
types (`VaultName`, `Id`, `ObjectSeed`, `Record`) and record JSON.
|
||||||
open are skipped, not treated as errors.
|
- `src/pipeline.rs` - the one bounded worker pool and the eight reusable
|
||||||
- `lock*.go` - vault lock on `DIR/.lock`: fcntl on unix, LockFileEx on
|
object buffers; coordinators run on the caller's thread.
|
||||||
Windows, an exclusive lock file elsewhere. Writers lock exclusively,
|
- `src/platform/` - all `unsafe` and OS specifics: `fs.rs` (no-clobber
|
||||||
readers shared, taken after the passphrase prompt.
|
publish, `Dir` confined extraction, epoch times, dir sync), `lock.rs`
|
||||||
- `docs/guide` - LaTeX user guide (`build-pdf.sh` via Tectonic; `shots.py`
|
(fcntl / LockFileEx / lock file), `memory.rs` (mlock, undumpable),
|
||||||
re-shoots screenshots from the real binary). Keep it in step with the CLI.
|
`term.rs` (prompts).
|
||||||
- `packaging/repo.sh [TAG]` - builds dixon-vault .deb/.rpm/Arch packages
|
- `src/net/` - `add URL` and `mirror` with limits; `url.rs` (Go `net/url`
|
||||||
from a tag with nfpm and publishes them, signed, to the apt/dnf/pacman
|
semantics), `html.rs` (tokenizer and link rewriting).
|
||||||
repo at repo.dixon.cx (same script, other project block, in ../mapsize).
|
- `src/absorb.rs` - tar (optionally gzipped) from stdin, streamed in.
|
||||||
- `build.sh` - cross-compiles every `go tool dist list` target except
|
- `src/secret.rs`, `src/rng.rs`, `src/util.rs`, `src/error.rs` - secret
|
||||||
iOS and android/386, amd64, arm (cgo-only).
|
buffers, randomness (seeded only under `cfg(test)`), Go-compatible
|
||||||
- `internal/pqcrypt` - file encryption (header + STREAM ChaCha20-Poly1305),
|
parsing/printing helpers, the error type.
|
||||||
suite registry, hybrid FrodoKEM-640 + ML-KEM-768 default suite.
|
- `oracle/go` - the frozen Go implementation (test-only reference, see
|
||||||
- `internal/format`, `internal/stream`, `internal/term` - header encoding,
|
`oracle/README.md`); `oracle/gen-vectors.sh` writes `tests/vectors`.
|
||||||
chunked payload encryption, terminal passphrase prompts.
|
- `tests/` - CLI and interoperability tests against the real binaries;
|
||||||
|
`fuzz/` - cargo-fuzz targets; `benches/micro.rs` - micro-benchmarks.
|
||||||
|
- `docs/` - man pages, `COMPATIBILITY.md`, `MIGRATION.md`, `PLATFORMS.md`,
|
||||||
|
`BENCHMARKS.md`, and the LaTeX guide (`docs/guide`, `build-pdf.sh` via
|
||||||
|
Tectonic; `shots.py` re-shoots screenshots from the real binary). Keep
|
||||||
|
them in step with the CLI.
|
||||||
|
- `build.sh` - cross-builds every supported target (see `PLATFORMS.md`).
|
||||||
|
- `packaging/package.sh` builds and inspects dixon-vault packages locally;
|
||||||
|
`packaging/repo.sh [TAG]` signs and publishes them to repo.dixon.cx.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go test ./...
|
cargo test --locked # needs go on PATH for interop tests
|
||||||
go build .
|
cargo clippy --locked --all-targets --all-features -- -D warnings
|
||||||
./build.sh # all platforms, into ./dist
|
cargo fmt --check
|
||||||
TARGETS="linux/amd64" ./build.sh # a subset
|
cargo build --release --locked
|
||||||
|
./build.sh # all buildable targets, into ./dist
|
||||||
|
TARGETS="linux/amd64" ./build.sh # a subset
|
||||||
|
cargo deny --locked check # advisories and licences
|
||||||
|
(cd fuzz && cargo +nightly fuzz run header)
|
||||||
```
|
```
|
||||||
|
|
||||||
End-to-end check:
|
End-to-end check:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
export VAULT_PASSPHRASE=test VAULT_KEY=$PWD/k VAULT_DIR=$PWD/s
|
export VAULT_PASSPHRASE=test VAULT_KEY=$PWD/k VAULT_DIR=$PWD/s
|
||||||
./vault genkey && ./vault init && ./vault add README.md docs && ./vault list
|
target/release/vault genkey && target/release/vault init && target/release/vault add README.md docs && target/release/vault list
|
||||||
```
|
```
|
||||||
|
|
||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
|
- The on-disk formats are frozen and shared with the Go version; check any
|
||||||
|
change against `docs/COMPATIBILITY.md` and the oracle vectors.
|
||||||
- Never write a seed or private key unencrypted (the agent gets it by pipe).
|
- Never write a seed or private key unencrypted (the agent gets it by pipe).
|
||||||
- Keep suite IDs, stanza tags, HKDF info strings and `fileKeySize` (32) stable.
|
- Keep suite IDs, stanza tags, HKDF info strings and the 32-byte file key stable.
|
||||||
- Every ciphertext's and shard directory's mtime is pinned to the epoch;
|
- Every ciphertext's and shard directory's mtime is pinned to the epoch;
|
||||||
records stay one fixed size; objects stay at most 4 MiB and bucket-padded.
|
records stay one fixed size; objects stay at most 4 MiB and bucket-padded.
|
||||||
These hide metadata from the storage host.
|
These hide metadata from the storage host.
|
||||||
- An object's plaintext starts with its own ID, binding it to its record
|
- An object's plaintext starts with its own ID, binding it to its record
|
||||||
and its position in the file.
|
and its position in the file.
|
||||||
- `get` publishes output only after every object has authenticated.
|
- `get` publishes output only after every object has authenticated, with a
|
||||||
|
no-replace primitive; never fall back to check-then-rename.
|
||||||
- Decoy deniability: both key slots stay the same size and are both always
|
- Decoy deniability: both key slots stay the same size and are both always
|
||||||
tried; `init` always writes filler; foreign records are skipped silently.
|
tried; `init` always writes filler; foreign records are skipped silently.
|
||||||
- Every command that changes the vault holds the exclusive lock.
|
- Every command that changes the vault holds the exclusive lock (POSIX
|
||||||
- Keep dependencies pure Go and CGO_ENABLED=0 builds working.
|
fcntl locks, compatible with Go's).
|
||||||
|
- Crash-safe order: objects before records, new records before removing old
|
||||||
|
ones, record removal (synced) before object removal.
|
||||||
|
- Secrets live in `SecretArray`/`SecretVec`/`LockedBuf`: no Clone, Debug or
|
||||||
|
serialisation. Production randomness is the OS only.
|
||||||
|
- `unsafe` only in `src/platform`, each block with a SAFETY comment.
|
||||||
- Diagnostics go to standard error, lower-case, no trailing full stops.
|
- Diagnostics go to standard error, lower-case, no trailing full stops.
|
||||||
|
|||||||
@@ -40,3 +40,7 @@ windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Storage
|
|||||||
codegen-units = 1
|
codegen-units = 1
|
||||||
lto = true
|
lto = true
|
||||||
strip = true
|
strip = true
|
||||||
|
|
||||||
|
[[bench]]
|
||||||
|
name = "micro"
|
||||||
|
harness = false
|
||||||
|
|||||||
@@ -23,16 +23,36 @@ Releases are packaged and published with `packaging/repo.sh [TAG]`.
|
|||||||
|
|
||||||
## Build
|
## Build
|
||||||
|
|
||||||
|
vault is written in Rust (the toolchain is pinned in `rust-toolchain.toml`;
|
||||||
|
`rustup` installs it on first use). Releases up to 0.5 were Go; the Go code
|
||||||
|
lives on, frozen, in `oracle/go` as the test reference. Nothing at run time
|
||||||
|
needs Go.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go build . # produces ./vault
|
cargo build --release --locked # produces target/release/vault
|
||||||
go test ./...
|
cargo test --locked # unit, CLI and Go-interoperability tests
|
||||||
./build.sh # every OS and CPU architecture, into ./dist
|
./build.sh # every target this host can build, into ./dist
|
||||||
|
TARGETS="linux/amd64 windows/amd64" ./build.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
`build.sh` cross-compiles static binaries for every target in
|
`build.sh` cross-compiles with `cargo-zigbuild` and `zig` (Linux, statically
|
||||||
`go tool dist list` except iOS and 32-bit or x86 Android, which need cgo. It builds 32-bit ARM Linux for
|
linked with musl, and Windows), on a Mac for macOS, and with `cross` and
|
||||||
both armv6 and armv7, and writes `dist/SHA256SUMS`. Set `TARGETS` to build a
|
Docker for the BSDs, illumos and Solaris. It names binaries as before
|
||||||
subset, for example `TARGETS="linux/amd64 darwin/arm64" ./build.sh`.
|
(`vault-linux-arm64`, `vault-windows-amd64.exe`, …), builds 32-bit ARM Linux
|
||||||
|
for both armv6 and armv7, and writes `dist/SHA256SUMS`. `docs/PLATFORMS.md`
|
||||||
|
lists every target, how it is built and tested, and the Go targets Rust
|
||||||
|
cannot build.
|
||||||
|
|
||||||
|
The interoperability tests build the Go oracle when `go` is installed and
|
||||||
|
are skipped (and say so) otherwise; set `VAULT_REQUIRE_ORACLE=1` to make a
|
||||||
|
missing oracle an error, as CI does. `oracle/gen-vectors.sh` regenerates the
|
||||||
|
Go test vectors. Packages are built and inspected locally, unsigned, with
|
||||||
|
`packaging/package.sh`, and published with `packaging/repo.sh [TAG]`.
|
||||||
|
|
||||||
|
Further reading: `SECURITY.md` (what is and is not protected, memory and
|
||||||
|
filesystem guarantees), `docs/COMPATIBILITY.md` (the formats),
|
||||||
|
`docs/MIGRATION.md` (what changed from the Go version) and
|
||||||
|
`docs/BENCHMARKS.md`.
|
||||||
|
|
||||||
## User guide
|
## User guide
|
||||||
|
|
||||||
@@ -82,12 +102,15 @@ vault lock # forget it now
|
|||||||
Commands that change the vault take an exclusive lock on `DIR/.lock`, and
|
Commands that change the vault take an exclusive lock on `DIR/.lock`, and
|
||||||
commands that read it take a shared one. A second process waits, and says
|
commands that read it take a shared one. A second process waits, and says
|
||||||
so, until the first finishes. The operating system releases the lock if a
|
so, until the first finishes. The operating system releases the lock if a
|
||||||
process dies. Plan 9 and WebAssembly have no file locks, so there the lock
|
process dies; the locks are the same ones the Go version used, so old and
|
||||||
file itself is the lock. After a crash on those platforms you remove it by
|
new versions exclude each other. WebAssembly (WASI) has no file locks, so
|
||||||
hand; the error message names it.
|
there the lock file itself is the lock. After a crash there you remove it
|
||||||
|
by hand; the error message names it.
|
||||||
|
|
||||||
Adding a name that already exists replaces it. `get` refuses to overwrite an
|
Adding a name that already exists replaces it. `get` never replaces an
|
||||||
existing file and writes nothing until the whole object has authenticated.
|
existing file (or symlink) and writes nothing until the whole file has
|
||||||
|
authenticated. `add URL` and `mirror` have size and time limits; see
|
||||||
|
`vault(1)`.
|
||||||
|
|
||||||
`--key PATH` and `--vault DIR` work with every command. They default to
|
`--key PATH` and `--vault DIR` work with every command. They default to
|
||||||
`$VAULT_KEY` and `$VAULT_DIR`, then to `~/.vault/key` and `~/.vault/store`.
|
`$VAULT_KEY` and `$VAULT_DIR`, then to `~/.vault/key` and `~/.vault/store`.
|
||||||
@@ -103,7 +126,8 @@ over a pipe and never touches the disk; the helper's memory is locked
|
|||||||
against swapping, it cannot be core-dumped (or, on Linux, debugged), and it
|
against swapping, it cannot be core-dumped (or, on Linux, debugged), and it
|
||||||
answers only on a socket in a directory private to your user. While it is
|
answers only on a socket in a directory private to your user. While it is
|
||||||
unlocked, anything running as you can use the vault, decoy or not, so lock
|
unlocked, anything running as you can use the vault, decoy or not, so lock
|
||||||
before handing the machine over.
|
before handing the machine over. If the helper cannot protect its memory it
|
||||||
|
refuses the key. `unlock` is not available on Windows.
|
||||||
|
|
||||||
## Decoy passphrase
|
## Decoy passphrase
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,79 @@
|
|||||||
|
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||||
|
|
||||||
|
//! The Rust side of scripts/bench.sh: the same operations as the Go
|
||||||
|
//! oracle's Benchmark* functions, timed as the median of repeated runs.
|
||||||
|
//! Run with `cargo bench --bench micro`.
|
||||||
|
|
||||||
|
use std::hint::black_box;
|
||||||
|
use std::io::{Read, Write};
|
||||||
|
use std::time::{Duration, Instant};
|
||||||
|
use vault::crypto::{fury, kem, stream};
|
||||||
|
|
||||||
|
fn pattern(n: usize, salt: u8) -> Vec<u8> {
|
||||||
|
(0..n).map(|i| (i.wrapping_mul(7) as u8) ^ salt).collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Runs `f` until about two seconds have passed (at least 5 times) and
|
||||||
|
/// prints the median and the spread.
|
||||||
|
fn bench(name: &str, bytes: usize, mut f: impl FnMut()) {
|
||||||
|
f(); // warm up
|
||||||
|
let mut times = Vec::new();
|
||||||
|
let start = Instant::now();
|
||||||
|
while times.len() < 5 || (start.elapsed() < Duration::from_secs(2) && times.len() < 10_000) {
|
||||||
|
let t = Instant::now();
|
||||||
|
f();
|
||||||
|
times.push(t.elapsed());
|
||||||
|
}
|
||||||
|
times.sort();
|
||||||
|
let med = times[times.len() / 2];
|
||||||
|
let (lo, hi) = (times[times.len() / 10], times[times.len() * 9 / 10]);
|
||||||
|
let rate = if bytes > 0 {
|
||||||
|
format!(" {:.0} MB/s", bytes as f64 / med.as_secs_f64() / 1e6)
|
||||||
|
} else {
|
||||||
|
String::new()
|
||||||
|
};
|
||||||
|
println!(
|
||||||
|
"{name:<24} median {:>10.3} ms (p10 {:.3}, p90 {:.3}, n={}){rate}",
|
||||||
|
med.as_secs_f64() * 1e3,
|
||||||
|
lo.as_secs_f64() * 1e3,
|
||||||
|
hi.as_secs_f64() * 1e3,
|
||||||
|
times.len()
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
fn main() {
|
||||||
|
let seed = pattern(32, 1);
|
||||||
|
bench("derive", 0, || {
|
||||||
|
black_box(kem::Identity::from_suite_seed(kem::SUITE_ID, &seed).unwrap());
|
||||||
|
});
|
||||||
|
let id = kem::Identity::from_suite_seed(kem::SUITE_ID, &seed).unwrap();
|
||||||
|
let r = id.recipient();
|
||||||
|
let fk: [u8; 32] = pattern(32, 2).try_into().unwrap();
|
||||||
|
bench("wrap", 0, || {
|
||||||
|
black_box(r.wrap(&fk, &mut vault::rng::os()));
|
||||||
|
});
|
||||||
|
let st = r.wrap(&fk, &mut vault::rng::os());
|
||||||
|
bench("unwrap", 0, || {
|
||||||
|
black_box(id.unwrap(std::slice::from_ref(&st)).unwrap());
|
||||||
|
});
|
||||||
|
let pt = pattern(4 << 20, 3);
|
||||||
|
bench("object 4MiB", pt.len(), || {
|
||||||
|
let mut ct = Vec::new();
|
||||||
|
fury::encrypt_to(&mut ct, &r, &pt, &mut vault::rng::os());
|
||||||
|
black_box(fury::decrypt_bounded(&ct[..], &id, pt.len()).unwrap());
|
||||||
|
});
|
||||||
|
let key = [0u8; 32];
|
||||||
|
let big = vec![0u8; 64 << 20];
|
||||||
|
bench("stream 64MiB", big.len(), || {
|
||||||
|
let mut w = stream::StreamWriter::new(&key, Vec::with_capacity(big.len() + (1 << 16)));
|
||||||
|
w.write_all(&big).unwrap();
|
||||||
|
let ct = w.finish().unwrap();
|
||||||
|
let mut out = Vec::with_capacity(big.len());
|
||||||
|
stream::StreamReader::new(&key, &ct[..]).read_to_end(&mut out).unwrap();
|
||||||
|
black_box(out);
|
||||||
|
});
|
||||||
|
let body = vault::keyfile::new_body(b"bench", b"", &mut vault::rng::os());
|
||||||
|
bench("unlock", 0, || {
|
||||||
|
black_box(vault::keyfile::unlock_body(&body, b"bench").unwrap());
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -73,11 +73,14 @@ out=$(cd -- "$out" && pwd -P)
|
|||||||
case $out in
|
case $out in
|
||||||
/ | "$(cd ~ && pwd -P)" | "$root") echo "refusing OUT=$out: use a dedicated directory" >&2; exit 2 ;;
|
/ | "$(cd ~ && pwd -P)" | "$root") echo "refusing OUT=$out: use a dedicated directory" >&2; exit 2 ;;
|
||||||
esac
|
esac
|
||||||
rm -f -- "$out"/vault-* "$out"/SHA256SUMS "$out"/*.log
|
rm -f -- "$out"/vault-* "$out"/SHA256SUMS
|
||||||
|
|
||||||
export CARGO_TARGET_DIR_BASE=$root/target/build-sh
|
export CARGO_TARGET_DIR_BASE=$root/target/build-sh
|
||||||
export RUSTFLAGS="${RUSTFLAGS:-} --remap-path-prefix=$root=vault --remap-path-prefix=${CARGO_HOME:-$HOME/.cargo}=cargo"
|
export RUSTFLAGS="${RUSTFLAGS:-} --remap-path-prefix=$root=vault --remap-path-prefix=${CARGO_HOME:-$HOME/.cargo}=cargo"
|
||||||
export OUT_DIR=$out JOBS_N=$jobs HOST_OS=$(uname -s)
|
export OUT_DIR=$out JOBS_N=$jobs HOST_OS=$(uname -s) root
|
||||||
|
logs=$root/target/build-sh-logs
|
||||||
|
mkdir -p -- "$logs"
|
||||||
|
export logs
|
||||||
|
|
||||||
build_one() {
|
build_one() {
|
||||||
local want=$1 row os triple method name ext="" tdir bin
|
local want=$1 row os triple method name ext="" tdir bin
|
||||||
@@ -115,15 +118,14 @@ build_one() {
|
|||||||
# zig's musl getrandom() never reaches the kernel on riscv64 (it spins);
|
# zig's musl getrandom() never reaches the kernel on riscv64 (it spins);
|
||||||
# make the getrandom crate issue the system call itself.
|
# make the getrandom crate issue the system call itself.
|
||||||
[[ $triple == riscv64*-musl ]] && flags+=' --cfg getrandom_backend="linux_raw"'
|
[[ $triple == riscv64*-musl ]] && flags+=' --cfg getrandom_backend="linux_raw"'
|
||||||
if ! RUSTFLAGS=$flags CARGO_TARGET_DIR=$tdir "${cmd[@]}" --release --locked --target "$triple" >"$OUT_DIR/$name.log" 2>&1; then
|
if ! RUSTFLAGS=$flags CARGO_TARGET_DIR=$tdir "${cmd[@]}" --release --locked --target "$triple" >"$logs/$name.log" 2>&1; then
|
||||||
echo "FAILED $want ($triple)" >&2
|
echo "FAILED $want ($triple)" >&2
|
||||||
tail -20 "$OUT_DIR/$name.log" >&2
|
tail -20 "$logs/$name.log" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
bin=$tdir/$triple/release/vault$ext
|
bin=$tdir/$triple/release/vault$ext
|
||||||
[[ $os == wasip1 ]] && bin=$tdir/$triple/release/vault.wasm
|
[[ $os == wasip1 ]] && bin=$tdir/$triple/release/vault.wasm
|
||||||
cp -- "$bin" "$OUT_DIR/$name"
|
cp -- "$bin" "$OUT_DIR/$name"
|
||||||
rm -f -- "$OUT_DIR/$name.log"
|
|
||||||
if [[ $triple == *-musl* ]] && command -v file >/dev/null; then
|
if [[ $triple == *-musl* ]] && command -v file >/dev/null; then
|
||||||
file -L "$OUT_DIR/$name" | grep -q 'static' || { echo "FAILED $want: not statically linked" >&2; return 1; }
|
file -L "$OUT_DIR/$name" | grep -q 'static' || { echo "FAILED $want: not statically linked" >&2; return 1; }
|
||||||
fi
|
fi
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# Migrating from the Go implementation
|
||||||
|
|
||||||
|
vault 0.6 is a rewrite in Rust of the Go vault (0.5 and earlier). It is an
|
||||||
|
implementation migration only: key files, vaults, the lock and the unlock
|
||||||
|
agent are the same formats, byte for byte where the format is
|
||||||
|
deterministic (see `COMPATIBILITY.md`). There is nothing to convert. Go and
|
||||||
|
Rust builds can be used on the same vault, even at the same time: they
|
||||||
|
honour each other's locks, and either can use an agent the other started.
|
||||||
|
|
||||||
|
To go back, install 0.5: everything 0.6 writes, 0.5 reads.
|
||||||
|
|
||||||
|
## Intentional behaviour changes
|
||||||
|
|
||||||
|
Safety and correctness:
|
||||||
|
|
||||||
|
- **`get` never replaces anything.** Output is published with an atomic
|
||||||
|
no-replace rename or link, so a file, a symlink (even a dangling one) or a
|
||||||
|
file created while `get` runs is left alone. Go checked first and then
|
||||||
|
renamed, which a concurrent creator could race.
|
||||||
|
- **`get -r` does not follow symlinks inside the output directory.** Each
|
||||||
|
directory below the chosen one is opened without following links; a link
|
||||||
|
or non-directory in the way stops the run. (Windows: checked just before
|
||||||
|
use, see `SECURITY.md`.)
|
||||||
|
- **Extracted files are synced** to disk before they appear under their
|
||||||
|
name, and their directory afterwards.
|
||||||
|
- **A source file that changes while being added is not stored.** If it
|
||||||
|
grows, shrinks, is modified or replaced during `add`/`migrate`, the
|
||||||
|
command fails and nothing is stored. Go stored whatever it read (for a
|
||||||
|
file that grew, a truncated copy).
|
||||||
|
- **`migrate` deletes an original only if it is still the file stored**
|
||||||
|
(same inode, size and time), after the record is durable.
|
||||||
|
- **Deletes are durable in order.** Removing a file syncs the removal of its
|
||||||
|
record before deleting its objects, so a crash cannot leave a record
|
||||||
|
pointing at deleted objects. A failed sync after a record is already in
|
||||||
|
place no longer deletes its objects.
|
||||||
|
- **Lock errors are reported.** Go let a reader continue without the lock on
|
||||||
|
any error; Rust does so only on a read-only filesystem.
|
||||||
|
- **Stricter records.** A record must have every field, with exactly the
|
||||||
|
field names Go writes, and nothing after the JSON. Go also accepted
|
||||||
|
differently-cased names, missing fields (as zero) and trailing bytes;
|
||||||
|
Go never writes such records, so no existing vault is affected.
|
||||||
|
- **Key files over 64 KiB** are rejected without reading them whole.
|
||||||
|
|
||||||
|
Agent (`unlock`):
|
||||||
|
|
||||||
|
- The agent **refuses the key if it cannot protect it**: locking its memory,
|
||||||
|
disabling core dumps and (Linux) becoming undumpable must all succeed. Go
|
||||||
|
ignored failures there.
|
||||||
|
- Its environment is stripped of `VAULT_PASSPHRASE` and
|
||||||
|
`VAULT_DECOY_PASSPHRASE`; it runs in `/` with standard error closed.
|
||||||
|
- Expiry no longer waits for the request in progress beyond its deadline,
|
||||||
|
and a key is never handed out after expiry.
|
||||||
|
- `unlock` waits at most ten seconds for the agent to start and kills it on
|
||||||
|
failure; the agent removes its socket only if it is still its own.
|
||||||
|
- **Windows: `unlock` is not available** (Go's agent there had no memory
|
||||||
|
protection). Every command asks for the passphrase.
|
||||||
|
|
||||||
|
Network (`add URL`, `mirror`):
|
||||||
|
|
||||||
|
- Explicit limits, all adjustable: connect 30 s, response headers 60 s,
|
||||||
|
whole request 2 h (`add`) or 5 min (`mirror`, per request), download size
|
||||||
|
16 GiB (`add`) or 4 GiB in total (`mirror`), 10 redirects, 10 000 mirror
|
||||||
|
pages, 16 MiB per HTML or CSS page. Bytes are counted after decompression.
|
||||||
|
New flags `--max-bytes`, `--timeout`, `--max-pages`. Go had no limits.
|
||||||
|
- Redirects go to http or https only; a `mirror` redirect to another host is
|
||||||
|
skipped (Go followed it).
|
||||||
|
- `mirror` resolves links against the final address after redirects and the
|
||||||
|
page's `<base href>`; it parses HTML instead of matching a pattern, so it
|
||||||
|
rewrites only `href`, `src`, `action` and `poster` on real tags, plus
|
||||||
|
`url(...)` in CSS, `<style>` and `style=`, keeping every other byte and
|
||||||
|
the original quoting. Text in scripts and comments is no longer touched.
|
||||||
|
- `mirror` never lets two different URLs in one crawl overwrite each other's
|
||||||
|
file; the second is skipped. It exits with status **3** when it stored
|
||||||
|
some resources but skipped others (Go always exited 0).
|
||||||
|
- Proxies follow `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` as in Go; local
|
||||||
|
addresses are never proxied.
|
||||||
|
|
||||||
|
Command line:
|
||||||
|
|
||||||
|
- New option `--workers N` (1–256) on every command.
|
||||||
|
- `--key`/`--vault` no longer need a home directory when given explicitly.
|
||||||
|
- Command-line arguments must be valid UTF-8 (Go accepted any bytes, but
|
||||||
|
vault names had to be UTF-8 already).
|
||||||
|
- `ls`/`list` piped into a reader that stops early end quietly with
|
||||||
|
status 141, as Go's did by `SIGPIPE`.
|
||||||
|
|
||||||
|
Platforms: see `PLATFORMS.md`. Some targets Go built have no stable Rust
|
||||||
|
equivalent; they are listed there and need a decision before release.
|
||||||
|
|
||||||
|
## Unchanged on purpose
|
||||||
|
|
||||||
|
Formats, limits (1024-byte names, 1 TiB files, 4 MiB objects, 4096-byte
|
||||||
|
records, 16 MiB headers, 1 MiB stanzas), Argon2id parameters, the hybrid
|
||||||
|
suite, filler, decoy behaviour, messages and exit statuses (apart from the
|
||||||
|
above), the shape of `ls` and `list` output, and the agent protocol.
|
||||||
+6
-3
@@ -7,7 +7,9 @@ sandbox path and file owner are shown as /home/sam and "sam sam", key lines
|
|||||||
are highlighted in amber for the reader, and "moment" frames show a prompt
|
are highlighted in amber for the reader, and "moment" frames show a prompt
|
||||||
with a cursor, exactly as vault prints it before you type.
|
with a cursor, exactly as vault prints it before you type.
|
||||||
|
|
||||||
Needs: go, python3, and Chrome or Chromium (set CHROME=/path if not found).
|
Needs: cargo (with the pinned toolchain), cargo-zigbuild and zig for the
|
||||||
|
build screenshot, python3, and Chrome or Chromium (set CHROME=/path if not
|
||||||
|
found).
|
||||||
Output: docs/guide/img/*.png
|
Output: docs/guide/img/*.png
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -75,7 +77,8 @@ class Sandbox:
|
|||||||
self.bin = os.path.join(self.root, "bin")
|
self.bin = os.path.join(self.root, "bin")
|
||||||
os.makedirs(self.home)
|
os.makedirs(self.home)
|
||||||
os.makedirs(self.bin)
|
os.makedirs(self.bin)
|
||||||
subprocess.run(["go", "build", "-o", os.path.join(self.bin, "vault"), "."], cwd=REPO, check=True)
|
subprocess.run(["cargo", "build", "--release", "--locked", "--quiet"], cwd=REPO, check=True)
|
||||||
|
shutil.copy2(os.path.join(REPO, "target", "release", "vault"), os.path.join(self.bin, "vault"))
|
||||||
|
|
||||||
def env(self, extra=None):
|
def env(self, extra=None):
|
||||||
e = {
|
e = {
|
||||||
@@ -406,7 +409,7 @@ def main():
|
|||||||
|
|
||||||
# building, into a temporary OUT so an existing dist/ is left alone
|
# building, into a temporary OUT so an existing dist/ is left alone
|
||||||
rel = tempfile.mkdtemp(prefix="vault-release-")
|
rel = tempfile.mkdtemp(prefix="vault-release-")
|
||||||
targets = "linux/amd64 linux/arm darwin/arm64 windows/amd64"
|
targets = "linux/amd64 linux/arm64 linux/armv7 windows/amd64"
|
||||||
t = Term(box, title="sam@laptop: ~/src/vault")
|
t = Term(box, title="sam@laptop: ~/src/vault")
|
||||||
out = subprocess.run(["./build.sh"], cwd=REPO, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
out = subprocess.run(["./build.sh"], cwd=REPO, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
|
||||||
text=True, env=dict(os.environ, TARGETS=targets, OUT=rel)).stdout
|
text=True, env=dict(os.environ, TARGETS=targets, OUT=rel)).stdout
|
||||||
|
|||||||
+36
-17
@@ -222,14 +222,17 @@ Linux on ARM64 & Pi 4/5 (64-bit OS), Graviton, Ampere & vault-linux-arm64 \\
|
|||||||
Raspberry Pi, 32-bit & Pi 2, 3, 4 on a 32-bit OS & vault-linux-armv7 \\
|
Raspberry Pi, 32-bit & Pi 2, 3, 4 on a 32-bit OS & vault-linux-armv7 \\
|
||||||
Raspberry Pi Zero / 1 & the tiny older ones & vault-linux-armv6 \\
|
Raspberry Pi Zero / 1 & the tiny older ones & vault-linux-armv6 \\
|
||||||
Android & in Termux & vault-android-arm64 \\
|
Android & in Termux & vault-android-arm64 \\
|
||||||
FreeBSD, OpenBSD, NetBSD & amd64, arm64, 386, arm\ldots & vault-freebsd-amd64 etc. \\
|
FreeBSD, NetBSD & amd64 (FreeBSD also 386) & vault-freebsd-amd64 etc. \\
|
||||||
Something exotic & RISC-V, POWER, s390x, MIPS, LoongArch & see \cmd{dist/} \\
|
Something exotic & RISC-V, POWER, s390x, LoongArch, illumos, Solaris & see \cmd{dist/} \\
|
||||||
\end{vtable}
|
\end{vtable}
|
||||||
|
|
||||||
\begin{squirrel}
|
\begin{squirrel}
|
||||||
The full release covers 43 platform builds, including AIX, Solaris, illumos,
|
The release covers some twenty platforms, from Windows on ARM to IBM
|
||||||
Plan~9 and even WebAssembly. Will you ever need \file{vault-plan9-arm}?
|
mainframes, and even WebAssembly. A few that the older Go builds reached,
|
||||||
Almost certainly not. Is it pleasing that it exists? Extremely.
|
such as Plan~9, OpenBSD and MIPS, have no stable Rust toolchain yet;
|
||||||
|
\file{docs/PLATFORMS.md} keeps the list. Will you ever need
|
||||||
|
\file{vault-linux-s390x}? Almost certainly not. Is it pleasing that it
|
||||||
|
exists? Extremely.
|
||||||
\end{squirrel}
|
\end{squirrel}
|
||||||
|
|
||||||
\begin{headsup}
|
\begin{headsup}
|
||||||
@@ -299,24 +302,29 @@ Run \cmd{vault help}. You should see the full command list:
|
|||||||
|
|
||||||
\section{Building it yourself}
|
\section{Building it yourself}
|
||||||
|
|
||||||
Prefer to build from source? You need Go 1.26.6 or newer. Then:
|
Prefer to build from source? You need Rust, through \cmd{rustup}, which
|
||||||
|
fetches the exact compiler version the project pins. Then:
|
||||||
|
|
||||||
\begin{shell}
|
\begin{shell}
|
||||||
$ go build . # just your platform, produces ./vault
|
$ cargo build --release --locked # just your platform: target/release/vault
|
||||||
$ ./build.sh # every platform, into ./dist, with SHA256SUMS
|
$ ./build.sh # every platform it can, into ./dist, with SHA256SUMS
|
||||||
\end{shell}
|
\end{shell}
|
||||||
|
|
||||||
\cmd{build.sh} cross-compiles every target Go supports without C libraries.
|
\cmd{build.sh} cross-compiles Linux and Windows with \cmd{cargo-zigbuild}
|
||||||
|
and \cmd{zig}, macOS on a Mac, and the BSDs, illumos and Solaris with
|
||||||
|
\cmd{cross} and Docker; it says which targets your machine can't build.
|
||||||
32-bit ARM Linux gets two builds, armv6 and armv7. Want just a few? Set
|
32-bit ARM Linux gets two builds, armv6 and armv7. Want just a few? Set
|
||||||
\env{TARGETS}. Want them somewhere else? Set \env{OUT}.
|
\env{TARGETS}. Want them somewhere else? Set \env{OUT}.
|
||||||
|
|
||||||
\shot{build}{A partial build, followed by checking every binary against SHA256SUMS}
|
\shot{build}{A partial build, followed by checking every binary against SHA256SUMS}
|
||||||
|
|
||||||
\begin{whyso}
|
\begin{whyso}
|
||||||
The binaries are \emph{static} and \emph{reproducible}: no C code, no system
|
The Linux binaries are \emph{static}: they need no system libraries at all.
|
||||||
libraries, file paths stripped, build IDs zeroed. Two people building the same
|
Every build uses the pinned compiler and the locked versions of every
|
||||||
commit get byte-identical files, so you can check a release against your own
|
dependency, with build paths stripped, so the result doesn't depend on whose
|
||||||
build.
|
machine or which folder built it. Built with the same compiler for the same
|
||||||
|
target, the same commit should give the same file, so you can check a
|
||||||
|
release against your own build.
|
||||||
\end{whyso}
|
\end{whyso}
|
||||||
|
|
||||||
\begin{pocket}
|
\begin{pocket}
|
||||||
@@ -324,7 +332,7 @@ build.
|
|||||||
\item One file per platform. Rename it \cmd{vault} and put it on your \env{PATH}.
|
\item One file per platform. Rename it \cmd{vault} and put it on your \env{PATH}.
|
||||||
\item Always check the SHA-256 against \file{SHA256SUMS}.
|
\item Always check the SHA-256 against \file{SHA256SUMS}.
|
||||||
\item macOS: clear the quarantine flag after checking.
|
\item macOS: clear the quarantine flag after checking.
|
||||||
\item \cmd{./build.sh} builds all 43 platform binaries from source.
|
\item \cmd{./build.sh} builds every platform your machine can, from source.
|
||||||
\end{pocketlist}
|
\end{pocketlist}
|
||||||
\end{pocket}
|
\end{pocket}
|
||||||
|
|
||||||
@@ -1071,8 +1079,9 @@ who's slow at typing never blocks you.
|
|||||||
\begin{whyso}
|
\begin{whyso}
|
||||||
\textbf{What if vault crashes while holding the lock?} On Linux, macOS, the
|
\textbf{What if vault crashes while holding the lock?} On Linux, macOS, the
|
||||||
BSDs and Windows, the operating system drops the lock the moment the process
|
BSDs and Windows, the operating system drops the lock the moment the process
|
||||||
dies. Nothing to clean up. On Plan~9 and WebAssembly, which have no file
|
dies. Nothing to clean up. Older Go builds of vault use the very same locks,
|
||||||
locks, vault uses a \file{.lock} file instead. If one's ever left behind,
|
so mixing versions is safe. On WebAssembly, which has no file locks, vault
|
||||||
|
uses a \file{.lock} file instead. If one's ever left behind,
|
||||||
vault tells you after a minute exactly which file to delete.
|
vault tells you after a minute exactly which file to delete.
|
||||||
\end{whyso}
|
\end{whyso}
|
||||||
|
|
||||||
@@ -1326,7 +1335,17 @@ could not read passphrase (set VAULT\_PASSPHRASE\ldots) & No terminal to ask on,
|
|||||||
agent did not start: \ldots & \cmd{unlock} couldn't start its helper. The reason follows; commands still work, asking for the passphrase. \\
|
agent did not start: \ldots & \cmd{unlock} couldn't start its helper. The reason follows; commands still work, asking for the passphrase. \\
|
||||||
nothing was unlocked & \cmd{vault lock} found nothing to lock. Not an error. \\
|
nothing was unlocked & \cmd{vault lock} found nothing to lock. Not an error. \\
|
||||||
waiting for another vault process to finish & Not an error. Another command is writing; this one continues when it's done. \\
|
waiting for another vault process to finish & Not an error. Another command is writing; this one continues when it's done. \\
|
||||||
vault still locked after a minute\ldots & Plan~9 and WebAssembly only. If nothing else is running, delete the file it names. \\
|
vault still locked after a minute\ldots & WebAssembly only. If nothing else is running, delete the file it names. \\
|
||||||
|
cannot lock the vault: \ldots & The lock file couldn't be opened or locked; the reason follows. Check the vault folder's permissions. \\
|
||||||
|
\ldots\ changed while being added & The file was being written while vault read it. Nothing was stored; try again when it's finished. \\
|
||||||
|
\ldots\ changed after it was stored; not deleting it & \cmd{migrate} stored the file, but it changed before it could be deleted. The original is kept; the vault has the version it read. \\
|
||||||
|
\ldots\ already exists & \cmd{get} never overwrites, not even a symlink. Pick another name with \cmd{-o}. \\
|
||||||
|
\ldots\ refusing to extract through it & \cmd{get -r} found a symlink (or a file) where a folder should be inside the output folder, and stopped rather than follow it. \\
|
||||||
|
cannot lock agent memory \ldots, cannot make the agent undumpable \ldots & \cmd{unlock}'s helper couldn't protect the key in memory, so it refused it. Commands still work, asking for the passphrase. \\
|
||||||
|
vault unlock is not supported on this platform\ldots & Windows: every command asks for the passphrase. \\
|
||||||
|
download exceeds \ldots\ bytes & A download passed its size limit. Raise it with \cmd{--max-bytes} if you meant it. \\
|
||||||
|
redirected to another host: \ldots & \cmd{mirror} only follows redirects within the site it is copying. \\
|
||||||
|
mirror incomplete: stored \ldots, skipped \ldots & Some pages failed; the reasons were printed as they happened. Exit status 3. \\
|
||||||
skipping unreadable record \ldots & A catalogue entry is damaged. Everything else still works; restore \file{records/} from backup to recover that file. \\
|
skipping unreadable record \ldots & A catalogue entry is damaged. Everything else still works; restore \file{records/} from backup to recover that file. \\
|
||||||
object does not match its catalogue record & The encrypted contents were swapped or damaged. Nothing was written. Restore \file{objects/} from backup. \\
|
object does not match its catalogue record & The encrypted contents were swapped or damaged. Nothing was written. Restore \file{objects/} from backup. \\
|
||||||
object size does not match its catalogue record & Same as above: damage or tampering, caught before any output appeared. \\
|
object size does not match its catalogue record & Same as above: damage or tampering, caught before any output appeared. \\
|
||||||
|
|||||||
+21
-2
@@ -1,4 +1,4 @@
|
|||||||
.TH VAULT-ADD 1 "September 2026" "vault" "User Commands"
|
.TH VAULT-ADD 1 "October 2026" "vault" "User Commands"
|
||||||
.SH NAME
|
.SH NAME
|
||||||
vault-add \- encrypt files, directories or URLs into the vault
|
vault-add \- encrypt files, directories or URLs into the vault
|
||||||
.SH SYNOPSIS
|
.SH SYNOPSIS
|
||||||
@@ -7,6 +7,10 @@ vault-add \- encrypt files, directories or URLs into the vault
|
|||||||
.IR PATH ]
|
.IR PATH ]
|
||||||
.RB [ \-\-vault
|
.RB [ \-\-vault
|
||||||
.IR DIR ]
|
.IR DIR ]
|
||||||
|
.RB [ \-\-max\-bytes
|
||||||
|
.IR SIZE ]
|
||||||
|
.RB [ \-\-timeout
|
||||||
|
.IR DURATION ]
|
||||||
.IR FILE | DIR | URL
|
.IR FILE | DIR | URL
|
||||||
.RI [ FOLDER ]
|
.RI [ FOLDER ]
|
||||||
.SH DESCRIPTION
|
.SH DESCRIPTION
|
||||||
@@ -17,7 +21,14 @@ and the key are skipped. A stored name that already exists is replaced.
|
|||||||
.PP
|
.PP
|
||||||
An \fBhttp://\fR or \fBhttps://\fR \fIURL\fR is downloaded with a Chrome user
|
An \fBhttp://\fR or \fBhttps://\fR \fIURL\fR is downloaded with a Chrome user
|
||||||
agent and stored under the last element of its final (post-redirect) path, or
|
agent and stored under the last element of its final (post-redirect) path, or
|
||||||
the host name if there is none. Only a 200 response is accepted.
|
the host name if there is none. Only a 200 response is accepted. Certificates
|
||||||
|
are always verified; at most 10 redirects are followed, to http or https
|
||||||
|
only. The download is staged in a private temporary file (it must be
|
||||||
|
complete before it can be stored), which is removed afterwards whatever
|
||||||
|
happens; it is plaintext on disk meanwhile.
|
||||||
|
.PP
|
||||||
|
A file that changes while it is being added (it grows, shrinks, or is
|
||||||
|
replaced) is not stored, and the command fails.
|
||||||
.PP
|
.PP
|
||||||
\fIFOLDER\fR places the result inside a folder in the vault. File contents
|
\fIFOLDER\fR places the result inside a folder in the vault. File contents
|
||||||
are cut into 4 MiB chunks, the last bucket-padded.
|
are cut into 4 MiB chunks, the last bucket-padded.
|
||||||
@@ -28,6 +39,14 @@ Key file. Default: $VAULT_KEY, else ~/.vault/key.
|
|||||||
.TP
|
.TP
|
||||||
.BI \-\-vault " DIR"
|
.BI \-\-vault " DIR"
|
||||||
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
|
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
|
||||||
|
.TP
|
||||||
|
.BI \-\-max\-bytes " SIZE"
|
||||||
|
Largest download accepted, counted after decompression, such as 500M or 50G.
|
||||||
|
Default 16G; at most 1T.
|
||||||
|
.TP
|
||||||
|
.BI \-\-timeout " DURATION"
|
||||||
|
Deadline for the whole download, such as 30m. Default 2h. Connecting must
|
||||||
|
take under 30 seconds and the response headers must arrive within a minute.
|
||||||
.SH ENVIRONMENT
|
.SH ENVIRONMENT
|
||||||
.TP
|
.TP
|
||||||
.B VAULT_PASSPHRASE
|
.B VAULT_PASSPHRASE
|
||||||
|
|||||||
+9
-4
@@ -1,4 +1,4 @@
|
|||||||
.TH VAULT-GET 1 "September 2026" "vault" "User Commands"
|
.TH VAULT-GET 1 "October 2026" "vault" "User Commands"
|
||||||
.SH NAME
|
.SH NAME
|
||||||
vault-get \- decrypt a stored file
|
vault-get \- decrypt a stored file
|
||||||
.SH SYNOPSIS
|
.SH SYNOPSIS
|
||||||
@@ -13,13 +13,18 @@ vault-get \- decrypt a stored file
|
|||||||
.I NAME
|
.I NAME
|
||||||
.SH DESCRIPTION
|
.SH DESCRIPTION
|
||||||
Decrypts \fINAME\fR. Nothing appears at the output path until every object has
|
Decrypts \fINAME\fR. Nothing appears at the output path until every object has
|
||||||
authenticated. The stored mode and modification time are restored. It refuses
|
authenticated, padding and end of file included; until then the output is a
|
||||||
to overwrite an existing file. Takes a shared lock.
|
private temporary file in the same directory, removed on failure. The stored
|
||||||
|
mode and modification time are restored. It never replaces anything already
|
||||||
|
at the output path, including a symbolic link (even a dangling one) or a
|
||||||
|
file created while it runs. Takes a shared lock.
|
||||||
.PP
|
.PP
|
||||||
With \fB\-r\fR, \fINAME\fR is a folder, and every file under it is
|
With \fB\-r\fR, \fINAME\fR is a folder, and every file under it is
|
||||||
decrypted into the output directory, keeping the folder structure. Each file
|
decrypted into the output directory, keeping the folder structure. Each file
|
||||||
is checked as above; a clash stops the run, leaving the files already
|
is checked as above; a clash stops the run, leaving the files already
|
||||||
written.
|
written. Directories are created as needed, mode 0700; a symbolic link (or
|
||||||
|
anything but a directory) where a directory belongs inside the output
|
||||||
|
directory stops the run rather than being followed.
|
||||||
.SH OPTIONS
|
.SH OPTIONS
|
||||||
.TP
|
.TP
|
||||||
.BI \-o " OUTPUT"
|
.BI \-o " OUTPUT"
|
||||||
|
|||||||
@@ -1,4 +1,4 @@
|
|||||||
.TH VAULT-MIGRATE 1 "September 2026" "vault" "User Commands"
|
.TH VAULT-MIGRATE 1 "October 2026" "vault" "User Commands"
|
||||||
.SH NAME
|
.SH NAME
|
||||||
vault-migrate \- move files into the vault, deleting the originals
|
vault-migrate \- move files into the vault, deleting the originals
|
||||||
.SH SYNOPSIS
|
.SH SYNOPSIS
|
||||||
@@ -13,7 +13,10 @@ vault-migrate \- move files into the vault, deleting the originals
|
|||||||
.SH DESCRIPTION
|
.SH DESCRIPTION
|
||||||
Works like
|
Works like
|
||||||
.BR vault\-add (1)
|
.BR vault\-add (1)
|
||||||
but deletes each original once it is stored, then any directories left empty.
|
but deletes each original once it is stored (and its record is on disk),
|
||||||
|
then any directories left empty. An original is deleted only if it is still
|
||||||
|
the same file, unchanged, that was stored; otherwise it is kept and the
|
||||||
|
command fails.
|
||||||
Directories still holding skipped files stay. A URL cannot be migrated.
|
Directories still holding skipped files stay. A URL cannot be migrated.
|
||||||
.PP
|
.PP
|
||||||
Before storing anything it lists the vault names that would be overwritten and
|
Before storing anything it lists the vault names that would be overwritten and
|
||||||
|
|||||||
+60
-17
@@ -1,4 +1,4 @@
|
|||||||
.TH VAULT-MIRROR 1 "September 2026" "vault" "User Commands"
|
.TH VAULT-MIRROR 1 "October 2026" "vault" "User Commands"
|
||||||
.SH NAME
|
.SH NAME
|
||||||
vault-mirror \- mirror a website into an encrypted vault
|
vault-mirror \- mirror a website into an encrypted vault
|
||||||
.SH SYNOPSIS
|
.SH SYNOPSIS
|
||||||
@@ -7,6 +7,12 @@ vault-mirror \- mirror a website into an encrypted vault
|
|||||||
.IR PATH ]
|
.IR PATH ]
|
||||||
.RB [ \-\-vault
|
.RB [ \-\-vault
|
||||||
.IR DIR ]
|
.IR DIR ]
|
||||||
|
.RB [ \-\-max\-pages
|
||||||
|
.IR N ]
|
||||||
|
.RB [ \-\-max\-bytes
|
||||||
|
.IR SIZE ]
|
||||||
|
.RB [ \-\-timeout
|
||||||
|
.IR DURATION ]
|
||||||
.I URL
|
.I URL
|
||||||
.RI [ FOLDER ]
|
.RI [ FOLDER ]
|
||||||
.SH DESCRIPTION
|
.SH DESCRIPTION
|
||||||
@@ -20,21 +26,30 @@ and stores every page and asset it finds in the vault, beneath
|
|||||||
if no folder is given).
|
if no folder is given).
|
||||||
.PP
|
.PP
|
||||||
The crawl is single threaded: one request at a time, in breadth-first order.
|
The crawl is single threaded: one request at a time, in breadth-first order.
|
||||||
It follows only links on the same host as
|
It follows only links on the same host (and port) as
|
||||||
.IR URL ,
|
.IR URL ,
|
||||||
and requests are sent with a Chrome user agent string.
|
and only redirects to that host; a redirect elsewhere skips the page.
|
||||||
|
Requests are sent with a Chrome user agent string, and certificates are
|
||||||
|
always verified.
|
||||||
.PP
|
.PP
|
||||||
Links in HTML
|
Links in HTML start tags
|
||||||
.RB ( href ,
|
.RB ( href ,
|
||||||
.BR src ,
|
.BR src ,
|
||||||
.BR action
|
.BR action
|
||||||
and
|
and
|
||||||
.B poster
|
.B poster
|
||||||
attributes) and in CSS
|
attributes) and CSS
|
||||||
.RB ( url(...) )
|
.RB ( url(...)
|
||||||
that point into the mirror are rewritten to relative paths, so the copy
|
in style sheets,
|
||||||
browses correctly once fetched out of the vault. Links to other hosts, and
|
.B <style>
|
||||||
fragments, are left as they were.
|
elements and
|
||||||
|
.B style
|
||||||
|
attributes) that point into the mirror are rewritten to relative paths, so
|
||||||
|
the copy browses correctly once fetched out of the vault. Relative links are
|
||||||
|
resolved against the page's final address and its
|
||||||
|
.B <base href>
|
||||||
|
if it has one. Links to other hosts, script and comment text, and every
|
||||||
|
other byte of the page are left as they were. Fragments are kept.
|
||||||
.PP
|
.PP
|
||||||
Pages are named after their URL path. A path that is empty or ends in
|
Pages are named after their URL path. A path that is empty or ends in
|
||||||
.B /
|
.B /
|
||||||
@@ -42,19 +57,39 @@ becomes
|
|||||||
.BR index.html ;
|
.BR index.html ;
|
||||||
a query string is appended as
|
a query string is appended as
|
||||||
.BI _ query
|
.BI _ query
|
||||||
(URL-escaped). Anything already stored under the same name is replaced.
|
(URL-escaped). A name already stored by an earlier run is replaced, but if
|
||||||
|
two different URLs in one crawl would get the same name, the second is
|
||||||
|
skipped rather than overwriting the first.
|
||||||
.PP
|
.PP
|
||||||
A page that fails to download, or answers with anything but
|
A page that fails to download, answers with anything but
|
||||||
.BR "200 OK" ,
|
.BR "200 OK" ,
|
||||||
is reported on standard error and skipped; the crawl carries on.
|
or exceeds a limit is reported on standard error as
|
||||||
The vault is locked exclusively for the whole run.
|
.BI "skipping " URL ": " reason
|
||||||
|
and skipped; the crawl carries on. HTML and CSS are rewritten in memory;
|
||||||
|
other files are staged in a private temporary file (plaintext on disk
|
||||||
|
meanwhile) that is always removed. The vault is locked exclusively for the
|
||||||
|
whole run.
|
||||||
.SH OPTIONS
|
.SH OPTIONS
|
||||||
.TP
|
.TP
|
||||||
|
.BI \-\-max\-pages " N"
|
||||||
|
Pages and assets to fetch at most. Default 10000.
|
||||||
|
.TP
|
||||||
|
.BI \-\-max\-bytes " SIZE"
|
||||||
|
Total bytes to download at most, counted after decompression. Default 4G.
|
||||||
|
.TP
|
||||||
|
.BI \-\-timeout " DURATION"
|
||||||
|
Deadline for each request. Default 5m.
|
||||||
|
.TP
|
||||||
.BI \-\-key " PATH"
|
.BI \-\-key " PATH"
|
||||||
Key file. Default: $VAULT_KEY, else ~/.vault/key.
|
Key file. Default: $VAULT_KEY, else ~/.vault/key.
|
||||||
.TP
|
.TP
|
||||||
.BI \-\-vault " DIR"
|
.BI \-\-vault " DIR"
|
||||||
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
|
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
|
||||||
|
.PP
|
||||||
|
A single HTML or CSS file larger than 16 MiB is skipped.
|
||||||
|
.SH EXIT STATUS
|
||||||
|
0 when everything found was stored, 3 when some of it was skipped, 1 on
|
||||||
|
an error that stops the crawl.
|
||||||
.SH ENVIRONMENT
|
.SH ENVIRONMENT
|
||||||
.TP
|
.TP
|
||||||
.B VAULT_PASSPHRASE
|
.B VAULT_PASSPHRASE
|
||||||
@@ -75,11 +110,19 @@ vault mirror https://example.com/ archive/2026
|
|||||||
vault get archive/2026/example.com/index.html
|
vault get archive/2026/example.com/index.html
|
||||||
.RE
|
.RE
|
||||||
.SH LIMITATIONS
|
.SH LIMITATIONS
|
||||||
Links are found by pattern matching, not by parsing HTML: URLs assembled by
|
URLs assembled by scripts, and
|
||||||
scripts, and
|
|
||||||
.B srcset
|
.B srcset
|
||||||
lists, are neither followed nor rewritten. There is no depth limit, rate
|
lists, are neither followed nor rewritten. In attribute values only numeric
|
||||||
limit or robots.txt handling; only mirror sites you may copy.
|
character references and
|
||||||
|
.BR & ,
|
||||||
|
.BR < ,
|
||||||
|
.BR > ,
|
||||||
|
.BR " ,
|
||||||
|
.B '
|
||||||
|
and
|
||||||
|
.B
|
||||||
|
are decoded. There is no rate limit or robots.txt handling; only mirror
|
||||||
|
sites you may copy.
|
||||||
.SH SEE ALSO
|
.SH SEE ALSO
|
||||||
.BR vault (1)
|
.BR vault (1)
|
||||||
.SH AUTHOR
|
.SH AUTHOR
|
||||||
|
|||||||
@@ -32,6 +32,14 @@ memory is locked against swapping and excluded from core dumps; on Linux the
|
|||||||
helper is also not debuggable. It answers only on a Unix socket inside
|
helper is also not debuggable. It answers only on a Unix socket inside
|
||||||
\fI$XDG_RUNTIME_DIR/vault-UID\fR (or the temporary directory), a directory
|
\fI$XDG_RUNTIME_DIR/vault-UID\fR (or the temporary directory), a directory
|
||||||
that must belong to this user with mode 0700. Rebooting ends it too.
|
that must belong to this user with mode 0700. Rebooting ends it too.
|
||||||
|
If the helper cannot lock its memory, disable core dumps or (on Linux) make
|
||||||
|
itself undebuggable, it refuses the key and
|
||||||
|
.B unlock
|
||||||
|
fails, saying why. The helper is the same program, started with the
|
||||||
|
passphrase variables removed from its environment. Helpers started by
|
||||||
|
earlier (Go) versions of vault are compatible and can be used and locked.
|
||||||
|
.PP
|
||||||
|
Not available on Windows, where every command asks for the passphrase.
|
||||||
.PP
|
.PP
|
||||||
While the key is unlocked, any program running as this user can use it, and
|
While the key is unlocked, any program running as this user can use it, and
|
||||||
it is whichever key the passphrase opened, real or decoy. Lock before
|
it is whichever key the passphrase opened, real or decoy. Lock before
|
||||||
|
|||||||
+23
-2
@@ -1,4 +1,4 @@
|
|||||||
.TH VAULT 1 "September 2026" "vault" "User Commands"
|
.TH VAULT 1 "October 2026" "vault" "User Commands"
|
||||||
.SH NAME
|
.SH NAME
|
||||||
vault \- post-quantum encrypted file vault
|
vault \- post-quantum encrypted file vault
|
||||||
.SH SYNOPSIS
|
.SH SYNOPSIS
|
||||||
@@ -77,6 +77,11 @@ Key file. Default: $VAULT_KEY, else ~/.vault/key.
|
|||||||
.TP
|
.TP
|
||||||
.BI \-\-vault " DIR"
|
.BI \-\-vault " DIR"
|
||||||
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
|
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
|
||||||
|
.TP
|
||||||
|
.BI \-\-workers " N"
|
||||||
|
Threads for encryption, decryption and file I/O, 1 to 256. Default: the
|
||||||
|
number of CPUs, at most 8. Up to eight 4 MiB objects are in flight at once
|
||||||
|
whatever the count.
|
||||||
.SH ENVIRONMENT
|
.SH ENVIRONMENT
|
||||||
.TP
|
.TP
|
||||||
.B VAULT_KEY
|
.B VAULT_KEY
|
||||||
@@ -92,6 +97,19 @@ Passphrase, instead of prompting on the terminal or using a key remembered by
|
|||||||
.B VAULT_DECOY_PASSPHRASE
|
.B VAULT_DECOY_PASSPHRASE
|
||||||
Decoy passphrase used by
|
Decoy passphrase used by
|
||||||
.BR "vault genkey" .
|
.BR "vault genkey" .
|
||||||
|
.TP
|
||||||
|
.BR HTTPS_PROXY ", " HTTP_PROXY ", " NO_PROXY
|
||||||
|
Proxies for
|
||||||
|
.B add
|
||||||
|
.I URL
|
||||||
|
and
|
||||||
|
.BR mirror ,
|
||||||
|
as other programs use them; local addresses are never proxied.
|
||||||
|
.TP
|
||||||
|
.BR XDG_RUNTIME_DIR ", " TMPDIR
|
||||||
|
Where
|
||||||
|
.BR vault\-unlock (1)
|
||||||
|
keeps its socket, and where downloads are staged.
|
||||||
.SH LOCKING
|
.SH LOCKING
|
||||||
Commands that change the vault hold an exclusive lock on
|
Commands that change the vault hold an exclusive lock on
|
||||||
.IR DIR/.lock ;
|
.IR DIR/.lock ;
|
||||||
@@ -106,7 +124,10 @@ and
|
|||||||
fills each vault with unreadable filler, so hidden files look like filler to
|
fills each vault with unreadable filler, so hidden files look like filler to
|
||||||
anyone holding only the decoy passphrase.
|
anyone holding only the decoy passphrase.
|
||||||
.SH EXIT STATUS
|
.SH EXIT STATUS
|
||||||
0 on success; 1 on error (diagnostics on standard error); 2 when run with no command.
|
0 on success; 1 on error (diagnostics on standard error); 2 when run with no
|
||||||
|
command; 3 when
|
||||||
|
.BR vault\-mirror (1)
|
||||||
|
stored some resources but had to skip others.
|
||||||
.SH EXAMPLES
|
.SH EXAMPLES
|
||||||
.nf
|
.nf
|
||||||
vault genkey
|
vault genkey
|
||||||
|
|||||||
@@ -0,0 +1,57 @@
|
|||||||
|
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||||
|
|
||||||
|
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
|
||||||
|
|
||||||
|
package pqcrypt
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"io"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func BenchmarkOracleDerive(b *testing.B) {
|
||||||
|
seed := pattern(32, 1)
|
||||||
|
for b.Loop() {
|
||||||
|
id, _ := NewPQIdentityFromSeed(0x0001, seed)
|
||||||
|
id.Close()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func BenchmarkOracleWrap(b *testing.B) {
|
||||||
|
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
|
||||||
|
r := id.Recipient()
|
||||||
|
fk := pattern(32, 2)
|
||||||
|
for b.Loop() {
|
||||||
|
r.Wrap(fk)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func BenchmarkOracleUnwrap(b *testing.B) {
|
||||||
|
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
|
||||||
|
st, _ := id.Recipient().Wrap(pattern(32, 2))
|
||||||
|
for b.Loop() {
|
||||||
|
if _, err := id.Unwrap(st); err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Whole-file encryption and decryption of 4 MiB: one vault object.
|
||||||
|
func BenchmarkOracleObject4MiB(b *testing.B) {
|
||||||
|
id, _ := NewPQIdentityFromSeed(0x0001, pattern(32, 1))
|
||||||
|
pt := pattern(4<<20, 3)
|
||||||
|
var ct bytes.Buffer
|
||||||
|
b.SetBytes(int64(len(pt)))
|
||||||
|
for b.Loop() {
|
||||||
|
ct.Reset()
|
||||||
|
w, _ := Encrypt(&ct, id.Recipient())
|
||||||
|
w.Write(pt)
|
||||||
|
w.Close()
|
||||||
|
r, err := Decrypt(bytes.NewReader(ct.Bytes()), id)
|
||||||
|
if err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
io.Copy(io.Discard, r)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||||
|
|
||||||
|
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
|
||||||
|
|
||||||
|
package stream
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"io"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// STREAM seal and open of 64 MiB, one goroutine (the sequential path).
|
||||||
|
func BenchmarkOracleStream64MiB(b *testing.B) {
|
||||||
|
key := make([]byte, 32)
|
||||||
|
pt := make([]byte, 64<<20)
|
||||||
|
var ct bytes.Buffer
|
||||||
|
b.SetBytes(int64(len(pt)))
|
||||||
|
for b.Loop() {
|
||||||
|
ct.Reset()
|
||||||
|
w, _ := NewEncryptWriter(key, &ct)
|
||||||
|
w.Write(pt)
|
||||||
|
w.Close()
|
||||||
|
r, _ := NewDecryptReader(key, bytes.NewReader(ct.Bytes()))
|
||||||
|
io.Copy(io.Discard, r)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
// SPDX-License-Identifier: GPL-3.0-or-later
|
||||||
|
|
||||||
|
// Test-only benchmarks for comparing with the Rust port (scripts/bench.sh).
|
||||||
|
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Unlocking a key file: Argon2id for both slots.
|
||||||
|
func BenchmarkOracleUnlock(b *testing.B) {
|
||||||
|
p := filepath.Join(b.TempDir(), "key")
|
||||||
|
if err := writeKeyFile(p, "bench", ""); err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
for b.Loop() {
|
||||||
|
if _, err := unlockKeyFile(p, "bench"); err != nil {
|
||||||
|
b.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Executable
+96
@@ -0,0 +1,96 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Compare the Go oracle and the Rust build on this machine, end to end.
|
||||||
|
#
|
||||||
|
# scripts/bench.sh [WORKDIR] (default: target/bench, on the repo's disk)
|
||||||
|
# RUNS=5 BIG_MB=2048 SMALL=5000 scripts/bench.sh
|
||||||
|
#
|
||||||
|
# For each implementation, in a fresh key and vault on the same filesystem:
|
||||||
|
# add one BIG_MB file, get it back, add SMALL small files (1-16 KiB) as a
|
||||||
|
# folder, list the catalogue (filler plus SMALL+1 records), and get one small
|
||||||
|
# file. Each step is timed RUNS times (median, min, max), with peak RSS and
|
||||||
|
# peak thread count. Every step includes unlocking the key (two Argon2id
|
||||||
|
# runs). Both implementations fsync every object and record, so `add`
|
||||||
|
# timings include durable writes. Page caches are warm: dropping them needs
|
||||||
|
# root, so cold-cache numbers are not measured here.
|
||||||
|
#
|
||||||
|
# Needs: go (to build the oracle), cargo, /usr/bin/time (GNU).
|
||||||
|
set -euo pipefail
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
root=$PWD
|
||||||
|
work=${1:-$root/target/bench}
|
||||||
|
runs=${RUNS:-3}
|
||||||
|
big_mb=${BIG_MB:-1024}
|
||||||
|
small=${SMALL:-2000}
|
||||||
|
|
||||||
|
mkdir -p -- "$work"
|
||||||
|
work=$(cd -- "$work" && pwd -P)
|
||||||
|
[[ $work == "$root" || $work == / ]] && { echo "refusing WORKDIR=$work" >&2; exit 2; }
|
||||||
|
|
||||||
|
cargo build --release --locked --quiet
|
||||||
|
(cd oracle/go && CGO_ENABLED=0 go build -trimpath -o "$work/vault-go" .)
|
||||||
|
cp target/release/vault "$work/vault-rust"
|
||||||
|
|
||||||
|
data=$work/data
|
||||||
|
if [[ ! -f $data/big || $(stat -c %s "$data/big") != $((big_mb << 20)) ]]; then
|
||||||
|
rm -rf -- "$data"
|
||||||
|
mkdir -p -- "$data/small"
|
||||||
|
head -c $((big_mb << 20)) /dev/urandom >"$data/big"
|
||||||
|
for i in $(seq -w 1 "$small"); do
|
||||||
|
head -c $((1024 + RANDOM % 15360)) /dev/urandom >"$data/small/f$i"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
export VAULT_PASSPHRASE=bench-passphrase
|
||||||
|
|
||||||
|
# measure LABEL CMD...: one timed run, printing "seconds rss_kib threads".
|
||||||
|
measure() {
|
||||||
|
local out=$work/time.$$ peak=0 pid t
|
||||||
|
/usr/bin/time -f '%e %M' -o "$out" "$@" >/dev/null 2>&1 &
|
||||||
|
pid=$!
|
||||||
|
while kill -0 $pid 2>/dev/null; do
|
||||||
|
for t in /proc/[0-9]*/status; do
|
||||||
|
if grep -q "^PPid:[[:space:]]*$pid$" "$t" 2>/dev/null; then
|
||||||
|
n=$(awk '/^Threads:/{print $2}' "$t" 2>/dev/null || echo 0)
|
||||||
|
((n > peak)) && peak=$n
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
sleep 0.05
|
||||||
|
done
|
||||||
|
wait $pid
|
||||||
|
echo "$(cat "$out") $peak"
|
||||||
|
rm -f -- "$out"
|
||||||
|
}
|
||||||
|
|
||||||
|
summarise() { # lines of "secs rss threads" -> median min max, max rss, max threads
|
||||||
|
sort -n | awk '{t[NR]=$1; if($2>r)r=$2; if($3>th)th=$3} END {printf "%8.2f s (min %.2f, max %.2f) rss %6.0f MiB threads %d\n", t[int((NR+1)/2)], t[1], t[NR], r/1024, th}'
|
||||||
|
}
|
||||||
|
|
||||||
|
for impl in go rust; do
|
||||||
|
bin=$work/vault-$impl
|
||||||
|
echo "== $impl ($(stat -c %s "$bin") bytes)"
|
||||||
|
for step in add-big get-big add-small list get-small; do
|
||||||
|
results=""
|
||||||
|
for run in $(seq 1 "$runs"); do
|
||||||
|
v=$work/run-$impl
|
||||||
|
rm -rf -- "$v"
|
||||||
|
mkdir -p -- "$v"
|
||||||
|
export VAULT_KEY=$v/key VAULT_DIR=$v/store
|
||||||
|
"$bin" genkey 2>/dev/null
|
||||||
|
"$bin" init 2>/dev/null
|
||||||
|
case $step in
|
||||||
|
get-big | list | get-small) "$bin" add "$data/big" 2>/dev/null; "$bin" add "$data/small" 2>/dev/null ;;
|
||||||
|
esac
|
||||||
|
case $step in
|
||||||
|
add-big) r=$(measure "$bin" add "$data/big") ;;
|
||||||
|
get-big) r=$(measure "$bin" get big -o "$v/big.out") ;;
|
||||||
|
add-small) r=$(measure "$bin" add "$data/small") ;;
|
||||||
|
list) r=$(measure "$bin" list) ;;
|
||||||
|
get-small) r=$(measure "$bin" get "small/f$(printf "%0${#small}d" 1)" -o "$v/small.out") ;;
|
||||||
|
esac
|
||||||
|
results+="$r"$'\n'
|
||||||
|
done
|
||||||
|
printf ' %-10s ' "$step"
|
||||||
|
printf '%s' "$results" | grep . | summarise
|
||||||
|
done
|
||||||
|
rm -rf -- "$work/run-$impl"
|
||||||
|
done
|
||||||
Reference in New Issue
Block a user