Docs for the Rust version; benchmarks; build.sh fixes

README, CLAUDE.md, man pages, the user guide and its screenshot script
now describe the Rust build and the intentional behaviour changes, which
docs/MIGRATION.md lists. Go and Rust micro-benchmarks for the same
operations, and scripts/bench.sh for end-to-end comparison. build.sh
exports its tree root to the per-target jobs and keeps logs out of the
output directory; riscv64 musl builds use getrandom's raw syscall
backend, since zig's musl getrandom never reaches the kernel there.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Andy Dixon
2026-10-08 11:50:55 +01:00
co-authored by Claude
parent cddde5ca52
commit 53968f8822
18 changed files with 659 additions and 109 deletions
+95
View File
@@ -0,0 +1,95 @@
# Migrating from the Go implementation
vault 0.6 is a rewrite in Rust of the Go vault (0.5 and earlier). It is an
implementation migration only: key files, vaults, the lock and the unlock
agent are the same formats, byte for byte where the format is
deterministic (see `COMPATIBILITY.md`). There is nothing to convert. Go and
Rust builds can be used on the same vault, even at the same time: they
honour each other's locks, and either can use an agent the other started.
To go back, install 0.5: everything 0.6 writes, 0.5 reads.
## Intentional behaviour changes
Safety and correctness:
- **`get` never replaces anything.** Output is published with an atomic
no-replace rename or link, so a file, a symlink (even a dangling one) or a
file created while `get` runs is left alone. Go checked first and then
renamed, which a concurrent creator could race.
- **`get -r` does not follow symlinks inside the output directory.** Each
directory below the chosen one is opened without following links; a link
or non-directory in the way stops the run. (Windows: checked just before
use, see `SECURITY.md`.)
- **Extracted files are synced** to disk before they appear under their
name, and their directory afterwards.
- **A source file that changes while being added is not stored.** If it
grows, shrinks, is modified or replaced during `add`/`migrate`, the
command fails and nothing is stored. Go stored whatever it read (for a
file that grew, a truncated copy).
- **`migrate` deletes an original only if it is still the file stored**
(same inode, size and time), after the record is durable.
- **Deletes are durable in order.** Removing a file syncs the removal of its
record before deleting its objects, so a crash cannot leave a record
pointing at deleted objects. A failed sync after a record is already in
place no longer deletes its objects.
- **Lock errors are reported.** Go let a reader continue without the lock on
any error; Rust does so only on a read-only filesystem.
- **Stricter records.** A record must have every field, with exactly the
field names Go writes, and nothing after the JSON. Go also accepted
differently-cased names, missing fields (as zero) and trailing bytes;
Go never writes such records, so no existing vault is affected.
- **Key files over 64 KiB** are rejected without reading them whole.
Agent (`unlock`):
- The agent **refuses the key if it cannot protect it**: locking its memory,
disabling core dumps and (Linux) becoming undumpable must all succeed. Go
ignored failures there.
- Its environment is stripped of `VAULT_PASSPHRASE` and
`VAULT_DECOY_PASSPHRASE`; it runs in `/` with standard error closed.
- Expiry no longer waits for the request in progress beyond its deadline,
and a key is never handed out after expiry.
- `unlock` waits at most ten seconds for the agent to start and kills it on
failure; the agent removes its socket only if it is still its own.
- **Windows: `unlock` is not available** (Go's agent there had no memory
protection). Every command asks for the passphrase.
Network (`add URL`, `mirror`):
- Explicit limits, all adjustable: connect 30 s, response headers 60 s,
whole request 2 h (`add`) or 5 min (`mirror`, per request), download size
16 GiB (`add`) or 4 GiB in total (`mirror`), 10 redirects, 10 000 mirror
pages, 16 MiB per HTML or CSS page. Bytes are counted after decompression.
New flags `--max-bytes`, `--timeout`, `--max-pages`. Go had no limits.
- Redirects go to http or https only; a `mirror` redirect to another host is
skipped (Go followed it).
- `mirror` resolves links against the final address after redirects and the
page's `<base href>`; it parses HTML instead of matching a pattern, so it
rewrites only `href`, `src`, `action` and `poster` on real tags, plus
`url(...)` in CSS, `<style>` and `style=`, keeping every other byte and
the original quoting. Text in scripts and comments is no longer touched.
- `mirror` never lets two different URLs in one crawl overwrite each other's
file; the second is skipped. It exits with status **3** when it stored
some resources but skipped others (Go always exited 0).
- Proxies follow `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` as in Go; local
addresses are never proxied.
Command line:
- New option `--workers N` (1–256) on every command.
- `--key`/`--vault` no longer need a home directory when given explicitly.
- Command-line arguments must be valid UTF-8 (Go accepted any bytes, but
vault names had to be UTF-8 already).
- `ls`/`list` piped into a reader that stops early end quietly with
status 141, as Go's did by `SIGPIPE`.
Platforms: see `PLATFORMS.md`. Some targets Go built have no stable Rust
equivalent; they are listed there and need a decision before release.
## Unchanged on purpose
Formats, limits (1024-byte names, 1 TiB files, 4 MiB objects, 4096-byte
records, 16 MiB headers, 1 MiB stanzas), Argon2id parameters, the hybrid
suite, filler, decoy behaviour, messages and exit statuses (apart from the
above), the shape of `ls` and `list` output, and the agent protocol.
+6 -3
View File
@@ -7,7 +7,9 @@ sandbox path and file owner are shown as /home/sam and "sam sam", key lines
are highlighted in amber for the reader, and "moment" frames show a prompt
with a cursor, exactly as vault prints it before you type.
Needs: go, python3, and Chrome or Chromium (set CHROME=/path if not found).
Needs: cargo (with the pinned toolchain), cargo-zigbuild and zig for the
build screenshot, python3, and Chrome or Chromium (set CHROME=/path if not
found).
Output: docs/guide/img/*.png
"""
@@ -75,7 +77,8 @@ class Sandbox:
self.bin = os.path.join(self.root, "bin")
os.makedirs(self.home)
os.makedirs(self.bin)
subprocess.run(["go", "build", "-o", os.path.join(self.bin, "vault"), "."], cwd=REPO, check=True)
subprocess.run(["cargo", "build", "--release", "--locked", "--quiet"], cwd=REPO, check=True)
shutil.copy2(os.path.join(REPO, "target", "release", "vault"), os.path.join(self.bin, "vault"))
def env(self, extra=None):
e = {
@@ -406,7 +409,7 @@ def main():
# building, into a temporary OUT so an existing dist/ is left alone
rel = tempfile.mkdtemp(prefix="vault-release-")
targets = "linux/amd64 linux/arm darwin/arm64 windows/amd64"
targets = "linux/amd64 linux/arm64 linux/armv7 windows/amd64"
t = Term(box, title="sam@laptop: ~/src/vault")
out = subprocess.run(["./build.sh"], cwd=REPO, stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
text=True, env=dict(os.environ, TARGETS=targets, OUT=rel)).stdout
+36 -17
View File
@@ -222,14 +222,17 @@ Linux on ARM64 & Pi 4/5 (64-bit OS), Graviton, Ampere & vault-linux-arm64 \\
Raspberry Pi, 32-bit & Pi 2, 3, 4 on a 32-bit OS & vault-linux-armv7 \\
Raspberry Pi Zero / 1 & the tiny older ones & vault-linux-armv6 \\
Android & in Termux & vault-android-arm64 \\
FreeBSD, OpenBSD, NetBSD & amd64, arm64, 386, arm\ldots & vault-freebsd-amd64 etc. \\
Something exotic & RISC-V, POWER, s390x, MIPS, LoongArch & see \cmd{dist/} \\
FreeBSD, NetBSD & amd64 (FreeBSD also 386) & vault-freebsd-amd64 etc. \\
Something exotic & RISC-V, POWER, s390x, LoongArch, illumos, Solaris & see \cmd{dist/} \\
\end{vtable}
\begin{squirrel}
The full release covers 43 platform builds, including AIX, Solaris, illumos,
Plan~9 and even WebAssembly. Will you ever need \file{vault-plan9-arm}?
Almost certainly not. Is it pleasing that it exists? Extremely.
The release covers some twenty platforms, from Windows on ARM to IBM
mainframes, and even WebAssembly. A few that the older Go builds reached,
such as Plan~9, OpenBSD and MIPS, have no stable Rust toolchain yet;
\file{docs/PLATFORMS.md} keeps the list. Will you ever need
\file{vault-linux-s390x}? Almost certainly not. Is it pleasing that it
exists? Extremely.
\end{squirrel}
\begin{headsup}
@@ -299,24 +302,29 @@ Run \cmd{vault help}. You should see the full command list:
\section{Building it yourself}
Prefer to build from source? You need Go 1.26.6 or newer. Then:
Prefer to build from source? You need Rust, through \cmd{rustup}, which
fetches the exact compiler version the project pins. Then:
\begin{shell}
$ go build . # just your platform, produces ./vault
$ ./build.sh # every platform, into ./dist, with SHA256SUMS
$ cargo build --release --locked # just your platform: target/release/vault
$ ./build.sh # every platform it can, into ./dist, with SHA256SUMS
\end{shell}
\cmd{build.sh} cross-compiles every target Go supports without C libraries.
\cmd{build.sh} cross-compiles Linux and Windows with \cmd{cargo-zigbuild}
and \cmd{zig}, macOS on a Mac, and the BSDs, illumos and Solaris with
\cmd{cross} and Docker; it says which targets your machine can't build.
32-bit ARM Linux gets two builds, armv6 and armv7. Want just a few? Set
\env{TARGETS}. Want them somewhere else? Set \env{OUT}.
\shot{build}{A partial build, followed by checking every binary against SHA256SUMS}
\begin{whyso}
The binaries are \emph{static} and \emph{reproducible}: no C code, no system
libraries, file paths stripped, build IDs zeroed. Two people building the same
commit get byte-identical files, so you can check a release against your own
build.
The Linux binaries are \emph{static}: they need no system libraries at all.
Every build uses the pinned compiler and the locked versions of every
dependency, with build paths stripped, so the result doesn't depend on whose
machine or which folder built it. Built with the same compiler for the same
target, the same commit should give the same file, so you can check a
release against your own build.
\end{whyso}
\begin{pocket}
@@ -324,7 +332,7 @@ build.
\item One file per platform. Rename it \cmd{vault} and put it on your \env{PATH}.
\item Always check the SHA-256 against \file{SHA256SUMS}.
\item macOS: clear the quarantine flag after checking.
\item \cmd{./build.sh} builds all 43 platform binaries from source.
\item \cmd{./build.sh} builds every platform your machine can, from source.
\end{pocketlist}
\end{pocket}
@@ -1071,8 +1079,9 @@ who's slow at typing never blocks you.
\begin{whyso}
\textbf{What if vault crashes while holding the lock?} On Linux, macOS, the
BSDs and Windows, the operating system drops the lock the moment the process
dies. Nothing to clean up. On Plan~9 and WebAssembly, which have no file
locks, vault uses a \file{.lock} file instead. If one's ever left behind,
dies. Nothing to clean up. Older Go builds of vault use the very same locks,
so mixing versions is safe. On WebAssembly, which has no file locks, vault
uses a \file{.lock} file instead. If one's ever left behind,
vault tells you after a minute exactly which file to delete.
\end{whyso}
@@ -1326,7 +1335,17 @@ could not read passphrase (set VAULT\_PASSPHRASE\ldots) & No terminal to ask on,
agent did not start: \ldots & \cmd{unlock} couldn't start its helper. The reason follows; commands still work, asking for the passphrase. \\
nothing was unlocked & \cmd{vault lock} found nothing to lock. Not an error. \\
waiting for another vault process to finish & Not an error. Another command is writing; this one continues when it's done. \\
vault still locked after a minute\ldots & Plan~9 and WebAssembly only. If nothing else is running, delete the file it names. \\
vault still locked after a minute\ldots & WebAssembly only. If nothing else is running, delete the file it names. \\
cannot lock the vault: \ldots & The lock file couldn't be opened or locked; the reason follows. Check the vault folder's permissions. \\
\ldots\ changed while being added & The file was being written while vault read it. Nothing was stored; try again when it's finished. \\
\ldots\ changed after it was stored; not deleting it & \cmd{migrate} stored the file, but it changed before it could be deleted. The original is kept; the vault has the version it read. \\
\ldots\ already exists & \cmd{get} never overwrites, not even a symlink. Pick another name with \cmd{-o}. \\
\ldots\ refusing to extract through it & \cmd{get -r} found a symlink (or a file) where a folder should be inside the output folder, and stopped rather than follow it. \\
cannot lock agent memory \ldots, cannot make the agent undumpable \ldots & \cmd{unlock}'s helper couldn't protect the key in memory, so it refused it. Commands still work, asking for the passphrase. \\
vault unlock is not supported on this platform\ldots & Windows: every command asks for the passphrase. \\
download exceeds \ldots\ bytes & A download passed its size limit. Raise it with \cmd{--max-bytes} if you meant it. \\
redirected to another host: \ldots & \cmd{mirror} only follows redirects within the site it is copying. \\
mirror incomplete: stored \ldots, skipped \ldots & Some pages failed; the reasons were printed as they happened. Exit status 3. \\
skipping unreadable record \ldots & A catalogue entry is damaged. Everything else still works; restore \file{records/} from backup to recover that file. \\
object does not match its catalogue record & The encrypted contents were swapped or damaged. Nothing was written. Restore \file{objects/} from backup. \\
object size does not match its catalogue record & Same as above: damage or tampering, caught before any output appeared. \\
+21 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT-ADD 1 "September 2026" "vault" "User Commands"
.TH VAULT-ADD 1 "October 2026" "vault" "User Commands"
.SH NAME
vault-add \- encrypt files, directories or URLs into the vault
.SH SYNOPSIS
@@ -7,6 +7,10 @@ vault-add \- encrypt files, directories or URLs into the vault
.IR PATH ]
.RB [ \-\-vault
.IR DIR ]
.RB [ \-\-max\-bytes
.IR SIZE ]
.RB [ \-\-timeout
.IR DURATION ]
.IR FILE | DIR | URL
.RI [ FOLDER ]
.SH DESCRIPTION
@@ -17,7 +21,14 @@ and the key are skipped. A stored name that already exists is replaced.
.PP
An \fBhttp://\fR or \fBhttps://\fR \fIURL\fR is downloaded with a Chrome user
agent and stored under the last element of its final (post-redirect) path, or
the host name if there is none. Only a 200 response is accepted.
the host name if there is none. Only a 200 response is accepted. Certificates
are always verified; at most 10 redirects are followed, to http or https
only. The download is staged in a private temporary file (it must be
complete before it can be stored), which is removed afterwards whatever
happens; it is plaintext on disk meanwhile.
.PP
A file that changes while it is being added (it grows, shrinks, or is
replaced) is not stored, and the command fails.
.PP
\fIFOLDER\fR places the result inside a folder in the vault. File contents
are cut into 4 MiB chunks, the last bucket-padded.
@@ -28,6 +39,14 @@ Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP
.BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.TP
.BI \-\-max\-bytes " SIZE"
Largest download accepted, counted after decompression, such as 500M or 50G.
Default 16G; at most 1T.
.TP
.BI \-\-timeout " DURATION"
Deadline for the whole download, such as 30m. Default 2h. Connecting must
take under 30 seconds and the response headers must arrive within a minute.
.SH ENVIRONMENT
.TP
.B VAULT_PASSPHRASE
+9 -4
View File
@@ -1,4 +1,4 @@
.TH VAULT-GET 1 "September 2026" "vault" "User Commands"
.TH VAULT-GET 1 "October 2026" "vault" "User Commands"
.SH NAME
vault-get \- decrypt a stored file
.SH SYNOPSIS
@@ -13,13 +13,18 @@ vault-get \- decrypt a stored file
.I NAME
.SH DESCRIPTION
Decrypts \fINAME\fR. Nothing appears at the output path until every object has
authenticated. The stored mode and modification time are restored. It refuses
to overwrite an existing file. Takes a shared lock.
authenticated, padding and end of file included; until then the output is a
private temporary file in the same directory, removed on failure. The stored
mode and modification time are restored. It never replaces anything already
at the output path, including a symbolic link (even a dangling one) or a
file created while it runs. Takes a shared lock.
.PP
With \fB\-r\fR, \fINAME\fR is a folder, and every file under it is
decrypted into the output directory, keeping the folder structure. Each file
is checked as above; a clash stops the run, leaving the files already
written.
written. Directories are created as needed, mode 0700; a symbolic link (or
anything but a directory) where a directory belongs inside the output
directory stops the run rather than being followed.
.SH OPTIONS
.TP
.BI \-o " OUTPUT"
+5 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT-MIGRATE 1 "September 2026" "vault" "User Commands"
.TH VAULT-MIGRATE 1 "October 2026" "vault" "User Commands"
.SH NAME
vault-migrate \- move files into the vault, deleting the originals
.SH SYNOPSIS
@@ -13,7 +13,10 @@ vault-migrate \- move files into the vault, deleting the originals
.SH DESCRIPTION
Works like
.BR vault\-add (1)
but deletes each original once it is stored, then any directories left empty.
but deletes each original once it is stored (and its record is on disk),
then any directories left empty. An original is deleted only if it is still
the same file, unchanged, that was stored; otherwise it is kept and the
command fails.
Directories still holding skipped files stay. A URL cannot be migrated.
.PP
Before storing anything it lists the vault names that would be overwritten and
+60 -17
View File
@@ -1,4 +1,4 @@
.TH VAULT-MIRROR 1 "September 2026" "vault" "User Commands"
.TH VAULT-MIRROR 1 "October 2026" "vault" "User Commands"
.SH NAME
vault-mirror \- mirror a website into an encrypted vault
.SH SYNOPSIS
@@ -7,6 +7,12 @@ vault-mirror \- mirror a website into an encrypted vault
.IR PATH ]
.RB [ \-\-vault
.IR DIR ]
.RB [ \-\-max\-pages
.IR N ]
.RB [ \-\-max\-bytes
.IR SIZE ]
.RB [ \-\-timeout
.IR DURATION ]
.I URL
.RI [ FOLDER ]
.SH DESCRIPTION
@@ -20,21 +26,30 @@ and stores every page and asset it finds in the vault, beneath
if no folder is given).
.PP
The crawl is single threaded: one request at a time, in breadth-first order.
It follows only links on the same host as
It follows only links on the same host (and port) as
.IR URL ,
and requests are sent with a Chrome user agent string.
and only redirects to that host; a redirect elsewhere skips the page.
Requests are sent with a Chrome user agent string, and certificates are
always verified.
.PP
Links in HTML
Links in HTML start tags
.RB ( href ,
.BR src ,
.BR action
and
.B poster
attributes) and in CSS
.RB ( url(...) )
that point into the mirror are rewritten to relative paths, so the copy
browses correctly once fetched out of the vault. Links to other hosts, and
fragments, are left as they were.
attributes) and CSS
.RB ( url(...)
in style sheets,
.B <style>
elements and
.B style
attributes) that point into the mirror are rewritten to relative paths, so
the copy browses correctly once fetched out of the vault. Relative links are
resolved against the page's final address and its
.B <base href>
if it has one. Links to other hosts, script and comment text, and every
other byte of the page are left as they were. Fragments are kept.
.PP
Pages are named after their URL path. A path that is empty or ends in
.B /
@@ -42,19 +57,39 @@ becomes
.BR index.html ;
a query string is appended as
.BI _ query
(URL-escaped). Anything already stored under the same name is replaced.
(URL-escaped). A name already stored by an earlier run is replaced, but if
two different URLs in one crawl would get the same name, the second is
skipped rather than overwriting the first.
.PP
A page that fails to download, or answers with anything but
A page that fails to download, answers with anything but
.BR "200 OK" ,
is reported on standard error and skipped; the crawl carries on.
The vault is locked exclusively for the whole run.
or exceeds a limit is reported on standard error as
.BI "skipping " URL ": " reason
and skipped; the crawl carries on. HTML and CSS are rewritten in memory;
other files are staged in a private temporary file (plaintext on disk
meanwhile) that is always removed. The vault is locked exclusively for the
whole run.
.SH OPTIONS
.TP
.BI \-\-max\-pages " N"
Pages and assets to fetch at most. Default 10000.
.TP
.BI \-\-max\-bytes " SIZE"
Total bytes to download at most, counted after decompression. Default 4G.
.TP
.BI \-\-timeout " DURATION"
Deadline for each request. Default 5m.
.TP
.BI \-\-key " PATH"
Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP
.BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.PP
A single HTML or CSS file larger than 16 MiB is skipped.
.SH EXIT STATUS
0 when everything found was stored, 3 when some of it was skipped, 1 on
an error that stops the crawl.
.SH ENVIRONMENT
.TP
.B VAULT_PASSPHRASE
@@ -75,11 +110,19 @@ vault mirror https://example.com/ archive/2026
vault get archive/2026/example.com/index.html
.RE
.SH LIMITATIONS
Links are found by pattern matching, not by parsing HTML: URLs assembled by
scripts, and
URLs assembled by scripts, and
.B srcset
lists, are neither followed nor rewritten. There is no depth limit, rate
limit or robots.txt handling; only mirror sites you may copy.
lists, are neither followed nor rewritten. In attribute values only numeric
character references and
.BR &amp; ,
.BR &lt; ,
.BR &gt; ,
.BR &quot; ,
.B &apos;
and
.B &nbsp;
are decoded. There is no rate limit or robots.txt handling; only mirror
sites you may copy.
.SH SEE ALSO
.BR vault (1)
.SH AUTHOR
+8
View File
@@ -32,6 +32,14 @@ memory is locked against swapping and excluded from core dumps; on Linux the
helper is also not debuggable. It answers only on a Unix socket inside
\fI$XDG_RUNTIME_DIR/vault-UID\fR (or the temporary directory), a directory
that must belong to this user with mode 0700. Rebooting ends it too.
If the helper cannot lock its memory, disable core dumps or (on Linux) make
itself undebuggable, it refuses the key and
.B unlock
fails, saying why. The helper is the same program, started with the
passphrase variables removed from its environment. Helpers started by
earlier (Go) versions of vault are compatible and can be used and locked.
.PP
Not available on Windows, where every command asks for the passphrase.
.PP
While the key is unlocked, any program running as this user can use it, and
it is whichever key the passphrase opened, real or decoy. Lock before
+23 -2
View File
@@ -1,4 +1,4 @@
.TH VAULT 1 "September 2026" "vault" "User Commands"
.TH VAULT 1 "October 2026" "vault" "User Commands"
.SH NAME
vault \- post-quantum encrypted file vault
.SH SYNOPSIS
@@ -77,6 +77,11 @@ Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP
.BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.TP
.BI \-\-workers " N"
Threads for encryption, decryption and file I/O, 1 to 256. Default: the
number of CPUs, at most 8. Up to eight 4 MiB objects are in flight at once
whatever the count.
.SH ENVIRONMENT
.TP
.B VAULT_KEY
@@ -92,6 +97,19 @@ Passphrase, instead of prompting on the terminal or using a key remembered by
.B VAULT_DECOY_PASSPHRASE
Decoy passphrase used by
.BR "vault genkey" .
.TP
.BR HTTPS_PROXY ", " HTTP_PROXY ", " NO_PROXY
Proxies for
.B add
.I URL
and
.BR mirror ,
as other programs use them; local addresses are never proxied.
.TP
.BR XDG_RUNTIME_DIR ", " TMPDIR
Where
.BR vault\-unlock (1)
keeps its socket, and where downloads are staged.
.SH LOCKING
Commands that change the vault hold an exclusive lock on
.IR DIR/.lock ;
@@ -106,7 +124,10 @@ and
fills each vault with unreadable filler, so hidden files look like filler to
anyone holding only the decoy passphrase.
.SH EXIT STATUS
0 on success; 1 on error (diagnostics on standard error); 2 when run with no command.
0 on success; 1 on error (diagnostics on standard error); 2 when run with no
command; 3 when
.BR vault\-mirror (1)
stored some resources but had to skip others.
.SH EXAMPLES
.nf
vault genkey