Andy DixonandClaude
53968f8822
README, CLAUDE.md, man pages, the user guide and its screenshot script now describe the Rust build and the intentional behaviour changes, which docs/MIGRATION.md lists. Go and Rust micro-benchmarks for the same operations, and scripts/bench.sh for end-to-end comparison. build.sh exports its tree root to the per-target jobs and keeps logs out of the output directory; riscv64 musl builds use getrandom's raw syscall backend, since zig's musl getrandom never reaches the kernel there. Co-Authored-By: Claude <noreply@anthropic.com>
130 lines
3.3 KiB
Groff
130 lines
3.3 KiB
Groff
.TH VAULT-MIRROR 1 "October 2026" "vault" "User Commands"
|
|
.SH NAME
|
|
vault-mirror \- mirror a website into an encrypted vault
|
|
.SH SYNOPSIS
|
|
.B vault mirror
|
|
.RB [ \-\-key
|
|
.IR PATH ]
|
|
.RB [ \-\-vault
|
|
.IR DIR ]
|
|
.RB [ \-\-max\-pages
|
|
.IR N ]
|
|
.RB [ \-\-max\-bytes
|
|
.IR SIZE ]
|
|
.RB [ \-\-timeout
|
|
.IR DURATION ]
|
|
.I URL
|
|
.RI [ FOLDER ]
|
|
.SH DESCRIPTION
|
|
.B vault mirror
|
|
crawls the website at
|
|
.I URL
|
|
and stores every page and asset it finds in the vault, beneath
|
|
.IR FOLDER / HOST
|
|
(or just
|
|
.I HOST
|
|
if no folder is given).
|
|
.PP
|
|
The crawl is single threaded: one request at a time, in breadth-first order.
|
|
It follows only links on the same host (and port) as
|
|
.IR URL ,
|
|
and only redirects to that host; a redirect elsewhere skips the page.
|
|
Requests are sent with a Chrome user agent string, and certificates are
|
|
always verified.
|
|
.PP
|
|
Links in HTML start tags
|
|
.RB ( href ,
|
|
.BR src ,
|
|
.BR action
|
|
and
|
|
.B poster
|
|
attributes) and CSS
|
|
.RB ( url(...)
|
|
in style sheets,
|
|
.B <style>
|
|
elements and
|
|
.B style
|
|
attributes) that point into the mirror are rewritten to relative paths, so
|
|
the copy browses correctly once fetched out of the vault. Relative links are
|
|
resolved against the page's final address and its
|
|
.B <base href>
|
|
if it has one. Links to other hosts, script and comment text, and every
|
|
other byte of the page are left as they were. Fragments are kept.
|
|
.PP
|
|
Pages are named after their URL path. A path that is empty or ends in
|
|
.B /
|
|
becomes
|
|
.BR index.html ;
|
|
a query string is appended as
|
|
.BI _ query
|
|
(URL-escaped). A name already stored by an earlier run is replaced, but if
|
|
two different URLs in one crawl would get the same name, the second is
|
|
skipped rather than overwriting the first.
|
|
.PP
|
|
A page that fails to download, answers with anything but
|
|
.BR "200 OK" ,
|
|
or exceeds a limit is reported on standard error as
|
|
.BI "skipping " URL ": " reason
|
|
and skipped; the crawl carries on. HTML and CSS are rewritten in memory;
|
|
other files are staged in a private temporary file (plaintext on disk
|
|
meanwhile) that is always removed. The vault is locked exclusively for the
|
|
whole run.
|
|
.SH OPTIONS
|
|
.TP
|
|
.BI \-\-max\-pages " N"
|
|
Pages and assets to fetch at most. Default 10000.
|
|
.TP
|
|
.BI \-\-max\-bytes " SIZE"
|
|
Total bytes to download at most, counted after decompression. Default 4G.
|
|
.TP
|
|
.BI \-\-timeout " DURATION"
|
|
Deadline for each request. Default 5m.
|
|
.TP
|
|
.BI \-\-key " PATH"
|
|
Key file. Default: $VAULT_KEY, else ~/.vault/key.
|
|
.TP
|
|
.BI \-\-vault " DIR"
|
|
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
|
|
.PP
|
|
A single HTML or CSS file larger than 16 MiB is skipped.
|
|
.SH EXIT STATUS
|
|
0 when everything found was stored, 3 when some of it was skipped, 1 on
|
|
an error that stops the crawl.
|
|
.SH ENVIRONMENT
|
|
.TP
|
|
.B VAULT_PASSPHRASE
|
|
Passphrase, instead of prompting on the terminal.
|
|
.SH EXAMPLES
|
|
Mirror a site into the vault under
|
|
.BR example.com/ :
|
|
.PP
|
|
.RS
|
|
vault mirror https://example.com/
|
|
.RE
|
|
.PP
|
|
Mirror it under a folder, then fetch the front page:
|
|
.PP
|
|
.RS
|
|
vault mirror https://example.com/ archive/2026
|
|
.br
|
|
vault get archive/2026/example.com/index.html
|
|
.RE
|
|
.SH LIMITATIONS
|
|
URLs assembled by scripts, and
|
|
.B srcset
|
|
lists, are neither followed nor rewritten. In attribute values only numeric
|
|
character references and
|
|
.BR & ,
|
|
.BR < ,
|
|
.BR > ,
|
|
.BR " ,
|
|
.B '
|
|
and
|
|
.B
|
|
are decoded. There is no rate limit or robots.txt handling; only mirror
|
|
sites you may copy.
|
|
.SH SEE ALSO
|
|
.BR vault (1)
|
|
.SH AUTHOR
|
|
Andy Dixon <vault@dixon.cx>, https://www.dixon.cx
|