Add a mirror command that crawls a site into the vault

Single-threaded, same host only, Chrome user agent. Links in HTML and
CSS are rewritten to relative paths. Includes a man page and a guide note.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
Andy Dixon
2026-09-29 15:02:12 +01:00
co-authored by Claude Sonnet 5.5
parent e40a803367
commit 10f2a88493
5 changed files with 315 additions and 0 deletions
+6
View File
@@ -561,6 +561,12 @@ downloads it (as Chrome, so fussy sites play along) and stores it under the
last part of the address, such as \file{report.pdf}. The download sits in a
private temporary file only until it's encrypted.
\subsection{Copying a website: \texttt{mirror}}
\cmd{mirror} takes an \file{http://} or \file{https://} address and copies
the whole site, one request at a time, into a folder named after its host.
Links inside pages and stylesheets are rewritten to relative paths, so the
copy still browses once you get it out. Only the same host is followed.
\subsection{Moving in: \texttt{migrate}}
\cmd{migrate} works exactly like \cmd{add}, but deletes each original once
it's safely stored, then tidies away any folders left empty. It won't take a
+86
View File
@@ -0,0 +1,86 @@
.TH VAULT-MIRROR 1 "September 2026" "vault" "User Commands"
.SH NAME
vault-mirror \- mirror a website into an encrypted vault
.SH SYNOPSIS
.B vault mirror
.RB [ \-\-key
.IR PATH ]
.RB [ \-\-vault
.IR DIR ]
.I URL
.RI [ FOLDER ]
.SH DESCRIPTION
.B vault mirror
crawls the website at
.I URL
and stores every page and asset it finds in the vault, beneath
.IR FOLDER / HOST
(or just
.I HOST
if no folder is given).
.PP
The crawl is single threaded: one request at a time, in breadth-first order.
It follows only links on the same host as
.IR URL ,
and requests are sent with a Chrome user agent string.
.PP
Links in HTML
.RB ( href ,
.BR src ,
.BR action
and
.B poster
attributes) and in CSS
.RB ( url(...) )
that point into the mirror are rewritten to relative paths, so the copy
browses correctly once fetched out of the vault. Links to other hosts, and
fragments, are left as they were.
.PP
Pages are named after their URL path. A path that is empty or ends in
.B /
becomes
.BR index.html ;
a query string is appended as
.BI _ query
(URL-escaped). Anything already stored under the same name is replaced.
.PP
A page that fails to download, or answers with anything but
.BR "200 OK" ,
is reported on standard error and skipped; the crawl carries on.
The vault is locked exclusively for the whole run.
.SH OPTIONS
.TP
.BI \-\-key " PATH"
Key file. Default: $VAULT_KEY, else ~/.vault/key.
.TP
.BI \-\-vault " DIR"
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
.SH ENVIRONMENT
.TP
.B VAULT_PASSPHRASE
Passphrase, instead of prompting on the terminal.
.SH EXAMPLES
Mirror a site into the vault under
.BR example.com/ :
.PP
.RS
vault mirror https://example.com/
.RE
.PP
Mirror it under a folder, then fetch the front page:
.PP
.RS
vault mirror https://example.com/ archive/2026
.br
vault get archive/2026/example.com/index.html
.RE
.SH LIMITATIONS
Links are found by pattern matching, not by parsing HTML: URLs assembled by
scripts, and
.B srcset
lists, are neither followed nor rewritten. There is no depth limit, rate
limit or robots.txt handling; only mirror sites you may copy.
.SH SEE ALSO
.BR vault (1)
.SH AUTHOR
Andy Dixon <vault@dixon.cx>, https://www.dixon.cx