Add a mirror command that crawls a site into the vault
Single-threaded, same host only, Chrome user agent. Links in HTML and CSS are rewritten to relative paths. Includes a man page and a guide note. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 5.5
parent
e40a803367
commit
10f2a88493
@@ -561,6 +561,12 @@ downloads it (as Chrome, so fussy sites play along) and stores it under the
|
||||
last part of the address, such as \file{report.pdf}. The download sits in a
|
||||
private temporary file only until it's encrypted.
|
||||
|
||||
\subsection{Copying a website: \texttt{mirror}}
|
||||
\cmd{mirror} takes an \file{http://} or \file{https://} address and copies
|
||||
the whole site, one request at a time, into a folder named after its host.
|
||||
Links inside pages and stylesheets are rewritten to relative paths, so the
|
||||
copy still browses once you get it out. Only the same host is followed.
|
||||
|
||||
\subsection{Moving in: \texttt{migrate}}
|
||||
\cmd{migrate} works exactly like \cmd{add}, but deletes each original once
|
||||
it's safely stored, then tidies away any folders left empty. It won't take a
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
.TH VAULT-MIRROR 1 "September 2026" "vault" "User Commands"
|
||||
.SH NAME
|
||||
vault-mirror \- mirror a website into an encrypted vault
|
||||
.SH SYNOPSIS
|
||||
.B vault mirror
|
||||
.RB [ \-\-key
|
||||
.IR PATH ]
|
||||
.RB [ \-\-vault
|
||||
.IR DIR ]
|
||||
.I URL
|
||||
.RI [ FOLDER ]
|
||||
.SH DESCRIPTION
|
||||
.B vault mirror
|
||||
crawls the website at
|
||||
.I URL
|
||||
and stores every page and asset it finds in the vault, beneath
|
||||
.IR FOLDER / HOST
|
||||
(or just
|
||||
.I HOST
|
||||
if no folder is given).
|
||||
.PP
|
||||
The crawl is single threaded: one request at a time, in breadth-first order.
|
||||
It follows only links on the same host as
|
||||
.IR URL ,
|
||||
and requests are sent with a Chrome user agent string.
|
||||
.PP
|
||||
Links in HTML
|
||||
.RB ( href ,
|
||||
.BR src ,
|
||||
.BR action
|
||||
and
|
||||
.B poster
|
||||
attributes) and in CSS
|
||||
.RB ( url(...) )
|
||||
that point into the mirror are rewritten to relative paths, so the copy
|
||||
browses correctly once fetched out of the vault. Links to other hosts, and
|
||||
fragments, are left as they were.
|
||||
.PP
|
||||
Pages are named after their URL path. A path that is empty or ends in
|
||||
.B /
|
||||
becomes
|
||||
.BR index.html ;
|
||||
a query string is appended as
|
||||
.BI _ query
|
||||
(URL-escaped). Anything already stored under the same name is replaced.
|
||||
.PP
|
||||
A page that fails to download, or answers with anything but
|
||||
.BR "200 OK" ,
|
||||
is reported on standard error and skipped; the crawl carries on.
|
||||
The vault is locked exclusively for the whole run.
|
||||
.SH OPTIONS
|
||||
.TP
|
||||
.BI \-\-key " PATH"
|
||||
Key file. Default: $VAULT_KEY, else ~/.vault/key.
|
||||
.TP
|
||||
.BI \-\-vault " DIR"
|
||||
Vault directory. Default: $VAULT_DIR, else ~/.vault/store.
|
||||
.SH ENVIRONMENT
|
||||
.TP
|
||||
.B VAULT_PASSPHRASE
|
||||
Passphrase, instead of prompting on the terminal.
|
||||
.SH EXAMPLES
|
||||
Mirror a site into the vault under
|
||||
.BR example.com/ :
|
||||
.PP
|
||||
.RS
|
||||
vault mirror https://example.com/
|
||||
.RE
|
||||
.PP
|
||||
Mirror it under a folder, then fetch the front page:
|
||||
.PP
|
||||
.RS
|
||||
vault mirror https://example.com/ archive/2026
|
||||
.br
|
||||
vault get archive/2026/example.com/index.html
|
||||
.RE
|
||||
.SH LIMITATIONS
|
||||
Links are found by pattern matching, not by parsing HTML: URLs assembled by
|
||||
scripts, and
|
||||
.B srcset
|
||||
lists, are neither followed nor rewritten. There is no depth limit, rate
|
||||
limit or robots.txt handling; only mirror sites you may copy.
|
||||
.SH SEE ALSO
|
||||
.BR vault (1)
|
||||
.SH AUTHOR
|
||||
Andy Dixon <vault@dixon.cx>, https://www.dixon.cx
|
||||
Reference in New Issue
Block a user