Docs: architecture, development and performance for the Rust implementation
This commit is contained in:
Andy Dixon
+81
-64
@@ -5,36 +5,40 @@ part relies on. It is kept in sync with the code; if they disagree, the code
|
||||
is wrong or this file is stale — fix one of them.
|
||||
|
||||
```
|
||||
cmd/mapsize CLI entry point, flag parsing, mode selection
|
||||
cmd/treegen synthetic test-tree generator
|
||||
internal/brand product name / file extensions (rename in one place)
|
||||
internal/inventory flat node store, aggregation, indexes, categories
|
||||
internal/scan bounded concurrent scanner (controller + workers)
|
||||
internal/platform OS-specific metadata (build-tagged files)
|
||||
internal/treemap squarified layout, LOD, spatial navigation (pure geometry)
|
||||
internal/filter query lexer / parser / evaluator
|
||||
internal/snapshot versioned binary snapshot format + comparison
|
||||
internal/duplicate staged duplicate detection
|
||||
internal/export JSON / CSV / text reports
|
||||
internal/config settings, themes, JSON config file
|
||||
internal/tui Bubble Tea model, canvas renderer, views, modals
|
||||
internal/textutil display sanitising, width-aware truncation, units
|
||||
src/main.rs CLI entry point, flag parsing
|
||||
src/app.rs mode selection: TUI, reports, snapshots, comparison
|
||||
src/brand.rs product name / file extensions (rename in one place)
|
||||
src/inventory/ flat node store, aggregation, indexes, categories
|
||||
src/scan.rs bounded concurrent scanner (controller + workers)
|
||||
src/cancel.rs cancellation token shared by scans and background work
|
||||
src/platform.rs OS-specific metadata and file operations (cfg blocks)
|
||||
src/treemap/ squarified layout, LOD, spatial navigation (pure geometry)
|
||||
src/filter/ query lexer / parser / evaluator
|
||||
src/snapshot/ versioned binary snapshot format + comparison
|
||||
src/duplicate.rs staged duplicate detection
|
||||
src/export.rs JSON / CSV / text reports
|
||||
src/config.rs settings, themes, JSON config file
|
||||
src/tui/ event loop, model, canvas renderer, views, modals
|
||||
src/textutil.rs display sanitising, width-aware truncation, units
|
||||
examples/treegen synthetic test-tree generator
|
||||
examples/bench in-process benchmarks
|
||||
```
|
||||
|
||||
Dependency direction is strictly downward: `tui` → (`treemap`, `filter`,
|
||||
`inventory`, `scan`, …); `scan` → (`inventory`, `platform`); `treemap` depends
|
||||
on nothing but the standard library. The TUI framework (Bubble Tea v2) is only
|
||||
imported by `internal/tui` and `cmd/mapsize`.
|
||||
on nothing but the standard library. The terminal library (crossterm) is only
|
||||
used by `tui`.
|
||||
|
||||
## Inventory representation
|
||||
|
||||
`inventory.Tree` is an append-only arena of `Node` records addressed by
|
||||
`NodeID` (`uint32`). Nodes are stored in fixed-size chunks
|
||||
(`[][]Node`, 65 536 nodes per chunk), so growing the arena never copies or
|
||||
`NodeId` (`u32`). Nodes are stored in fixed-size chunks
|
||||
(`Vec<Vec<Node>>`, 65 536 nodes per chunk), so growing the arena never copies or
|
||||
moves existing nodes and peak memory never doubles during growth.
|
||||
|
||||
A node holds: parent ID, first-child / next-sibling IDs (an intrusive child
|
||||
list — no per-directory slices), name (the single path component, not the full
|
||||
list — no per-directory vectors), name (the single path component as raw
|
||||
bytes, not the full
|
||||
path), kind, flags, own logical size, own allocated size, mtime, mode,
|
||||
uid/gid, link count, and aggregates (total logical, total allocated, file
|
||||
count, directory count, error count). Directory-only data — per-category byte
|
||||
@@ -42,7 +46,7 @@ totals used for colouring — lives in a side table indexed by a directory index
|
||||
so files do not pay for it.
|
||||
|
||||
Full paths are never stored; `Tree.Path(id)` walks parents. Extensions are
|
||||
interned into a small table and stored as a `uint16`. All sizes are `int64`.
|
||||
interned into a small table and stored as a `u16`. All sizes are `i64`.
|
||||
|
||||
Node IDs are stable for the lifetime of a `Tree`: nodes are never removed or
|
||||
reordered. A rescan produces a new `Tree`. This is what makes selection
|
||||
@@ -51,11 +55,11 @@ persistence trivial (see below).
|
||||
## Scanner worker architecture
|
||||
|
||||
```
|
||||
+------------------ controller goroutine ------------------+
|
||||
+------------------ controller thread ---------------------+
|
||||
| owns: Tree (write lock), pending FIFO of dir NodeIDs, |
|
||||
| hard-link table, visited-dir table |
|
||||
+----------------------------------------------------------+
|
||||
| jobs (bounded chan, cap = workers*2) ^ results (bounded chan)
|
||||
| jobs (bounded channel, cap = workers*2) ^ results (bounded channel)
|
||||
v |
|
||||
worker 1 .. worker N (fixed pool, N from --workers / mode)
|
||||
```
|
||||
@@ -77,9 +81,9 @@ while the only consumer of that queue is also blocked. mapsize avoids it by
|
||||
construction:
|
||||
|
||||
* Only the controller produces jobs. Workers only produce *results*.
|
||||
* The controller's main loop is a single `select` that simultaneously offers
|
||||
the next pending job to the bounded `jobs` channel *and* receives from the
|
||||
bounded `results` channel. It never performs a blocking send without also
|
||||
* The controller's main loop is a single `crossbeam_channel::Select` that
|
||||
simultaneously offers the next pending job to the bounded `jobs` channel
|
||||
*and* receives from the bounded `results` channel. It never performs a blocking send without also
|
||||
being ready to receive. Therefore a worker blocked on `results <-` is always
|
||||
eventually served, and the controller is never blocked on `jobs <-` alone.
|
||||
* Discovered-but-unscheduled directories wait in the controller's pending FIFO
|
||||
@@ -94,11 +98,12 @@ directory with 5 M entries neither balloons worker memory nor stalls the UI.
|
||||
|
||||
Completion: the controller tracks `inflight` (dispatched directories whose
|
||||
final chunk has not arrived). The scan is finished when `pending` is empty and
|
||||
`inflight == 0`. Cancellation (`context.Context`) stops dispatch immediately;
|
||||
workers check the context between chunks and between entries, and the
|
||||
controller drains outstanding results so no goroutine leaks.
|
||||
`inflight == 0`. Cancellation (`cancel::Cancel`, which the controller also
|
||||
selects on) stops dispatch immediately; workers check the token between
|
||||
chunks, and the controller drains outstanding results and joins every
|
||||
worker, so no thread leaks.
|
||||
|
||||
Stress tests (`internal/scan/scan_test.go`) run with 1 worker and job/result
|
||||
Stress tests (`scan::tests`) run with 1 worker and job/result
|
||||
channel capacity 1 over trees with thousands of directories to prove progress
|
||||
under maximum backpressure.
|
||||
|
||||
@@ -112,26 +117,27 @@ for a 1 M-file tree at depth 20 that is ~20 K ancestor updates instead of
|
||||
~20 M. There is no reconciliation pass: totals are exact at every moment for
|
||||
the data seen so far.
|
||||
|
||||
The UI reads the tree under `Tree.RLock`; the controller applies batches of
|
||||
results under `Tree.Lock`, holding it for at most a few milliseconds (it
|
||||
The tree is shared as `Arc<RwLock<Tree>>`. The UI reads it under the read
|
||||
lock; the controller applies batches of results under the write lock, holding it for at most a few milliseconds (it
|
||||
releases the lock between batches). This is the only lock in the data path.
|
||||
|
||||
**Concurrency invariants**
|
||||
|
||||
1. Only the scan controller (or a snapshot loader before publication) mutates
|
||||
a `Tree`.
|
||||
2. Every read of `Tree` node data from another goroutine holds `RLock`.
|
||||
2. Every read of `Tree` node data from another thread holds the read lock.
|
||||
3. Nodes are never deleted; a `NodeID` once observed stays valid.
|
||||
4. Workers share nothing mutable except the bounded channels and atomic
|
||||
progress counters.
|
||||
5. No goroutine ever takes `RLock` while already holding it. Go's
|
||||
`RWMutex` blocks new readers once a writer waits, so a recursive read
|
||||
lock can deadlock against the scan controller. The TUI takes the lock
|
||||
once per entry point (`View`, `handleKey`, mouse handlers); helpers they
|
||||
call assume it is held. Background commands (filter, duplicates,
|
||||
snapshot save) take their own lock on their own goroutine.
|
||||
6. After a scan finishes, the UI goroutine may become the writer (move to
|
||||
trash updates aggregates under `Lock`). Trash is refused while scanning.
|
||||
5. No thread ever takes the read lock while already holding it. The
|
||||
standard `RwLock` may block new readers once a writer waits, so a
|
||||
recursive read lock can deadlock against the scan controller. The TUI
|
||||
takes the lock once per entry point (render, key and mouse handlers) and
|
||||
passes `&Tree` to the helpers it calls. Background tasks (filter,
|
||||
duplicates, snapshot save) take their own lock on their own thread.
|
||||
6. After a scan finishes, the UI thread may become the writer (move to
|
||||
trash updates aggregates under the write lock). Trash is refused while
|
||||
scanning.
|
||||
|
||||
## Hard links and sizes
|
||||
|
||||
@@ -166,31 +172,41 @@ releases the lock between batches). This is the only lock in the data path.
|
||||
## TUI event flow
|
||||
|
||||
```
|
||||
scan controller ──(applies under Lock)──> Tree
|
||||
scan controller ──(applies under write lock)──> Tree
|
||||
│ atomic progress counters
|
||||
▼
|
||||
tea.Tick (10 Hz while scanning) ──> tickMsg ──> Update marks frame dirty
|
||||
key/mouse/resize ──> Update ──> state change ──> frame dirty
|
||||
View(): if dirty → RLock tree → layout (if layout dirty) → paint canvas → string
|
||||
tick (10 Hz while scanning) ─────┐
|
||||
crossterm events (input thread) ─┼─> select ─> Model::update ─> state change ─> frame dirty
|
||||
background task results ─────────┘
|
||||
render: if dirty → read-lock tree → layout (if layout dirty) → paint canvas → write changed lines
|
||||
```
|
||||
|
||||
There is never one message per file: the scanner does not talk to the UI at
|
||||
all except for a single `scanDoneMsg`. The UI polls at a fixed rate.
|
||||
`tui::run` puts the terminal in raw mode on the alternate screen and selects
|
||||
over three crossbeam channels: terminal events from a dedicated input thread,
|
||||
a timer tick, and messages from background tasks (filter, duplicates,
|
||||
snapshot save, scan done). There is never one message per file: the scanner
|
||||
does not talk to the UI at all except for the scan-done notification. The UI
|
||||
polls its progress counters at a fixed rate.
|
||||
|
||||
`View()` is called by Bubble Tea after *every* `Update`. The model therefore
|
||||
caches the rendered frame and only repaints when something visible changed.
|
||||
The model caches the rendered frame and only repaints when something
|
||||
visible changed; several updates in a row cost at most one render.
|
||||
|
||||
Expensive per-directory work (sorting children by size) is cached per data
|
||||
version. During a scan the version advances at most every 400 ms, so the
|
||||
layout does not jitter at frame rate even though labels and counters update
|
||||
at the 10 Hz refresh.
|
||||
|
||||
A panic hook restores the terminal (leaves the alternate screen, disables
|
||||
raw mode and mouse reporting) before the panic is reported; SIGTERM quits
|
||||
cleanly.
|
||||
|
||||
## Rendering: the canvas
|
||||
|
||||
All drawing goes to an in-memory `Canvas` of cells (grapheme, fg, bg, attrs).
|
||||
Views and modals paint into the canvas; the canvas is serialised to a string
|
||||
with minimal SGR changes. Bubble Tea's renderer diffs frames and downsamples
|
||||
true-colour to the detected profile (256 / 16 / none), so the drawing code
|
||||
Views and modals paint into the canvas; the canvas is serialised with minimal
|
||||
SGR changes, converted to the detected colour profile (truecolour / 256 / 16 /
|
||||
none), and only the changed span of each changed line is written, inside a
|
||||
synchronized-output block where the terminal supports it. The drawing code
|
||||
uses RGB throughout. For 256-colour terminals the canvas quantizes colours
|
||||
itself (redmean distance over the 6×6×6 cube and grey ramp, never the
|
||||
palette-dependent first 16 entries): the stock conversion maps dark tints to
|
||||
@@ -226,7 +242,7 @@ visible. Groups are selectable and inspectable.
|
||||
|
||||
## Spatial navigation algorithm
|
||||
|
||||
`treemap.Neighbour(rects, selected, dir)` is pure geometry:
|
||||
`treemap::neighbour(blocks, selected, dir)` is pure geometry:
|
||||
|
||||
1. Candidates must lie in the requested half-plane: for RIGHT, `c.X ≥
|
||||
sel.X + sel.W − tolerance` (with the candidate's centre to the right of the
|
||||
@@ -241,18 +257,18 @@ visible. Groups are selectable and inspectable.
|
||||
|
||||
## Terminal resize architecture
|
||||
|
||||
`tea.WindowSizeMsg` only records the new size and marks layout dirty. The next
|
||||
`View()` recomputes panel geometry for the new size class (large / medium /
|
||||
A resize event only records the new size and marks layout dirty. The next
|
||||
render recomputes panel geometry for the new size class (large / medium /
|
||||
small / too-small), reruns the treemap layout for the new treemap rectangle
|
||||
(never stretches old coordinates), and repaints. Because work happens only in
|
||||
`View()` and the frame cache absorbs repeated calls, a storm of resize events
|
||||
(never stretches old coordinates), and repaints. Because work happens only
|
||||
at render time and the frame cache absorbs repeated calls, a storm of resize events
|
||||
costs at most one layout per rendered frame. Modal geometry is derived from
|
||||
the current size every frame, clamped to the screen, and becomes scrollable
|
||||
when content does not fit.
|
||||
|
||||
## Selection persistence
|
||||
|
||||
Selection is a `NodeID` (or, for the synthetic LOD group, `-1 - parentID`).
|
||||
Selection is a `NodeId` (or, for the synthetic LOD group, `-1 - parent`).
|
||||
Until the user moves the selection it follows the largest block, which
|
||||
changes while a scan runs; after that it is sticky. After any relayout the renderer looks the ID up in the
|
||||
new rectangle list. If it is absent (filtered out, collapsed into a group,
|
||||
@@ -261,8 +277,8 @@ nearest visible ancestor, then the largest block.
|
||||
|
||||
## Modal architecture
|
||||
|
||||
A modal is a small interface: `Title()`, `Lines(width)`, `Footer()`, and
|
||||
`HandleKey`. A generic frame renders any modal centred, clamped to the screen
|
||||
A modal is a small trait: `title()`, `content(width)`, `footer()`, and
|
||||
`key`. A generic frame renders any modal centred, clamped to the screen
|
||||
(min 1-cell margin), with a scrollable viewport when content exceeds height.
|
||||
The base screen is dimmed before the modal is painted. While a modal is open
|
||||
it receives all keys; `Esc` always closes it.
|
||||
@@ -286,7 +302,7 @@ No other package switches on `runtime.GOOS`.
|
||||
Filenames and snapshot contents are untrusted.
|
||||
|
||||
* Every string that reaches the terminal passes through
|
||||
`textutil.Sanitize`, which replaces C0/C1 control characters (including
|
||||
`textutil::sanitize`, which replaces C0/C1 control characters (including
|
||||
ESC, CSI, BEL, DEL) with visible escapes (`\x1b`), replaces invalid UTF-8
|
||||
with U+FFFD, and neutralises bidi override characters. The canvas also
|
||||
refuses control characters as a second line of defence. Internal names keep
|
||||
@@ -299,9 +315,10 @@ Filenames and snapshot contents are untrusted.
|
||||
(≤ 1 EiB) and node count (at most one node per byte of snapshot file, 1 Mi floor); caps the decompressed body at 64× the file size
|
||||
(decompression bombs fail fast); and verifies a SHA-256 trailer.
|
||||
Aggregates are recomputed with saturating arithmetic rather than trusted.
|
||||
* No shell is ever invoked. "Open/reveal" and trash use direct `exec` with
|
||||
argument vectors or native APIs; paths are passed as single arguments.
|
||||
* No shell is ever invoked. "Open/reveal" and trash use `std::process::Command`
|
||||
with argument vectors or native APIs; paths are passed as single arguments.
|
||||
* Permanent delete (headless only) checks the item's owner and type against
|
||||
the scan and removes it through an `os.Root` on the parent directory, so
|
||||
a symlink swapped in since the scan cannot redirect it.
|
||||
the scan and removes it through descriptors opened from the parent
|
||||
directory without following symlinks (`openat`/`unlinkat` with
|
||||
`O_NOFOLLOW`), so a symlink swapped in since the scan cannot redirect it.
|
||||
* `--read-only` disables every mutating action in the UI.
|
||||
|
||||
Reference in New Issue
Block a user