Initial Go project structure, docs, build and CI
This commit is contained in:
Andy Dixon
+279
@@ -0,0 +1,279 @@
|
||||
# mapsize architecture
|
||||
|
||||
This document describes how mapsize is put together and the invariants each
|
||||
part relies on. It is kept in sync with the code; if they disagree, the code
|
||||
is wrong or this file is stale — fix one of them.
|
||||
|
||||
```
|
||||
cmd/mapsize CLI entry point, flag parsing, mode selection
|
||||
cmd/treegen synthetic test-tree generator
|
||||
internal/brand product name / file extensions (rename in one place)
|
||||
internal/inventory flat node store, aggregation, indexes, categories
|
||||
internal/scan bounded concurrent scanner (controller + workers)
|
||||
internal/platform OS-specific metadata (build-tagged files)
|
||||
internal/treemap squarified layout, LOD, spatial navigation (pure geometry)
|
||||
internal/filter query lexer / parser / evaluator
|
||||
internal/snapshot versioned binary snapshot format + comparison
|
||||
internal/duplicate staged duplicate detection
|
||||
internal/export JSON / CSV / text reports
|
||||
internal/config settings, themes, JSON config file
|
||||
internal/tui Bubble Tea model, canvas renderer, views, modals
|
||||
internal/textutil display sanitising, width-aware truncation, units
|
||||
```
|
||||
|
||||
Dependency direction is strictly downward: `tui` → (`treemap`, `filter`,
|
||||
`inventory`, `scan`, …); `scan` → (`inventory`, `platform`); `treemap` depends
|
||||
on nothing but the standard library. The TUI framework (Bubble Tea v2) is only
|
||||
imported by `internal/tui` and `cmd/mapsize`.
|
||||
|
||||
## Inventory representation
|
||||
|
||||
`inventory.Tree` is an append-only arena of `Node` records addressed by
|
||||
`NodeID` (`uint32`). Nodes are stored in fixed-size chunks
|
||||
(`[][]Node`, 65 536 nodes per chunk), so growing the arena never copies or
|
||||
moves existing nodes and peak memory never doubles during growth.
|
||||
|
||||
A node holds: parent ID, first-child / next-sibling IDs (an intrusive child
|
||||
list — no per-directory slices), name (the single path component, not the full
|
||||
path), kind, flags, own logical size, own allocated size, mtime, mode,
|
||||
uid/gid, link count, and aggregates (total logical, total allocated, file
|
||||
count, directory count, error count). Directory-only data — per-category byte
|
||||
totals used for colouring — lives in a side table indexed by a directory index,
|
||||
so files do not pay for it.
|
||||
|
||||
Full paths are never stored; `Tree.Path(id)` walks parents. Extensions are
|
||||
interned into a small table and stored as a `uint16`. All sizes are `int64`.
|
||||
|
||||
Node IDs are stable for the lifetime of a `Tree`: nodes are never removed or
|
||||
reordered. A rescan produces a new `Tree`. This is what makes selection
|
||||
persistence trivial (see below).
|
||||
|
||||
## Scanner worker architecture
|
||||
|
||||
```
|
||||
+------------------ controller goroutine ------------------+
|
||||
| owns: Tree (write lock), pending FIFO of dir NodeIDs, |
|
||||
| hard-link table, visited-dir table |
|
||||
+----------------------------------------------------------+
|
||||
| jobs (bounded chan, cap = workers*2) ^ results (bounded chan)
|
||||
v |
|
||||
worker 1 .. worker N (fixed pool, N from --workers / mode)
|
||||
```
|
||||
|
||||
* Workers do I/O only: open a directory, read entries in chunks of up to 1024,
|
||||
`lstat` each entry (relative to the directory fd where the platform allows),
|
||||
and send a `result` chunk back. Workers never touch the `Tree`.
|
||||
* The controller is the only writer of the `Tree`. It applies result chunks,
|
||||
creates child nodes, assigns IDs, propagates aggregate deltas, and decides
|
||||
which subdirectories become new jobs.
|
||||
* Worker count: `conservative` = 2, `balanced` = min(8, 2×CPU),
|
||||
`aggressive` = min(64, 8×CPU), or an explicit number. More is not always
|
||||
faster — on rotational media 2–4 is often best.
|
||||
|
||||
## Bounded directory scheduling (no deadlock under backpressure)
|
||||
|
||||
The classic deadlock is: workers block sending new work into a full queue
|
||||
while the only consumer of that queue is also blocked. mapsize avoids it by
|
||||
construction:
|
||||
|
||||
* Only the controller produces jobs. Workers only produce *results*.
|
||||
* The controller's main loop is a single `select` that simultaneously offers
|
||||
the next pending job to the bounded `jobs` channel *and* receives from the
|
||||
bounded `results` channel. It never performs a blocking send without also
|
||||
being ready to receive. Therefore a worker blocked on `results <-` is always
|
||||
eventually served, and the controller is never blocked on `jobs <-` alone.
|
||||
* Discovered-but-unscheduled directories wait in the controller's pending FIFO
|
||||
as 4-byte NodeIDs (paths are rebuilt at dispatch). Its size is bounded by the
|
||||
number of directory nodes already in the inventory, so it adds O(1) bytes per
|
||||
directory on top of memory we already spend.
|
||||
* FIFO order means the scan is roughly breadth-first: every top-level directory
|
||||
starts growing early, which is what makes the live treemap informative.
|
||||
|
||||
Huge directories are streamed: a worker sends a result per 1024 entries, so a
|
||||
directory with 5 M entries neither balloons worker memory nor stalls the UI.
|
||||
|
||||
Completion: the controller tracks `inflight` (dispatched directories whose
|
||||
final chunk has not arrived). The scan is finished when `pending` is empty and
|
||||
`inflight == 0`. Cancellation (`context.Context`) stops dispatch immediately;
|
||||
workers check the context between chunks and between entries, and the
|
||||
controller drains outstanding results so no goroutine leaks.
|
||||
|
||||
Stress tests (`internal/scan/scan_test.go`) run with 1 worker and job/result
|
||||
channel capacity 1 over trees with thousands of directories to prove progress
|
||||
under maximum backpressure.
|
||||
|
||||
## Concurrent aggregation strategy
|
||||
|
||||
Because the controller is the single writer, aggregation needs no atomics.
|
||||
For each result chunk it sums the chunk's file sizes, allocated sizes, counts
|
||||
and per-category bytes **once**, then walks the ancestor chain adding that one
|
||||
delta. Cost is O(depth) per 1024-entry chunk rather than O(depth) per file —
|
||||
for a 1 M-file tree at depth 20 that is ~20 K ancestor updates instead of
|
||||
~20 M. There is no reconciliation pass: totals are exact at every moment for
|
||||
the data seen so far.
|
||||
|
||||
The UI reads the tree under `Tree.RLock`; the controller applies batches of
|
||||
results under `Tree.Lock`, holding it for at most a few milliseconds (it
|
||||
releases the lock between batches). This is the only lock in the data path.
|
||||
|
||||
**Concurrency invariants**
|
||||
|
||||
1. Only the scan controller (or a snapshot loader before publication) mutates
|
||||
a `Tree`.
|
||||
2. Every read of `Tree` node data from another goroutine holds `RLock`.
|
||||
3. Nodes are never deleted; a `NodeID` once observed stays valid.
|
||||
4. Workers share nothing mutable except the bounded channels and atomic
|
||||
progress counters.
|
||||
|
||||
## Hard links and sizes
|
||||
|
||||
* *Logical size* = `st_size`. *Allocated size* = `st_blocks × 512` on Unix.
|
||||
On Windows allocated size is reported as unknown (equal to logical, flagged)
|
||||
because getting it costs an extra handle-open per file.
|
||||
* For regular files with link count > 1, the controller keys `(dev, inode)`.
|
||||
The first path seen contributes to directory aggregates; later paths are
|
||||
flagged `HardlinkDup` and contribute **zero** to aggregates (their own size
|
||||
is still shown). Which path is "first" depends on scan order and is not
|
||||
deterministic. This matches `du`'s default semantics.
|
||||
* Windows does not expose link counts from directory enumeration, so hard
|
||||
links are not deduplicated there. This is documented, not hidden.
|
||||
|
||||
## Symlinks, mount points, exclusions
|
||||
|
||||
* `--follow-symlinks=none` (default): symlinks are leaf nodes counted at their
|
||||
own (tiny) size. `same-filesystem` / `all` follow directory symlinks; every
|
||||
descended directory's `(dev, inode)` is recorded and a directory already
|
||||
visited is never descended twice — loops terminate and nothing is counted
|
||||
twice.
|
||||
* A directory whose device differs from its parent is flagged `MountPoint`.
|
||||
With `--one-file-system` it is not descended. On Linux, virtual filesystems
|
||||
(proc, sysfs, cgroup, debugfs, …) are never descended — `/proc/kcore` would
|
||||
otherwise dominate any scan of `/` — and are reported in scan info.
|
||||
* `--exclude PATTERN`: patterns containing a path separator match full paths;
|
||||
others match the entry name with glob semantics. Excluded entries are
|
||||
counted and the UI shows that exclusions are active.
|
||||
|
||||
## TUI event flow
|
||||
|
||||
```
|
||||
scan controller ──(applies under Lock)──> Tree
|
||||
│ atomic progress counters
|
||||
▼
|
||||
tea.Tick (10 Hz while scanning) ──> tickMsg ──> Update marks frame dirty
|
||||
key/mouse/resize ──> Update ──> state change ──> frame dirty
|
||||
View(): if dirty → RLock tree → layout (if layout dirty) → paint canvas → string
|
||||
```
|
||||
|
||||
There is never one message per file: the scanner does not talk to the UI at
|
||||
all except for a single `scanDoneMsg`. The UI polls at a fixed rate.
|
||||
|
||||
`View()` is called by Bubble Tea after *every* `Update`. The model therefore
|
||||
caches the rendered frame and only repaints when something visible changed.
|
||||
|
||||
## Rendering: the canvas
|
||||
|
||||
All drawing goes to an in-memory `Canvas` of cells (grapheme, fg, bg, attrs).
|
||||
Views and modals paint into the canvas; the canvas is serialised to a string
|
||||
with minimal SGR changes. Bubble Tea's renderer diffs frames and downsamples
|
||||
true-colour to the detected profile (256 / 16 / none), so the drawing code
|
||||
uses RGB throughout. Selection never relies on colour alone: the selected
|
||||
rectangle uses a double-line border, bold reversed title and a status-line
|
||||
description.
|
||||
|
||||
## Treemap algorithm
|
||||
|
||||
`treemap.Squarify` implements Bruls, Huizing & van Wijk's squarified layout
|
||||
on float rectangles: items sorted by size descending are added to the current
|
||||
row while the worst aspect ratio improves; then the row is laid out along the
|
||||
shorter side. Float rectangles are snapped to the cell grid by rounding the
|
||||
*edges* (not the widths), so neighbouring rectangles share edges exactly — no
|
||||
gaps, no overlaps. Layout is deterministic (stable sort by size, then ID).
|
||||
|
||||
Nested levels are laid out recursively inside each block's interior (inside
|
||||
its border) while the interior is large enough to be useful.
|
||||
|
||||
## Level-of-detail model
|
||||
|
||||
A 160×50 terminal has 8 000 cells; the tree may have millions of entries.
|
||||
For each directory being laid out, children are reduced to the top K by size
|
||||
using a bounded heap (O(n log K)). Children whose proportional area would be
|
||||
smaller than a minimum useful block (a few cells), or beyond K, are merged
|
||||
into one synthetic *group* block ("312 smaller items, 8.4 GiB"). Zooming into
|
||||
a directory gives its children the whole screen, so they become individually
|
||||
visible. Groups are selectable and inspectable.
|
||||
|
||||
## Spatial navigation algorithm
|
||||
|
||||
`treemap.Neighbour(rects, selected, dir)` is pure geometry:
|
||||
|
||||
1. Candidates must lie in the requested half-plane: for RIGHT, `c.X ≥
|
||||
sel.X + sel.W − tolerance` (with the candidate's centre to the right of the
|
||||
selected centre).
|
||||
2. Each candidate is scored by the primary-axis gap (edge to edge) plus a
|
||||
penalty for the orthogonal offset. Candidates overlapping the selected
|
||||
rectangle's orthogonal span get the offset term set to zero and are
|
||||
preferred over any non-overlapping candidate.
|
||||
3. Among overlapping candidates, the one whose overlap is largest relative to
|
||||
the source, then nearest to the source's centre line, wins.
|
||||
4. Ties are broken by top-left position, then NodeID — fully deterministic.
|
||||
|
||||
## Terminal resize architecture
|
||||
|
||||
`tea.WindowSizeMsg` only records the new size and marks layout dirty. The next
|
||||
`View()` recomputes panel geometry for the new size class (large / medium /
|
||||
small / too-small), reruns the treemap layout for the new treemap rectangle
|
||||
(never stretches old coordinates), and repaints. Because work happens only in
|
||||
`View()` and the frame cache absorbs repeated calls, a storm of resize events
|
||||
costs at most one layout per rendered frame. Modal geometry is derived from
|
||||
the current size every frame, clamped to the screen, and becomes scrollable
|
||||
when content does not fit.
|
||||
|
||||
## Selection persistence
|
||||
|
||||
Selection is a `NodeID` (plus, for the synthetic LOD group, the group's parent
|
||||
ID and a group marker). After any relayout the renderer looks the ID up in the
|
||||
new rectangle list. If it is absent (filtered out, collapsed into a group,
|
||||
zoom changed) selection falls back to: the group containing it, then its
|
||||
nearest visible ancestor, then the largest block.
|
||||
|
||||
## Modal architecture
|
||||
|
||||
A modal is a small interface: `Title()`, `Lines(width)`, `Footer()`, and
|
||||
`HandleKey`. A generic frame renders any modal centred, clamped to the screen
|
||||
(min 1-cell margin), with a scrollable viewport when content exceeds height.
|
||||
The base screen is dimmed before the modal is painted. While a modal is open
|
||||
it receives all keys; `Esc` always closes it.
|
||||
|
||||
## OS abstraction
|
||||
|
||||
`internal/platform` exposes `ReadDir(path, func(chunk []Entry) error)` and
|
||||
`Meta` structs. Implementations:
|
||||
|
||||
* `readdir_linux.go` — `getdents` via `os.File.ReadDir` + `statx` relative to
|
||||
the directory fd (one syscall per entry, gives birth time).
|
||||
* `readdir_unix.go` — darwin/BSD: `fstatat` relative to the directory fd.
|
||||
* `readdir_windows.go` — `os.File.ReadDir` + `Lstat`; reparse points
|
||||
(junctions, symlinks) are never descended by default.
|
||||
* `fs_*.go` — virtual-filesystem detection, owner name lookup.
|
||||
|
||||
No other package switches on `runtime.GOOS`.
|
||||
|
||||
## Security handling for hostile filenames
|
||||
|
||||
Filenames and snapshot contents are untrusted.
|
||||
|
||||
* Every string that reaches the terminal passes through
|
||||
`textutil.Sanitize`, which replaces C0/C1 control characters (including
|
||||
ESC, CSI, BEL, DEL) with visible escapes (`\x1b`), replaces invalid UTF-8
|
||||
with U+FFFD, and neutralises bidi override characters. The canvas also
|
||||
refuses control characters as a second line of defence. Internal names keep
|
||||
the raw bytes.
|
||||
* Text is clipped by display width (grapheme clusters, East-Asian wide
|
||||
characters) so labels never overflow their rectangle.
|
||||
* Snapshot decoding validates magic, version, every length, every parent
|
||||
reference (a parent must precede its child, so cycles and recursion are
|
||||
impossible), caps decompressed size and node count, and verifies a SHA-256
|
||||
trailer. Aggregates are recomputed rather than trusted.
|
||||
* No shell is ever invoked. "Open/reveal" and trash use direct `exec` with
|
||||
argument vectors or native APIs; paths are passed as single arguments.
|
||||
* `--read-only` disables every mutating action in the UI.
|
||||
Reference in New Issue
Block a user