mapsize/ARCHITECTURE.md

mapsize architecture

This document describes how mapsize is put together and the invariants each part relies on. It is kept in sync with the code; if they disagree, the code is wrong or this file is stale — fix one of them.

cmd/mapsize        CLI entry point, flag parsing, mode selection
cmd/treegen        synthetic test-tree generator
internal/brand     product name / file extensions (rename in one place)
internal/inventory flat node store, aggregation, indexes, categories
internal/scan      bounded concurrent scanner (controller + workers)
internal/platform  OS-specific metadata (build-tagged files)
internal/treemap   squarified layout, LOD, spatial navigation (pure geometry)
internal/filter    query lexer / parser / evaluator
internal/snapshot  versioned binary snapshot format + comparison
internal/duplicate staged duplicate detection
internal/export    JSON / CSV / text reports
internal/config    settings, themes, JSON config file
internal/tui       Bubble Tea model, canvas renderer, views, modals
internal/textutil  display sanitising, width-aware truncation, units

Dependency direction is strictly downward: tui → (treemap, filter, inventory, scan, …); scan → (inventory, platform); treemap depends on nothing but the standard library. The TUI framework (Bubble Tea v2) is only imported by internal/tui and cmd/mapsize.

Inventory representation

inventory.Tree is an append-only arena of Node records addressed by NodeID (uint32). Nodes are stored in fixed-size chunks ([][]Node, 65 536 nodes per chunk), so growing the arena never copies or moves existing nodes and peak memory never doubles during growth.

A node holds: parent ID, first-child / next-sibling IDs (an intrusive child list — no per-directory slices), name (the single path component, not the full path), kind, flags, own logical size, own allocated size, mtime, mode, uid/gid, link count, and aggregates (total logical, total allocated, file count, directory count, error count). Directory-only data — per-category byte totals used for colouring — lives in a side table indexed by a directory index, so files do not pay for it.

Full paths are never stored; Tree.Path(id) walks parents. Extensions are interned into a small table and stored as a uint16. All sizes are int64.

Node IDs are stable for the lifetime of a Tree: nodes are never removed or reordered. A rescan produces a new Tree. This is what makes selection persistence trivial (see below).

Scanner worker architecture

               +------------------ controller goroutine ------------------+
               |  owns: Tree (write lock), pending FIFO of dir NodeIDs,   |
               |        hard-link table, visited-dir table               |
               +----------------------------------------------------------+
                  | jobs (bounded chan, cap = workers*2)   ^ results (bounded chan)
                  v                                        |
             worker 1 .. worker N  (fixed pool, N from --workers / mode)

• Workers do I/O only: open a directory, read entries in chunks of up to 1024, lstat each entry (relative to the directory fd where the platform allows), and send a result chunk back. Workers never touch the Tree.
• The controller is the only writer of the Tree. It applies result chunks, creates child nodes, assigns IDs, propagates aggregate deltas, and decides which subdirectories become new jobs.
• Worker count: conservative = 2, balanced = min(8, 2×CPU), aggressive = min(64, 8×CPU), or an explicit number. More is not always faster — on rotational media 2–4 is often best.

Bounded directory scheduling (no deadlock under backpressure)

The classic deadlock is: workers block sending new work into a full queue while the only consumer of that queue is also blocked. mapsize avoids it by construction:

• Only the controller produces jobs. Workers only produce results.
• The controller's main loop is a single select that simultaneously offers the next pending job to the bounded jobs channel and receives from the bounded results channel. It never performs a blocking send without also being ready to receive. Therefore a worker blocked on results <- is always eventually served, and the controller is never blocked on jobs <- alone.
• Discovered-but-unscheduled directories wait in the controller's pending FIFO as 4-byte NodeIDs (paths are rebuilt at dispatch). Its size is bounded by the number of directory nodes already in the inventory, so it adds O(1) bytes per directory on top of memory we already spend.
• FIFO order means the scan is roughly breadth-first: every top-level directory starts growing early, which is what makes the live treemap informative.

Huge directories are streamed: a worker sends a result per 1024 entries, so a directory with 5 M entries neither balloons worker memory nor stalls the UI.

Completion: the controller tracks inflight (dispatched directories whose final chunk has not arrived). The scan is finished when pending is empty and inflight == 0. Cancellation (context.Context) stops dispatch immediately; workers check the context between chunks and between entries, and the controller drains outstanding results so no goroutine leaks.

Stress tests (internal/scan/scan_test.go) run with 1 worker and job/result channel capacity 1 over trees with thousands of directories to prove progress under maximum backpressure.

Concurrent aggregation strategy

Because the controller is the single writer, aggregation needs no atomics. For each result chunk it sums the chunk's file sizes, allocated sizes, counts and per-category bytes once, then walks the ancestor chain adding that one delta. Cost is O(depth) per 1024-entry chunk rather than O(depth) per file — for a 1 M-file tree at depth 20 that is ~20 K ancestor updates instead of ~20 M. There is no reconciliation pass: totals are exact at every moment for the data seen so far.

The UI reads the tree under Tree.RLock; the controller applies batches of results under Tree.Lock, holding it for at most a few milliseconds (it releases the lock between batches). This is the only lock in the data path.

Concurrency invariants

1. Only the scan controller (or a snapshot loader before publication) mutates a Tree.
2. Every read of Tree node data from another goroutine holds RLock.
3. Nodes are never deleted; a NodeID once observed stays valid.
4. Workers share nothing mutable except the bounded channels and atomic progress counters.

Hard links and sizes

• Logical size = st_size. Allocated size = st_blocks × 512 on Unix. On Windows allocated size is reported as unknown (equal to logical, flagged) because getting it costs an extra handle-open per file.
• For regular files with link count > 1, the controller keys (dev, inode). The first path seen contributes to directory aggregates; later paths are flagged HardlinkDup and contribute zero to aggregates (their own size is still shown). Which path is "first" depends on scan order and is not deterministic. This matches du's default semantics.
• Windows does not expose link counts from directory enumeration, so hard links are not deduplicated there. This is documented, not hidden.

Symlinks, mount points, exclusions

• --follow-symlinks=none (default): symlinks are leaf nodes counted at their own (tiny) size. same-filesystem / all follow directory symlinks; every descended directory's (dev, inode) is recorded and a directory already visited is never descended twice — loops terminate and nothing is counted twice.
• A directory whose device differs from its parent is flagged MountPoint. With --one-file-system it is not descended. On Linux, virtual filesystems (proc, sysfs, cgroup, debugfs, …) are never descended — /proc/kcore would otherwise dominate any scan of / — and are reported in scan info.
• --exclude PATTERN: patterns containing a path separator match full paths; others match the entry name with glob semantics. Excluded entries are counted and the UI shows that exclusions are active.

TUI event flow

scan controller ──(applies under Lock)──> Tree
       │ atomic progress counters
       ▼
tea.Tick (10 Hz while scanning) ──> tickMsg ──> Update marks frame dirty
key/mouse/resize ──> Update ──> state change ──> frame dirty
View(): if dirty → RLock tree → layout (if layout dirty) → paint canvas → string

There is never one message per file: the scanner does not talk to the UI at all except for a single scanDoneMsg. The UI polls at a fixed rate.

View() is called by Bubble Tea after every Update. The model therefore caches the rendered frame and only repaints when something visible changed.

Rendering: the canvas

All drawing goes to an in-memory Canvas of cells (grapheme, fg, bg, attrs). Views and modals paint into the canvas; the canvas is serialised to a string with minimal SGR changes. Bubble Tea's renderer diffs frames and downsamples true-colour to the detected profile (256 / 16 / none), so the drawing code uses RGB throughout. Selection never relies on colour alone: the selected rectangle uses a double-line border, bold reversed title and a status-line description.

Treemap algorithm

treemap.Squarify implements Bruls, Huizing & van Wijk's squarified layout on float rectangles: items sorted by size descending are added to the current row while the worst aspect ratio improves; then the row is laid out along the shorter side. Float rectangles are snapped to the cell grid by rounding the edges (not the widths), so neighbouring rectangles share edges exactly — no gaps, no overlaps. Layout is deterministic (stable sort by size, then ID).

Nested levels are laid out recursively inside each block's interior (inside its border) while the interior is large enough to be useful.

Level-of-detail model

A 160×50 terminal has 8 000 cells; the tree may have millions of entries. For each directory being laid out, children are reduced to the top K by size using a bounded heap (O(n log K)). Children whose proportional area would be smaller than a minimum useful block (a few cells), or beyond K, are merged into one synthetic group block ("312 smaller items, 8.4 GiB"). Zooming into a directory gives its children the whole screen, so they become individually visible. Groups are selectable and inspectable.

Spatial navigation algorithm

treemap.Neighbour(rects, selected, dir) is pure geometry:

1. Candidates must lie in the requested half-plane: for RIGHT, c.X ≥ sel.X + sel.W − tolerance (with the candidate's centre to the right of the selected centre).
2. Each candidate is scored by the primary-axis gap (edge to edge) plus a penalty for the orthogonal offset. Candidates overlapping the selected rectangle's orthogonal span get the offset term set to zero and are preferred over any non-overlapping candidate.
3. Among overlapping candidates, the one whose overlap is largest relative to the source, then nearest to the source's centre line, wins.
4. Ties are broken by top-left position, then NodeID — fully deterministic.

Terminal resize architecture

tea.WindowSizeMsg only records the new size and marks layout dirty. The next View() recomputes panel geometry for the new size class (large / medium / small / too-small), reruns the treemap layout for the new treemap rectangle (never stretches old coordinates), and repaints. Because work happens only in View() and the frame cache absorbs repeated calls, a storm of resize events costs at most one layout per rendered frame. Modal geometry is derived from the current size every frame, clamped to the screen, and becomes scrollable when content does not fit.

Selection persistence

Selection is a NodeID (plus, for the synthetic LOD group, the group's parent ID and a group marker). After any relayout the renderer looks the ID up in the new rectangle list. If it is absent (filtered out, collapsed into a group, zoom changed) selection falls back to: the group containing it, then its nearest visible ancestor, then the largest block.

Modal architecture

A modal is a small interface: Title(), Lines(width), Footer(), and HandleKey. A generic frame renders any modal centred, clamped to the screen (min 1-cell margin), with a scrollable viewport when content exceeds height. The base screen is dimmed before the modal is painted. While a modal is open it receives all keys; Esc always closes it.

OS abstraction

internal/platform exposes ReadDir(path, func(chunk []Entry) error) and Meta structs. Implementations:

• readdir_linux.go — getdents via os.File.ReadDir + statx relative to the directory fd (one syscall per entry, gives birth time).
• readdir_unix.go — darwin/BSD: fstatat relative to the directory fd.
• readdir_windows.go — os.File.ReadDir + Lstat; reparse points (junctions, symlinks) are never descended by default.
• fs_*.go — virtual-filesystem detection, owner name lookup.

No other package switches on runtime.GOOS.

Security handling for hostile filenames

Filenames and snapshot contents are untrusted.

• Every string that reaches the terminal passes through textutil.Sanitize, which replaces C0/C1 control characters (including ESC, CSI, BEL, DEL) with visible escapes (\x1b), replaces invalid UTF-8 with U+FFFD, and neutralises bidi override characters. The canvas also refuses control characters as a second line of defence. Internal names keep the raw bytes.
• Text is clipped by display width (grapheme clusters, East-Asian wide characters) so labels never overflow their rectangle.
• Snapshot decoding validates magic, version, every length, every parent reference (a parent must precede its child, so cycles and recursion are impossible), caps decompressed size and node count, and verifies a SHA-256 trailer. Aggregates are recomputed rather than trusted.
• No shell is ever invoked. "Open/reveal" and trash use direct exec with argument vectors or native APIs; paths are passed as single arguments.
• --read-only disables every mutating action in the UI.