andy/mapsize
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity

Docs: architecture, development and performance for the Rust implementation

Browse Source
This commit is contained in:
Andy Dixon
2026-10-08 19:56:43 +01:00
parent 01785850bf
commit c3ed83250b
5 changed files with 209 additions and 263 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE.md
+81 -64
View File
@@ -5,36 +5,40 @@ part relies on. It is kept in sync with the code; if they disagree, the code
is wrong or this file is stale — fix one of them. is wrong or this file is stale — fix one of them.
``` ```
cmd/mapsize CLI entry point, flag parsing, mode selection src/main.rs CLI entry point, flag parsing
cmd/treegen synthetic test-tree generator src/app.rs mode selection: TUI, reports, snapshots, comparison
internal/brand product name / file extensions (rename in one place) src/brand.rs product name / file extensions (rename in one place)
internal/inventory flat node store, aggregation, indexes, categories src/inventory/ flat node store, aggregation, indexes, categories
internal/scan bounded concurrent scanner (controller + workers) src/scan.rs bounded concurrent scanner (controller + workers)
internal/platform OS-specific metadata (build-tagged files) src/cancel.rs cancellation token shared by scans and background work
internal/treemap squarified layout, LOD, spatial navigation (pure geometry) src/platform.rs OS-specific metadata and file operations (cfg blocks)
internal/filter query lexer / parser / evaluator src/treemap/ squarified layout, LOD, spatial navigation (pure geometry)
internal/snapshot versioned binary snapshot format + comparison src/filter/ query lexer / parser / evaluator
internal/duplicate staged duplicate detection src/snapshot/ versioned binary snapshot format + comparison
internal/export JSON / CSV / text reports src/duplicate.rs staged duplicate detection
internal/config settings, themes, JSON config file src/export.rs JSON / CSV / text reports
internal/tui Bubble Tea model, canvas renderer, views, modals src/config.rs settings, themes, JSON config file
internal/textutil display sanitising, width-aware truncation, units src/tui/ event loop, model, canvas renderer, views, modals
src/textutil.rs display sanitising, width-aware truncation, units
examples/treegen synthetic test-tree generator
examples/bench in-process benchmarks
``` ```
Dependency direction is strictly downward: `tui` → (`treemap`, `filter`, Dependency direction is strictly downward: `tui` → (`treemap`, `filter`,
`inventory`, `scan`, …); `scan` → (`inventory`, `platform`); `treemap` depends `inventory`, `scan`, …); `scan` → (`inventory`, `platform`); `treemap` depends
on nothing but the standard library. The TUI framework (Bubble Tea v2) is only on nothing but the standard library. The terminal library (crossterm) is only
imported by `internal/tui` and `cmd/mapsize`. used by `tui`.
## Inventory representation ## Inventory representation
`inventory.Tree` is an append-only arena of `Node` records addressed by `inventory.Tree` is an append-only arena of `Node` records addressed by
`NodeID` (`uint32`). Nodes are stored in fixed-size chunks `NodeId` (`u32`). Nodes are stored in fixed-size chunks
(`[][]Node`, 65 536 nodes per chunk), so growing the arena never copies or (`Vec<Vec<Node>>`, 65 536 nodes per chunk), so growing the arena never copies or
moves existing nodes and peak memory never doubles during growth. moves existing nodes and peak memory never doubles during growth.
A node holds: parent ID, first-child / next-sibling IDs (an intrusive child A node holds: parent ID, first-child / next-sibling IDs (an intrusive child
list — no per-directory slices), name (the single path component, not the full list — no per-directory vectors), name (the single path component as raw
bytes, not the full
path), kind, flags, own logical size, own allocated size, mtime, mode, path), kind, flags, own logical size, own allocated size, mtime, mode,
uid/gid, link count, and aggregates (total logical, total allocated, file uid/gid, link count, and aggregates (total logical, total allocated, file
count, directory count, error count). Directory-only data — per-category byte count, directory count, error count). Directory-only data — per-category byte
@@ -42,7 +46,7 @@ totals used for colouring — lives in a side table indexed by a directory index
so files do not pay for it. so files do not pay for it.
Full paths are never stored; `Tree.Path(id)` walks parents. Extensions are Full paths are never stored; `Tree.Path(id)` walks parents. Extensions are
interned into a small table and stored as a `uint16`. All sizes are `int64`. interned into a small table and stored as a `u16`. All sizes are `i64`.
Node IDs are stable for the lifetime of a `Tree`: nodes are never removed or Node IDs are stable for the lifetime of a `Tree`: nodes are never removed or
reordered. A rescan produces a new `Tree`. This is what makes selection reordered. A rescan produces a new `Tree`. This is what makes selection
@@ -51,11 +55,11 @@ persistence trivial (see below).
## Scanner worker architecture ## Scanner worker architecture
``` ```
+------------------ controller goroutine ------------------+ +------------------ controller thread ---------------------+
| owns: Tree (write lock), pending FIFO of dir NodeIDs, | | owns: Tree (write lock), pending FIFO of dir NodeIDs, |
| hard-link table, visited-dir table | | hard-link table, visited-dir table |
+----------------------------------------------------------+ +----------------------------------------------------------+
| jobs (bounded chan, cap = workers*2) ^ results (bounded chan) | jobs (bounded channel, cap = workers*2) ^ results (bounded channel)
v | v |
worker 1 .. worker N (fixed pool, N from --workers / mode) worker 1 .. worker N (fixed pool, N from --workers / mode)
``` ```
@@ -77,9 +81,9 @@ while the only consumer of that queue is also blocked. mapsize avoids it by
construction: construction:
* Only the controller produces jobs. Workers only produce *results*. * Only the controller produces jobs. Workers only produce *results*.
* The controller's main loop is a single `select` that simultaneously offers * The controller's main loop is a single `crossbeam_channel::Select` that
the next pending job to the bounded `jobs` channel *and* receives from the simultaneously offers the next pending job to the bounded `jobs` channel
bounded `results` channel. It never performs a blocking send without also *and* receives from the bounded `results` channel. It never performs a blocking send without also
being ready to receive. Therefore a worker blocked on `results <-` is always being ready to receive. Therefore a worker blocked on `results <-` is always
eventually served, and the controller is never blocked on `jobs <-` alone. eventually served, and the controller is never blocked on `jobs <-` alone.
* Discovered-but-unscheduled directories wait in the controller's pending FIFO * Discovered-but-unscheduled directories wait in the controller's pending FIFO
@@ -94,11 +98,12 @@ directory with 5 M entries neither balloons worker memory nor stalls the UI.
Completion: the controller tracks `inflight` (dispatched directories whose Completion: the controller tracks `inflight` (dispatched directories whose
final chunk has not arrived). The scan is finished when `pending` is empty and final chunk has not arrived). The scan is finished when `pending` is empty and
`inflight == 0`. Cancellation (`context.Context`) stops dispatch immediately; `inflight == 0`. Cancellation (`cancel::Cancel`, which the controller also
workers check the context between chunks and between entries, and the selects on) stops dispatch immediately; workers check the token between
controller drains outstanding results so no goroutine leaks. chunks, and the controller drains outstanding results and joins every
worker, so no thread leaks.
Stress tests (`internal/scan/scan_test.go`) run with 1 worker and job/result Stress tests (`scan::tests`) run with 1 worker and job/result
channel capacity 1 over trees with thousands of directories to prove progress channel capacity 1 over trees with thousands of directories to prove progress
under maximum backpressure. under maximum backpressure.
@@ -112,26 +117,27 @@ for a 1 M-file tree at depth 20 that is ~20 K ancestor updates instead of
~20 M. There is no reconciliation pass: totals are exact at every moment for ~20 M. There is no reconciliation pass: totals are exact at every moment for
the data seen so far. the data seen so far.
The UI reads the tree under `Tree.RLock`; the controller applies batches of The tree is shared as `Arc<RwLock<Tree>>`. The UI reads it under the read
results under `Tree.Lock`, holding it for at most a few milliseconds (it lock; the controller applies batches of results under the write lock, holding it for at most a few milliseconds (it
releases the lock between batches). This is the only lock in the data path. releases the lock between batches). This is the only lock in the data path.
**Concurrency invariants** **Concurrency invariants**
1. Only the scan controller (or a snapshot loader before publication) mutates 1. Only the scan controller (or a snapshot loader before publication) mutates
a `Tree`. a `Tree`.
2. Every read of `Tree` node data from another goroutine holds `RLock`. 2. Every read of `Tree` node data from another thread holds the read lock.
3. Nodes are never deleted; a `NodeID` once observed stays valid. 3. Nodes are never deleted; a `NodeID` once observed stays valid.
4. Workers share nothing mutable except the bounded channels and atomic 4. Workers share nothing mutable except the bounded channels and atomic
progress counters. progress counters.
5. No goroutine ever takes `RLock` while already holding it. Go's 5. No thread ever takes the read lock while already holding it. The
`RWMutex` blocks new readers once a writer waits, so a recursive read standard `RwLock` may block new readers once a writer waits, so a
lock can deadlock against the scan controller. The TUI takes the lock recursive read lock can deadlock against the scan controller. The TUI
once per entry point (`View`, `handleKey`, mouse handlers); helpers they takes the lock once per entry point (render, key and mouse handlers) and
call assume it is held. Background commands (filter, duplicates, passes `&Tree` to the helpers it calls. Background tasks (filter,
snapshot save) take their own lock on their own goroutine. duplicates, snapshot save) take their own lock on their own thread.
6. After a scan finishes, the UI goroutine may become the writer (move to 6. After a scan finishes, the UI thread may become the writer (move to
trash updates aggregates under `Lock`). Trash is refused while scanning. trash updates aggregates under the write lock). Trash is refused while
scanning.
## Hard links and sizes ## Hard links and sizes
@@ -166,31 +172,41 @@ releases the lock between batches). This is the only lock in the data path.
## TUI event flow ## TUI event flow
``` ```
scan controller ──(applies under Lock)──> Tree scan controller ──(applies under write lock)──> Tree
│ atomic progress counters │ atomic progress counters
▼ ▼
tea.Tick (10 Hz while scanning) ──> tickMsg ──> Update marks frame dirty tick (10 Hz while scanning) ─────┐
key/mouse/resize ──> Update ──> state change ──> frame dirty crossterm events (input thread) ─┼─> select ─> Model::update ─> state change ─> frame dirty
View(): if dirty → RLock tree → layout (if layout dirty) → paint canvas → string background task results ─────────┘
render: if dirty → read-lock tree → layout (if layout dirty) → paint canvas → write changed lines
``` ```
There is never one message per file: the scanner does not talk to the UI at `tui::run` puts the terminal in raw mode on the alternate screen and selects
all except for a single `scanDoneMsg`. The UI polls at a fixed rate. over three crossbeam channels: terminal events from a dedicated input thread,
a timer tick, and messages from background tasks (filter, duplicates,
snapshot save, scan done). There is never one message per file: the scanner
does not talk to the UI at all except for the scan-done notification. The UI
polls its progress counters at a fixed rate.
`View()` is called by Bubble Tea after *every* `Update`. The model therefore The model caches the rendered frame and only repaints when something
caches the rendered frame and only repaints when something visible changed. visible changed; several updates in a row cost at most one render.
Expensive per-directory work (sorting children by size) is cached per data Expensive per-directory work (sorting children by size) is cached per data
version. During a scan the version advances at most every 400 ms, so the version. During a scan the version advances at most every 400 ms, so the
layout does not jitter at frame rate even though labels and counters update layout does not jitter at frame rate even though labels and counters update
at the 10 Hz refresh. at the 10 Hz refresh.
A panic hook restores the terminal (leaves the alternate screen, disables
raw mode and mouse reporting) before the panic is reported; SIGTERM quits
cleanly.
## Rendering: the canvas ## Rendering: the canvas
All drawing goes to an in-memory `Canvas` of cells (grapheme, fg, bg, attrs). All drawing goes to an in-memory `Canvas` of cells (grapheme, fg, bg, attrs).
Views and modals paint into the canvas; the canvas is serialised to a string Views and modals paint into the canvas; the canvas is serialised with minimal
with minimal SGR changes. Bubble Tea's renderer diffs frames and downsamples SGR changes, converted to the detected colour profile (truecolour / 256 / 16 /
true-colour to the detected profile (256 / 16 / none), so the drawing code none), and only the changed span of each changed line is written, inside a
synchronized-output block where the terminal supports it. The drawing code
uses RGB throughout. For 256-colour terminals the canvas quantizes colours uses RGB throughout. For 256-colour terminals the canvas quantizes colours
itself (redmean distance over the 6×6×6 cube and grey ramp, never the itself (redmean distance over the 6×6×6 cube and grey ramp, never the
palette-dependent first 16 entries): the stock conversion maps dark tints to palette-dependent first 16 entries): the stock conversion maps dark tints to
@@ -226,7 +242,7 @@ visible. Groups are selectable and inspectable.
## Spatial navigation algorithm ## Spatial navigation algorithm
`treemap.Neighbour(rects, selected, dir)` is pure geometry: `treemap::neighbour(blocks, selected, dir)` is pure geometry:
1. Candidates must lie in the requested half-plane: for RIGHT, `c.X ≥ 1. Candidates must lie in the requested half-plane: for RIGHT, `c.X ≥
sel.X + sel.W − tolerance` (with the candidate's centre to the right of the sel.X + sel.W − tolerance` (with the candidate's centre to the right of the
@@ -241,18 +257,18 @@ visible. Groups are selectable and inspectable.
## Terminal resize architecture ## Terminal resize architecture
`tea.WindowSizeMsg` only records the new size and marks layout dirty. The next A resize event only records the new size and marks layout dirty. The next
`View()` recomputes panel geometry for the new size class (large / medium / render recomputes panel geometry for the new size class (large / medium /
small / too-small), reruns the treemap layout for the new treemap rectangle small / too-small), reruns the treemap layout for the new treemap rectangle
(never stretches old coordinates), and repaints. Because work happens only in (never stretches old coordinates), and repaints. Because work happens only
`View()` and the frame cache absorbs repeated calls, a storm of resize events at render time and the frame cache absorbs repeated calls, a storm of resize events
costs at most one layout per rendered frame. Modal geometry is derived from costs at most one layout per rendered frame. Modal geometry is derived from
the current size every frame, clamped to the screen, and becomes scrollable the current size every frame, clamped to the screen, and becomes scrollable
when content does not fit. when content does not fit.
## Selection persistence ## Selection persistence
Selection is a `NodeID` (or, for the synthetic LOD group, `-1 - parentID`). Selection is a `NodeId` (or, for the synthetic LOD group, `-1 - parent`).
Until the user moves the selection it follows the largest block, which Until the user moves the selection it follows the largest block, which
changes while a scan runs; after that it is sticky. After any relayout the renderer looks the ID up in the changes while a scan runs; after that it is sticky. After any relayout the renderer looks the ID up in the
new rectangle list. If it is absent (filtered out, collapsed into a group, new rectangle list. If it is absent (filtered out, collapsed into a group,
@@ -261,8 +277,8 @@ nearest visible ancestor, then the largest block.
## Modal architecture ## Modal architecture
A modal is a small interface: `Title()`, `Lines(width)`, `Footer()`, and A modal is a small trait: `title()`, `content(width)`, `footer()`, and
`HandleKey`. A generic frame renders any modal centred, clamped to the screen `key`. A generic frame renders any modal centred, clamped to the screen
(min 1-cell margin), with a scrollable viewport when content exceeds height. (min 1-cell margin), with a scrollable viewport when content exceeds height.
The base screen is dimmed before the modal is painted. While a modal is open The base screen is dimmed before the modal is painted. While a modal is open
it receives all keys; `Esc` always closes it. it receives all keys; `Esc` always closes it.
@@ -286,7 +302,7 @@ No other package switches on `runtime.GOOS`.
Filenames and snapshot contents are untrusted. Filenames and snapshot contents are untrusted.
* Every string that reaches the terminal passes through * Every string that reaches the terminal passes through
`textutil.Sanitize`, which replaces C0/C1 control characters (including `textutil::sanitize`, which replaces C0/C1 control characters (including
ESC, CSI, BEL, DEL) with visible escapes (`\x1b`), replaces invalid UTF-8 ESC, CSI, BEL, DEL) with visible escapes (`\x1b`), replaces invalid UTF-8
with U+FFFD, and neutralises bidi override characters. The canvas also with U+FFFD, and neutralises bidi override characters. The canvas also
refuses control characters as a second line of defence. Internal names keep refuses control characters as a second line of defence. Internal names keep
@@ -299,9 +315,10 @@ Filenames and snapshot contents are untrusted.
(≤ 1 EiB) and node count (at most one node per byte of snapshot file, 1 Mi floor); caps the decompressed body at 64× the file size (≤ 1 EiB) and node count (at most one node per byte of snapshot file, 1 Mi floor); caps the decompressed body at 64× the file size
(decompression bombs fail fast); and verifies a SHA-256 trailer. (decompression bombs fail fast); and verifies a SHA-256 trailer.
Aggregates are recomputed with saturating arithmetic rather than trusted. Aggregates are recomputed with saturating arithmetic rather than trusted.
* No shell is ever invoked. "Open/reveal" and trash use direct `exec` with * No shell is ever invoked. "Open/reveal" and trash use `std::process::Command`
argument vectors or native APIs; paths are passed as single arguments. with argument vectors or native APIs; paths are passed as single arguments.
* Permanent delete (headless only) checks the item's owner and type against * Permanent delete (headless only) checks the item's owner and type against
the scan and removes it through an `os.Root` on the parent directory, so the scan and removes it through descriptors opened from the parent
a symlink swapped in since the scan cannot redirect it. directory without following symlinks (`openat`/`unlinkat` with
`O_NOFOLLOW`), so a symlink swapped in since the scan cannot redirect it.
* `--read-only` disables every mutating action in the UI. * `--read-only` disables every mutating action in the UI.
DEVELOPMENT.md
+58 -44
View File
@@ -1,61 +1,77 @@
# Development # Development
Requires Go (see `go.mod`). No CGO, no external tools. Requires a stable Rust toolchain (`rustup`); the dependencies are pinned in
`Cargo.lock`. No C toolchain beyond what Rust itself needs.
```sh ```sh
make build # bin/mapsize, bin/treegen make build # target/release/mapsize
make test # go test ./... make test # cargo test --release
make race # go test -race ./... make lint # clippy, warnings are errors
make vet fmt-check make fmt-check # rustfmt
make benchmark # go test -bench make benchmark # in-process benchmarks (layout, render, filter): examples/bench.rs
make fuzz # short fuzz runs of snapshot / treemap / filter make treegen # synthetic test-tree generator: examples/treegen.rs
make release # release archives and packages in bin/release make release # local release assets in dist/release (see below)
``` ```
CI (`.github/workflows/ci.yml`) runs rustfmt and clippy, the tests on Linux
(x86-64 and ARM64), macOS (Intel and Apple silicon) and Windows, and
cross-builds every release target, running the ARM and RISC-V Linux
binaries under qemu.
## Platforms
`scripts/build.sh` builds `dist/mapsize-OS-ARCH`:
| OS | Architectures | Rust target | Built with |
|---|---|---|---|
| Linux | amd64, arm64, 386, arm (v7), riscv64 | `*-unknown-linux-musl` (static) | cargo-zigbuild + zig |
| Windows | amd64, arm64 | `x86_64-pc-windows-gnu`, `aarch64-pc-windows-gnullvm` | cargo-zigbuild + zig |
| macOS | amd64, arm64 | `*-apple-darwin` | cargo on a Mac (or zigbuild) |
| FreeBSD, NetBSD | amd64 | `x86_64-unknown-{freebsd,netbsd}` | cross + Docker |
Platform differences live in `src/platform.rs` (and `cfg` blocks where the
terminal code needs them); nothing else switches on the OS.
## Releasing ## Releasing
Example for 1.2.0; substitute the real version throughout. Example for 1.2.0; substitute the real version throughout.
1. Update the version in `internal/brand/brand.go` and `docs/mapsize.1`, 1. Update the version in `Cargo.toml` (then `cargo check` to refresh
and add a `## 1.2.0` section to CHANGELOG.md. Commit. `Cargo.lock`) and `docs/mapsize.1`, and turn `## Unreleased` in
CHANGELOG.md into `## 1.2.0 — DATE`. Commit.
2. Tag and push: 2. Tag and push:
```sh ```sh
git tag -a v1.2.0 -m "mapsize 1.2.0" git tag -a v1.2.0 -m "mapsize 1.2.0"
git push origin master v1.2.0 git push origin master v1.2.0
``` ```
3. Build everything (archives, .deb/.rpm/.apk/Arch packages, SHA256SUMS 3. The tag starts `.github/workflows/release.yml`, which builds every
into `bin/release`; the first run downloads `nfpm`): target, packages them (archives, .deb/.rpm/.apk/Arch packages,
```sh SHA256SUMS) and publishes the GitHub release with the changelog section
make release VERSION=v1.2.0 as its notes. Watch it with `gh run watch`. `make release` builds the
``` same assets locally into `dist/release`, for whichever targets the host
4. Publish the GitHub release with the changelog section as notes: can build (`scripts/build.sh`, then `scripts/release.sh`).
```sh 4. Check the release page once the workflow has finished.
awk '/^## 1.2.0/{f=1;next} /^## /{f=0} f' CHANGELOG.md > /tmp/notes.md
gh release create v1.2.0 --title v1.2.0 --notes-file /tmp/notes.md bin/release/*
```
5. Update Homebrew. This needs the tag pushed (step 2), because it 5. Update Homebrew. This needs the tag pushed (step 2), because it
checksums GitHub's source tarball for that tag: checksums GitHub's source tarball for that tag:
```sh ```sh
make brew-formula VERSION=v1.2.0 # writes bin/homebrew/mapsize.rb make brew-formula VERSION=v1.2.0 # writes dist/homebrew/mapsize.rb
git clone https://github.com/andydixon/homebrew-tap /tmp/tap git clone https://github.com/andydixon/homebrew-tap /tmp/tap
cp bin/homebrew/mapsize.rb /tmp/tap/Formula/mapsize.rb cp dist/homebrew/mapsize.rb /tmp/tap/Formula/mapsize.rb
git -C /tmp/tap commit -am "mapsize 1.2.0" && git -C /tmp/tap push git -C /tmp/tap commit -am "mapsize 1.2.0" && git -C /tmp/tap push
``` ```
Users get it with `brew upgrade mapsize`. Template: Users get it with `brew upgrade mapsize`. Template:
`packaging/homebrew/mapsize.rb.in`. `packaging/homebrew/mapsize.rb.in`.
6. Publish to the apt, dnf and pacman repository at repo.dixon.cx. This 6. Publish to the apt, dnf and pacman repository at repo.dixon.cx. This
builds signed .deb, .rpm and Arch packages from the tag and rebuilds the builds signed .deb, .rpm and Arch packages from the tag and rebuilds the
repository indexes (needs docker, gpg, apt-ftparchive and the signing key repository indexes (needs cargo-zigbuild and zig, docker, gpg,
in `~/.config/repo.dixon.cx/gnupg`): apt-ftparchive and the signing key in `~/.config/repo.dixon.cx/gnupg`):
```sh ```sh
packaging/repo.sh v1.2.0 packaging/repo.sh v1.2.0
``` ```
The same script, with its own project block, publishes vault. The same script, with its own project block, publishes vault.
Always pass `VERSION=`: otherwise it comes from `git describe`, which only The binary reports the version in `Cargo.toml`; `scripts/release.sh`
matches the tag when HEAD is exactly the tagged commit. `make release` and refuses to package a tag that does not match it.
the Homebrew formula stamp the version into the binary; `make build` and
`go install` use the value in `brand.go`, which is why step 1 updates it.
The snapshot format is stable at version 1: a format change must bump The snapshot format is stable at version 1: a format change must bump
`snapshot.Version` and keep the reader for version 1. `snapshot.Version` and keep the reader for version 1.
@@ -65,17 +81,16 @@ The snapshot format is stable at version 1: a format change must bump
`treegen` creates synthetic trees: `treegen` creates synthetic trees:
```sh ```sh
bin/treegen --out /tmp/t --directories 50000 --files 1000000 --max-depth 20 --sparse --unicode cargo run --release --example treegen -- --out /tmp/t --directories 50000 --files 1000000 --max-depth 20 --sparse --unicode
bin/mapsize /tmp/t target/release/mapsize /tmp/t
``` ```
## Benchmarks ## Benchmarks
```sh ```sh
make benchmark # Go micro-benchmarks (layout, render, filter, scan) make benchmark # in-process: layout, render, navigation, filters
scripts/bench-scan.sh /path 1 2 4 8 # wall time and peak RSS per worker count scripts/bench-scan.sh /path 1 2 4 8 # wall time and peak RSS per worker count
sudo DROP_CACHES=1 scripts/bench-scan.sh /path # cold-cache runs sudo DROP_CACHES=1 scripts/bench-scan.sh /path # cold-cache runs
MAPSIZE_MEM_ROOT=/path go test ./internal/scan -run TestMemoryPerNode -v
``` ```
Paste results into PERFORMANCE.md with the machine description. Never edit Paste results into PERFORMANCE.md with the machine description. Never edit
@@ -84,26 +99,25 @@ numbers by hand.
## Terminal restoration checks ## Terminal restoration checks
```sh ```sh
MAPSIZE_DEBUG_PANIC=ui bin/mapsize . # panics on the first key press MAPSIZE_DEBUG_PANIC=ui target/release/mapsize . # panics on the first key press
MAPSIZE_DEBUG_PANIC=scanner bin/mapsize / # panics in a scanner goroutine MAPSIZE_DEBUG_PANIC=scanner target/release/mapsize / # panics in a scanner thread (aborts)
stty -a # echo/icanon must be back on stty -a # echo/icanon must be back on
``` ```
## Profiling ## Profiling
```sh Build with symbols (`CARGO_PROFILE_RELEASE_STRIP=false
bin/mapsize --no-ui --cpuprofile cpu.prof --memprofile mem.prof /path CARGO_PROFILE_RELEASE_DEBUG=1 cargo build --release`) and use the platform
go tool pprof -http=: cpu.prof profiler, for example `perf record -g target/release/mapsize --no-ui /path`
``` on Linux or Instruments on macOS. `examples/bench.rs` times the hot UI paths
in-process.
`--log FILE --log-level debug` writes a log without disturbing the TUI.
Paths are logged only at `debug`/`trace` level.
## Layout ## Layout
See ARCHITECTURE.md. Rules of thumb: See ARCHITECTURE.md. Rules of thumb:
* `internal/treemap` and `internal/filter` must not import UI packages. * `treemap` and `filter` must not depend on `tui`.
* Only `internal/platform` may contain OS-specific code (build tags). * OS-specific code lives in `platform` (`cfg` attributes); the terminal code
* All strings derived from filenames go through `textutil.Sanitize` before has the only other `cfg` blocks.
* All strings derived from filenames go through `textutil::sanitize` before
reaching a terminal. reaching a terminal.
PERFORMANCE.md
+69 -61
View File
@@ -3,107 +3,115 @@
All numbers below were measured, not estimated. Re-run them on your own All numbers below were measured, not estimated. Re-run them on your own
hardware before relying on them; filesystem caches dominate scan timings. hardware before relying on them; filesystem caches dominate scan timings.
## Test machine (2026-10-02) ## Test machine (2026-10-08)
| | | | | |
|---|---| |---|---|
| CPU | Intel Core i7-7700 @ 3.60 GHz, 4 cores / 8 threads | | CPU | Intel Core i7-7700 @ 3.60 GHz, 4 cores / 8 threads |
| RAM | 32 GiB | | RAM | 32 GiB |
| OS | Linux 7.0, Go 1.27.1 | | OS | Linux 7.0, Rust 1.91.1 (release profile: LTO, one codegen unit) |
| Filesystem | ext4 on Linux software RAID (`/dev/md2`) over two 7200 rpm SATA HDDs | | Filesystem | ext4 on Linux software RAID (`/dev/md2`) over two 7200 rpm SATA HDDs |
The host is shared; its load average was 5–9 during these runs. Each figure
is the median (and best) of 7 runs.
## Scanner ## Scanner
Dataset: `~/go/pkg/mod` — 388,307 files + 65,518 directories (458,510 Dataset: `~/go/pkg/mod`, a large module cache: 388,307 files + 65,518
inventory nodes), 10.5 GiB allocated. **Warm cache only**: dropping the page directories (458,510 inventory nodes), 10.5 GiB allocated. **Warm cache
cache needs root, which was not available, so these measure CPU and syscall only**: dropping the page cache needs root, so these measure CPU and syscall
cost, not disk seeks. Cold-cache behaviour on HDDs will be dominated by seek cost, not disk seeks. Cold-cache behaviour on HDDs will be dominated by seek
time; expect fewer workers to be optimal there. time; expect fewer workers to be optimal there.
`mapsize --no-ui --workers N ~/go/pkg/mod`, wall time, 3 runs each `mapsize --no-ui --workers N PATH`:
(`/usr/bin/time`):
| workers | run 1 | run 2 | run 3 | entries/s (best) | peak RSS | | workers | median | best | peak RSS |
|---:|---:|---:|---:|---:|---:| |---:|---:|---:|---:|
| 1 | 1.77 s | 1.67 s | 1.65 s | ~278,000 | 117 MB | | 1 | 1.52 s | 1.44 s | 75 MB |
| 2 | 1.06 s | 0.99 s | 1.00 s | ~463,000 | 120 MB | | 8 (balanced default) | 0.47 s | 0.43 s | 76 MB |
| 4 | 0.59 s | 0.59 s | 0.62 s | ~777,000 | 117 MB | | 32 | 0.46 s | 0.43 s | 80 MB |
| 8 (balanced default) | 0.48 s | 0.50 s | 0.51 s | ~955,000 | 121 MB |
| 16 | 0.45 s | 0.44 s | 0.43 s | ~1,066,000 | 121 MB |
| 32 | 0.43 s | 0.44 s | 0.42 s | ~1,092,000 | 122 MB |
For comparison, `du -s` on the same tree (warm): 1.59 s, 34 MB RSS. At 8 workers that is about 1,000,000 entries a second, in under 80 MB for
458k nodes. Most of the time is spent in the `statx` and `getdents` system
calls.
CPU profile at 8 workers: ~60 % of samples in `statx`/`getdents` syscalls; | other runs over the same tree | median | peak RSS |
path reconstruction for job dispatch ~7 %; aggregation is negligible. |---|---:|---:|
| scan + `--largest-files 50` | 0.60 s | 76 MB |
### Memory | scan + `--json` (streamed to /dev/null) | 0.86 s | 76 MB |
`go test ./internal/scan -run TestMemoryPerNode` with
`MAPSIZE_MEM_ROOT=~/go/pkg/mod`: 458,510 nodes, 69.1 MiB live heap,
**~158 bytes per node** (node record ~112 bytes plus name storage and
directory side tables). Extrapolated, 10 M entries need ~1.6 GiB.
## Snapshots ## Snapshots
Same 458,510-node tree: | operation | median | peak RSS |
|---|---:|---:|
| scan + save | 0.75 s | 77 MB |
| load (`mapsize --no-ui file.msz`) | 0.23 s | 78 MB |
| load a snapshot written by mapsize 1.2.0 | 0.26 s | 78 MB |
| operation | result | Loading decompresses and verifies SHA-256 on one thread while another parses.
|---|---:| Against a warm cache that is about twice as fast as rescanning; against a
| snapshot file size (gzip level 1) | 6.7 MB (~15 bytes/node) | cold cache or a network filesystem, where scans are dominated by I/O
| scan + save | 1.03 s | latency, the gain is far larger (not yet measured here).
| load (`mapsize --no-ui file.msz`), 3 runs | 0.28 / 0.29 / 0.32 s, 81 MB RSS |
| warm-cache rescan for comparison | 0.55 s |
Loading decompresses and verifies SHA-256 on a separate goroutine, overlapping
with parsing. Against a warm cache the win is ~1.9×; against a cold cache or
a network filesystem, where scans are dominated by I/O latency, it is far
larger (not yet measured here).
## UI ## UI
`go test ./internal/tui -bench Big` on a synthetic in-memory tree of `make benchmark` (`examples/bench.rs`), best of 5 runs, on a synthetic
1,008,201 nodes (200 directories × 40 subdirectories × 100 files, plus one in-memory tree of 1,008,201 nodes (200 directories × 40 subdirectories × 100
flat directory with 200,000 files), 200×60 terminal: files, plus one flat directory with 200,000 files), 200×60 terminal:
| operation | time per op | | operation | time per op |
|---|---:| |---|---:|
| cold frame at root (sort children, LOD, layout incl. nested previews, paint) | 33.8 ms | | cold frame at root (sort children, LOD, layout incl. nested previews, paint) | 11.34 ms |
| resize to a new size (layout + paint, child lists cached) | 2.0 ms | | resize to a new size (layout + paint, child lists cached) | 1.50 ms |
| arrow-key move + re-render | 1.5 ms | | arrow-key move + re-render | 1.31 ms |
| zoomed into the 200,000-entry directory, cold | 29.5 ms | | zoomed into the 200,000-entry directory, cold | 9.37 ms |
| render of a small tree (2.4 K nodes) at 200×60 | 0.66 ms |
| resize relayout of the small tree | 0.45 ms |
During a scan the size caches are invalidated at most every 400 ms, so a During a scan the size caches are invalidated at most every 400 ms, so even
cold frame (worst case above ~34 ms) costs under 10 % of one core; other a cold frame costs a few percent of one core. Resize events only record the
frames reuse caches. Resize events only record the new size; layout happens new size; layout happens once per rendered frame, so a resize storm cannot
once per rendered frame, so a resize storm cannot queue up work. queue up work.
### Geometry ### Geometry
`go test ./internal/treemap -bench .`:
| operation | time | | operation | time |
|---|---:| |---|---:|
| squarified layout, 300 items | 89 µs | | squarified layout, 300 items | 8.4 µs |
| spatial neighbour lookup among 300 blocks | 2.1 µs | | spatial neighbour lookup among 300 blocks | 1.1 µs |
### Filtering (1,008,201 nodes) ### Filtering (1,008,201 nodes)
| query | time | | query | time |
|---|---:| |---|---:|
| `size > 1MB` | 31 ms | | `size > 1MB` | 26.75 ms |
| `*.dat AND size > 500k` | 119 ms | | `*.dat AND size > 500k` | 67.84 ms |
| `path contains sub3` | 112 ms | | `path contains sub3` | 63.80 ms |
Filters run on a background goroutine with cancellation and a 120 ms Filters run on a background thread with cancellation and a 120 ms keystroke
keystroke debounce; the UI never waits for them. debounce; the UI never waits for them. Path predicates build each path from
the cached parent directory path into a reused buffer, and `*.ext` globs are
a suffix test.
## Compared with mapsize 1.2.0
mapsize 1.2.0 was the last release of the previous implementation, written
in Go. Both were run alternately on the same machine, over the same tree, in
the same session:
| | 1.2.0 | current | |
|---|---:|---:|---|
| scan, 8 workers (median) | 0.63 s | 0.47 s | 1.34× faster |
| scan, 1 worker (median) | 1.89 s | 1.52 s | 1.24× faster |
| peak RSS while scanning | 117 MB | 76 MB | |
| scan + save snapshot | 1.22 s | 0.75 s | 1.63× faster |
| load snapshot | 0.34 s | 0.23 s | 1.48× faster |
| scan + JSON export | 1.27 s, 153 MB | 0.86 s, 76 MB | 1.48× faster |
| cold frame, 1M nodes | 36.01 ms | 11.34 ms | 3.18× faster |
| squarified layout, 300 items | 84.3 µs | 8.4 µs | 10.07× faster |
| arrow key + re-render | 1.50 ms | 1.31 ms | 1.14× faster |
| filter `*.dat AND size > 500k` | 121.40 ms | 67.84 ms | 1.79× faster |
## Known costs / future work ## Known costs / future work
* Path predicates now build each path from the cached parent directory path
(244 ms → 112 ms); globs are dominated by `filepath.Match`.
* Node records could shrink (owner/group/mode/link count are only needed for * Node records could shrink (owner/group/mode/link count are only needed for
display and some filters) if memory on 10 M+ entry trees matters. display and some filters) if memory on 10 M+ entry trees matters.
* Cold-cache HDD benchmarks still need to be run with root access. * Cold-cache HDD benchmarks still need to be run with root access.
RUST-PORT-NOTES.md
-93
View File
@@ -1,93 +0,0 @@
# mapsize — Rust port
A port of the Go implementation (repo root) to Rust, with the same CLI, TUI,
keys, themes and snapshot format. `.msz` snapshots are byte-compatible in
both directions. The code is Linux/Unix only; the Windows code paths were
not ported.
```
cd rust
cargo build --release # target/release/mapsize
cargo test --release # 69 tests (ported from the Go suites, fuzz tests excepted)
```
Module layout mirrors `internal/`: `inventory`, `scan`, `platform`, `treemap`,
`filter`, `snapshot` (+ `compare`), `duplicate`, `export`, `config`,
`textutil`, `tui`, `app`. The tree is shared as `Arc<RwLock<Tree>>` with the
same single-writer invariants as ARCHITECTURE.md; `cancel::Cancel` replaces
`context.Context`; crossterm replaces Bubble Tea as the event loop (the
canvas renderer is ported as-is, plus line-diffed frame output).
## Known differences from Go
* `--log`/`--log-level` are accepted but there is no debug log yet.
* A scanner thread panic restores the terminal and aborts the process instead
of returning an "internal error" message.
* The colour profile is detected from `NO_COLOR`/`TERM`/`COLORTERM` only (no
terminal query); `--color` still overrides.
* JSON export does not HTML-escape `<>&`; `started` is written in UTC.
* The filter lexer no longer hangs on queries containing bytes Go treats as
white space but never skips (e.g. `voilà`, whose UTF-8 contains 0xA0).
* Huge sizes saturate rather than wrap in filters and snapshot comparison.
## Benchmarks
Machine: Intel Core i7-7700 (4C/8T), 32 GiB, ext4 on md RAID over 7200 rpm
HDDs. Go 1.27.1, Rust 1.91.1 (`lto`, `codegen-units = 1`). The host is
shared, and its load average was 5–9 on 8 threads during the runs. Go's
absolute times are close to those in PERFORMANCE.md. Go and Rust runs were
interleaved; the tables report best-of (micro) or median and best
(end-to-end). To re-run:
```
go test -c -o /tmp/tui.test ./internal/tui && go test -c -o /tmp/treemap.test ./internal/treemap
(cd rust && cargo build --release --example bench)
rust/bench/micro.py /tmp/tui.test /tmp/treemap.test rust/target/release/examples/bench 5
rust/bench/compare.py bin/mapsize rust/target/release/mapsize ~/go/pkg/mod 7
```
### In-process (Go `go test -bench` suite vs `examples/bench.rs`)
The tree is 1,008,201 synthetic nodes, rendered at 200×60.
| benchmark | Go | Rust | speed-up |
|---|---:|---:|---:|
| treemap/Squarify(300) | 84.3 µs | 8.4 µs | 10.07× |
| treemap/Neighbour(300) | 2.1 µs | 1.1 µs | 1.94× |
| tui/BigColdFrame | 36.01 ms | 11.34 ms | 3.18× |
| tui/BigResize | 2.02 ms | 1.50 ms | 1.35× |
| tui/BigFlatDirCold | 32.00 ms | 9.37 ms | 3.41× |
| tui/BigNavigate | 1.50 ms | 1.31 ms | 1.14× |
| tui/BigFilter/size > 1MB | 29.44 ms | 26.75 ms | 1.10× |
| tui/BigFilter/*.dat AND size > 500k | 121.40 ms | 67.84 ms | 1.79× |
| tui/BigFilter/path contains sub3 | 101.86 ms | 63.80 ms | 1.60× |
### End to end (`~/go/pkg/mod`: 388k files + 65k dirs, warm cache)
| scenario | Go median | Rust median | Go best | Rust best | Go RSS | Rust RSS | speed-up |
|---|---:|---:|---:|---:|---:|---:|---:|
| scan, 1 workers | 1.89 s | 1.52 s | 1.72 s | 1.44 s | 112 MB | 75 MB | 1.24× |
| scan, 8 workers | 0.63 s | 0.47 s | 0.58 s | 0.43 s | 117 MB | 76 MB | 1.34× |
| scan, 32 workers | 0.60 s | 0.46 s | 0.48 s | 0.43 s | 118 MB | 80 MB | 1.30× |
| scan + save snapshot | 1.22 s | 0.75 s | 1.13 s | 0.71 s | 119 MB | 77 MB | 1.63× |
| load own snapshot | 0.34 s | 0.23 s | 0.30 s | 0.23 s | 85 MB | 78 MB | 1.48× |
| load Go snapshot | 0.36 s | 0.26 s | 0.30 s | 0.23 s | 85 MB | 78 MB | 1.38× |
| scan + JSON export | 1.27 s | 0.86 s | 1.04 s | 0.68 s | 153 MB | 76 MB | 1.48× |
| scan + top 50 largest files | 0.70 s | 0.60 s | 0.64 s | 0.47 s | 114 MB | 76 MB | 1.17× |
Reading the numbers:
* **Scanning** is 1.2–1.3× faster at every worker count. It is syscall-bound
(`statx`/`getdents`), so the gain comes from lower per-entry overhead in the
controller (no GC, no per-chunk allocation churn), not from faster I/O.
* **Memory**: peak RSS is about a third lower (76 MB vs 117 MB while
scanning 458k nodes). The JSON export halves memory (76 MB vs 153 MB) because
it streams without per-node allocations.
* **Snapshots**: scan + save is 1.6× faster and load is 1.4–1.5× faster. The
two binaries load each other's files.
* **UI**: cold frames are 3.2–3.4× faster, and squarified layout is about 10×
faster. Navigation and resize gain less (1.1–1.35×) because they mostly
reuse cached layout.
* **Filters**: 1.1–1.8× faster. This required caching lower-cased directory
paths in a `Vec` with reused buffers, and a `*.ext` fast path in the
`filepath.Match` port.
scripts/bench-scan.sh
+1 -1
View File
@@ -10,7 +10,7 @@ set -euo pipefail
path=${1:?usage: bench-scan.sh PATH [workers...]} path=${1:?usage: bench-scan.sh PATH [workers...]}
shift || true shift || true
workers=${*:-1 2 4 8 16} workers=${*:-1 2 4 8 16}
bin=${MAPSIZE:-bin/mapsize} bin=${MAPSIZE:-target/release/mapsize}
[[ -x $bin ]] || make build >/dev/null [[ -x $bin ]] || make build >/dev/null
fs=$(df -T "$path" | awk 'NR==2 {print $2}') fs=$(df -T "$path" | awk 'NR==2 {print $2}')