From 31dbdc7009f6f43588e98c3b5b5ccb06eec690a2 Mon Sep 17 00:00:00 2001 From: Andy Dixon Date: Thu, 8 Oct 2026 14:15:30 +0100 Subject: [PATCH] Benchmark results: Go 0.5 against Rust 0.6 Co-Authored-By: Claude --- docs/BENCHMARKS.md | 85 ++++++++++++++++++++++++++++++++++++++++++++++ scripts/bench.sh | 47 +++++++++++++++++-------- 2 files changed, 117 insertions(+), 15 deletions(-) create mode 100644 docs/BENCHMARKS.md diff --git a/docs/BENCHMARKS.md b/docs/BENCHMARKS.md new file mode 100644 index 0000000..1a6688f --- /dev/null +++ b/docs/BENCHMARKS.md @@ -0,0 +1,85 @@ +# Benchmarks: Go 0.5 against Rust 0.6 + +Measured on 2026-10-08 on one machine: Intel Core i7-7700 (4 cores, 8 +threads), 31 GiB RAM, ext4 on a two-disk RAID 1 (`/dev/md2`), Linux 7.0. +Both binaries were built from this tree: the Go oracle (`oracle/go`, Go +1.27.1, `CGO_ENABLED=0`) and the Rust release build (Rust 1.99.0, LTO). + +**Limitation:** the machine was shared. Other users' Docker builds kept the +load average between 4 and 12 throughout, so absolute times are noisy. +Go and Rust were run alternately under the same conditions, and every +figure is a median with its spread, but small differences mean nothing. +Page caches were warm (dropping them needs root), so cold-cache numbers +were not measured. + +## Primitives + +`cargo bench --bench micro` and +`(cd oracle/go && go test -run XXX -bench Oracle -benchtime 2s ./ ./internal/pqcrypt ./internal/stream)`, +run alternately twice (load 4.4 to 5.4). Medians in milliseconds; the Rust +harness's p10 to p90 range is in brackets. + +| Operation | Go | Rust | +| --- | --- | --- | +| Derive hybrid identity from seed | 5.6 | 3.0 (2.9 to 4.6) | +| Wrap file key (both encapsulations) | 7.4 to 8.0 | 3.1 (3.0 to 4.8) | +| Unwrap file key (both decapsulations) | 7.6 to 8.3 | 3.1 (3.0 to 3.6) | +| Encrypt and decrypt one 4 MiB object | 19.9 to 20.1 | 19.0 to 19.5 | +| STREAM seal and open, 64 MiB, streaming API | 89 to 92 | 187 to 188 | +| Unlock key file (Argon2id for both slots) | 121 to 125 | 129 (125 to 147) | + +- The FrodoKEM port is about twice as fast as CIRCL's Go code, mostly from + multiplying A row by row instead of walking it by column. +- ChaCha20-Poly1305 is slower: RustCrypto reaches about 1.2 GB/s on this CPU + (`cargo bench --bench aead_probe`), Go's assembly about twice that. The + 64 MiB row uses the streaming reader and writer, which copy through + 64 KiB buffers. The vault itself uses the in-memory path (`seal_all` + 0.85 GB/s, `open_all` 1.4 GB/s), and per 4 MiB object the two are + level because the KEM work is cheaper. +- Unlock takes 180 ms with Argon2id lanes computed in turn. Computing the + four lanes in parallel (argon2's `parallel` feature) brings it to 129 ms, + close to Go. The parameters are unchanged. + +## End to end + +`RUNS=3 BIG_MB=1024 SMALL=500 scripts/bench.sh` (load 8 to 13). Every step +includes unlocking the key. The 1 GiB file and the 500 small files (1 to +16 KiB each) are each added to a fresh vault. The reads use one vault +holding both, plus filler. Times are medians of 3 (min to max). + +| Step | Go | Rust | +| --- | --- | --- | +| add 1 GiB file | 41.7 s (32.0 to 50.3) | 29.6 s (28.5 to 31.0) | +| add 500 small files | 123.9 s (113.3 to 156.2) | 79.0 s (77.3 to 133.3) | +| get 1 GiB file | 9.8 s (8.8 to 10.0) | 16.2 s (15.5 to 16.7) | +| list (about 530 records) | 4.95 s (4.89 to 5.20) | 0.74 s (0.71 to 0.77) | +| get one small file | 5.24 s (4.88 to 6.00) | 0.74 s (0.70 to 0.81) | +| Peak RSS | 137 to 140 MiB | 132 MiB | +| Peak threads | 18 | 17 | +| Binary size | 11.5 MB | 3.5 MB (static musl: 3.4 MB) | + +- **Catalogue reads** (`list`, and every command that looks a name up) are + about 6 times faster. Records are decrypted in parallel on the worker pool + (bounded at twice the worker count in flight), and each FrodoKEM + decapsulation is cheaper. +- **Adds** are faster. Both implementations fsync every object, its shard + directory and every record. Rust overlaps up to eight objects' encryption + and syncs on the pool. +- **`get` of a large file is slower, on purpose.** Rust fsyncs the output + file before giving it its name, so a crash just after `get` cannot leave + the name pointing at an empty or partial file. Go did not. On an idle + rerun (load about 8, output on the same disk), Go took 3.6 s and Rust + 15.5 s. Of Rust's time, 13.3 s was the single `fsync` of 1 GiB of + freshly written data (measured with `strace -T`). The other 2 s or so is + unlock, decryption and writing. This is the cost of the + durability barrier on this RAID 1 array, and it scales with the disk's + write speed. +- Peak memory is dominated by Argon2id: 64 MiB for each of the two slots, + computed at the same time in both implementations. Rust's object pipeline + adds at most eight 4 MiB plaintext buffers and their ciphertexts. Peak + threads: Rust has 8 workers, 8 Argon2 threads while unlocking, and the + main thread. + +Reproduce with `scripts/bench.sh` (it builds both binaries, creates +synthetic data in `target/bench`, and prints the tables' numbers) and the +two micro-benchmark commands above. diff --git a/scripts/bench.sh b/scripts/bench.sh index 32b2aa5..4cda314 100755 --- a/scripts/bench.sh +++ b/scripts/bench.sh @@ -5,9 +5,9 @@ # RUNS=5 BIG_MB=2048 SMALL=5000 scripts/bench.sh # # For each implementation, in a fresh key and vault on the same filesystem: -# add one BIG_MB file, get it back, add SMALL small files (1-16 KiB) as a -# folder, list the catalogue (filler plus SMALL+1 records), and get one small -# file. Each step is timed RUNS times (median, min, max), with peak RSS and +# add one BIG_MB file, and add SMALL small files (1-16 KiB) as a folder, each +# into a fresh vault; then, in one vault holding both, get the big file, +# list the catalogue (filler plus SMALL+1 records), and get one small file. Each step is timed RUNS times (median, min, max), with peak RSS and # peak thread count. Every step includes unlocking the key (two Argon2id # runs). Both implementations fsync every object and record, so `add` # timings include durable writes. Page caches are warm: dropping them needs @@ -20,7 +20,7 @@ root=$PWD work=${1:-$root/target/bench} runs=${RUNS:-3} big_mb=${BIG_MB:-1024} -small=${SMALL:-2000} +small=${SMALL:-500} mkdir -p -- "$work" work=$(cd -- "$work" && pwd -P) @@ -65,25 +65,42 @@ summarise() { # lines of "secs rss threads" -> median min max, max rss, max thre sort -n | awk '{t[NR]=$1; if($2>r)r=$2; if($3>th)th=$3} END {printf "%8.2f s (min %.2f, max %.2f) rss %6.0f MiB threads %d\n", t[int((NR+1)/2)], t[1], t[NR], r/1024, th}' } +fresh() { # fresh key and vault at $1 + rm -rf -- "$1" + mkdir -p -- "$1" + export VAULT_KEY=$1/key VAULT_DIR=$1/store + "$bin" genkey 2>/dev/null + "$bin" init 2>/dev/null +} + for impl in go rust; do bin=$work/vault-$impl echo "== $impl ($(stat -c %s "$bin") bytes)" - for step in add-big get-big add-small list get-small; do + # Writes: a fresh vault for every run. + for step in add-big add-small; do results="" for run in $(seq 1 "$runs"); do - v=$work/run-$impl - rm -rf -- "$v" - mkdir -p -- "$v" - export VAULT_KEY=$v/key VAULT_DIR=$v/store - "$bin" genkey 2>/dev/null - "$bin" init 2>/dev/null - case $step in - get-big | list | get-small) "$bin" add "$data/big" 2>/dev/null; "$bin" add "$data/small" 2>/dev/null ;; - esac + fresh "$work/run-$impl" case $step in add-big) r=$(measure "$bin" add "$data/big") ;; - get-big) r=$(measure "$bin" get big -o "$v/big.out") ;; add-small) r=$(measure "$bin" add "$data/small") ;; + esac + results+="$r"$'\n' + done + printf ' %-10s ' "$step" + printf '%s' "$results" | grep . | summarise + done + # Reads: one populated vault, read repeatedly. + v=$work/run-$impl + fresh "$v" + "$bin" add "$data/big" 2>/dev/null + "$bin" add "$data/small" 2>/dev/null + for step in get-big list get-small; do + results="" + for run in $(seq 1 "$runs"); do + rm -f -- "$v/big.out" "$v/small.out" + case $step in + get-big) r=$(measure "$bin" get big -o "$v/big.out") ;; list) r=$(measure "$bin" list) ;; get-small) r=$(measure "$bin" get "small/f$(printf "%0${#small}d" 1)" -o "$v/small.out") ;; esac