cqlite M0 — disk & SSTable read-path profile

Server-direct full-scan IO characterization, v0.16.0 (Arm 1) → 0.17-dev (Arm 2). Focus: what the SSTable read path actually asks the block device for, and why it is (still) the standing throughput lever. Test bed: i4i.xlarge (4 vCPU / 2 physical cores, XFS on the i4i NVMe nvme1n1), LZ4-compressed keyvalue corpus (chunk_length_in_kb=16, ~3.4M partitions, ~650 MB Data.db), flight-loadgen --shape full, no Trino. Tracked on cqlite #2818. Measured 2026-07-26/27.

The one-line disk finding. Across both arms the scan reads the SSTables in ~4.3 KB block requests — one LZ4 chunk faulted a page at a time — and that did not change. 0.17-dev nearly doubled throughput by fixing the CPU/pipeline path, so the device now does fewer tiny reads per second (util 74% → 53%), but the request size is unmoved. The read path is still not doing large sequential IO. That is the standing lever, and nothing in 0.17 addresses it.

~4.3 KB

mean block read (both arms)

99.3–99.7%

reads in [4K, 8K)

0.10 ms

device r_await (disk is fast)

15% → 28%

of fio seq ceiling (711.8 MB/s)

1. Block-read size distribution — the headline, and it did not move

bpftrace on tracepoint:block:block_rq_complete (reads only), full cold scan. bcc's xfsslower will not compile on the lab's HWE 7.0.0-aws kernel, so bpftrace is the substitute (per CASSANDRA-15452 method).

Arm 1 — v0.16.0 (498,759 reads, 2.12 GB)

[512, 1K)         2
[4K,  8K)   497,417  ████████████████████████████████████████████  99.7%
[8K, 16K)        64
[16K,32K)        76
[32K,64K)        47
[64K,128K)    1,045
[128K,256K)     108              mean read = 4,260 B (4.2 KB)

Arm 2 — 0.17-dev (192,514 reads, 0.87 GB)

[4K,  8K)   191,180  ████████████████████████████████████████████  99.3%
[8K, 16K)        67
[16K,32K)        72
[32K,64K)        47
[64K,128K)    1,114
[128K,256K)      34              mean read = 4,494 B (4.4 KB)

Interpretation. The corpus is LZ4-compressed at a 16 KiB chunk length. A chunk is decompressed as a unit, but the read of the compressed chunk off disk lands as ~4 KB page-sized requests — i.e. the read path faults roughly a page at a time and never coalesces adjacent chunks into a large sequential request. #2877 (per-chunk read coalescing) shipped in 0.17-dev; it does not change the device-level request size on this narrow-row corpus, which is the expected null result — the reads were already chunk-granular. The absent bars ≥64 KB are what a sequential SSTable scan would produce with readahead.

2. iostat — the device is starved by tiny requests, not saturated

metric (`nvme1n1`, cold scan)	Arm 1 — v0.16.0	Arm 2 — 0.17-dev	read
`rareq-sz` (avg request)	4.2 KB	4.4 KB	unchanged — tiny
`r/s` (read IOPS)	~9,600	~5,400	fewer (CPU path faster)
`r_await` (latency)	0.11 ms	0.10 ms	device is fast
`%util`	~74%	~53%	more headroom
device MB/s served	~39 MB/s	~23 MB/s	limited by IOPS × 4KB

The device answers each request in ~0.1 ms and never exceeds ~74% busy — it is not the bottleneck by bandwidth. It is starved: every read is ~4 KB, so throughput is bounded by request count, not device capability. Arm 2 issues fewer such requests per second (the CPU pipeline feeds them less often now that decode/merge got faster), which is why %util dropped even as end-to-end MB/s rose.

3. Headroom vs the raw device (fio)

fio sequential read (bs=1M iodepth=32 direct=1, cache dropped) on the same i4i NVMe: 711.8 MB/s / 712 IOPS per node (identical on all 3). Against that ceiling:

single-stream scan (warm, Arrow wire)	MB/s	% of 711.8 MB/s device ceiling
fio bs=1M sequential (the ceiling)	711.8	100%
Arm 2 — 0.17-dev	199.3	28%
Arm 1 — v0.16.0	106.8	15%

0.17-dev nearly doubled the fraction of the disk the read path can actually use (15% → 28%), but the remaining 72% gap is not disk bandwidth — the device will do 711 MB/s at bs=1M and only ~40 MB/s worth of work is being asked of it at 4 KB. Closing the gap means larger reads, not faster storage.

4. Read path, resolved (cqlite #2818 ask)

At SSTable open the reader builds a dedicated MADV_RANDOM point-read mmap per Data.db (issue #2210) — this is the Dedicated MADV_RANDOM point-read mapping log line, and it is for point reads, not scans.

scan-path Data.db mapping	Arm 1 — v0.16.0	Arm 2 — 0.17-dev
`/proc/<pid>/maps` Data.db regions during scan	0 (scan read via positioned/buffered path)	5 `r--s` shared regions
mechanism	single read plane	#2876 read-plane split → distinct scan-side mapping

So the two Arm-1 statements that looked contradictory resolve to: the MADV_RANDOM point mapping is created at open; in 0.16 the scan never mapped Data.db, in 0.17-dev it does — a separate scan read plane. Either way, the device-level read stays ~4 KB.

5. Where the cold-IO cost actually lands — #2819 sub-phase timers (0.17-dev)

0.17-dev adds five in-stream sub-phase timers on cqlite.rpc.phase.duration (do_get only). They run on concurrent pipeline threads and overlap — they do not sum to stream. Mean ms/RPC:

sub-phase	cold	warm	cold − warm	what it is
stream_cold_fault	4437.9	105.8	−4332 ms (42×)	cold body-chunk page-in (the cold-IO term)
stream_merge	7351.7	7224.4	~0	k-way merge + reconcile + materialize (CPU)
stream_encode	1868.3	1842.9	~0	Arrow RecordBatch encode (CPU)
stream_decompress	149.2	149.6	~0	LZ4 chunk decompress (producer thread)
stream_grpc_write	5.8	5.9	~0	egress send (client-paced)

The disk cost is real and isolable. stream_cold_fault is 4.4 s/scan cold and collapses 42× to 106 ms warm — that is the tiny-read disk penalty, in production-observable form, and it is purely a cold-cache effect. The persistent (warm) bound is CPU: stream_merge + stream_encode are identical cold vs warm. So on a cold or partly-cold node — the real field condition at 3.4M partitions — the 4 KB read pattern costs seconds per scan that a large-sequential-read path would largely erase.

6. Disk-access knobs don't help — the 4 KB size is intrinsic

Zero-cost prelude A/B (single-stream, cold each), Arm 2 — mirrors Arm 1 exactly:

config	MB/s
`baseline` (Auto)	204.6
`CQLITE_PREFETCH=willneed`	209.8
`CQLITE_PREFETCH=sequential`	205.8
`CQLITE_DISK_ACCESS_MODE=buffered`	204.9
`CQLITE_DISK_ACCESS_MODE=mmap`	121.4

No existing knob moves the needle; forcing mmap is worse. The 4 KB granularity is not a tunable — it needs a code change.

The disk verdict & the lever

0.17-dev did not change the disk read pattern. Reads are ~4.3 KB in both arms (99.3–99.7% in [4K,8K)), device util actually fell (74%→53%) because the faster CPU path needs fewer IOPS, and the scan still uses only 28% of a 711 MB/s device. The throughput doubling came entirely from the CPU/pipeline fixes (#2876 read-plane split, #2825 batching), not from the disk path.

The standing lever is scan-side readahead across adjacent compressed chunks — a ≥256 KiB scan-only read buffer (gated on > chunk_length), exactly the CASSANDRA-15452 fix. It is not in the 0.17 manifest. Predicted payoff: the histogram mass moves to ≥64 KB, stream_cold_fault shrinks, and cold/partly-cold single-stream MB/s rises toward the 711 MB/s device instead of plateauing at ~28% of it.

Server-direct, unstripped self-built cqlite-flight + flight-loadgen: Arm 1 = v0.16.0 tag; Arm 2 = origin/main ed3ac7e (0.17-dev, --features observability). Same LZ4 corpus (chunk_length_in_kb=16), same method. Disk stats: bpftrace block_rq_complete, iostat -x 1, /proc/<pid>/io (read_bytes / rchar / syscr reported separately), fio bs=1M ceiling. samply recorded 0 samples on this HWE kernel; CPU decomposition via perf record. Full method + non-disk results on cqlite #2818. Raw artifacts in the easy-db-lab rig at clusters/cqlite-m0-arm{1,2}-artifacts/.