Flight metrics reference

CQLite Flight metrics — operator reference

GENERATED from the observability catalog (cqlite-core/src/observability/catalog.rs + operator_docs.rs) by cargo run -p cqlite-core --example gen_operator_metrics_doc. Do NOT edit by hand — edit the catalog/annotations and regenerate. The operator-metrics-doc agent-gate component fails if this file drifts from the catalog (issue #2426).

Operator-facing reference for every cqlite.* metric name in CQLite’s observability catalog, covering Arrow Flight and the storage/write/compaction paths. Names, units, and bounded attribute sets are generated from the code.

Two populations. Most catalogued names are LIVE OTel instruments: registered with the meter, and scrapeable once they have recorded a value. The rest are stats-only — no instrument is ever registered, so they are never on a scrape at all; read them from the in-process Database::stats().memory_stats snapshot. The two are listed in separate sections below.

Related: the Flight/Trino operator docs (docs/flight-trino/) and the round scoreboard template (issue #2399) link back to the entries here.

Catalogued metrics: 91 — 85 live OTel instruments and 6 stats-only (not scrapeable).

Live instruments

Registered with the OTel meter; each appears on a Prometheus scrape / OTel collector export once it has recorded a value.

Metric	Type	Unit	Attributes	Operator meaning	Healthy vs alarming
cqlite.compaction.budget.consumed	histogram	s	(none)	Maintenance budget actually consumed per maintenance_step call.	Consumed materially exceeding requested means maintenance is overrunning its budget.
cqlite.compaction.budget.requested	histogram	s	(none)	Maintenance budget requested per maintenance_step call.	Compare against budget.consumed to confirm the scheduler honors its ~10% tolerance.
cqlite.compaction.bytes_written	counter	By	(none)	Bytes written to compaction output SSTables.	Compare with write.bytes/flush.bytes to reason about write amplification from compaction.
cqlite.compaction.duration	histogram	s	(none)	Compaction run durations.	Use with compaction.rows_merged for merge throughput; a growing tail means compaction is falling behind.
cqlite.compaction.finalize.duration	histogram	s	(none)	Compaction finalize (atomic rename / publication-barrier) durations.	Normally small; a growing tail points at slow filesystem rename/barrier operations.
cqlite.compaction.lag	gauge	{sstable}	(none)	L0 SSTables currently pending compaction (compaction lag).	A stable low level is healthy; a monotonic rise means compaction cannot keep up with flush.
cqlite.compaction.rows_merged	counter	{row}	(none)	Rows emitted by the k-way merge across all compactions.	With compaction.duration this yields rows-merged-per-second; a drop signals compaction stalling.
cqlite.compaction.sstables_in	counter	{sstable}	(none)	Input SSTables consumed by compactions.	In vs out ratio is the consolidation factor; in >> out is healthy consolidation.
cqlite.compaction.sstables_out	counter	{sstable}	(none)	Output SSTables produced by compactions.	See compaction.sstables_in; out approaching in means little consolidation is happening.
cqlite.compaction.tombstones_emitted	counter	{tombstone}	(none)	Tombstone markers retained (carried forward, not purgeable) into merge output.	Expected while data is within gc_grace; a persistent climb means tombstones are not aging out.
cqlite.compaction.tombstones_purged	counter	{tombstone}	(none)	Tombstones genuinely purged (gc-grace/overlap-safe) during compaction.	Rising means space is being reclaimed; zero while tombstones accumulate means purge is blocked (gc_grace/overlap).
cqlite.compaction.tombstones_suppressed	counter	{tombstone}	(none)	Live cells/rows shadowed by a tombstone during merge reconciliation.	Suppression without a matching purge or retained marker is the resurrection-risk smell.
cqlite.compression.ratio	histogram	1	cqlite.compression	WRITE-SIDE ONLY: per-chunk compression ratio recorded by the SSTable writer as it compresses each chunk (compressed/uncompressed; <=1.0 means the chunk shrank). There is no read-side emission, so it says nothing about the SSTables being read.	Ratios near 1.0 mean incompressible data; a sudden rise means less benefit from the configured codec. Silent unless compressed writing is in use (#1406: the production write surface emits uncompressed SSTables).
cqlite.errors.total	counter	{error}	cqlite.error.category cqlite.subsystem cqlite.flight.abort_reason	Canonical error-rate signal, keyed by bounded {category, subsystem}; flight do_get aborts also carry flight.abort_reason; eagerly registered at 0 on startup.	Absent metric name = error counting isn’t wired (never ‘no errors’); any sustained increase is the error-rate alarm; on subsystem=flight, split abort_reason to separate benign aborts (client_cancel/superseded_split/snapshot_retired/admission_shed) from genuine internal faults.
cqlite.flight.admission.in_use	gauge	1	(none)	do_get admission permits currently held (admitted, in-flight scans only).	Below the limit is healthy headroom; sitting at admission.limit with a non-zero waiting gauge is saturation.
cqlite.flight.admission.limit	gauge	1	(none)	The configured do_get admission ceiling K (—max-concurrent-scans); constant while the server runs.	Chart admission.in_use against this ceiling; in_use pinned at the limit means the server is saturated.
cqlite.flight.admission.rejected_total	counter	1	(none)	do_get requests rejected (gRPC UNAVAILABLE) because no permit freed within the wait timeout.	Should stay 0 under normal load; any sustained increase means clients are being shed and should fail over.
cqlite.flight.admission.wait_seconds	histogram	s	(none)	How long a do_get waited on acquire before it was admitted or rejected.	A near-zero distribution is healthy; a rising tail means requests are increasingly queuing for permits.
cqlite.flight.admission.waiting	gauge	1	(none)	do_get requests parked waiting for an admission permit — the backpressure signal.	Zero is healthy; a sustained non-zero value means offered concurrency exceeds the ceiling and requests are queuing.
cqlite.flight.blocking_tasks_in_use	gauge	{thread}	(none)	Flight spawn_blocking tasks currently outstanding (flight-managed proxy, NOT the global tokio pool queue depth).	Rises with concurrent do_get scans and returns to baseline; a level pinned near the blocking-pool size with flat rpc.rows is blocking-pool saturation. Distinct from admission.in_use.
cqlite.flight.tables_discovered	gauge	{entry}	(none)	Table dirs visible under —data-dir, re-sampled by a readdir-only walk on the ~2s saturation tick (no SSTable opens). Undersampled at ~2s vs a longer scrape interval (#2661).	Rises when a table appears on disk and falls when one is removed; a wrong or empty —data-dir reads 0 immediately (an inert mount, visible before the first query errors).
cqlite.flight.warm_tables	gauge	{entry}	(none)	Tables with a live warm reader set in the flight WarmTableRegistry (atomic-backed at the registry mutation sites, not sampler-driven).	Rises on the first serve of a previously-unseen table and falls on eviction/retirement; pinned at capacity with steady eviction churn means the warm byte budget is small vs the working set. Distinct from tables_discovered (visible on disk).
cqlite.flush.bytes	counter	By	(none)	Data.db bytes produced by memtable flushes.	Write volume landing on disk from flush; use with flush.duration for flush throughput.
cqlite.flush.duration	histogram	s	(none)	Memtable-to-SSTable flush durations.	A rising tail alongside a climbing memtable size means flush cannot drain the memtable fast enough.
cqlite.flush.rows	counter	{row}	(none)	Rows flushed from the memtable to L0 SSTables.	Should track write.mutations over time; a lag is the flush-behind signal.
cqlite.flush.sstables	counter	{sstable}	(none)	L0 SSTables created by memtable flushes.	Feeds compaction.lag; a high rate with rising lag means L0 is accumulating faster than compaction clears it.
cqlite.memtable.rows	gauge	{row}	(none)	Buffered rows in the active memtable.	Same sawtooth shape as memtable.size_bytes; a rising floor signals flush pressure.
cqlite.memtable.size_bytes	gauge	By	(none)	Current approximate in-memory size of the active memtable.	Sawtooth (rise then drop at flush) is healthy; a monotonic climb means flush is not keeping up.
cqlite.merge.active_merges	gauge	{merge}	(none)	Live concurrent k-way merges — the divisor of the adaptive egress budget (#2765).	Per-channel capacity is clamp(EGRESS_ROW_BUDGET/active, MIN_CAP, 256); a level well above budget/256 means concurrent merges are being throttled toward MIN_CAP.
cqlite.merge.egress_channel_depth	gauge	{entry}	(none)	Live occupancy, in ENTRIES (rows), of the bounded merge egress sync_channel feeding do_get / compaction — batched since #2820 (up to 256 rows per message). Its per-source ceiling is CHANNEL-RESIDENT rows, 2 x rows_cap = 512 at the default #2765 budget — NOT the 4 x rows_cap = 1024 total-in-flight MEMORY bound, which also counts the consumer-held and producer-blocked batches this gauge never sees.	Near zero = consumer keeping up (or a stalled producer); riding near the per-source channel-resident bound (512 at the default) = producer outrunning a slower consumer (back-pressured egress). Threshold alerts on 2 x rows_cap, never on 1024: a 1024 threshold is UNREACHABLE per source and can never fire. Process-globally the gauge is the SUM over concurrent sources, so it can exceed any single source ceiling. The unit is ENTRIES, so a batch of n rows moves it by n, never by 1.
cqlite.merge.producer_threads	gauge	{thread}	(none)	Live OS producer threads the k-way merge currently holds (one per input SSTable).	Bounded by O(M) inputs and returns to baseline at merge completion; a stuck-high value is a thread leak (#2316).
cqlite.merge.rows_in	counter	{row}	(none)	Input rows consumed at the k-way merge reconcile boundary (once per merge).	Compare with merge.rows_out; the gap is rows removed by last-write-wins collapse and tombstone suppression.
cqlite.merge.rows_out	counter	{row}	(none)	Rows emitted by the merge reconcile boundary post-reconciliation.	See merge.rows_in; rows_out much smaller than rows_in means heavy reconciliation.
cqlite.proc.fds	gauge	{fd}	(none)	Process open fd count sampled from /proc/self/fd (Linux; absent off-/proc, never 0).	Rises as concurrent scans open SSTables (no reader pool, #815); a level nearing the container ulimit is the fd-exhaustion binding point.
cqlite.proc.rss_bytes	gauge	By	(none)	Process resident set size from /proc/self/status VmRSS (Linux; absent off-/proc, never 0).	Rises with concurrent scan payloads (Flight bypasses the result-byte budget); a level nearing the memory limit is the OOMKill risk.
cqlite.proc.threads	gauge	{thread}	(none)	Process OS thread count sampled from /proc/self/task (Linux; absent off-/proc, never 0).	Rises with concurrent scans and settles to baseline; a level climbing toward the thread ceiling is thread/scheduler collapse.
cqlite.query.degraded_path.total	counter	1	cqlite.query.fallback_reason	SELECTs that took a soundness fallback to a full-scan read path, tagged by fallback reason.	Should stay near 0 for targeted queries; a climbing value means queries are silently degrading to full scans.
cqlite.query.duration	histogram	s	cqlite.subsystem	End-to-end query execution durations.	Watch p99; a rising tail is the top-level query-latency alarm.
cqlite.query.rows	counter	{row}	cqlite.query.access_path cqlite.query.plan_type	Rows returned to callers by the query engine.	Compare with query.rows_scanned; a large gap is read amplification.
cqlite.query.rows_scanned	counter	{row}	cqlite.query.access_path	Rows examined by the SELECT scan step before filter/projection/LIMIT.	≈ query.rows for a point lookup; far larger means a full_scan is doing the work.
cqlite.read.bloom.checks	counter	1	cqlite.result cqlite.sstable.format	Bloom-filter / BTI-trie present-or-absent checks; miss = definitely absent.	A high miss-rate is healthy pruning; pairing with partition_lookup reveals the bloom false-positive rate.
cqlite.read.bloom.false_negatives	counter	1	cqlite.sstable.format	Opt-in soundness alarm: keys found on an authoritative scan that the presence oracle said were absent.	MUST stay 0. Any non-zero value is a corruption/soundness alarm (only emitted when verification is enabled).
cqlite.read.bti.rows_root_rejected	counter	{partition}	cqlite.read.rows_root_reject_reason	Clustering-slice reads that decoded a WHOLE BTI partition because its Rows.db row-index root failed structural validation, tagged by the violated invariant.	Should be 0. Non-zero explains clustering-read latency with no narrowing: the rows are correct but the row index is unusable — re-flush/re-compact a Rows.db written by CQLite <= 0.16 (#3002).
cqlite.read.bytes	counter	By	cqlite.compression	Total Data.db bytes read (post-decompression), counted once per chunk decode (compressed or not).	Track against read.rows to spot read amplification; a spike with flat rows means wide scans. The decompressed-chunk cache does NOT reduce it: the scan feed reads each chunk off disk BEFORE consulting that cache, so a warm re-scan counts the same bytes again (the cache saves decompression, not I/O). Counted for EVERY subsystem that reads an SSTable, compaction included, so a bytes spike with flat read.rows can also be a compaction running.
cqlite.read.duration	histogram	s	cqlite.sstable.format	Distribution of single read/scan operation durations, at the core query read boundaries (manager scans, point reads, streaming handles).	Watch p99; a growing tail is the read-latency alarm.
cqlite.read.partition_access.accesses	counter	1	cqlite.read.repeat_bucket	Accesses attributable to each repeat-access bucket, incremented once per closed probe window (#2827) — the sum of the repeat counts of the partitions in that bucket.	CUMULATIVE across windows, like its siblings. accesses minus distinct_partitions is the number of reads a cache holding that bucket could have served; equal values mean every access was a first touch (nothing to cache).
cqlite.read.partition_access.bytes	counter	By	cqlite.read.repeat_bucket	Distinct-partition on-disk bytes per repeat-access bucket, incremented once per closed probe window (#2827), measured as each partition’s successor gap; a partition read ten times counts its bytes once.	CUMULATIVE across windows. Uncompressed on-disk bytes, the correct input for a decoded-size multiplier; zero only when every access reported size_source=unavailable. Never an estimate.
cqlite.read.partition_access.distinct_partitions	counter	{partition}	cqlite.read.repeat_bucket cqlite.read.size_source	Distinct partitions per repeat-access bucket, incremented once per closed window of the default-OFF access-distribution probe (#2827) — the workload’s concentration shape, with no per-key label.	Absent unless an operator enabled CQLITE_PARTITION_ACCESS_PROBE. CUMULATIVE across windows — do not price a cache off a scrape of this after several windows have closed; use the in-process WindowSummary. Mass in bucket 1 is a uniform, cache-hostile pattern; mass in 9-16/17+ is a hot set. A non-zero size_source=unavailable share means the byte total beside it is incomplete and the procedure refuses the window.
cqlite.read.partition_access.dropped_accesses	counter	1	(none)	Accesses the access-distribution probe could not seat in its fixed counting table (#2827).	Zero on a healthy window. Non-zero INVALIDATES the window: only keys not already in the table can be dropped, so a loss suppresses the singleton bucket and overstates concentration — the cache-sizing procedure refuses such a window.
cqlite.read.partition_access.sample_denominator	gauge	1	(none)	Sampling scale in force when the probe window closed (#2827): 1 is a census, 2^k means the recorder downsampled k times to stay inside its fixed 3 MiB table.	1 is ideal. A large value means the window touched far more distinct partitions than the table holds; bucket fractions stay unbiased but absolute counts must be scaled by it.
cqlite.read.partition_access.sampling_floor	gauge	1	(none)	1 when the last closed probe window reached its sampling-prefix cap, 0 otherwise (#2827).	0 is healthy. 1 means the window is a ~1-in-a-million sample and is statistically worthless; the cache-sizing procedure refuses it. Read alongside sample_denominator.
cqlite.read.partition_access.window_dropped_accesses	gauge	1	(none)	Accesses the LAST CLOSED probe window could not seat, reset every window (#2827).	A window is CLEAN exactly when this and sampling_floor both read 0. Use this, not the cumulative dropped_accesses counter, to judge the current window: a counter that ever incremented reads non-zero for the life of the process.
cqlite.read.partition_lookup.total	counter	1	cqlite.result cqlite.read.lookup_route cqlite.sstable.format	Partition point lookups attempted, tagged hit/miss so a dashboard computes the hit ratio from one series.	A healthy point-read workload is dominated by hits; a miss-heavy ratio means keys are absent or mis-routed.
cqlite.read.partitions	counter	{partition}	cqlite.sstable.format	Partitions a read DELIVERED rows from on core read paths; partitions SCANNED on Flight’s k-way merge arm.	Rising in step with read.rows is healthy; many partitions for few rows indicates a full scan. Mind the producer difference when comparing series: core paths derive partition boundaries from EMITTED row keys, so a partition whose every row is tombstone-shadowed or TTL-expired contributes ZERO, while Flight’s merge arm counts it as scanned. The gap is exactly the fully-suppressed partitions, so an amplification dashboard mixing the two producers can under-count partitions on the core side.
cqlite.read.phase.decode	histogram	s	(none)	Wall time ONE completed scan spent decoding rows/cells from already-resident bytes, accumulated per PARTITION at the parse boundary (never per row).	Usually the largest phase of a WARM full scan — that is healthy, it is the CPU work of the read. Alarming when it grows relative to the rows delivered: wide partitions, many collection/UDT cells, or a schema-less fallback decode doing more work per row. SURFACE COVERAGE IS PARTIAL (cqlite.read.phase.io lists the instrumented and uninstrumented surfaces): an absent read.phase.* breakdown beside a rising read.duration means the phase was NOT MEASURED on that read path, never that it was fast.
cqlite.read.phase.decompress	histogram	s	(none)	Wall time ONE completed scan spent decompressing chunk payloads, measured in the single chunk-decode plane around the compressor call only.	Proportional to compressed bytes read; a growing share points at chunk length / compressor choice, or at re-decompressing the same chunks (a decompressed-chunk cache too small for the scan’s window). ABSENT — not zero — for an uncompressed SSTable, which decompresses nothing. Absence and 0.0 are DIFFERENT statements: absence means the phase did not run, a 0.0 means it ran and measured zero (decompression too fast for the clock). SURFACE COVERAGE IS PARTIAL (cqlite.read.phase.io lists the instrumented and uninstrumented surfaces): an absent read.phase.* breakdown beside a rising read.duration means the phase was NOT MEASURED on that read path, never that it was fast.
cqlite.read.phase.io	histogram	s	(none)	Wall time ONE completed scan spent reading Data.db bytes (positional chunk/piece reads incl. CRC verify, decompression excluded) — exactly one sample per scan, never per chunk.	Read this FIRST when a scan is slow: io dominating the four read.phase.* series means the scan is disk/page-cache bound, so decode or merge tuning cannot help it. Small on a warm page cache. The phases run on concurrent pipeline threads, so they OVERLAP and do not sum to read.duration — compare them against EACH OTHER, not against wall time. SURFACE COVERAGE IS PARTIAL: io is recorded ONLY by the streaming scan surfaces (scan_stream / scan_stream_batched over a chunk-stitching reader) — the streaming cross-generation merge records merge and decompress but NO io, because its producer threads read through a route that has no io seam; the materializing SSTableManager::scan, the BIG reverse-clustering scan, the BTI trie walk, point reads and compaction reads emit read.duration with NO phase series at all — so an absent read.phase.* breakdown beside a rising read.duration means the phase was NOT MEASURED on that read path, never that it was fast.
cqlite.read.phase.merge	histogram	s	(none)	Wall time ONE completed cross-generation read spent in the k-way merge/reconcile step, with the blocking merge-input recv-wait SUBTRACTED so producer starvation is not counted as merge CPU.	Recorded ONLY on the cross-generation merge route: a single-generation scan has nothing to merge and records no sample, so absence means “nothing to merge”, never “merge was free”. A 0.0 here is NOT absence — it means a merge ran and the recv-wait subtraction consumed the whole step, i.e. the merge thread spent all its time waiting on starved producers. Grows with the number of overlapping generations and with reconcile work (tombstones, LWW collapse) — a merge-dominated read delivering few rows is the compaction-lag smell, so cross-check cqlite.compaction.lag. SURFACE COVERAGE IS PARTIAL: the STREAMING cross-generation merge records this; the materializing merge_generations_for_read beneath SSTableManager::scan records nothing — so absence here is “nothing to merge” only when the other read.phase.* series ARE present for the same traffic, and otherwise means NOT MEASURED on that read path, never that it was fast.
cqlite.read.rows	counter	{row}	cqlite.sstable.format	Rows a read DELIVERED to its consumer (climbs incrementally during a long Flight merge scan).	Steadily rising under load is healthy; flat while a scan is in flight suggests a stall. It counts DELIVERED rows, not every row decoded: an abandoned stream reports the rows the consumer polled plus those already enqueued, so a row the producer was mid-send with when the consumer went away is excluded. The series can therefore read marginally below what a producer decoded, but never above the rows a query could have returned.
cqlite.read.scan.window_refill	counter	1	(none)	Windowed streaming-scan refills at a compression-chunk boundary (a partition straddles the chunk edge and the driver awaits the next decompressed chunk).	Non-zero proves the multi-chunk stitch path was exercised; it stays 0 for single-chunk SSTables. A runaway value means excessive chunk-boundary straddling.
cqlite.read.sstables_pruned	counter	{sstable}	cqlite.sstable.format	SSTables skipped by a definitive presence-oracle negative (bloom/BTI-trie miss).	Higher is better — it is the dashboard-honest prune signal; zero on a point read means no pruning happened.
cqlite.reader.fds.open	gauge	{fd}	(none)	OS file descriptors the SSTable readers currently hold, reported by the readers themselves (an atomic incremented where a descriptor is really minted and decremented in the matching Drop) — no /proc read.	Rises with concurrent scans and falls as they complete (there is no reader pool, by design). A level climbing toward the process ulimit -n is EMFILE pressure, visible here BEFORE an open fails. Counts DESCRIPTORS, so an mmap-backed source contributes 0 and an Arc clone of an existing handle contributes nothing. Distinct from cqlite.proc.fds, the whole-process /proc sample (sockets and the WAL included): proc.fds minus this is roughly the non-reader footprint.
cqlite.rpc.bytes	counter	By	cqlite.rpc.method	Record-batch payload bytes streamed to clients by do_get (pre-IPC-framing).	Tracks egress volume; pair with rpc.rows to see batch sizing.
cqlite.rpc.duration	histogram	s	cqlite.rpc.method cqlite.rpc.status	Arrow Flight RPC handler durations (includes admission wait time).	Watch do_get p99; localize a rising tail with rpc.phase.duration.
cqlite.rpc.in_flight	gauge	1	cqlite.rpc.method	Arrow Flight RPCs currently being handled.	A sustained-high value with flat rpc.rows is a stall; distinct from admission.in_use (all RPCs, not just admitted scans).
cqlite.rpc.phase.active	gauge	1	cqlite.rpc.method cqlite.rpc.phase	In-flight phase a do_get is CURRENTLY in (1 on entry, 0 on exit) — visible before the phase completes.	stream=1 for a whole hang exposes a wedged do_get that phase.duration would never record; admission=1 exposes an admission queue wait.
cqlite.rpc.phase.duration	histogram	s	cqlite.rpc.method cqlite.rpc.phase	Per-phase do_get wall time: the top-level validate/admission/resolve/merge_setup/stream, plus (within stream, do_get only) the in-stream sub-phases stream_cold_fault/stream_decompress/stream_merge/stream_encode/stream_encode_framing/stream_grpc_write.	Localizes WHERE a slow do_get spent time: piling up in merge_setup, queued in admission, or stuck parsing in validate. Within stream the sub-phases run on concurrent pipeline threads and OVERLAP (they do NOT sum to stream) — read the cold-warm delta on stream_cold_fault as the cold-IO latency bucket; stream_merge is merge CPU ONLY (the blocking merge-input recv wait is subtracted, so producer starvation does not inflate it); stream_grpc_write is CLIENT-PACED (egress park/wake), not server cost. stream_encode covers, on the merge-consumer thread, the Arrow array build AND (since issue #3552) the push-time cell resolution, per-row width charge and column-major commit that were folded into the row loop — so it is NOT COMPARABLE ACROSS #3552 IN EITHER DIRECTION: it gained that push-time work and simultaneously lost the flush-time transpose, which moved INTO it. Neither stream_encode NOR merge+encode is comparable across that boundary — the per-row width charge sat in NO sub-phase before #3552 and is inside stream_encode now, so the SUM gains it too. Use throughput (rows/s) or a per-row-normalised measurement instead. While stream_encode_framing (issue #3096) covers the arrow-flight encoder stage on the async gRPC task — IPC serialization, dictionary hydration and encoder-target re-slicing — so a framing change is attributable separately from an array-build change; stream_encode_framing is poll time on that task, so it also contains the cheap non-blocking channel poll that returns Pending while the merge has produced nothing yet. CAVEAT: stream_cold_fault/stream_decompress are SUMMED across the N concurrent per-SSTable scan threads, so on a multi-generation table they can EXCEED the stream phase duration (compare the cold-warm DELTA, never the absolute vs stream); and they are recorded only on the instrumented scan read paths (Summary-guided + full-ring fallback), so a near-zero value can mean ‘this route was not measured’, not ‘cold IO negligible’. The six stream_* sub-phases are emitted only on the STREAMING do_get route; an aggregating do_get (materialized per-group output) emits none of them.
cqlite.rpc.requests	counter	1	cqlite.rpc.method cqlite.rpc.status	Arrow Flight RPC requests served, tagged by method + ok/error.	Compute per-method error rate from the status arm; a rising error fraction is the RPC-health alarm.
cqlite.rpc.rows	counter	{row}	cqlite.rpc.method	Rows returned to clients by do_get (emitted per record batch).	Climbing = healthy long scan; flat while rpc.in_flight>0 = a stalled do_get.
cqlite.sstable.index_interval_parses_total	counter	1	(none)	Bounded Summary-guided Index.db interval parses (one per point lookup); distinct from whole-file parses.	Grows ~1 per point lookup; a cold lazy open adds 0. Whole-file spikes stay visible on index_parses_total.
cqlite.sstable.index_parses_total	counter	1	(none)	Whole-file BIG/NB Index.db parses; the scale-free probe for the #2383 resolve-phase CPU spin.	Healthy ≈ sum of generations per query; climbing far past that means redundant re-parses (a #2383-class regression).
cqlite.sstables.open	gauge	{sstable}	cqlite.sstable.format	SSTables currently held open.	A stable level is healthy; unbounded growth is a handle/file-descriptor leak.
cqlite.storage.open.bytes	counter	By	(none)	Total on-disk Data.db bytes across the SSTables discovered at open.	The dataset size an open must account for; use with open.sstables to size cold-start cost.
cqlite.storage.open.sstables	counter	{sstable}	(none)	SSTables discovered and opened per StorageEngine open, summed over process lifetime.	Tracks how much on-disk state each open touches; sudden growth signals compaction lag or fan-out.
cqlite.storage.open.tables	counter	1	(none)	Logical tables represented by the SSTables discovered at open.	Baseline inventory count; unexpected changes mean schema/keyspace drift on disk.
cqlite.wal.recovery.duration	histogram	s	(none)	How long write-ahead-log recovery at engine open took, in seconds — both halves of it: the CRC validation scan of the log (plus any torn-tail trim) and the replay of its entries into the memtable. Normally ONE sample per process (recovery runs exactly once per open), so read it as a value rather than a distribution.	Near-zero after a clean shutdown; seconds means a crash left a large log to validate and replay, which directly delays open — pair it with cqlite.wal.size, whose growth makes this grow. On a large or corrupt-tail WAL the validation scan is the dominant half, so the timer spans the whole open-and-recover window. A histogram rather than a gauge because the gauge plane is i64 and a sub-second recovery would truncate to a fabricated 0. Recorded at EVERY open, including when there was nothing to recover (a fresh WAL genuinely took ~0s, which is a real measurement) and including a corrupt-WAL open, so absence means no engine was opened in this process, never that recovery was skipped.
cqlite.wal.size	gauge	By	(none)	Current on-disk size of the active write-ahead log, reported by the write engine after each successful mutation (a value the engine already tracks — nothing is stat’ed).	Healthy shape is a saw-tooth: grows with writes, drops back when a flush truncates/rotates the log. A level that only climbs means flushes are not keeping up — a disk-space problem AND a startup-latency one, because next open’s recovery cost is a function of this size (cross-check cqlite.wal.recovery.duration).
cqlite.wal.sync.duration	histogram	s	(none)	WAL fsync durations.	A growing tail points at slow/contended disk — the write-durability latency alarm.
cqlite.warm.cache.evicts	counter	1	(none)	Warm generations evicted by LRU byte-budget pressure or removal on disk.	A high evict rate with a low hit ratio means the warm byte budget is too small for the working set.
cqlite.warm.cache.hits	counter	1	(none)	Flight warm-handle cache hits served from warm parsed state with zero reader-open.	A high hit ratio (hits vs misses) is the warm-cache win; a low ratio means generations keep changing.
cqlite.warm.cache.misses	counter	1	(none)	Flight warm-handle cache misses (no cached entry, or generation set changed).	Pair with warm.cache.hits for the hit ratio; a rising miss rate erodes the warm-open speedup.
cqlite.warm.cache.refresh	counter	1	cqlite.warm.refresh_outcome	Warm-handle refresh outcomes, tagged unchanged/rebuilt_delta/fail_closed_retained.	Mostly unchanged/rebuilt_delta is healthy; a rising fail_closed_retained arm means refreshes are failing and serving stale-but-safe state.
cqlite.write.bytes	counter	By	(none)	Data.db bytes produced by the SSTable writer across all outputs.	Total on-disk write volume; use with compaction.bytes_written to reason about write amplification.
cqlite.write.mutations	counter	{row}	(none)	Mutations accepted by the write path (one per successful memtable insert).	Tracks write throughput; flatlining under offered load means writes are blocked.
cqlite.write.partitions	counter	{partition}	(none)	Partitions written by the SSTable writer (flush + compaction output).	Baseline write-shape signal; interpret alongside write.bytes.

Stats-only metrics — NOT OTel instruments

These 6 catalogued names have NO OTel instrument: nothing registers them with a meter, so a Prometheus scrape or OTel collector will never show them. Read them from the in-process Database::stats() snapshot at the field named in the last column. This is enforced, not documentation: catalog::STATS_ONLY_METRICS is what this section is generated from, and the stats_only_metrics_are_catalogued_and_never_otel_registered guard fails the build if one of them ever does get an instrument (issue #1705).

Metric	Type	Unit	Attributes	Operator meaning	Healthy vs alarming	Read it from
cqlite.cache.key.capacity_bytes	gauge	By	(none)	Surfaced only via Database::stats().memory_stats — NOT emitted as a live OTel instrument (not scrapeable from Prometheus/an OTel collector). The global key cache’s fixed byte budget (0 when block caching is disabled).	A fixed named constant inside the <128MB envelope; not a user knob. Zero means the read caches are disabled.	Database::stats().memory_stats.key_cache_capacity_bytes
cqlite.cache.key.evictions	counter	1	(none)	Surfaced only via Database::stats().memory_stats — NOT emitted as a live OTel instrument (not scrapeable from Prometheus/an OTel collector). Global key-cache entries evicted to stay within the byte budget (budget-driven; distinct from invalidations).	Sustained growth means the hot location set exceeds the fixed budget; expected to be flat on a small working set.	Database::stats().memory_stats.key_cache_evictions
cqlite.cache.key.hits	counter	1	(none)	Surfaced only via Database::stats().memory_stats — NOT emitted as a live OTel instrument (not scrapeable from Prometheus/an OTel collector). Hits on the process-global key→partition-offset cache (a point read skips the Index.db interval parse / trie descent).	A high hit rate means hot partitions are resolving from cache; near-zero on a cold or highly-random point-read workload.	Database::stats().memory_stats.key_cache_hits
cqlite.cache.key.invalidations	counter	1	(none)	Surfaced only via Database::stats().memory_stats — NOT emitted as a live OTel instrument (not scrapeable from Prometheus/an OTel collector). Global key-cache entries dropped on generation removal/compaction/warm-evict (distinct from budget evictions).	Tracks generation turnover (compaction/refresh); a #2383 rebind over a byte-identical generation does NOT invalidate.	Database::stats().memory_stats.key_cache_invalidations
cqlite.cache.key.misses	counter	1	(none)	Surfaced only via Database::stats().memory_stats — NOT emitted as a live OTel instrument (not scrapeable from Prometheus/an OTel collector). Misses on the global key cache (incl. fail-closed identity mismatch); each pays one interval parse / trie descent then populates.	Rises with working-set churn or eviction pressure; a persistently high miss ratio suggests the budget is small vs the hot set.	Database::stats().memory_stats.key_cache_misses
cqlite.cache.key.resident_bytes	gauge	By	(none)	Surfaced only via Database::stats().memory_stats — NOT emitted as a live OTel instrument (not scrapeable from Prometheus/an OTel collector). Approximate resident footprint of the process-global key cache.	Bounded above by capacity_bytes; a single global cap regardless of how many readers are open.	Database::stats().memory_stats.key_cache_resident_bytes

Round-scoreboard mapping (issue #2399)

Metrics a #2399 round-template scoreboard item consumes. Round handoffs (#2367-style) link the item to its metric here instead of re-explaining it.

Metric	Scoreboard item
cqlite.cache.key.capacity_bytes	read-path perf (#2059)
cqlite.cache.key.evictions	read-path perf (#2059)
cqlite.cache.key.hits	read-path perf (#2059)
cqlite.cache.key.invalidations	read-path perf (#2059)
cqlite.cache.key.misses	read-path perf (#2059)
cqlite.cache.key.resident_bytes	read-path perf (#2059)
cqlite.compaction.lag	compaction-lag watch (#2399)
cqlite.errors.total	error-rate watch (#2193/#2399/#2681)
cqlite.flight.admission.in_use	admission saturation (#2420/#2399)
cqlite.flight.admission.limit	admission saturation (#2420/#2399)
cqlite.flight.admission.rejected_total	admission saturation (#2420/#2399)
cqlite.flight.admission.wait_seconds	admission saturation (#2420/#2399)
cqlite.flight.admission.waiting	admission saturation (#2420/#2399)
cqlite.flight.blocking_tasks_in_use	blocking-pool pressure watch (#2419/#2313)
cqlite.flight.tables_discovered	inert-mount watch (#2684)
cqlite.flight.warm_tables	warm-working-set watch (#2684)
cqlite.merge.active_merges	egress-backpressure watch (#2765/#2367)
cqlite.merge.egress_channel_depth	egress backpressure watch (#2419/#2399)
cqlite.merge.producer_threads	thread-budget watch (#2313/#2399)
cqlite.proc.fds	fd-exhaustion watch (#2419/#2313)
cqlite.proc.rss_bytes	memory-pressure watch (#2419/#2313)
cqlite.proc.threads	thread-budget watch (#2419/#2313)
cqlite.rpc.duration	do_get latency (#2367/#2399)
cqlite.rpc.phase.active	do_get hang detection (#2361/#2399)
cqlite.rpc.phase.duration	do_get phase + in-stream sub-phase breakdown (#2398/#2399/#2819/#3096)
cqlite.rpc.requests	rpc-error-rate watch (#2399)
cqlite.rpc.rows	do_get progress (#2367/#2399)
cqlite.sstable.index_interval_parses_total	index-parse audit (#2367/#2399)
cqlite.sstable.index_parses_total	index-parse audit (#2367/#2399)
cqlite.warm.cache.hits	warm-cache hit ratio (#2310/#2399)
cqlite.warm.cache.misses	warm-cache hit ratio (#2310/#2399)