cqlite-flight — v0.15.0 Milestone Verdict

v0.15.0 milestone snapshot (condensed) · flight v0.15.0@sha256:30c2b10c (multi-arch INDEX) · connector 0.15.0 · Trino 481 · Cassandra 5.0 RF=3 · 3× i4i.xlarge · ~1.93M partitions/node, 2 SSTable gens · 2026-07-17

Claims table

Headline — throughput & latency (90-min steady state, 32 threads)

Grafana latency panels are shown for shape only — they derive from rate() over histogram _bucket counters and are subject to the same pipeline bug. The authoritative p50/p99 are the driver numbers in the table above.

Findings (report, not pass/fail)

1. Fan-out skew — the throughput ceiling FINDING

One flight pod carried the bulk of the work (~650m CPU / 297Mi) while the other two sat at 140–300m / 28–39Mi. All three participated (0 client errors), but the load is single-pod-heavy — the same token-range-dependent pattern seen in R9/R10b. This is why throughput landed ~39 qps, short of the Lane-B B2 goal (≥100 qps @ 32 threads). Not a correctness or stability problem; a distribution-evenness one.

2. do_get server-side errors — 0.89%, client-invisible FINDING

0 client-visible errors the entire run (loadtest final: errors: 0). But the server-side counter shows do_get error = 8,563 vs ok = 951,762 = 0.89% (R12 was ~1.2%), all in category other. What the investigation found:

• Load-proportional, not overload-driven — a steady ~1,300 errors/5min held at a constant 0.9% through both the 32-thread and 80-thread phases; the burst did not raise the error fraction.
• Concentrated on the busy pod — killing the fan-out-heavy pod in Phase 5 dropped the counter ~20k (28,864→8,563), so the errors live where the work lives.
• Metric-only — flight pods logged 27–45 lines total with zero abort/cancel/error lines; these never surface outside the counter.

Best interpretation (consistent with, not proven in-field): aborted / superseded scan splits — when a multi-split scan has enough data, the coordinator abandons the remaining in-flight do_get streams and the server records them as errors. Load-proportional + client-invisible + busy-pod-concentrated all fit that. Why it can't be closed here: category other is too coarse to name the cause, and there is no log line to trace. To reach 0: finer error categorization + a debug trace on the do_get abort path, then stop counting superseded-split teardown as an error if that is the cause. Recommended as a P2 follow-up.

(The 0.89% is computed from raw counter totals — do_get ok 951,762 vs error 8,563 — not from the rate() panels below, which are shape-only per the metrics bug.)

Metrics-pipeline bug — rate() panels inflate ~1000× FIXED

The "RPC Requests/sec" panel (and the other rate()-based panels) show a rising, accumulating line reaching ~2.35 million/s. Actual throughput was ~39 qps (~176 do_get/s). Investigation:

• No per-pod label. cqlite_rpc_requests_total exports one series per flight pod (3 total) but with no distinguishing label — identical cluster/job/service_name. VictoriaMetrics merges all three into a single series identity.
• Cumulative counters with staggered starts. Each pod's OTLP cumulative counter has an independent start-time; merged into one series they jump backwards constantly. resets(...[90m]) = 90 on a counter that should never reset.
• Result: rate() reads each backward jump as a counter reset and extrapolates a fresh 0→N ramp — hence the ~1000× inflation and the "ramp then freeze" shape (it freezes when load stops because no new samples arrive to trigger further phantom resets).