Memtable Freshness

Status: Proposed spike.

Analytical reads are already correct for flushed SSTables. The freshness gap is the unflushed Cassandra memtable tail: CQLite discovers files, and a live memtable is not a file. The preferred spike exports the tail as ordinary SSTable generations so the existing CQLite merge path can read it without a second correctness model.

Public summary

The merge engine is not the gap

CQLite already performs the k-way last-write-wins merge over SSTables. The missing piece is a trustworthy source for live Cassandra writes before flush.

Materialize the tail

Exporting a live tail as real gen-* SSTable components keeps the integration path-based and reuses the reader, filters, and Arrow pipeline.

Keep Cassandra in charge

Cassandra still owns commit log durability, memtable lifecycle, flush, and compaction. CQLite only reads the durable foundation plus the exported tail.

Freshness diagram

flowchart TB
  Write["Application write"] --> Cassandra["Cassandra commit log + memtable"]
  Cassandra --> Flush["Flush / compaction"]
  Flush --> SSTables["Durable SSTables"]
  SSTables --> CQLite["CQLite SSTable reader"]
  CQLite --> Query["Flight / Trino analytical read"]

  Cassandra -. current gap .-> Invisible["Unflushed tail is not a file"]
  Cassandra -. proposed export .-> Tail["Tail gen-* SSTables"]
  Tail --> Merge["Existing CQLite k-way LWW merge"]
  SSTables --> Merge
  Merge --> FreshQuery["Fresh analytical read"]

Freshness becomes a source-enumeration problem: include exported tail generations alongside flushed SSTables, then let the existing merge path reconcile them.

Mermaid source for the freshness diagram

flowchart TB
  Write["Application write"] --> Cassandra["Cassandra commit log + memtable"]
  Cassandra --> Flush["Flush / compaction"]
  Flush --> SSTables["Durable SSTables"]
  SSTables --> CQLite["CQLite SSTable reader"]
  CQLite --> Query["Flight / Trino analytical read"]

  Cassandra -. current gap .-> Invisible["Unflushed tail is not a file"]
  Cassandra -. proposed export .-> Tail["Tail gen-* SSTables"]
  Tail --> Merge["Existing CQLite k-way LWW merge"]
  SSTables --> Merge
  Merge --> FreshQuery["Fresh analytical read"]

Decision map

flowchart LR
  subgraph Use["Use now / spike"]
    direction TB
    Correct["Flush -> snapshot -> read"]
    Spike["CEP-11 tail export"]
    Correct --> Spike
  end

  subgraph Hold["Hold or avoid"]
    direction TB
    Fallback["CDC / commitlog tail fallback"]
    Avoid["Avoid StorageHook, JVM agent, blind flush"]
    Fallback --> Avoid
  end

  Spike --> Fallback

The preferred spike is a CEP-11 memtable export path that produces normal SSTable generations. CDC remains a useful fallback, while StorageHook and forced-flush postures should not steer the first implementation.

Mermaid source for the freshness decision map

flowchart LR
  subgraph Use["Use now / spike"]
    direction TB
    Correct["Flush -> snapshot -> read"]
    Spike["CEP-11 tail export"]
    Correct --> Spike
  end

  subgraph Hold["Hold or avoid"]
    direction TB
    Fallback["CDC / commitlog tail fallback"]
    Avoid["Avoid StorageHook, JVM agent, blind flush"]
    Fallback --> Avoid
  end

  Spike --> Fallback

Decision	Status	Rationale
Export live tail as SSTable generations	Proposed spike	It preserves CQLite’s existing reader and merge semantics.
Flush then snapshot	Accepted baseline	It is correct today, but freshness is bounded by flush timing.
CDC or commitlog tail into CQLite	Deferred fallback	It avoids in-JVM code but accepts bounded lag and CDC operational cost.
StorageHook as the freshness seam	Rejected for now	It wraps per-SSTable iterator creation and cannot see live memtables.
Tiny memtables or blind flush cadence	Rejected for now	It trades OLTP health for analytics freshness.

Research sources

Memtable freshness report The option analysis for exposing Cassandra's unflushed tail to CQLite analytical reads.

CEP-11 plugin design The spike design for exporting a Cassandra memtable tail as real SSTable generations.

CQLite tail seam map The CQLite-side inventory for adding a tail directory without rewriting the merge engine.

Primary source paths:

docs/storage engine/report-1-memtable-freshness.md
docs/storage engine/memtable-plugin-design.md
docs/storage engine/cassandra-index/research-cqlite-tail-seam.md
docs/storage engine/cassandra-index/research-export-format.md
docs/storage engine/cassandra-index/memtable-api.md