Appendix H — Cassandra 5.0 CommitLog Segment Format

Verified byte-for-byte against a real Cassandra 5.0.2 segment (issue #2389). The reader lives in cqlite-core/src/storage/commitlog/; fixtures + ground truth are generated by test-data/scripts/generate-commitlog-fixtures.sh.

Scope: uncompressed segments, descriptor version 7 (VERSION_40, Cassandra 4.0/5.0). Note Cassandra 5.0 writes version 7 only under the default storage_compatibility_mode: CASSANDRA_4; with the mode set to NONE or UPGRADING it writes version 8 (VERSION_50), which the reader’s exact-match-7 version gate rejects with a typed Error::UnsupportedCommitLogVersion (a contained, honest failure — not a bug). Compressed/encrypted segments are detected from the descriptor and fail closed (design decision D3). All integers are big-endian; all CRC32s are java.util.zip.CRC32 (IEEE, = crc32fast).

1. Segment layout

[ descriptor header ]
[ sync marker 0 ][ record ][ record ]…            section 0
[ sync marker 1 ][ record ][ record ]…            section 1
…
[ zero-filled pre-allocation to segment size ]

A segment is pre-allocated to commitlog_segment_size_in_mb (32 MB default) and zero-filled; only the used prefix is meaningful.

2. Descriptor header (CommitLogDescriptor.writeHeader)

Field	Type	Notes
version	int32	7 (VERSION_40) for 5.0 under CASSANDRA_4 mode; 8 (VERSION_50) under NONE/UPGRADING (rejected)
id	int64	segment id (also in the filename)
paramsLen	int16	length of the params JSON
params	bytes	UTF-8 JSON; {} when uncompressed/unencrypted
crc	int32	CRC32 over the header (see below)

Header CRC input order (each int fed big-endian, per FBUtilities.updateChecksumInt): version, id low 32 bits, id high 32 bits, paramsLen as int32, then the raw params bytes. header_len = 14 + paramsLen + 4.

Compression/encryption are read authoritatively from params using the real Cassandra keys — a non-null compressionClass (a plain STRING, e.g. "LZ4Compressor", not a nested {"class":…} object) ⇒ compressed; any non-null encCipher / encKeyAlias / encIV ⇒ encrypted (Cassandra never writes an "encryption"/"encryptionContext" key). A present-but-null value (e.g. {"compressionClass":null}) is neither. Never sniffed from payload bytes (no-heuristics, issue #28). CQLite rejects invalid UTF-8, malformed JSON, non-object parameters, and a non-string/non-null compressionClass as corrupt metadata; a valid header checksum does not make those shapes interpretable.

3. Sync marker (8 bytes, CommitLogSegment.writeSyncMarker)

Field	Type	Notes
nextMarker	int32	file offset of the next marker = section end
crc	int32	CRC32 over id-low, id-high, markerPos

The marker CRC covers the segment id (low then high int32) and the marker’s own byte offset — not nextMarker. nextMarker == 0 or an invalid CRC over a zero region marks the clean end of the written data.

4. Record framing (CommitLog.add, ENTRY_OVERHEAD_SIZE = 12)

Field	Type	Notes
size	int32	serialized mutation length
sizeCrc	int32	CRC32 over the 4 size bytes
body	size bytes	serialized mutation (messaging form)
bodyCrc	int32	CRC32 over size bytes then body

A size <= 0 within a section is zero padding ⇒ end of written data.

Truncation vs corruption

• Torn tail (record/marker bytes extend past EOF): tolerated — prior records are returned and the walk stops, surfacing truncated. Mirrors Cassandra’s tolerateTruncation.
• Corruption (a fully-present record whose CRC does not match): a typed Error::CorruptCommitLogFrame.

5. Mutation body (messaging form, isForSSTable() == false)

uvint numUpdates
per update:
  tableId (16 bytes: MSB int64, LSB int64)
  UnfilteredRowIterator:
    writeWithVIntLength(partitionKey)   # uvint len + key bytes
    iterFlags (1 byte)
    EncodingStats: 3 uvints (minTimestamp, minLocalDeletionTime, minTTL deltas)
    [static columns]  if HAS_STATIC_ROW
    regular columns:  uvint count, then count × (uvint len + name bytes)
    [partition deletion]  if HAS_PARTITION_DELETION
    [static row]          if HAS_STATIC_ROW
    [rowEstimate uvint]   if HAS_ROW_ESTIMATE
    unfiltereds… terminated by END_OF_PARTITION (row flag 0x01)

iterFlags: IS_EMPTY 0x01, IS_REVERSED 0x02, HAS_PARTITION_DELETION 0x04, HAS_STATIC_ROW 0x08, HAS_ROW_ESTIMATE 0x10 (UnfilteredRowIteratorSerializer.java:110-150).

The static-columns block is gated on HAS_STATIC_ROW, per-iterator — not on the table having static columns. UnfilteredRowIteratorSerializer.serialize computes hasStatic = staticRow != Rows.EMPTY_STATIC_ROW, uses it to set the flag, and passes that same boolean to SerializationHeader.serializer.serializeForMessaging(header, selection, out, hasStatic) (UnfilteredRowIteratorSerializer.java:129-139), which writes the statics Columns block only if (hasStatic) (SerializationHeader.java:391-406). An IS_EMPTY partition returns immediately after the flag byte — no EncodingStats, no columns (UnfilteredRowIteratorSerializer.java:120-124).

Note — the messaging form omits rowBodySize/previousUnfilteredSize (those are SSTable-only). This is the key difference from the Data.db row layout.

Row (UnfilteredSerializer)

flags (1 byte): END_OF_PARTITION 0x01, IS_MARKER 0x02, HAS_TIMESTAMP 0x04, HAS_TTL 0x08, HAS_DELETION 0x10, HAS_ALL_COLUMNS 0x20, HAS_COMPLEX_DELETION 0x40, EXTENSION_FLAG 0x80. Then: [ext flags], clustering values, [ts delta], [ttl+ldt], [row deletion], [column subset], then one cell per present column.

Cell (Cell.serializer)

flags (1 byte): IS_DELETED 0x01, IS_EXPIRING 0x02, HAS_EMPTY_VALUE 0x04, USE_ROW_TIMESTAMP 0x08, USE_ROW_TTL 0x10. Then [ts delta], [ldt], [ttl], [complex path], then the value — which is present only when HAS_EMPTY_VALUE is clear (Cell.java:303-304, reader :310); when set there is no length and no bytes.

Values need the schema

Clustering and cell values are written by AbstractType.writeValue: fixed-length types carry no length prefix (int=4, bigint/timestamp=8, uuid=16, …), variable types are uvint-length-prefixed. Decoding values therefore requires the column types — CommitLogReader::open_with_schemas supplies them (schema-aware, no guessing). Without a schema the reader still decodes table id, partition key, and column names structurally.

6. Reader coverage (v1) — fail closed on unmodeled constructs

Fully decoded: the common insert path — a table with no clustering columns, simple scalar column types, all columns present, no static row, no partition deletion, no range-tombstone marker, no complex deletion. Everything else fails closed: the reader returns the structural facts it read authoritatively and marks the row decode as not done, rather than guessing at bytes it cannot align. Guessing here would be a no-heuristics violation (issue #28) and a silent-corruption source, since misreading one field’s width misaligns every field after it.

The public API exposes three completeness signals:

Signal	Meaning when false
PartitionUpdate::columns_read	the regular column-name block was not read; column_names is empty but does not assert a parsed empty list
PartitionUpdate::rows_decoded	rows/cells were not decoded for this update; rows is empty (never partially filled)
Mutation::updates_complete	a batch declared more partition updates than updates holds — the walk stopped at the first update whose body could not be fully consumed, because the next update’s offset is unknowable without finishing this one

A partially decoded final update can leave updates_complete true: every declared update is represented, while its rows_decoded remains false.

What still is authoritative when rows_decoded == false: table_id, partition_key, has_partition_deletion (read straight from the iterFlags bit before any early return), and column_names when columns_read is true. A true flag with an empty list means the reader parsed a zero-column block. Static-row bailouts and the empty-partition fast path do not read that block and leave columns_read false.

The read-commitlog JSON output carries columns_read on each update. Its mutation_count counts decoded mutation records encountered, including a record whose display was cut short. --limit bounds both that count and emitted updates, whichever is reached first. When it omits an already-decoded update from the last mutation, last_mutation_partial is true and all_updates_complete is false. Reaching the limit exactly after the last update in a mutation does not set last_mutation_partial; limited still reports that the segment tail was not inspected. Decoder incompleteness is reported independently by all_updates_complete, even without a display limit. These are output limits: the reader still loads the segment and decodes each encountered mutation before the CLI applies the emitted-update bound.

Bail sites, each with its reason (all in cqlite-core/src/storage/commitlog/mutation.rs):

Construct	Site	Why it cannot be guessed
Static row (HAS_STATIC_ROW)	:216-232	bails before reading any columns block, so column_names is never populated with misparsed bytes
Partition deletion (HAS_PARTITION_DELETION)	:237-239	the partition-level DeletionTime body is not decoded; the flag itself is still reported
No schema for the table id	:245-248	values are type-width-dependent (see Values need the schema) — structural-only decode
Clustering columns present	:288-290	ClusteringPrefix.Serializer writes an unsigned-VInt presence header (2 bits per column, batched per 32) before the values; a naive per-column loop would read the header as the first value
Complex column (collection/tuple/UDT/vector) in the schema	:301-306	a complex column is an optional complex-deletion time plus a count-prefixed set of (cell-path, cell) pairs — an entirely different wire shape from a simple cell. Checked once as a schema-level property
Range-tombstone marker (IS_MARKER)	:313-316	marker bodies are not modeled
Row-level complex deletion (HAS_COMPLEX_DELETION)	:345-347	ditto
Column subset (HAS_ALL_COLUMNS clear)	:350-352	the subset encoding is not modeled, so column↔cell correspondence is unknown

Note that row deletions ARE decoded (HAS_DELETION yields DecodedRow::is_row_deletion == true, :339-344) — it is partition deletions that bail.

Because each record is independently CRC-framed (§4), a partial body decode never disturbs the neighboring records: the walk resumes at the next framed record. And because decode_mutation rejects a declared update count larger than the body length outright (:147-152), an untrusted count cannot spin the update loop.

Extending full value decode to these constructs is a follow-up, not a correctness gap in the sense of producing wrong data — the reader reports what it does not know.