Appendix H — Cassandra 5.0 CommitLog Segment Format

Verified byte-for-byte against a real Cassandra 5.0.2 segment (issue #2389). The reader lives in cqlite-core/src/storage/commitlog/; fixtures + ground truth are generated by test-data/scripts/generate-commitlog-fixtures.sh.

Scope: uncompressed segments, descriptor version 7 (VERSION_40, Cassandra 4.0/5.0). Note Cassandra 5.0 writes version 7 only under the default storage_compatibility_mode: CASSANDRA_4; with the mode set to NONE or UPGRADING it writes version 8 (VERSION_50), which the reader’s exact-match-7 version gate rejects with a typed Error::UnsupportedCommitLogVersion (a contained, honest failure — not a bug). Compressed/encrypted segments are detected from the descriptor and fail closed (design decision D3). All integers are big-endian; all CRC32s are java.util.zip.CRC32 (IEEE, = crc32fast).

1. Segment layout

[ descriptor header ]
[ sync marker 0 ][ record ][ record ]…            section 0
[ sync marker 1 ][ record ][ record ]…            section 1
…
[ zero-filled pre-allocation to segment size ]

A segment is pre-allocated to commitlog_segment_size_in_mb (32 MB default) and zero-filled; only the used prefix is meaningful.

2. Descriptor header (`CommitLogDescriptor.writeHeader`)

Field	Type	Notes
`version`	int32	7 (`VERSION_40`) for 5.0 under `CASSANDRA_4` mode; 8 (`VERSION_50`) under `NONE`/`UPGRADING` (rejected)
`id`	int64	segment id (also in the filename)
`paramsLen`	int16	length of the params JSON
`params`	bytes	UTF-8 JSON; `{}` when uncompressed/unencrypted
`crc`	int32	CRC32 over the header (see below)

Header CRC input order (each int fed big-endian, per FBUtilities.updateChecksumInt): version, id low 32 bits, id high 32 bits, paramsLen as int32, then the raw params bytes. header_len = 14 + paramsLen + 4.

Compression/encryption are read authoritatively from params using the real Cassandra keys — a non-null compressionClass (a plain STRING, e.g. "LZ4Compressor", not a nested {"class":…} object) ⇒ compressed; any non-null encCipher / encKeyAlias / encIV ⇒ encrypted (Cassandra never writes an "encryption"/"encryptionContext" key). A present-but-null value (e.g. {"compressionClass":null}) is neither. Never sniffed from payload bytes (no-heuristics, issue #28).

3. Sync marker (8 bytes, `CommitLogSegment.writeSyncMarker`)

Field	Type	Notes
`nextMarker`	int32	file offset of the next marker = section end
`crc`	int32	CRC32 over id-low, id-high, `markerPos`

The marker CRC covers the segment id (low then high int32) and the marker’s own byte offset — not nextMarker. nextMarker == 0 or an invalid CRC over a zero region marks the clean end of the written data.

4. Record framing (`CommitLog.add`, `ENTRY_OVERHEAD_SIZE = 12`)

Field	Type	Notes
`size`	int32	serialized mutation length
`sizeCrc`	int32	CRC32 over the 4 `size` bytes
`body`	`size` bytes	serialized mutation (messaging form)
`bodyCrc`	int32	CRC32 over `size` bytes then `body`

A size <= 0 within a section is zero padding ⇒ end of written data.

Truncation vs corruption

Torn tail (record/marker bytes extend past EOF): tolerated — prior records are returned and the walk stops, surfacing truncated. Mirrors Cassandra’s tolerateTruncation.
Corruption (a fully-present record whose CRC does not match): a typed Error::CorruptCommitLogFrame.

5. Mutation body (messaging form, `isForSSTable() == false`)

uvint numUpdates
per update:
  tableId (16 bytes: MSB int64, LSB int64)
  UnfilteredRowIterator:
    writeWithVIntLength(partitionKey)   # uvint len + key bytes
    iterFlags (1 byte)
    EncodingStats: 3 uvints (minTimestamp, minLocalDeletionTime, minTTL deltas)
    [static columns]  if HAS_STATIC_ROW
    regular columns:  uvint count, then count × (uvint len + name bytes)
    [partition deletion]  if HAS_PARTITION_DELETION
    [static row]          if HAS_STATIC_ROW
    [rowEstimate uvint]   if HAS_ROW_ESTIMATE
    unfiltereds… terminated by END_OF_PARTITION (row flag 0x01)

iterFlags: IS_EMPTY 0x01, IS_REVERSED 0x02, HAS_PARTITION_DELETION 0x04, HAS_STATIC_ROW 0x08, HAS_ROW_ESTIMATE 0x10 (UnfilteredRowIteratorSerializer.java:110-150).

The static-columns block is gated on HAS_STATIC_ROW, per-iterator — not on the table having static columns. UnfilteredRowIteratorSerializer.serialize computes hasStatic = staticRow != Rows.EMPTY_STATIC_ROW, uses it to set the flag, and passes that same boolean to SerializationHeader.serializer.serializeForMessaging(header, selection, out, hasStatic) (UnfilteredRowIteratorSerializer.java:129-139), which writes the statics Columns block only if (hasStatic) (SerializationHeader.java:391-406). An IS_EMPTY partition returns immediately after the flag byte — no EncodingStats, no columns (UnfilteredRowIteratorSerializer.java:120-124).

Note — the messaging form omits rowBodySize/previousUnfilteredSize (those are SSTable-only). This is the key difference from the Data.db row layout.

Row (`UnfilteredSerializer`)

flags (1 byte): END_OF_PARTITION 0x01, IS_MARKER 0x02, HAS_TIMESTAMP 0x04, HAS_TTL 0x08, HAS_DELETION 0x10, HAS_ALL_COLUMNS 0x20, HAS_COMPLEX_DELETION 0x40, EXTENSION_FLAG 0x80. Then: [ext flags], clustering values, [ts delta], [ttl+ldt], [row deletion], [column subset], then one cell per present column.

Cell (`Cell.serializer`)

flags (1 byte): IS_DELETED 0x01, IS_EXPIRING 0x02, HAS_EMPTY_VALUE 0x04, USE_ROW_TIMESTAMP 0x08, USE_ROW_TTL 0x10. Then [ts delta], [ldt], [ttl], [complex path], then the value — which is present only when HAS_EMPTY_VALUE is clear (Cell.java:303-304, reader :310); when set there is no length and no bytes.

Values need the schema

Clustering and cell values are written by AbstractType.writeValue: fixed-length types carry no length prefix (int=4, bigint/timestamp=8, uuid=16, …), variable types are uvint-length-prefixed. Decoding values therefore requires the column types — CommitLogReader::open_with_schemas supplies them (schema-aware, no guessing). Without a schema the reader still decodes table id, partition key, and column names structurally.

6. Reader coverage (v1) — fail closed on unmodeled constructs

Fully decoded: the common insert path — a table with no clustering columns, simple scalar column types, all columns present, no static row, no partition deletion, no range-tombstone marker, no complex deletion. Everything else fails closed: the reader returns the structural facts it read authoritatively and marks the row decode as not done, rather than guessing at bytes it cannot align. Guessing here would be a no-heuristics violation (issue #28) and a silent-corruption source, since misreading one field’s width misaligns every field after it.

The two honesty signals, both on the public API (cqlite-core/src/storage/commitlog/mutation.rs:60-95):

Signal	Meaning when `false`
`PartitionUpdate::rows_decoded`	rows/cells were not decoded for this update; `rows` is empty (never partially filled)
`Mutation::updates_complete`	a batch declared more partition updates than `updates` holds — the walk stopped at the first update whose body could not be fully consumed, because the next update’s offset is unknowable without finishing this one

What still is authoritative when rows_decoded == false: table_id, partition_key, has_partition_deletion (read straight from the iterFlags bit before any early return), and column_names when the reader got far enough to read the columns block.

Bail sites, each with its reason (all in cqlite-core/src/storage/commitlog/mutation.rs):

Construct	Site	Why it cannot be guessed
Static row (`HAS_STATIC_ROW`)	`:216-232`	bails before reading any columns block, so `column_names` is never populated with misparsed bytes
Partition deletion (`HAS_PARTITION_DELETION`)	`:237-239`	the partition-level `DeletionTime` body is not decoded; the flag itself is still reported
No schema for the table id	`:245-248`	values are type-width-dependent (see Values need the schema) — structural-only decode
Clustering columns present	`:288-290`	`ClusteringPrefix.Serializer` writes an unsigned-VInt presence header (2 bits per column, batched per 32) before the values; a naive per-column loop would read the header as the first value
Complex column (collection/tuple/UDT/vector) in the schema	`:301-306`	a complex column is an optional complex-deletion time plus a count-prefixed set of (cell-path, cell) pairs — an entirely different wire shape from a simple cell. Checked once as a schema-level property
Range-tombstone marker (`IS_MARKER`)	`:313-316`	marker bodies are not modeled
Row-level complex deletion (`HAS_COMPLEX_DELETION`)	`:345-347`	ditto
Column subset (`HAS_ALL_COLUMNS` clear)	`:350-352`	the subset encoding is not modeled, so column↔cell correspondence is unknown

Note that row deletions ARE decoded (HAS_DELETION yields DecodedRow::is_row_deletion == true, :339-344) — it is partition deletions that bail.

Because each record is independently CRC-framed (§4), a partial body decode never disturbs the neighboring records: the walk resumes at the next framed record. And because decode_mutation rejects a declared update count larger than the body length outright (:147-152), an untrusted count cannot spin the update loop.

Extending full value decode to these constructs is a follow-up, not a correctness gap in the sense of producing wrong data — the reader reports what it does not know.

Appendix H — Cassandra 5.0 CommitLog Segment Format

1. Segment layout

2. Descriptor header (CommitLogDescriptor.writeHeader)

3. Sync marker (8 bytes, CommitLogSegment.writeSyncMarker)

4. Record framing (CommitLog.add, ENTRY_OVERHEAD_SIZE = 12)