Cassandra SSTable Format Parsing

This skill helps with parsing and understanding Cassandra 5.0+ SSTable file formats.

When to Use This Skill

• Parsing Data.db, Index.db, Statistics.db files
• Debugging binary format mismatches
• Analyzing hex dumps of SSTable data
• Working with compression (LZ4, Snappy, Deflate)
• Investigating offset calculation errors
• Understanding BTI (Big Table Index) format
• Validating partition boundaries

Key SSTable Components

Data.db

Contains the actual row data with:

• Partition headers
• Row data (clustering + cells)
• Compression blocks
• Checksums

Index.db

Contains partition index entries with:

• BTI (Big Table Index) format in Cassandra 5.0+
• Partition key → file offset mapping
• Promoted index entries

Statistics.db

Contains serialization metadata:

• Encoding stats (min/max timestamps, TTLs)
• Column definitions
• Schema information
• Compression parameters

Summary.db

Contains sampling of index entries for faster lookups

Format References

Primary Source of Truth: docs/sstables-definitive-guide/

Key chapters:

• Ch.5: Data.db Format - Row layout, flags, V5CompressedLegacy
• Ch.6: Index.db and Summary.db - Partition lookups
• Ch.9: CompressionInfo.db - Compression metadata, chunking
• Ch.17: BTI Formats - Trie-based indexes
• Appendix B: Encoding Cheat Sheet - VInt, cell flags
• Appendix F: Known Limitations - What doesn't work yet

Common Debugging Techniques

Hex Dump Analysis

When debugging parsing errors:

1. Extract hex at specific offset:

   hexdump -C Data.db -s <offset> -n 64
   

2. Compare with expected format:

   • Check magic bytes (if applicable)
   • Verify VInt encoding
   • Validate flag bytes

3. Look for patterns:

   • Repeated byte sequences may indicate arrays/collections
   • All zeros may indicate padding
   • Non-zero high bytes suggest multi-byte integers

Offset Validation

Track byte consumption at each parsing stage:

• Clustering prefix (may be 0 bytes)
• Row sizes (2 VInts)
• Liveness info (conditional)
• Deletion info (conditional)
• Column bitmap (conditional)
• Cell data

Zero-Copy Considerations

When implementing parsers:

• Use Bytes crate for buffer sharing
• Avoid copying large cell values
• Keep references to original buffer
• Use byte slices not owned Vecs

Integration with Rust Code

Current implementation in cqlite-core/src/storage/sstable/reader/parsing/:

• v5_compressed_legacy.rs - Main V5 format parser (1997 lines)
• Uses zero-copy patterns with Bytes
• Handles compression transparently

PRD Alignment

Supports Milestone M1 (Core Reading Library):

• 100% Cassandra 5 SSTable format support
• All compression formats (LZ4, Snappy, Deflate)
• Zero-copy deserialization
• Memory target: <128MB for large files

Quick Reference

Flag Bytes (Row)

• 0x01: HAS_IS_MARKER
• 0x02: HAS_ALL_COLUMNS (inverted - 0x20 means all present)
• 0x04: HAS_TIMESTAMP
• 0x08: HAS_TTL
• 0x10: HAS_DELETION
• 0x20: HAS_ALL_COLUMNS (flag set = all columns present)
• 0x40: IS_STATIC
• 0x80: EXTENSION_FLAG

VInt Encoding

Variable-length integer encoding:

• First byte indicates length
• Subsequent bytes contain value
• Used for row sizes, timestamps, offsets

Next Steps

When parser encounters issues:

1. Log byte offsets at each stage
2. Compare against Java source (UnfilteredSerializer.java)
3. Validate against sstabledump output
4. Check compression block boundaries
5. Verify delta encoding calculations