next.js
d52877ef - turbo-persistence: drop key compression dictionary from SST files (#90608)

Commit
2 days ago
turbo-persistence: drop key compression dictionary from SST files (#90608) ## Summary - Remove the zstd key compression dictionary from SST files, simplifying the on-disk format - Key blocks now use plain LZ4 compression (same as value blocks), eliminating the zstd dependency - Update the turbo-persistence README to reflect the format changes ## Why The key compression dictionary added complexity (zstd dependency, dictionary computation, dual decompression paths) with diminishing returns. Plain LZ4 is already used for value blocks and provides sufficient compression for key blocks. This simplifies both the SST file format and the compaction/inspection tooling. This main goal is to unblock streaming SST writes. ## What Changed **On-disk format (breaking):** - SST files no longer start with a serialized key compression dictionary - Meta file entries no longer include the `key_compression_dictionary_length` field **Code:** - Removed `zstd` dependency from `turbo-persistence` - Removed dictionary computation in `StaticSortedFileBuilder` - Simplified `sst_inspect` tool (no more dictionary-aware decompression) - Removed `key_compression_dictionary_length` from `StaticSortedFileMetaData`, `MetaEntry`, and `MetaFileBuilder` **Docs:** - Updated README meta file format (removed dictionary length field) - Updated README SST file format (removed dictionary region) - Updated compression description to reference LZ4 directly ## Test Plan - Existing turbo-persistence tests and benchmarks pass with the format changes - `sst_inspect` tool updated to work without dictionary logic --------- Co-authored-by: Tobias Koppers <tobias.koppers@googlemail.com>
Author
Parents
Loading