next.js
11266613 - Delete blob files during compaction when entries are superseded (#91314)

Commit
129 days ago
Delete blob files during compaction when entries are superseded (#91314) ### What? During compaction in `turbo-persistence`, when entries are dropped (superseded by newer values or pruned by tombstones), blob files referenced by those entries are now marked for deletion. ### Why? Previously, compaction would merge SST files and correctly drop stale entries, but blob files referenced by those dropped entries were leaked on disk (marked with a TODO at the time). Over time this would cause unbounded disk usage growth for databases that overwrite or delete blob-sized values. ### How? When the compaction merge loop skips an entry (because `skip_remaining_for_this_key` is `true`), it now checks if the dropped entry is a `LookupValue::Blob` and, if so, pushes its sequence number to `blob_seq_numbers_to_delete`. The existing `commit()` infrastructure already handles the rest — writing `.del` files and removing the actual `.blob` files after the CURRENT pointer is updated. The change is minimal (4 lines of logic in `db.rs`): - Made `blob_seq_numbers_to_delete` mutable - Added an `else` branch to collect blob sequence numbers from dropped entries This covers both cases: - **SingleValue**: After the first (newest) entry for a key is written, all older entries are skipped. Blob references in those older entries are marked for deletion. - **MultiValue**: After a tombstone is encountered, all older entries for that key are skipped. Blob references in those older entries are marked for deletion. ### Tests Added 4 new tests: - `compaction_deletes_superseded_blob` — blob overwritten by smaller value → blob deleted after compaction - `compaction_deletes_blob_on_tombstone` — blob deleted via tombstone → blob deleted after compaction - `compaction_deletes_blob_multi_value_tombstone` — MultiValue: tombstone prunes blob → blob deleted - `compaction_preserves_active_blob` — blob still referenced → blob preserved after compaction All existing compaction tests (23) and full turbo-persistence test suite (60) continue to pass. Co-authored-by: Tobias Koppers <sokra@users.noreply.github.com> Co-authored-by: Claude <noreply@anthropic.com>
Author
Parents
Loading