next.js
09135201 - [turbopack] Reduce memory usage during compaction (#90229)

Commit
1 day ago
[turbopack] Reduce memory usage during compaction (#90229) ### What? Optimizes memory usage, I/O patterns, and CPU efficiency during database compaction by: * clearing block caches and SST mmaps before compaction starts * opening SST files independently with madvise(SEQUENTIAL) for compaction * making the SST iterator own the file so mmaps are dropped as as soon as the iterator is are consumed * moving used-key-hash filter deserialization into per-family scope, * also mering AMQF filters into a single pre-sized filter per family, reducing used-key classification from O(N filters) to O(1) per entry * flushing entries early once collectors reach 50% capacity to reduce peak overhead while still avoiding small files * adding a lightweight single-entry value block cache on the iterator (replacing the global `BlockCache` for sequential scans, this reaps all the benefits while not accumulating too many blocks that will never be visited again) * boxing MergeIter heap elements so BinaryHeap sifts swap 8-byte pointers instead of ~312-byte structs ### Why? Database compaction was consuming excessive memory by maintaining block caches and memory-mapped files that weren't needed during the sequential merge operations. The existing approach used random access patterns optimized for lookups, but compaction performs sequential scans that benefit from different memory management strategies. Profiling revealed two additional CPU bottlenecks: 1. **AMQF query overhead (44% of compaction time):** With 128 commits, `compact_internal` iterated all 128 AMQF filters per entry (16M entries × 128 filters = ~2B probes). Most filters were empty in write-only workloads but still deserialized. Filtering empty filters and merging the rest into a single filter eliminates this entirely. 2. **MergeIter heap sift overhead (21% inclusive):** `ActiveIterator<StaticSortedFileIter>` is ~312 bytes. BinaryHeap sifts moved these large structs at each level (log2(128) = 7 levels per pop/push). Boxing reduces each swap to 8 bytes, cutting MergeIter inclusive time from 21% to 12%. ### Performance results Compaction benchmark (16Mi entries, 128 commits): | Scenario | Before | After | Improvement | |---|---|---|---| | Partial compaction | 6.79s | 3.71s | -45% | | Full compaction | 8.07s | 3.79s | -53% |
Author
Parents
Loading