next.js
7640d0bd - [turbopack] hashing improvements to turbo-persistence (#90936)

Commit
6 days ago
[turbopack] hashing improvements to turbo-persistence (#90936) ## Summary Reduces CPU overhead in turbo-persistence SST reads through three optimizations: 1. **Switch from `twox-hash` to `xxhash-rust`** — `twox_hash::XxHash3_64::with_seed(0)` heap-allocates a 192-byte secret buffer on every call. `xxhash_rust::xxh3::Xxh3Default::new()` is a `const fn` with zero allocation. Applied to both `turbo-persistence` and `turbo-tasks-hash`. 2. **Bulk `write()` in `KeyBase::hash` impls** — The hash implementations were feeding bytes one at a time via `write_u8()`, preventing xxh3 from using its optimized bulk ingestion path. Changed all slice-backed types (`&[u8]`, `[u8; N]`, `Vec<u8>`, `Box<[u8]>`) to use a single `state.write(slice)` call. 3. **Binary search improvements in SST lookups:** - **Index block lookup**: Replaced hand-rolled binary search with `as_chunks::<10>()` + stdlib `binary_search_by`, making the code simpler and leveraging the stdlib's optimized search. - **Duplicate key scans**: When collecting all entries with the same key (FIND_ALL mode), the backward/forward scans now use the binary search bounds (`l..m` and `m+1..r`) instead of scanning from `0` to `entry_count`. The binary search already proved entries outside these bounds are different, so this avoids unnecessary comparisons. - **`entry_matches_key` helper**: Equality checks during duplicate scans now use a dedicated function that avoids recomputing the full hash when no hash is stored in the block (block type 2/4). Previously `compare_hash_key` would call `hash_key()` to compute the full hash even when only an equality check was needed. ## Benchmark results Compared against the `fixed_size_blocks` branch using `cargo bench -p turbo-persistence -- 'read/get/key_8'` (8-byte keys, 4-byte values): | Benchmark | Change | |-----------|--------| | 10.67Mi/compacted/miss/cached | **-45%** | | 10.67Mi/compacted/hit/cached | **-35%** | | 10.67Mi/uncompacted/hit/cached | **-35%** | | 10.67Mi/uncompacted/hit/uncached | **-22%** | | 10.67Mi/uncompacted/miss/uncached | **-22%** | | 10.67Mi/uncompacted/miss/cached | **-21%** | | 10.67Mi/20commits/hit/cached | **-26%** | | 10.67Mi/20commits/miss/* | -13% | | 85.33Mi/compacted/hit/uncached | **-56%** | | 85.33Mi/compacted/hit/cached | **-22%** | | 85.33Mi/compacted/miss/* | **-22%** | | 85.33Mi/uncompacted/hit/cached | -10% | | 85.33Mi/20commits/hit/uncached | -12% | | 85.33Mi/20commits/hit/cached | -6% | | 85.33Mi/20commits/miss/* | -1 to -3% | Largest wins on cached paths with smaller datasets where CPU cost (hashing, search) dominates over I/O.
Author
Parents
Loading