[Turbopack] Custom persistence layer #73029
sokra
merged
76 commits into
canary
from
sokra/turbo-persistance
sokra
changed the base branch from
sokra/invalidation-improvements
to
graphite-base/73029
1 year ago
sokra
changed the title
sokra
changed the title 
mischnic
commented
on 2024-12-01
Initial implementation of a custom persistence layer
7a022a75
generic key
65c6a196
bugfixes
cda0a19a
integrate turbo-persistence into the backend
66c4c3fe
fixup
c2786a5d
smaller value blocks
f1028262
put verification check behind feature flag
04173e07
change block count and index block block indicies to 2 bytes
17dca521
WIP: log hit miss
89299f88
Revert "WIP: log hit miss"
2c7f7c70
use 2 bytes block index in key blocks
f7f02315
add medium sized values that are placed in a separate value block
572b3fd6
adjust block sizes and small value size field
ee6b6f6e
prehashing
871edc1c
more test cases
71fc940c
add strict checks
f2fdc47a
fixup verify_sst_content
86406454
support large keys
7ea181cc
add stats
64b97c1a
reorder test cases
69d22341
print stats on shutdown
e74105ae
add separate index block cache, cache all aqmfs
a025d11d
smaller blocks
213ed6f7
avoid caching medium values
829df126
increase SST file size, parellel finishing
bccde2dc
more stats
42c9b86d
fix feature flagged import
75643e08
store hashes in sst files
fffbebaf
refactoring to capture hash during collecting
932eb2f3
sort sst by hash, index block contains hashes only
a6114e8d
move min/max hash to file header and test that first
c60917b8
start with last block
fe2dfe0e
add compaction
0b2b5162
parallel compaction
11f5a3aa
fixup compaction
6e2c89ae
increase aqmf cache
eb7da897
reuse write batches and collectors
b6246a97
allow partial compaction
5dbe9e34
run compaction after persisting
216fc022
improve test cases
9b1b9646
spread out compacted files
d4e0aea9
handle *.del files on startup
60e51627
rename
d9e2e467
add support for key families
5438dfaa
trait compression dict
512f3b52
compact less files at once
1f31d9bf
Revert "trait compression dict"
20096d8a
improve stats output
300515f2
fix sst deletion
2956f457
fixup
491d8ef6
longer chains
4313771f
more concurrent writing
38f698f7
skip task_type serialization for an empty database
1b5ebe89
lint
3aaceb68
remove compaction until it's working well
7d29c73f
clippy
03fd9857
improve compaction
8848c5e2
run compaction in background
2cc85483
bugfix, naming
2d9fe165
update full_compact
ed37f809
wait for compaction to finish on shutdown
81aea918
improve compaction
8848c5e2
update full_compact
ed37f809
use constants
cd4aa816
larger compaction
2c01f8da
add documentation of compaction
9afdef44
add lmdb feature
764197b3
clippy
e9af79ad
fix unit test
db524716
train compression dict
376195db
add min sample sizes
24f670a0
use more samples then the dictionary size
883fc4dd
mischnic
commented
on 2024-12-02
mischnic
commented
on 2024-12-02
add comments
d018c496
mischnic
approved these changes
on 2024-12-02
typo
ba804fdf
add explainer for key families
d30eb6c8
Login to write a write a comment.
Login via GitHub