ruff
Reduce memory usage by dropping token-excess capacity and improve performance by approximating the initial tokens vec size
#25354
Merged
Merged
Reduce memory usage by dropping token-excess capacity and improve performance by approximating the initial tokens vec size #25354
MichaReiser
merged
8 commits into
main
from
micha/view-benchmark-token-capacity
MichaReiser
changed the title
Use estimated capacity for parser token source
5b636fa8
Shrink token buffer after parsing
744cf832
Use conservative token capacity estimate
7aed0ac8
Change to 10%
18746e82
Go up to 15% again
5879173e
charliermarsh
approved these changes
on 2026-05-25
Use remaining source for token capacity estimates (#25382)
2edc8c7b
dhruvmanila
approved these changes
on 2026-05-26
MichaReiser
commented
on 2026-05-26
Apply suggestion from @MichaReiser
68d03f8a
Use token growth buckets for initial capacity
1e4aafa9
MichaReiser
changed the title
Login to write a write a comment.
Login via GitHub