text-generation-inference
Attempt for cleverer auto batch_prefill values (some simplifications).
#2808
Merged
Merged
Attempt for cleverer auto batch_prefill values (some simplifications). #2808
Narsil
merged
7 commits into
main
from
update_max_prefill_auto_with_vram_reqs
Attempt for cleverer auto batch_prefill values (some simplifications).
037ea55a
Less flaky tests.
a0003a62
Fixing typo insertion.
5b04d6c4
Update launcher/src/main.rs
36ed43c9
Adding small comment for source of calculation.
d701f9e8
Adding L40.
908dec63
Adding L40s.
14d19738
drbh
approved these changes
on 2024-12-09
Login to write a write a comment.
Login via GitHub