transformers
Reduce by 2 the memory requirement in `generate()` 🔥🔥🔥
#30536
Merged
Merged
Reduce by 2 the memory requirement in `generate()` 🔥🔥🔥 #30536
ArthurZucker
merged
34 commits into
huggingface:main
from
Cyrilvallez:cache-efficiency
ArthurZucker
commented
on 2024-04-30
ArthurZucker
commented
on 2024-05-08
gante
approved these changes
on 2024-05-21
Fix contrastive_search for new cache structure, and improve performan…
750955d6
Fix _contrastive_search for non-standard cache using ellipsis slicing
c8a43a8d
Fix all outputs.logits memory leaks for all decoding strategies!
9342fff1
Fix small error in _contrastive_search()
15e8615e
Make all necessary change and revert for the new class
c0e40d4c
Apply coding style
1de31480
Remove pipes in type hints for compatibility
e170e962
correct type hint
d7c43594
apply style
3986f1dc
Use DynamicCache by default and solve conflicts
b9f7e04e
Fix rebase issues
614e0525
Add `_supports_dynamic_cache_class` in models for models that support…
241b851b
Create generation config to return legacy format by default, or to ch…
d31eea46
style
17525ab5
Fix case when use_cache is False
c47e6ce5
Remove default DynamicCache in assiste_decoding if assistant_model do…
b9bbfd9b
Update prepare_inputs_for_generation() for case with empty DynamicCache
3b59cf62
Correct return of args in _assisted_decoding
2a809ad1
Remove EfficientDynamicCache as it is no longer needed
e96adcbe
Correct mistake in generation config
20174eca
Move cache logic of assisted decoding to AssistedCandidateGenerator._…
7e39b928
change DynamicCache function names from "split" to "batch_split" for …
f3e31611
Remove `_supports_dynamic_cache_class` attribute after rebase
8abe0551
Correct missing line lost in conflict resolution during rebasing
e9d0b25d
Add special case for Jamba
c902dc1d
Fix jamba test
2f838676
Coding style
2c51e031
coding style
3c0999b7
Correct missing import in rebasing
b494dd57
Simplify _validate_model_kwargs based on removal of _supports_dynamic…
70a01852
Simplify code paths in _contrastive_search
d38a9660
coding style
c8edaef3
ArthurZucker
commented
on 2024-06-03
Update docstrings of cache methods
1e020d68
Update prepare_inputs_for_generation() -> past_key_values are always …
7bd2e3ee
ArthurZucker
approved these changes
on 2024-06-05
Login to write a write a comment.
Login via GitHub