transformers
Substantially reduce memory usage in _update_causal_mask for large batches by using .expand instead of .repeat
#29413
Merged
Overview
Commits
13
Changes
View On GitHub

Substantially reduce memory usage in _update_causal_mask for large batches by using .expand instead of .repeat #29413

ArthurZucker merged 13 commits into huggingface:main from nqgl:causal_mask_fix
nqgl
nqgl try to fix gemma mem use
51c8efde
nqgl fix: handle attention mask dim==2 case
ac01cea0
nqgl remove logits=logits.float()
5cd54d0d
nqgl clean up + add llama
92d46824
nqgl nqgl changed the title Substantially reduce memory usage in _update_causal_mask for large batches by using .expand instead of .repeat [WIP - need tests+sanity check] Substantially reduce memory usage in _update_causal_mask for large batches by using .expand instead of .repeat [needs tests+sanity check] 2 years ago
nqgl apply formatting
23935d95
nqgl readability edit: swap order of items being multiplied
7ed6b7f1
nqgl
ArthurZucker
ArthurZucker commented on 2024-03-04
ArthurZucker
nqgl
nqgl Merge branch 'huggingface:main' into causal_mask_fix
3420f9ab
nqgl revert change unrelated to PR
473f6cb4
nqgl revert black autoformat
7dc28df6
nqgl Merge pull request #2 from nqgl/remove_repeat_to
b2a1c99c
nqgl switch to one .to
654f6a5a
nqgl
nqgl
nqgl Merge pull request #3 from nqgl/remove_repeat_to
16753281
ArthurZucker
ArthurZucker approved these changes on 2024-03-05
ArthurZucker
nqgl Accept style edits
0a907afe
nqgl
ArthurZucker
ArthurZucker ArthurZucker merged 2a939f20 into main 2 years ago
ArthurZucker
nqgl nqgl changed the title Substantially reduce memory usage in _update_causal_mask for large batches by using .expand instead of .repeat [needs tests+sanity check] Substantially reduce memory usage in _update_causal_mask for large batches by using .expand instead of .repeat 2 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum