FEAT / Optim: Add GaLore optimizer #29588
younesbelkada
merged
44 commits into
huggingface:main
from
younesbelkada:add-galore-optimizer
add galore v1
b31ce79c
muellerzr
commented
on 2024-03-11
add import
58169f15
add tests and doc
9032635c
fix doctest
136f104c
forward contrib credits from discussions
a5483b36
forward contrib credits from discussions
887d3adc
younesbelkada
changed the title
Apply suggestions from code review
d6f119fb
Merge remote-tracking branch 'upstream/main' into HEAD
3fae2290
fix failing tests'
c8c50f80
amyeroberts
commented
on 2024-03-11
pacman100
commented
on 2024-03-12
Merge remote-tracking branch 'upstream/main' into add-galore-optimizer
2bdda681
switch to `optim_target_modules` and clarify docs
630bd13c
more clarification
a871b75a
Merge remote-tracking branch 'upstream/main' into add-galore-optimizer
cb6cd7e9
enhance lookup logic
51b7b292
update a test to add peak memory
3da3b90d
add regex, all-linear and single string support
9115c94d
add layer-wise optimization through DummyOptimizers and LRSchedulers
0b4ba838
forward contrib credits from discussions and original idea
3e5930ef
add a section about DDP not supported in layerwise
a16d3a87
muellerzr
commented
on 2024-03-13
Update src/transformers/trainer.py
29e7e94f
fix self
18ea144a
check only if layer_wise
7800bf1f
peterjc123
commented
on 2024-03-13
peterjc123
commented
on 2024-03-13
PenutChen
commented
on 2024-03-14
Update src/transformers/training_args.py
e022bdda
oops
830c68df
make use of intervals
b640e980
clarify comment
14a89b2f
add matching tests
6f7102db
GaLoRe -> GaLore
c11cb63e
move to `get_scheduler`
36782019
add note on docs
fdc4b2a2
add a warning
e7ce9b7d
adapt a bit the docs
91d64368
update docstring
b9e338a0
support original API
6ff37620
hiyouga
commented
on 2024-03-17
younesbelkada
commented
on 2024-03-17
Update docs/source/en/trainer.md
0d0440a1
BenjaminBossan
commented
on 2024-03-18
slightly refactor
832f2be9
matthewdouglas
commented
on 2024-03-18
Update docs/source/en/trainer.md
898a3c5a
Update src/transformers/training_args.py
ed3ad4ad
fix args parsing and add tests
57e7096e
remove warning for regex
64ccfa6b
Merge remote-tracking branch 'upstream/main' into add-galore-optimizer
4413f074
fix type hint
73dcabb8
add note about extra args
1987b7ae
make `is_regex` return optional
db2bf219
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub