accelerate
Torchao float8 training
#3348
Merged
Overview
Commits
25
Changes
View On GitHub

Torchao float8 training #3348

muellerzr merged 25 commits into main from torchao-float8
muellerzr
muellerzr muellerzr requested a review from SunMarc SunMarc 1 year ago
vkuzo
vkuzo commented on 2025-01-16
vkuzo
vkuzo commented on 2025-01-16
vkuzo
vkuzo commented on 2025-01-16
muellerzr muellerzr changed the title Torchao float8 training [WIP/MVP] Torchao float8 training 364 days ago
muellerzr Bookmark
febc6344
muellerzr bookmark
a7663c51
muellerzr Add torchao base example
ed1adb1d
muellerzr Currently broken
3d34f8ec
muellerzr Clean
a032f1a5
muellerzr DDP varient working
71979436
muellerzr FSDP as well
dc797fde
muellerzr Works for all but zero3
145dec2c
muellerzr Bookmark: currently zero3 is underperforming
676d5ac1
muellerzr Bookmark
92b3d9bb
muellerzr muellerzr force pushed from f8058764 to 92b3d9bb 340 days ago
muellerzr Another diff
660b6b59
HuggingFaceDocBuilderDev
muellerzr Fin
ca45f463
muellerzr Fin
a0193ce2
muellerzr Add req huggingface suite
b271b139
SunMarc
SunMarc commented on 2025-02-12
muellerzr update tests for fp8/torchao/ddp
a5d2c29f
muellerzr Log FP8 backend used and adjust typing
002c4be2
muellerzr add documentation for convert_to_float8_training
95fbb8dc
muellerzr Rename to convert_model_to_fp8_ao
ac222966
muellerzr Call superinit"
06642ca1
muellerzr Add types
d46b0a1d
muellerzr Clean
62881acb
SunMarc
SunMarc approved these changes on 2025-02-12
muellerzr Use filter_first_and_last_linear_layers
f3ceb37b
muellerzr Update usage guide docs
14f6d04e
muellerzr Actually loop through the zero stages
a8b6b8c6
muellerzr Clean
f222d721
SunMarc
SunMarc approved these changes on 2025-02-17
muellerzr muellerzr merged 8039158d into main 333 days ago
muellerzr muellerzr deleted the torchao-float8 branch 333 days ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum