exclude fsdp from delay_optimizer_creation #34140
exclude fsdp from delay_optimizer_creation
Nice :) Can we add a test in tests/test_trainer.py? We can set env variables to configure Accelerate properly (ACCELERATE_MIXED_PRECISION="fp8" will auto-use TE)
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
The required tests are distributed tests. We need to verify FSDP functionality with and without FP8 mixed precision. The appropriate test file might be tests/trainer/test_trainer_fsdp.py.
add test case for trainer: FSDP mode and fp8 as mixed precision
rearrange imports
ruff formatted
Is the test/trainer folder included in the CI tests? Where can I check the results for test_trainer_fsdp.py? @muellerzr @SunMarc
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
@eljandoubi we can't run them on the normal CI since GPU runners are not part of PR's.
Instead when ready I'll pull the PR down and run it myself
@muellerzr Thank you for the information. I have tested the branch in my code on a multi-node, multi-GPU setup using FSDP mode, both with and without FP8 mixed precision, and it worked as expected. Please let me know if you encounter any issues on your end.
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
adapt _init_fsdp to fp8
use _init_fsdp only when resume_from_checkpoint
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
In case of FDP, self.layer will be CheckpointWrapper which has no lenβ¦
delete _init_fsdp
solve conflict
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
fix conflict
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
@muellerzr Any updates regarding this PR?
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Thanks! Can you do pip install -e .[quality] followed by make fixup? I'll then pull it locally to test on my 4090 system and we should be set!
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'main' into fix_fsdp_with_fp8_in_trainer
make fixup
Merge branch 'huggingface:main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'main' into fix_fsdp_with_fp8_in_trainer
@muellerzr I have done make fixup.
Merge branch 'main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'main' into fix_fsdp_with_fp8_in_trainer
Merge branch 'main' into fix_fsdp_with_fp8_in_trainer
Don't worry we'll merge as is, failing tests are unrelated!
Login to write a write a comment.
What does this PR do?
It passes the model and the optimizer to accelerate.prepare in order to enable fp8 mixed precision, if any.
Fixes #34024
Who can review?
Library:
-->