[Core] support saving and loading of sharded checkpoints #7830
sayakpaul
merged
63 commits into
main
from
feat-save-sharded-ckpt
feat: support saving a model in sharded checkpoints.
b566c95b
feat: make loading of sharded checkpoints work.
86059090
add tests
885d5b68
cleanse the loading logic a bit more.
560fe32e
more resilience while loading from the Hub.
fc5d837b
parallelize shard downloads by using snapshot_download()/
0d3b9e12
default to a shard size.
df8e9454
more fix
6eff632c
Empty-Commit
ed83244b
debug
642ee39a
fix
36de0c48
uality
cc5656ec
more debugging
88987176
fix more
2dfb9a11
Merge branch 'main' into feat-save-sharded-ckpt
179495f0
sayakpaul
commented
on 2024-05-01
sayakpaul
commented
on 2024-05-01
merge main and fix conflicts.
7e2c09b1
SunMarc
approved these changes
on 2024-05-14
resolve conflicts.
35357016
initial comments from Benjamin
5ae8e464
move certain methods to loading_utils
aefd0db4
add test to check if the correct number of shards are present.
80005be5
add a test to check if loading of sharded checkpoints from the Hub is…
d144526f
BenjaminBossan
approved these changes
on 2024-05-15
clarify the unit when passed as an int.
8e52c6d1
Merge branch 'main' into feat-save-sharded-ckpt
9d2f19a8
Merge branch 'main' into feat-save-sharded-ckpt
a8f5c03f
Merge branch 'main' into feat-save-sharded-ckpt
c917be25
Wauplin
requested changes
on 2024-05-24
Merge branch 'main' into feat-save-sharded-ckpt
7cf53404
use hf_hub for sharding.
d6f9b179
remove unnecessary code
1ae5987a
Merge branch 'main' into feat-save-sharded-ckpt
fda5d995
SunMarc
approved these changes
on 2024-05-27
remove unnecessary function
2ec27d50
Merge branch 'main' into feat-save-sharded-ckpt
c98d7790
lucain's comments.
ed289c9c
fixes
9acbbeaa
address high-level comments.
7326e25d
fix test
0706cae4
sayakpaul
commented
on 2024-05-28
subfolder shenanigans./
711fd505
sayakpaul
commented
on 2024-05-28
Wauplin
commented
on 2024-05-28
Update src/diffusers/utils/hub_utils.py
32419acc
Merge branch 'main' into feat-save-sharded-ckpt
b03e13c1
Apply suggestions from code review
cbfd70f9
remove _huggingface_hub_version as not needed.
868cfb62
address more feedback.
13fd0635
Merge branch 'main' into feat-save-sharded-ckpt
0d32c45f
add a test for local_files_only=True/
bad44c0f
need hf hub to be at least 0.23.2
c7796188
style
302d59d5
Merge branch 'main' into feat-save-sharded-ckpt
ab3a5aab
Merge branch 'main' into feat-save-sharded-ckpt
7f887420
Wauplin
approved these changes
on 2024-05-29
final comment.
a7fc2aed
Merge branch 'main' into feat-save-sharded-ckpt
f74fc674
clean up subfolder.
38749fc4
deal with suffixes in code.
edbd8de5
_add_variant default.
2ecd4dad
use weights_name_pattern
d51d0b90
remove add_suffix_keyword
c2a71a05
clean up downloading of sharded ckpts.
a70e9271
don't return something special when using index.json
65da7dc9
fix more
5599388f
don't use bare except
7cdf9589
remove comments and catch the errors better
16dcdf88
fix a couple of things when using is_file()
737e6272
resolve conflicts
491e1e20
yiyixuxu
commented
on 2024-06-07
yiyixuxu
approved these changes
on 2024-06-07
empty
2a185181
Merge branch 'main' into feat-save-sharded-ckpt
51906c6f
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub