pytorch-lightning
Enhance remote checkpoint loading with parallel mmap chunk downloading (#21868)
#21869
Open
Open
Enhance remote checkpoint loading with parallel mmap chunk downloading (#21868) #21869
yuxin00j
wants to merge
10 commits into
Lightning-AI:master
from
yuxin00j:mmap-remote-checkpoint-load
Enhance remote checkpoint fetching using mmap and multiprocessing
47909ba6
Address PR reviews: flush mmap and enforce aggressive cache cleanup o…
d63c56ed
test: add unit tests for remote checkpoint mmap parallel downloading …
6e8c4bcb
docs: update changelog and remote filesystem doc for mmap parallel do…
1a20f1cb
![pre-commit-ci[bot]](https://avatars.githubusercontent.com/in/68672?v=4&s=60)
[pre-commit.ci] auto fixes from pre-commit.com hooks
504b35c4
fix: resolve ruff formatting, mypy type errors, md link check, and gu…
73614baf
test: add tests for shm caching and info exception fallback for 100% …
84af602d
test: mock os.path.join for /dev/shm in test_cloud_io to fix macOS an…
dc811c9d
fix: disable mmap=True in torch.load on Windows (sys.platform == 'win…
a99943c6
fix: use atomic staging (.tmp file + os.replace) for remote checkpoin…
77449f82
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub