DeepSpeed
ZeRO-2 CPU offload: accumulate in the compact partition between requests
#9044
Merged
Merged
ZeRO-2 CPU offload: accumulate in the compact partition between requests #9044
sfc-gh-truwase
merged
13 commits into
master
from
truwase/zero-offload-free-compact-partition
ZeRO-2 CPU offload: accumulate in the compact partition between requests
00bcc411
chatgpt-codex-connector
commented
on 2026-11-09
Format grad_accum_dtype check with yapf
d53ddea3
Allow fp32 grad accumulation with low-precision master weights
be4ef875
ZeRO-2 offload: check overflow on the host-dtype gradient; fix vacuou…
bca22e21
Fix fp16 host overflow test to use managed accumulation
b1859ac8
ZeRO-2 offload: use the compact partition buffer only for fp32 accumu…
9960ecb7
Test ZeRO-1/2 offload window-sum precision and set_full_grad residency
ebf01123
Add zero_optimization.release_compact_grad_partition
7273449e
Rename release_compact_grad_partition to free_hbm_grad_partition_betw…
fb2afb9c
Rename free_hbm_grad_partition_between_requests to free_hbm_grad_part…
f6d73eac
Rename to retain_hbm_grad_partition (default false)
ef34708f
stas00
commented
on 2026-11-10
Rename to retain_gpu_grad_partition_until_step (default false)
5dd42923
Merge branch 'master' into truwase/zero-offload-free-compact-partition
8cb7109a
stas00
approved these changes
on 2026-11-10
Login to write a write a comment.
Login via GitHub