DeepSpeed
use ```non_reentrant_checkpoint``` fix requires_grad of input must be true for activation checkpoint layer in pipeline train.
#4224
Merged
Merged
use ```non_reentrant_checkpoint``` fix requires_grad of input must be true for activation checkpoint layer in pipeline train. #4224
tohtana
merged
36 commits into
deepspeedai:master
from
inkcherry:use_reentrant
feat: add `non_reentrant_checkpoint`
a20c79ca
feat: add missing output postprocess and change the hook to record le…
8aeba5f6
fix: make the multi_grad_hook registered after graph construction
ee04fa8f
fix: backward compatibility for multi_tensor_hook
51f833d4
fix: nonlocal reference error of deepspeed_saved_tensors
b29c1efa
fix: reduce repeating hook registration
37e7c234
Merge branch 'microsoft:master' into feat/non-reentrant-checkpoint
d7c5440c
test: add test for `activation_checkpointing.checkpointing.non_reentr…
e22c4877
Pass correct node size for ZeRO++ (#4085)
4d2a274b
add deepspeed chat arxiv report (#4110)
d4d070b8
style: change flake8 detected style missmatch
aaf309e6
test: hack to clone the `test_activation_checkpointing` module for re…
a9109221
doc: explain the introduction of `non_reentrant_checkpoint`
fc919b17
doc: explain the test of `non_reentrant_checkpoint`
b6a0a44b
Merge branch 'microsoft:master' into feat/non-reentrant-checkpoint
8ec86a4c
Merge branch 'master' into feat/non-reentrant-checkpoint
78c0d65d
Merge branch 'master' into feat/non-reentrant-checkpoint
e4eff23a
Merge branch 'master' into feat/non-reentrant-checkpoint
a6c78715
Merge branch 'master' into feat/non-reentrant-checkpoint
fbbb7604
Merge branch 'master' into feat/non-reentrant-checkpoint
a338097c
Merge branch 'master' into feat/non-reentrant-checkpoint
a00cff1e
Merge branch 'master' into feat/non-reentrant-checkpoint
c17cc3d0
Merge branch 'master' into feat/non-reentrant-checkpoint
a680399f
Merge branch 'master' into feat/non-reentrant-checkpoint
13e766de
Merge branch 'master' into feat/non-reentrant-checkpoint
a46e3260
Merge branch 'master' into feat/non-reentrant-checkpoint
b5c03f42
Merge branch 'master' into feat/non-reentrant-checkpoint
13a026d4
apply non_reentrant_checkpoint in pipeline parallel training
0c18dda3
ut pass
71421bfc
Merge branch 'master' into use_reentrant
18d64d32
fix ci
d9edc63e
Merge branch 'master' into use_reentrant
d98790d2
reduce check level for ci
c27d3345
reduce check level for ci
79b427dd
Merge branch 'master' into use_reentrant
80beb040
Merge branch 'master' into use_reentrant
bdc9db0d
tohtana
approved these changes
on 2023-09-06
Login to write a write a comment.
Login via GitHub