DeepSpeed
feat(activation_checkpointing): add `non_reentrant_checkpoint` to support inputs require no grad
#4118

Merged

feat(activation_checkpointing): add `non_reentrant_checkpoint` to support inputs require no grad #4118

tjruwase merged 27 commits into deepspeedai:master from hughpu:feat/non-reentrant-checkpoint

feat: add `non_reentrant_checkpoint`

a20c79ca

feat: add missing output postprocess and change the hook to record le…

8aeba5f6

fix: make the multi_grad_hook registered after graph construction

ee04fa8f

fix: backward compatibility for multi_tensor_hook

51f833d4

fix: nonlocal reference error of deepspeed_saved_tensors

b29c1efa

fix: reduce repeating hook registration

37e7c234

Merge branch 'microsoft:master' into feat/non-reentrant-checkpoint

d7c5440c

test: add test for `activation_checkpointing.checkpointing.non_reentr…

e22c4877

Pass correct node size for ZeRO++ (#4085)

4d2a274b

add deepspeed chat arxiv report (#4110)

d4d070b8

style: change flake8 detected style missmatch

aaf309e6

test: hack to clone the `test_activation_checkpointing` module for re…

a9109221

doc: explain the introduction of `non_reentrant_checkpoint`

fc919b17

doc: explain the test of `non_reentrant_checkpoint`

b6a0a44b

Merge branch 'microsoft:master' into feat/non-reentrant-checkpoint

8ec86a4c

hughpu requested a review from

jeffra 2 years ago

hughpu requested a review from

tjruwase 2 years ago

hughpu requested a review from