PR #9447 Optimize python overhead of APEX amp

Optimize python overhead of APEX amp #9447

pengwa merged 12 commits into master from pengwa/opt_amp

optimize python overhead of _post_amp_backward

f7bee7cd

pengwa requested a review from

baijumeswani 4 years ago

pengwa requested a review from

liqunfu 4 years ago

pengwa requested a review from

SherlockNoMad 4 years ago

pengwa requested a review from

thiagocrepaldi 4 years ago

pengwa requested a review from

tlh20 4 years ago

pengwa requested a review from

xadupre 4 years ago

pengwa added training

gradually warm up the unscale kernel launches to reduce CUDA stream i…

fff81aa5

xadupre commented on 2021-10-20

overwrite apex amp's zero_grad for faster implementation

e1db2615

baijumeswani force pushed from 1f372a38 to e1db2615 4 years ago

Update check_requirement functions

2001f7bb

baijumeswani dismissed these changes on 2021-10-21

move unscale_fp16_grads_into_fp32_grads into C++ impl

7f9575c5

Merge branch 'pengwa/opt_amp' of https://github.com/microsoft/onnxrun…

4da10440

pengwa dismissed their stale review via 4da10440 4 years ago

refine a bit

4149d251

format

d5aeb6e9

make the heavy at::empty_like run in parallel

30e06728

improve the efficiency furthur, reducing 3.5ms to 1.7ms for unilm.

ba613669

unilm 1.7ms to 338us: 1). optimize python list <==> std::vector copy,…

0846a6c7

refine the logic a bit after validating

d97392d9

baijumeswani approved these changes on 2021-10-25

pengwa merged b125446f into master 4 years ago

pengwa deleted the pengwa/opt_amp branch 4 years ago

Reviewers

baijumeswani

xadupre

liqunfu

SherlockNoMad

thiagocrepaldi

tlh20

Assignees

No one assigned

Labels

training

Milestone

No milestone

onnxruntime Optimize python overhead of APEX amp #9447 Merged

Optimize python overhead of APEX amp #9447

onnxruntime
Optimize python overhead of APEX amp
#9447

Merged