DeepSpeed
NCCL based 1-bit Implementation + Refactor to add communication backends
#593

Merged

NCCL based 1-bit Implementation + Refactor to add communication backends #593

awan-10 merged 29 commits into staging-1bit-nccl-v1 from amawa/1-bit-refactor

add nccl 1-bit optim.

47788326

temporary commit to save stuff.

567232be

Use dist collectives instead of mpi routines.

79f64049

Merge branch 'master' into amawa/1bit-adam-nccl

39b5949d

remove old code for comm.

57ab220a

Fix bugs. still does not work.

ebec1fee

modify to test the nccl side code path

3e6974d1

Initial gather impl. Works intra-node.

a72049b6

Updates to comm. phase 2. nccl comm. passed the tests.

1bf1c275

refactor code to introduce nccl/mpi as backends for onebit adam.

886ebb52

Refactor updates to test/engine.

a38351ec

Merge branch 'master' into amawa/1-bit-refactor

716ac132

Fix compile/runtime errors.

be75d885

simplify support for nccl/mpi backends.

7b7f122b

Add missign file

fd2c366f

Add compression backend in constructor. Revert later.

df8c40d3

modify test with some perf counting.

f29ea3f3

Implement a true non-blocking gather for nccl side.

170ef020

Revert "Add compression backend in constructor. Revert later."

e2ddf489

improve the 1-bit adam test.

dbd3cff5

Refactor comm. and compression backend in 1-bit adam.

7edc3ab2

Fix the test.

0813d117

Fix runtime errors and typos in nccl backend

4c3c7772

fix mpi backend. modify tests.

d495c7a2

modify nccl perf test.

60f3344b

fix mpi side errors.

c1ab39e0

Add an mpi perf test

70938e17

Merge branch 'master' into amawa/1-bit-refactor

de634979