PR #392 Pipeline parallel training engine.

Pipeline parallel training engine. #392

ShadenSmith merged 45 commits into deepspeedai:master from ShadenSmith:pipeline-parallelism

improved pipe test + merge

a1c0aee7

fixed model ckpt naming

e77704eb

Improves checkpoint config and tensorboard.

28487b9a

less noise

d29f45e4

tb directory creation

e26bd33b

Improved activation checkpointing error checkpoing

7cad50f4

preparing for large perf tests

a1d4b6af

Pipeline parallelism (squash) (#82)

6704ebdd

completing renames

2eafdd39

code formatter

5c1449b6

moving over updates from staging v1 review

4e0f140b

Tied module indexing bugfix.

a59c2775

Train and inference pipeline schedules.

5c677218

adds check for no_grad()

0b3b42a1

doctring improvements

03a460ff

documentation improvements

59e132db

engine documentation

fc45bf64

Move code quality tests to Azure-hosted agents. (#368)

ed716a61

documentation

71afb3fd

verbose pipeline unit test

211d23b4

extends module to support torch.nn.Sequential

42a3c56e

docstring tweak

191fe4b9

catching up unit tests

917afebc

Pipeline staging v2 PR #1 - scheduler (#83)

13c64416

merge conflicts

d3546be7

outlining tutorial

1b7a0b85

Support torch.nn.Sequential (#84)

10c1af46

checking in tutorial

fa9d0871

edits for PP tutorial

01ff1a4c

checkpointing tutorial

689b1e6b

Merge branch 'shaden/pp-tutorial' into shaden/pipe-tutorial

55b55f33

Merge branch 'staging-pp-v2' into shaden/pp-tutorial

9ea0d398

init exports

81251406

Merge branch 'shaden/pp-tutorial' of github.com:microsoft/DeepSpeed-i…

51f91671

tutorial edits

51d3f4b0

section title

ff6b1b32

property to method

67d0463b

Merge branch 'shaden/pp-tutorial' of github.com:microsoft/DeepSpeed-i…

e1017bef

exporting module types from pipeline

2917ca0d

better pipe schedule figure

908b9b48

relative path fix

f840ddff

Preparing pipeline parallelism pull request.

0b9bf2a5

cleaning up merge

0bec1052

finishing merge

8a84062e

tutorial edits

25acb5cf

ShadenSmith added enhancement