microsoft/DeepSpeed

Pull Requests Commits

GPT-3 finetuning seems to be working

samyam committed 4 years ago

302eb67c

Added assertions to check the F.linear parameters have ds_id attributes to ensure they are deepspeed parameters

samyam committed 4 years ago

319afab0

Changing the order of pre_module hook registration so that the step_id is reset at the beginning of forward and not after the fetching of parameters of the first module. Partitioning parameters after GatheredParameters exit regardless of whether parameters are modified or not

samyam committed 4 years ago

f4802aa5

1. allow not resetting gpu_sum when calling has_overflow, 2. Clone the param grad before putting it in the swap out gradient buffer since underlying param.grad buffer is reused. 3. Do not return -1 for norm when total norm is NaN Inf. Just return the computed value

samyam committed 4 years ago

a3cee49b

clip grad only for non deepspeed optimizers that do not clip grads inside

samyam committed 4 years ago

1947b38d

Gradient calcualation and clipping bug for not fp16/non-zero code paths

samyam committed 4 years ago

5131fa56

Supporting multiple datatypes in param offload to nvme

samyam committed 4 years ago

95ec643e

Adding a linear layer that does not transpose weights but assumes that the weights are already transposed

samyam committed 4 years ago

9ac6a36f

Changes needed for GPT-3 FineTune to Work

samyam committed 4 years ago

7b64b10e

ZeRO-Infinity: support swapping misaligned sized fp16 tensors (#1076)

tjruwase committed 4 years ago

Verified d88d9279

ZeRO2-Offload: Load balance gradient copying to CPU (#1067)

tjruwase committed 4 years ago

Verified ee4deabd

Seeded unit tests (#1072)

Shaden Smith committed 4 years ago

Verified 46f4573b

Get correct fp16 reuse buffer size (#1071)

tjruwase committed 4 years ago

Verified 6b49b60e

ensure only ds params are gathered (#1044)

stas00 committed 4 years ago

Verified 29b444b6

[configure_distributed_model] improve assert (#1053)

stas00 committed 4 years ago

Verified 1eb9b218

[docs] unused parameter handling (#1060)

tjruwase committed 4 years ago

Verified 63c5070e

Improve flops profiler functionality (#1065)

cli99 committed 4 years ago

Verified 4544b7d2

[docs] Rename train_step_batch_size to train_micro_batch_size_per_gpu (#1066)

wbuchwalter committed 4 years ago

Verified 1f82ab78

Auto-tuning aio parameters (#1059)

tjruwase committed 4 years ago

Verified 6124eb37

Avoid unused parameters assert by default (#1039)

tjruwase committed 4 years ago

Verified 5b393f15

Added community tutorials to README (#1052)

msaroufim committed 4 years ago

Verified fb1213fc

Align optimizer swap buffer sizes (#1036)

tjruwase committed 4 years ago

Verified be5cd76a

bump 0.3.17 for next release

jeffra committed 4 years ago

Verified e4139a7d

fix assert message (#1040)

stas00 committed 4 years ago

Verified 47ec97eb

Change methods to be static (#1038)

janEbert committed 4 years ago

Verified 21047072

fix format (#1032)

cli99 committed 4 years ago

Verified 962dbc63

[Stage][Fix] Add additional conditions when checking types of output from the model (#1026)

Sean Naren committed 4 years ago

Verified b3870363

Bump rexml from 3.2.4 to 3.2.5 in /docs (#1025)

dependabot[bot] committed 4 years ago

Verified c4f4ef5d

Validate env; remove write size from logs (#1013)

tjruwase committed 4 years ago

Verified 820515e4

jeffra committed 4 years ago

a12aaedc

Older