PR #3962 add lm_head and embed_out tensor parallel

add lm_head and embed_out tensor parallel #3962

tjruwase merged 4 commits into deepspeedai:master from Yejing-Lai:lyj/lmhead_tp

Yejing-Lai requested a review from

RezaYazdaniAminabadi 2 years ago

Yejing-Lai requested a review from

jeffra 2 years ago

Yejing-Lai requested a review from

mrwyattii 2 years ago

Yejing-Lai requested a review from

awan-10 2 years ago

Yejing-Lai requested a review from

cmikeh2 2 years ago

Yejing-Lai requested a review from

arashb 2 years ago

RezaYazdaniAminabadi closed this 2 years ago

RezaYazdaniAminabadi reopened this 2 years ago

delock commented on 2023-07-24

dc3671 force pushed from 3036650d to 92303cb2 2 years ago

dc3671 commented on 2023-08-16

RezaYazdaniAminabadi approved these changes on 2023-09-29

RezaYazdaniAminabadi enabled auto-merge 2 years ago

disabled auto-merge 2 years ago
Manually disabled by user

dc3671 force pushed from ea7fb280 to 10fbdbc3 2 years ago

add lm_head and embed_out tensor parallel

fecbfdd8

fix load lm_head.weight name issue

3ef32311

replace all_reduce with inference_all_reduce

e3e60e5b

refactor lm_head tensor parallel

fc64ef5b

dc3671 force pushed from ecc676ad to fc64ef5b 2 years ago

tjruwase merged 6763e2de into master 2 years ago

Reviewers

RezaYazdaniAminabadi

delock

dc3671

jeffra

mrwyattii

awan-10

cmikeh2

arashb

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

DeepSpeed add lm_head and embed_out tensor parallel #3962 Merged

add lm_head and embed_out tensor parallel #3962

DeepSpeed
add lm_head and embed_out tensor parallel
#3962

Merged