DeepSpeed
0d3bb77b - Add chatglm2 & chatglm3 autotp (#5540)

Commit
1 year ago
Add chatglm2 & chatglm3 autotp (#5540) This PR aims to enable chatglm2 & chatglm3 autotp. Similar to the phi3, this model uses the chunk MLP layer, so we adjust the weight order by 'shard_mlp_chunk' func. Please kindly review~ Thanks! --------- Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com> Co-authored-by: Logan Adams <114770087+loadams@users.noreply.github.com> Co-authored-by: Heyang Qin <heyangqin@microsoft.com>
Author
Parents
Loading