transformers
parallelism goes brrr
#37877
Merged
Overview
Commits
59
Changes
View On GitHub

parallelism goes brrr #37877

ArthurZucker merged 59 commits into main from nouamane/nanotron
nmntz
nmntz accept custom device_mesh
3d90a99d
HuggingFaceDocBuilderDev
nmntz fix device_map
df1eaee8
nmntz assert that num_heads % tp_size == 0
b9298864
Rocketknight1
nmntz todo.
1df751bc
nmntz ReplicateParallel
5887ffc1
nmntz handle tied weights
924cceec
nmntz handle dtensor in save_pretrained with safe_serialization
cfacec55
nmntz tp test works
98333058
nmntz doesnt work
7d7b3636
ArthurZucker ArthurZucker added Tensor Parallel
ArthurZucker ArthurZucker added Core: Modeling
ArthurZucker
ArthurZucker commented on 2025-05-01
S1ro1
S1ro1 commented on 2025-05-01
nmntz fix shard_and_distribute_module's rank should be local_rank
11f02a59
nmntz tp=4 is correct
317c0276
nmntz dp+tp is broken
f3b4ae81
nmntz todo allreduce with dtensors on another dim is annoying
f6a49ee8
nmntz workaround to sync dp grads when using dtensors
eaa65921
nmntz loading a checkpoint works
7c6219bc
nmntz wandb and compare losses with different tp/dp
6ceabe01
nmntz cleaning
a9a15925
nmntz nmntz requested a review from ArthurZucker ArthurZucker 1 year ago
nmntz nmntz requested a review from S1ro1 S1ro1 1 year ago
nmntz nmntz marked this pull request as ready for review 1 year ago
nmntz cleaning
4e323a51
qubvel
qubvel commented on 2025-05-02
nmntz .
7f327b13
nmntz .
c3e5c5ed
nmntz logs
810bd51a
nmntz CP2 DP2 no mask works after commenting attn_mask and is_causal from s…
82348732
nmntz DP=2 TP=2 now works even with tied embeddings
29c2a9ca
nmntz model.parameters() and model.module.parameters() are empty..
8fa760be
nmntz reformat sanity_check_tensor_sync
610e6bb0
nmntz set atol=1e-4 for CP to pass
75cad51d
nmntz try populate _parameters from named_modules
b816a3cc
nmntz refactors
688107c0
nmntz is_causal=True and pack sequences, no attn mask, and preshuffle dataset
cfe688b4
nmntz fix packing
83095210
nmntz CP=4 doesn't work
c0f616ee
nmntz fix labels and position_ids for CP
011d981e
nmntz DP CP works with transformers 🥳🥳🥳
265f90dc
kmehant
ArthurZucker refactor
afa72e24
ArthurZucker add example cp
75176794
ArthurZucker fixup
835726da
ArthurZucker revert sdpa changes
0ad2a156
ArthurZucker example cleared
5b119645
ArthurZucker add CP, DP to the mesh init
7855d102
ArthurZucker nit
0b2bd157
ArthurZucker
ArthurZucker commented on 2025-05-15
nmntz clean
c82d39ce
ArthurZucker use `ALL_PARALLEL_STYLES`
957c351e
ArthurZucker Merge branch 'nouamane/nanotron' of github.com:huggingface/transforme…
6d462e9f
ArthurZucker style
43c175d0
nmntz FSDP works
378b2e7b
nmntz log on 1 rank
30752c63
nmntz .
9c1e1fc2
ArthurZucker fix?
3f683b6e
ArthurZucker Merge branch 'nouamane/nanotron' of github.com:huggingface/transforme…
d36acced
nmntz FSDP1 also has .parameters() bug
780d74d3
nmntz reported gradnorm when using FSDP1 is wrong, but loss is correct so i…
9e549694
nmntz .
ba01287a
nmntz
ArthurZucker style and fixup
677ce533
ArthurZucker move stuff around
81c21de9
ArthurZucker Merge branch 'main' of github.com:huggingface/transformers into nouam…
656277c5
ArthurZucker fix tests
e27ddb85
ArthurZucker style
d702d94d
ArthurZucker let's make it a check
5083c0b0
ArthurZucker
ArthurZucker warning should be an info
67a81826
ArthurZucker ArthurZucker enabled auto-merge (squash) 1 year ago
disabled auto-merge 1 year ago
Manually disabled by user
ArthurZucker ArthurZucker merged 1c2f36b4 into main 1 year ago
ArthurZucker ArthurZucker deleted the nouamane/nanotron branch 1 year ago
manueldeprada
manueldeprada
LysandreJik LysandreJik restored the head branch 1 year ago
ArthurZucker
ydshieh

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
Core: Modeling Tensor Parallel
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum