DeepSpeed
[AutoTP] Make AutoTP work when num_heads not divisible by number of workers
#4011
Merged
Overview
Commits
55
Changes
View On GitHub
Commits
  • allow number of heads not divisible by number of ranks
    delock committed 2 years ago
  • get num_heads from model config, more robust
    delock committed 2 years ago
  • simplify logic where num_head itself is sharded
    delock committed 2 years ago
  • name tweaks
    delock committed 2 years ago
  • make code more robust where num_attention_heads may not be defined in model_config
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    loadams committed 2 years ago
  • support num_key_value_heads < num_attention_heads which is used by llama2
    delock committed 2 years ago
  • add test for 5 ranks
    delock committed 2 years ago
  • change odd rank # to 3 to avoid test skip
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    tjruwase committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • add get_shard_size function
    delock committed 2 years ago
  • modify sharding mechanism according to latest auto TP
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • fix accuracy issue
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    molly-smith committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    tjruwase committed 2 years ago
  • fix format
    delock committed 2 years ago
  • skip tests with fusedqkv
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • remove skip of fusedqkv tests
    delock committed 2 years ago
  • skip test fusedqkv with odd number of ranks
    delock committed 2 years ago
  • support model with n_heads in model_config
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    molly-smith committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • fix TestInjectionPolicy::test[fp32-t5]
    delock committed 2 years ago
  • fix uneven_heads on some fusedqkv types (#12)
    inkcherry committed 2 years ago
  • better fix when activation size cannot be divided by number of heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads_rebase
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    molly-smith committed 2 years ago
  • move tp_shard.py under module_inject
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Add get_num_kv_heads in tp_shard.py
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    molly-smith committed 2 years ago
  • Refine according to comments
    delock committed 2 years ago
  • remove old comment
    mrwyattii committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    mrwyattii committed 2 years ago
  • fix bug in getting num_kv_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    molly-smith committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    tjruwase committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'up-master' into gma/uneven_heads
    delock committed 2 years ago
  • support uneven sharding of lm_head tensor parallel
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
  • Merge branch 'master' into gma/uneven_heads
    delock committed 2 years ago
FAQ Terms Privacy Refunds Impressum
Loading