transformers
f0d3a726 - Find the expert modules by the marker the sharding style sets

Commit
68 days ago
Find the expert modules by the marker the sharding style sets `apply_fully_sharded_data_parallelism` scanned for modules holding DTensor parameters to find the experts, relying on an invariant established in another file (only the experts are sharded under token dispatch) that stops holding as soon as anything else is sharded. `MoEParamShard.shard_param` already marks the modules it shards; read that marker.
Author
Parents
Loading