transformers
4c2daf7f - Fix WeightConverter regex incorrectly matching shared_experts as experts in DeepSeek V4 (#45782)

Commit
85 days ago
Fix WeightConverter regex incorrectly matching shared_experts as experts in DeepSeek V4 (#45782) Fix WeightConverter regex matching shared_experts as experts in DeepSeek V4 The `WeightConverter` patterns for DeepSeek V4 routed experts used the bare substring `experts.` in source and target patterns. Since `shared_experts` contains the substring `experts`, the regex `experts.down_proj` also matches `model.layers.X.mlp.shared_experts.down_proj.weight`. During `save_pretrained`, `revert_weight_conversion` reverses these converters, causing the `down_proj.weight` [hidden_size, intermediate_size] tensor to be incorrectly split into `hidden_size` individual row-slices named `shared_experts.{i}.w2.weight.weight` instead of being saved as a single `shared_experts.down_proj.weight`. This bloats the state dict from 180 keys to 3738 keys. Fix: prefix patterns with `mlp.` so `mlp.experts` no longer matches `mlp.shared_experts`. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Author
Parents
Loading