transformers
9a6925a9 - Fine-grained FP8: DeepSeek-V4's expert_dtype in the quantization config (#49848)

Commit
31 days ago
Fine-grained FP8: DeepSeek-V4's expert_dtype in the quantization config (#49848) * Read DeepSeek-V4.1's expert_dtype from its quantization config V4 names its experts' dtype on the model config, and V4.1 in its quantization config. The FP8 experts read the model config, so DeepseekV4Config.__post_init__ moves V4.1's there. A model config value wins. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Keep DeepSeek-V4's expert_dtype in the quantization config The experts' format describes the quantization. DeepseekV4Config.__post_init__ moves V4's model-config expert_dtype into the quantization config, where V4.1 ships it, and FineGrainedFP8Config reads it. FP8Experts no longer reads the model config. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Run fine-grained FP8 expert parallelism as masked expert parallelism update_parallel_plan rewrites ep_dispatch_experts to moe_tp_experts with an ep_router. Each rank runs its own experts on every token, and an all-reduce sums the outputs, so the quantized experts need no all-to-all dispatch. Mega MoE keeps its own styles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Inline the masked expert parallelism rewrite into update_parallel_plan Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Drop the expert_dtype config test Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Shorten comments, drop an unused import Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Move masked expert parallelism to its own PR Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Parents
Loading