transformers

fa6c8308 - DeepGEMM BF16 + mixed FP8/FP4 + MegaMoE + refactor (#45634)

DeepGEMM BF16 + mixed FP8/FP4 + MegaMoE + refactor (#45634) * init * style * full support * support EP better using offsets ! * comments * get rid of neutralize_ep_sentinels * remove deepgemm stuff * fix * prefix * move * fix * remove comment * fix unintilized outputs leaking * revert unnecessary changes * more unnecessary changes * revert downcast * keep it simple * guard deepgemm cuda version * fix style * update * add deepgemm testing * moe sentinel support * fix * compilable sonicmoe * mega moe kernel support attempt * use package for now * skip ep router and experts pre/post processing * simpler * fix Co-authored-by: Copilot <copilot@github.com> * fix * fix * dtensor support * more dtensor * simpler * remove comment * revert * bc order * revert extra indent * revert unnecessary change * update * less defensive * allow all kernels * alow all kernels * hub only * fix * fix * test * test * sync * check nvcc * probe * fix * test psum * test * test * probe * fix * test * nan issue * repro * repro * fix * simplifications * fix * fix * fix * fix * fix * fix * fix * empty * simplify * test deepseek * dsv4 only * download dsv4 * fix test * push * test * fix * fix ep plan * fix attempt * debug * attempt * debug * fixes in modeling * more modeling changes * more modeling changes * initial megamoe works but wrong output, deepgemm fully functional * mega moe works ! * simplification * comments * fix merge * fix rank divergence * use _keep_in_fp32_modules * simplify quantizer * introduce indexerscorer * remove testing file * style * weight rename for scorer * style * address review comments * style * align with laguna * stale comments and styling * use grouped finegrained fp8 for grouped linear Co-authored-by: sywangyi yi.a.wang@intel.com * comments and docs * Revert "[DeepSeek V4] Fix MoE converter substring-matching FP8 scale companions (#45930)" This reverts commit ecc7e32814750cfaae8de7fd02f462d337ae419c. * fix quantizer and use upstream deepgemm * remove force al kernels * [deepseek_v4] fix FP8 save-side substring match + fp32 gate through softmax * core_model_loading: WeightConverter.reverse_transform now appends a (?=\.|$) token-boundary lookahead to each reversed source so they can't substring-match longer sibling tokens. Re-fixes #45794 (the FP8Experts scale companion `mlp.experts.gate_up_proj_scale_inv` was being routed into the weight-only [Chunk, SplitModulelist] ops on save) without needing `$` on every model-specific converter — that was #45930's approach and broke load via `process_source_pattern`. * Add `ConversionOps.supports_round_trip` opt-in flag; the on-the-fly `quantization_operation` check in reverse_transform now uses it instead of a blanket bail-out. Flagged the ops with audited reverse pairs (Chunk/Concatenate, MergeModulelist/SplitModulelist, Transpose, Conv3d↔Linear, Ernie pair, _IdentityOp, Fp8Quantize/Fp8Dequantize). * deepseek_v4 compressors (HCA / Indexer / CSA): drop the trailing `.to(chunk_gate.dtype)` so the gate stays fp32 from the `+ position_bias` through the softmax — matches the reference Compressor.forward which upcasts x to fp32 at entry. * deepseek_v4 config: only set `attention_factor=1.0` when the compress group resolves to `rope_type="yarn"`. Suppresses the noisy "Unrecognized keys in rope_parameters for rope_type=default" warning on the second post_init pass (where yarn params have already been consumed). * Refresh slow integration test expectations to match the fp32-gate output. Verified on the published DeepSeek-V4-Flash checkpoint: RUN_SLOW=1 pytest tests/models/deepseek_v4 -k Integration -> 2 passed. * always floats scal inv with triton * minimal changes i quantizer * force stable abi build for now * support transposed * fix yarn * unnecessary change * add flash base test and fix the incorrect and crash issue during enab… (#46076) * fix incorrect output in tests/models/deepseek_v4/test_modeling_deepseek_v4.py::DeepseekV4IntegrationTest::test_v4_flash_fp8_generation and add test for flash-base Signed-off-by: Wang, Yi <yi.a.wang@intel.com> * update Signed-off-by: Wang, Yi <yi.a.wang@intel.com> --------- Signed-off-by: Wang, Yi <yi.a.wang@intel.com> * merge * better errors across deepgemm * more better errors for deepgemm * ep tests * Apply suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * scope supports_round_trip to quantization ops only Drop the flag from the regular shape ops (Chunk, Concatenate, MergeModulelist, SplitModulelist, Transpose, Conv3d↔Linear, _IdentityOp, the Ernie pair) — those reverse themselves via their own `reverse_op` property, no flag needed. The flag only gates `WeightConverter.reverse_transform`'s check on the converter's `quantization_operation`, so today it's just Fp8Quantize / Fp8Dequantize. Other quant backends (bnb, torchao, eetq, mxfp4, quanto, ...) stay opt-out until their round-trip is audited. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Apply suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Update src/transformers/core_model_loading.py * style: ruff format Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * custom ep/tp plan for megamoe * process megamoe after weight loading * fix megamoe process * up * the actual fix??? * up * nit comment for futur * style: drop unused exception variable in Fp8Dequantize._dequantize_one * up * style: apply make fix-repo docstring formatting * simplify after merge * review comments * revert unncessary * avoid name contraction * fixes * dynamic tp/ep plan swap * fix deepgemm single error and device assertion docs * properly support bias in grouped linear * simplify * fix dist error handling and min cuda compatbility * fix least cuda compat * get alignment from deepgemm and cache it --------- Signed-off-by: Wang, Yi <yi.a.wang@intel.com> Co-authored-by: Copilot <copilot@github.com> Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com> Co-authored-by: Wang, Yi <yi.a.wang@intel.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>