transformers
Integrate finegrained kernels: SOTA performance and per-module quantization
#49658
Open
Open
Integrate finegrained kernels: SOTA performance and per-module quantization #49658
IlyasMoutawwakil
wants to merge
210 commits into
main
from
finegrained-quantizer-redesign
finegrained kernels
64708a24
deepgemm and tp/ep fixes
7b21bdb7
finegrained: adopt the interleaved gate|up layout
05bd0f35
finegrained: keep DeepGEMM Mega MoE on the stacked gate|up layout
686238ed
fix
641fcb2c
Merge remote-tracking branch 'origin/main' into finegrained-unified
e6b20c4a
finegrained: build the per-call kernel objects instead of caching them
cb04cbe0
finegrained: fuse both MoE biases into the GEMMs
e1c7a921
Merge origin/main into finegrained-unified
0a1b49b7
Merge remote-tracking branch 'origin/main' into finegrained-unified
90413201
finegrained: expert layouts as conversion ops; experts forwards on th…
eac0ee87
style
1d94e557
finegrained: on-the-fly quantization for every format; loader keeps c…
fb16472d
finegrained: pass input_recipe to the dispatchers, drop the Quantizat…
07dae500
Revert "finegrained: pass input_recipe to the dispatchers, drop the Q…
0b6ca4c9
finegrained: pass activation_format straight through; let deep-gemm s…
513e2314
hub_kernels: finegrained-kernels ships as version 0
240abc9b
hub_kernels: load finegrained-kernels from the kernels-staging build …
30ab2c21
finegrained: consume modelopt's calibrated globals, and fuse a model'…
a562fb8c
finegrained: address the integration review
89267bf3
finegrained: freeze the mxfp4 and nvfp4 integrations, and serve what …
09ea21c0
Merge branch 'main' into finegrained-unified
d59f9037
fix style
e139d2a0
FineGrainedFP8Config is its own class again, not an alias of the new one
c6656441
FineGrainedFP8Config: drop the frozen-notice docstring line
29cca042
Tolerate an experts module without the norm, and a machine without CUDA
f63ad97d
Pin that every modelopt converter target is a slot the module holds
1265f0cc
is_sm100: answer False without CUDA, and fake both halves in the tests
97dfa106
fix
788d02dd
finegrained: drop the DTensor unwrap, and pass the kernel operands by…
53a8970a
deepgemm: only Mega MoE cannot apply a post-expert norm
e8e6b879
Shard quantized expert companions under EP and TP
43b9a748
fixes for the new model
fe0b7182
fix multimodal and add some testing
22b1001c
static
f9302cc2
Load a calibrated checkpoint's input_scale, and fix what per-tensor w…
daa24205
fix
b4dcb7b0
Finish the finegrained integration: reload, TP sharding, dsv4 norms
f7081ecf
Fix a destructive Qwen3 TP-plan override; split out the conversions
9ddb2140
Collapse the conversion-op boilerplate; shorten the longest units
e83eadf5
Key on capability not name; drop the last op boilerplate
60f904e6
Quantize per module group; split the arms by producer, not by format
4cb5d9cb
Carry the quantized-load fixes onto the per-scheme arms, and the same…
a67e7acf
Address the per-scheme review
9fc683de
trim
95b3e44f
fixes
2392e6a6
last
f7a8e543
Cover the groups feature, and export what it asks callers to build
4cbbe7ff
Point everything at the fine-grained integration; deprecate the froze…
f6878604
Bring two comment blocks under the noisy-comment limits
fab7f741
Bring two comment blocks under the noisy-comment limits
21c0a070
Do not let the tied-head plan raise in post_init
12f5aa8e
Do not let the tied-head plan raise in post_init
7abcedf9
Address the second review round
db85a446
Merge finegrained-replace-frozen-fp8 into finegrained-per-scheme-demo
d5c361cf
Carry the review's comment and lint fixes to the sites it did not name
9f7326c6
Make tests/kernels a package so the moved fp8 test can import its fix…
f1b593dc
Declare GLM-4v-MoE's head plan instead of deriving it in the mixin
0067189d
cleanup guards and post expert norm in deepgemm
bb24085b
not now
39eb1d19
minimal peft integration preparation
bf0e7ec4
addressing review comments
5b2b7f00
Tiny checkpoints for every shipped layout, and the save/load fixes th…
f3c7cc8d
Keep pre-quantized scale dtypes through the dtype plan
6a7c92e3
Update src/transformers/distributed/sharding_utils.py
2d1f0bc7
Update src/transformers/distributed/sharding_utils.py
f37d1d46
adapters
5ea2a3ea
Fine-grained layout ops: follow the experts module each tensor fills
c766206b
Revert "adapters"
7d44b81e
trimming
eb9c09fe
Fine-grained NVFP4: exact per-expert gate_up activation globals
77f135cd
Fine-grained quantization: one quantizer per format, and FineGrainedC…
8dc70c4d
Block-scaled quantization: config review fixes, scoped checkpoint ren…
0b3eba2f
Merge remote-tracking branch 'origin/main' into finegrained-per-schem…
6ccaac76
Block-scaled quantization: name per-tensor scales `replicate` in the …
d89e0c4b
Borrow the expert-parallel dispatch wait for raw-pointer kernels
ae746d55
Merge origin/main into finegrained-quantizer-redesign
504d2c38
Merge main into finegrained-quantizer-redesign
6c162bca
Add Muse Spark NVFP4 checkpoint test
e8886aba
Apply repository consistency fixes
30b036fd
Merge main and simplify fine-grained quantizers
79af6725
IlyasMoutawwakil
changed the title
Limit kernel correctness tests to one autotuning trial
7a7e5fd2
IlyasMoutawwakil
changed the title 
Clarify autotuning trial budget and remaining refinement
9dba82da
Restore fused dense scale containers and align MiniMax dtype expectat…
363e4198
Use temporary masked EP and test compiled static-cache decoding
5b2cb429
Apply minimal tuning to standalone generation tests and fix style
5d234d3d
Reuse tuning context in distributed quantization tests
040d4a9c
Simplify finegrained conversions and separate distributed test coverage
aea13cac
Merge main and drop unrelated GLM4V-MoE tensor parallel changes
6ccd9832
Remove scale container conversions and only quantize weights
e1f3eae4
Fix finegrained loading regressions and map ModelOpt mixed precision
6624d7ad
Reject conflicting producer quantization group targets
903b70c8
Preserve ModelOpt native exclusions and trailing-dot module prefixes
3af3c2b9
Refresh Muse NVFP4 fixture expectations and verify normalized FP8 scales
63e39561
Keep format rules in their quantizers and load packed NVFP4 embeddings
0a193c0b
Dispatch quantized embedding lookups through the required kernel API
4622833f
Reject unsupported quantized embedding vocabulary sharding
abe8b7c2
Disable extra autotuning phases explicitly in correctness tests
4272b401
Use finegrained refinement iteration limit in tests
3606b29c
Route rowwise FP8 through finegrained and migrate Llama4 experts
319658d7
Fix finegrained CI docstrings, TP fixtures, and MuseSpark expert plans
1d744428
Merge origin/main into finegrained-quantizer-redesign
f9048ed8
Fix the fine-grained review's bugs
86e5eaf3
Rename BlockFp8Config to Fp8Config
f16b3256
Drop rowwise FP8, the FBGEMM routing and the Llama4 experts migration…
f66c1bea
Fuse MuseSpark's post-expert norm
0f1d4057
Keep each format's producer layouts in one table
edcb6f57
Pass a format its experts' up projections instead of setting them on it
05d449ad
Build the dequantize chains directly
385944ed
Hold what each format fixes as its config's class attributes
9d043497
Name a model's modules back in the checkpoint's namespace in one place
f54f0f54
Split the quantizer's parallel plan update into named steps
f4f4fc04
Make prefers_deepgemm_linear the single gate of the DeepGEMM linear
d37b720a
Declare each experts backend's layout once and check it on switch
55415b4b
Resolve the MoE operands' static half once and name a projection's te…
a805c4d0
Share the formats' storage facts and the slot names between modules a…
d42d3a7a
Stop requiring finegrained-kernels symbols nothing calls
76d1482c
Read the shared gate_up activation global's scope from the quantizati…
8a146662
Make the shared NVFP4 gate_up activation global an `NVFP4Config` option
9ab9a644
Keep only the per-tensor FP8 embedding tables finegrained-fp8 supported
22b7ec74
Read other tools' checkpoints through a registry of config readers
ada7743d
Pass the model to `get_weight_conversions`
b118cc5e
Keep a capturing group's escapes when reversing a conversion pattern
0fb81a0b
Let a model's config adapt the quantization config its checkpoints de…
fe3418f9
Name DeepSeek-V4's block scales through its producer layout, not a mo…
efa0c159
Resolve a checkpoint's skip-list in the model's names in core, and ma…
38c8be5e
Let related quantization config classes lend each other their loading…
4152cf26
Let core save a dequantized load through the model's own conversions
9ae01bdb
Read experts layout flags from the instance or class in use_experts_i…
e75f48a5
Declare each experts implementation's requirements in the ExpertsInte…
43a26a13
Declare a module's local-tensor TP forward with a public tp_uses_loca…
f79fdfa5
Make the intra-expert TP styles and 0-dim replication generic in dist…
9d7cd1b5
Match a skip-list's layer index whole, and keep its other matches as …
08ce100a
Make the block-scaled tests that could not fail able to fail
b86ce0ae
Declare the experts weight layouts on the fine-grained interface
a38525fa
Drop tests/kernels/conftest.py
00a9a377
Drop block-scaled tests that duplicate others or only pin internals
e84c6e10
Make patch_finegrained_kernel a context manager
47fa055d
Table-drive the repeated block-scaled test patterns
864a2173
Make the fine-grained page the hub of the block-scaled format docs
c9c1885b
Remove dead branches and leftovers in the fine-grained quantizers
7b00974d
Say why update_weight_conversions keeps for_save
36057ef5
Trim the fine-grained integration's comments and docstrings
836fd56b
Trim the fine-grained quantizers' comments and docstrings
af98b63a
Shrink the DeepGEMM and hub-kernel diffs against main
f29c0286
Simplify the FP8 table placement exclusion and the plan resolution check
5b791ba7
Cut the slow block-scaled runs: four mixed-scheme pairs, fewer checkp…
30da1693
Run the FP8 Hub tests without a gated checkpoint
601a0abd
Keep main's activation_scale argument on deepgemm_fp8_fp4_linear
22163f1c
Merge branch 'fg-cleanup-tests' into finegrained-quantizer-redesign
f6e7141a
Drop weight-only FP8, which no kernel serves
b0ac2e71
Point the MiniMax-M3-VL docs at the public checkpoint and drop the ol…
9ed300bc
Merge origin/main into finegrained-quantizer-redesign (split target)
1590677a
Use FINEGRAINED_AUTOTUNE_CONFIG=fast for cheap kernel tuning
558204f3
Test the fine-grained machinery against test-only formats
8cfa85ef
Match a skip-list entry from the start only up to a module boundary
9d753a45
Add new tests at the end of their existing test classes
23868896
[fix-skip-list-index] Shorten the should_convert_module docstring
1e529c91
[fix-reverse-escaping] Place the escaping comment next to the unescap…
9e87bcf9
[1-weight-loading] Test the loader fixes and let WeightConverter.copy…
8a3817ea
[2-quantizer-api] Give the legacy FP8 quantizer the checkpoint renami…
8f7a8243
[2-quantizer-api] Drop the reader registration and always adapt a con…
14ed3b26
[3-experts-tp] Drop the legacy FP8 copies of the plan rewrites and te…
97a93dfa
[4-machinery] Follow the private _apply_experts_parallel_styles
d583819b
[4-machinery] Quantize nothing on the fly below the required compute …
eadd1a30
[4-machinery] Check the DeepGEMM experts calls in plain code and test…
59a85854
[4-machinery] Spell out the expert operands and keep internal helpers…
9d008014
[8-mxfp8] Patch the finish helpers where the quantizer imports them
716fa601
[5-fp8] Keep KV-cache checkpoints on compressed-tensors and warn abou…
d7cd4215
[5-fp8] Keep the legacy FP8 tests while the legacy path ships
22b00b73
[8-mxfp8] Shorten the compressed-tensors note and say KV-cache checkp…
9019ac6f
[4-machinery] Let a single-format quantizer own every module it built
eb5311fe
[5-fp8] Quantize embedding tables in one place with a per-tensor FP8 …
180f2a3b
[4-machinery] Align the overview table row
583adb9b
[6-mxfp4] Align the overview table row
ef3701df
[7-nvfp4] Align the overview table row
f9515ce8
[8-mxfp8] Align the overview table row
c0740875
[4-machinery] Dequantize a quantized checkpoint when the kernels pack…
f823cb96
[4-machinery] Apply one fallback rule when quantized compute is not p…
d99f5b9f
[6-mxfp4] Restore main's MXFP4 behaviours and the tests that covered …
49b8c458
[6-mxfp4] Build DeepSeek-V4's plan in its config and fix the docs tha…
bf094703
[4-machinery] Add the per-format activation-format check and drop the…
a860ae9a
[6-mxfp4] Test that Mxfp4Config takes main's positional arguments
a47b68da
[7-nvfp4] Drop the shared gate_up activation global and take the skip…
419d2d38
[7-nvfp4] Delete the frozen NVFP4 integration
f5f07f9c
[7-nvfp4] Test MuseSpark's TP plan with the common expert-sharding test
c9ec382f
[7-nvfp4] Show NVFP4 on-the-fly quantization and tidy the NVFP4 names…
ed00a9bf
[8-mxfp8] Point the MiniMax-M3-VL speed example at the bf16 repo
7839921c
[8-mxfp8] Build the MiniMax-M3-VL parallel fixture from its model tester
796da64f
[1-weight-loading] Write the comments and docstrings in short plain s…
478865c2
[2-quantizer-api] Write the comments and docstrings in short plain se…
f3ac21dd
[3-experts-tp] Write the comments and docstrings in short plain sente…
191cfa07
[4-machinery] Write the comments and docstrings in short plain sentences
dc931650
[5-fp8] Write the comments and docstrings in short plain sentences
44ef6070
[6-mxfp4] Write the comments and docstrings in short plain sentences
08d81e75
[7-nvfp4] Write the comments and docstrings in short plain sentences
72c202d8
[8-mxfp8] Write the comments and docstrings in short plain sentences
79a51891
Sync with the stack: one 8.9 minimum, main's config merge, no W8A4
ddaa2023
Pin MiniMax-M3-VL's routing in the parallel fixture and restore its d…
8bb91a5c
Quantize partial blocks in the legacy FP8 integration
2d1dcbfc
Transpose a dequantized expert stack back for a transposed module
128e6ec8
Keep expert scale grids row-major when the kernels cannot read them s…
6bbad813
Re-record the DeepSeek-V3 FP8 logits slice for the fixed block-FP8 ke…
650b5399
Give the DeepGEMM refusal test's experts two K bands so MX scales swi…
0f695533
Move the skip-list test to a general quantizer utils file
8b2f5392
Sync with the fg-split stack: fold in the review fixes and drop the M…
2cdb10c1
Split the reverse key renaming into explicit converter and renaming s…
fd8d7fd2
Explain why each piece of the fine-grained stack exists
e7ff882c
Keep main's Mega MoE plan rewrite and drop the experts requirements API
f19e8e76
Sync with the stack: the experts layout follows the GPU
8ab6e03f
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub