transformers
Inkling2 Mini: native NVFP4 / MXFP8 experts on MPS
#51062
Open

Inkling2 Mini: native NVFP4 / MXFP8 experts on MPS #51062

ArthurZucker wants to merge 167 commits into main from inkling2-metal-fp4
ArthurZucker
ArthurZucker [compressed-tensors] dequantize while loading through the conversion …
6030455c
ArthurZucker [compressed-tensors] dequantize=True drops w8a8 activation quantizati…
ac8b3c96
ArthurZucker [compressed-tensors] dequantize is the user's flag: base picks the lo…
eaadfac3
ArthurZucker [compressed-tensors] static serialized names and load op, op resolves…
0501fe0f
ArthurZucker [compressed-tensors] deserialize through the conversion mapping: weig…
93b11334
ArthurZucker [quantization] per-layer quantization plan: mixed NVFP4 + FP8 checkpo…
6309e6ea
ArthurZucker [quantization] plan: TypeGuard for the plans
387320fc
ArthurZucker [quantization] no left name boundary: own-name converters use 3 name …
749a2632
ArthurZucker [compressed-tensors] dequantize while loading through the conversion …
14796da4
ArthurZucker [compressed-tensors] dequantize=True drops w8a8 activation quantizati…
81ff283f
ArthurZucker [compressed-tensors] dequantize is the user's flag: base picks the lo…
3acfe149
ArthurZucker [compressed-tensors] static serialized names and load op, op resolves…
acc40ca4
ArthurZucker [compressed-tensors] deserialize through the conversion mapping: weig…
20b3c02e
ArthurZucker [compressed-tensors] keep weights compressed unless dequantize=True
7cf9e8ae
ArthurZucker [quantization] quantize on the fly through a "quantize" load operation
142380e8
ArthurZucker [quantization] bnb dequantizes through the "dequantize" load operatio…
1a3ccb59
ArthurZucker [quantization] every config takes `dequantize`, a loading attribute o…
9cad2c38
ArthurZucker [quantization] gemma dequantizes through the "dequantize" load operation
1f799fc1
ArthurZucker [quantization] remove post-load model.dequantize(): load with dequant…
4e221ebd
ArthurZucker [quantization] "dequantize" load operation for every quantizer
8c93f47c
ArthurZucker [loading] source patterns start on a name boundary; don't dequantize …
d621a7cd
ArthurZucker [quantization] saving goes through the "serialize" load operation (to…
ae153968
ArthurZucker [quantization] hqq "quantize" op; higgs repacks in "deserialize"; fbg…
cfccc9c0
ArthurZucker [loading] non-persistent buffers holding a value keep it when moved o…
baf46a18
ArthurZucker [quantization] sinq saves through "serialize"; fbgemm input_scale_ub …
d07429eb
ArthurZucker [quantization] dequantize fused params stored under their own name; n…
cbb64d83
ArthurZucker [quantization] quantizers import without torch; doc headings without …
2e7276ff
ArthurZucker [quantization] review: `dequantizing` property, one FP4 table, leftov…
47e4ec2c
ArthurZucker [quantization] `dequantizing` holds for configs without the attribute
0e4799db
ArthurZucker [quantization] get_weight_conversions reads `dequantizing`
104943c3
ArthurZucker [quantization] leave NVFP4, fused-experts dequantize, the fp8 chain f…
a5fcfdfe
ArthurZucker [loading] keep WeightTransform.quantization_operation for peft
0e162f00
ArthurZucker [quantization] port bnb dequantize tests to dequantize=True
9311c929
ArthurZucker [quantization] bnb dequantize: undo Conv1D transpose, cast to the den…
4f110910
ArthurZucker [compressed-tensors] test the default load keeps the compressed forma…
c1884c25
ArthurZucker [quantization] say dequantized / unquantized instead of dense
fec5a1f2
ArthurZucker [hqq] swap in empty HqqLinear before loading, load ops fill it in place
3d0dd750
ArthurZucker [quantization] swap modules in one pass through a per-quantizer `quan…
103d28ba
ArthurZucker [compressed-tensors] docs: default load keeps the compressed format a…
ac06ef59
ArthurZucker Merge remote-tracking branch 'origin/compressed-tensors-dequantize-on…
dbfba366
ArthurZucker [quantization] plan: one update_weight_conversions, quantizer sets ea…
5e2c8471
ArthurZucker [quantization] plan: delegate after-load hooks, drop dequantized patt…
165c1f42
ArthurZucker [eetq] dequantize without the kernel: undo the CUTLASS layout in torc…
5e01aa6b
ArthurZucker [quantization] quantizer_for -> get_quantizer
33194b48
ArthurZucker [quantization] dequantize without kernels: torch bnb 4-bit dequant, s…
5ac253ad
ArthurZucker Merge remote-tracking branch 'origin/compressed-tensors-dequantize-on…
49d23434
ArthurZucker Merge remote-tracking branch 'origin/quantize-load-operation' into HEAD
546564b6
ArthurZucker [fp8] bring back replace_with_fp8_embedding
e5314e79
ArthurZucker [fp8] drop replace_with_fp8_embedding again: quantized_module swaps t…
96923370
ArthurZucker [quantization] load ModelOpt per-layer checkpoints (NVFP4, FP8, MXFP4…
d27969b6
ArthurZucker [quantization plan] cleanup: core back near main, NVFP4 experts forwa…
8333cb22
ArthurZucker Merge quantize-load-operation (#50240): quantize is a load operation,…
41722334
ArthurZucker Merge main into mixed-nvfp4-fp8-plan
959185c0
ArthurZucker Split: quantize / dequantize as load operations move to their own PR …
4d4e8b89
ArthurZucker [docs] add a quantized format: from a FineGrainedFP8 config to a new …
c8e86f58
ArthurZucker [fp8] deserialize ue8m0 scales saved as bytes by their bits (MXFP8)
a2ae59f4
ArthurZucker [quantization] docs follow the plan; ModelOpt single quant_algo loads…
6e4953a7
ArthurZucker human text
3a23dd5c
ArthurZucker not a fan yes of this shit
706bebd6
ArthurZucker nits
034c0c37
ArthurZucker unbloat the fuck up
809e9e71
ArthurZucker more unbloat
0377ebe8
ArthurZucker is it simpler?
8d3639d1
ArthurZucker one line
2e6066a2
ArthurZucker trying to simplify
2431bfeb
ArthurZucker simplify ?
adb262b9
ArthurZucker nits
5e175d00
ArthurZucker will this work?
a4f26d2c
ArthurZucker cleanup?
8bc15305
ArthurZucker this should make it simpler overall
5243cca0
ArthurZucker reject stuffs that need more work for now
035a195f
ArthurZucker nits
d5141f8d
ArthurZucker renames
647fb5e4
ArthurZucker nit
aca01928
ArthurZucker super nit
85c8e9e7
ArthurZucker doc and rename
efa360a2
ArthurZucker space
417f08d1
ArthurZucker update
769a6ffb
ArthurZucker up
488376e9
ArthurZucker Update src/transformers/core_model_loading.py
5564888e
ArthurZucker Merge remote-tracking branch 'origin/main' into mixed-nvfp4-fp8-plan
47a6551f
ArthurZucker Merge remote-tracking branch 'origin/mixed-nvfp4-fp8-plan' into mixed…
cf8a3e8b
ArthurZucker Sort the integrations init (init_isort)
5e4d4fbc
ArthurZucker Kimi-K2.5: name the qkv params its converter unfuses
7b1faa50
ArthurZucker ai update for most review comments, cleaning up now
94909526
ArthurZucker Address review: plan out of Auto, schema readers, per-format experts …
94c9abc2
ArthurZucker FP8: drop the MXFP4 paths, MXFP4 experts run on their own interface
ef08fc2e
ArthurZucker Rename renamed_names to checkpoint_renames; TODO for MiniMax M3's moe…
3e87b6b6
ArthurZucker Shorten the FP8 DeepGEMM dispatch comments (noisy_comments)
baabd050
ArthurZucker FP8 dequantize round trip (single format), NVFP4 experts via experts …
4cf548b3
ArthurZucker cleanup some shit
04f6f81c
ArthurZucker up
b9c16953
ArthurZucker Dequantized models save unquantized again (TODO: quantize back)
5a253392
IlyasMoutawwakil Merge remote-tracking branch 'origin/main' into mixed-nvfp4-fp8-plan
aabbd560
IlyasMoutawwakil Quantization plan fixes: bitsandbytes loading, NVFP4 parallelism and …
bb5fc01b
IlyasMoutawwakil Docstrings, spacing and an NVFP4 limitations note; NVFP4 EP test runs…
2baa90ff
guarin add initial tests
78762b20
guarin init draft version
0bab3c2a
guarin Merge branch 'main' into add-inkling2-loop
01147ff1
guarin update tests
a69e0178
guarin more todo
02ee7260
guarin update cache layer handling
39b88b8d
guarin cleanup
aa0cc269
guarin split layers
ef5f4bba
guarin cleanup message passing
3f2a85ff
guarin update gates
f8482977
guarin refactor message passing
aea15837
guarin cleanup
7434736b
guarin update
1c79cce9
guarin cleanup
93542507
guarin remove defensive checks
abe71273
guarin move to correct place
e2abf0f4
guarin dequant step 1
29b216e2
guarin flatten loop config
737f9809
guarin merge message queries
6fa00506
guarin fix
6f3482c6
guarin norm
1aa26ba4
guarin update init weights
5a3dab44
guarin split layer loop
a6a4af27
guarin more splitting
2983e57b
guarin fix tests
c1846e6f
guarin rename
ff6dc648
guarin update inkling2 attentions
d03e8d35
guarin remove unneeded layers
95075700
guarin more cleanup
06661c00
guarin cleanup
ef20ef1b
guarin refactor relative logits
e23ef0ff
guarin merge k/v proj
908b61b9
guarin move things
cb6894c9
guarin fix packed sequences
1a8cad7c
guarin add Inkling2GatedOutputProjection
cc4f3074
guarin drop sdpa
d425f551
guarin refactor scales
6427d4fe
guarin refactor rel pos bias
6620ca61
guarin minor
eb892bfc
guarin fix repo
32049ef7
guarin handle unpadded_vocab_size
ed58c546
guarin merge kv proj
3add8847
guarin fix repo
6e657c3b
guarin fix head loading
9a940316
guarin fix mxfp8 loading
3b759727
guarin use attn interface
b01cd5fc
guarin text_hidden_size is now in the config
48141e7c
guarin use dory caching
1a15086b
guarin move norm to proj
c9ccbdb4
guarin move test
9aed9db2
guarin update docs
e843e8f1
guarin add todo
9ed3671b
guarin update tests
18c8ac85
guarin update docs
3ed39b6e
guarin add blog link
46c36e56
guarin tiny checkpoint
6b1044a9
guarin Merge branch 'main' into add-inkling2-loop
9a1ff825
guarin cleanup
eb4b4a37
guarin make linter happy
6819273b
guarin cleanup
8d628c77
guarin add a10 expectations
09c356e8
guarin skip export tests
f0fd853c
guarin Merge branch 'main' into add-inkling2-mini
e916f57d
guarin update links
c5b376ce
guarin update links
e18dccc2
guarin Merge branch 'add-inkling2-mini' of https://github.com/huggingface/tr…
e3a65e2c
guarin update tests
6173a2f6
guarin fix fp32 loading
bc3f7155
guarin add kernels note
a10ebc3c
ArthurZucker Merge the per-layer quantization plan (#50245)
db239281
ArthurZucker Native NVFP4 / MXFP8 experts on MPS through the quantization plan
7f80f888
github-actions
Base automatically changed from add-inkling2-mini to main 2 hours ago

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
Milestone