transformers
Inkling2 Mini: native NVFP4 / MXFP8 experts on MPS
#51062
Open
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Overview
Commits
167
Changes
View On
GitHub
Inkling2 Mini: native NVFP4 / MXFP8 experts on MPS
#51062
ArthurZucker
wants to merge 167 commits into
main
from
inkling2-metal-fp4
[compressed-tensors] dequantize while loading through the conversion …
6030455c
[compressed-tensors] dequantize=True drops w8a8 activation quantizati…
ac8b3c96
[compressed-tensors] dequantize is the user's flag: base picks the lo…
eaadfac3
[compressed-tensors] static serialized names and load op, op resolves…
0501fe0f
[compressed-tensors] deserialize through the conversion mapping: weig…
93b11334
[quantization] per-layer quantization plan: mixed NVFP4 + FP8 checkpo…
6309e6ea
[quantization] plan: TypeGuard for the plans
387320fc
[quantization] no left name boundary: own-name converters use 3 name …
749a2632
[compressed-tensors] dequantize while loading through the conversion …
14796da4
[compressed-tensors] dequantize=True drops w8a8 activation quantizati…
81ff283f
[compressed-tensors] dequantize is the user's flag: base picks the lo…
3acfe149
[compressed-tensors] static serialized names and load op, op resolves…
acc40ca4
[compressed-tensors] deserialize through the conversion mapping: weig…
20b3c02e
[compressed-tensors] keep weights compressed unless dequantize=True
7cf9e8ae
[quantization] quantize on the fly through a "quantize" load operation
142380e8
[quantization] bnb dequantizes through the "dequantize" load operatio…
1a3ccb59
[quantization] every config takes `dequantize`, a loading attribute o…
9cad2c38
[quantization] gemma dequantizes through the "dequantize" load operation
1f799fc1
[quantization] remove post-load model.dequantize(): load with dequant…
4e221ebd
[quantization] "dequantize" load operation for every quantizer
8c93f47c
[loading] source patterns start on a name boundary; don't dequantize …
d621a7cd
[quantization] saving goes through the "serialize" load operation (to…
ae153968
[quantization] hqq "quantize" op; higgs repacks in "deserialize"; fbg…
cfccc9c0
[loading] non-persistent buffers holding a value keep it when moved o…
baf46a18
[quantization] sinq saves through "serialize"; fbgemm input_scale_ub …
d07429eb
[quantization] dequantize fused params stored under their own name; n…
cbb64d83
[quantization] quantizers import without torch; doc headings without …
2e7276ff
[quantization] review: `dequantizing` property, one FP4 table, leftov…
47e4ec2c
[quantization] `dequantizing` holds for configs without the attribute
0e4799db
[quantization] get_weight_conversions reads `dequantizing`
104943c3
[quantization] leave NVFP4, fused-experts dequantize, the fp8 chain f…
a5fcfdfe
[loading] keep WeightTransform.quantization_operation for peft
0e162f00
[quantization] port bnb dequantize tests to dequantize=True
9311c929
[quantization] bnb dequantize: undo Conv1D transpose, cast to the den…
4f110910
[compressed-tensors] test the default load keeps the compressed forma…
c1884c25
[quantization] say dequantized / unquantized instead of dense
fec5a1f2
[hqq] swap in empty HqqLinear before loading, load ops fill it in place
3d0dd750
[quantization] swap modules in one pass through a per-quantizer `quan…
103d28ba
[compressed-tensors] docs: default load keeps the compressed format a…
ac06ef59
Merge remote-tracking branch 'origin/compressed-tensors-dequantize-on…
dbfba366
[quantization] plan: one update_weight_conversions, quantizer sets ea…
5e2c8471
[quantization] plan: delegate after-load hooks, drop dequantized patt…
165c1f42
[eetq] dequantize without the kernel: undo the CUTLASS layout in torc…
5e01aa6b
[quantization] quantizer_for -> get_quantizer
33194b48
[quantization] dequantize without kernels: torch bnb 4-bit dequant, s…
5ac253ad
Merge remote-tracking branch 'origin/compressed-tensors-dequantize-on…
49d23434
Merge remote-tracking branch 'origin/quantize-load-operation' into HEAD
546564b6
[fp8] bring back replace_with_fp8_embedding
e5314e79
[fp8] drop replace_with_fp8_embedding again: quantized_module swaps t…
96923370
[quantization] load ModelOpt per-layer checkpoints (NVFP4, FP8, MXFP4…
d27969b6
[quantization plan] cleanup: core back near main, NVFP4 experts forwa…
8333cb22
Merge quantize-load-operation (#50240): quantize is a load operation,…
41722334
Merge main into mixed-nvfp4-fp8-plan
959185c0
Split: quantize / dequantize as load operations move to their own PR …
4d4e8b89
[docs] add a quantized format: from a FineGrainedFP8 config to a new …
c8e86f58
[fp8] deserialize ue8m0 scales saved as bytes by their bits (MXFP8)
a2ae59f4
[quantization] docs follow the plan; ModelOpt single quant_algo loads…
6e4953a7
human text
3a23dd5c
not a fan yes of this shit
706bebd6
nits
034c0c37
unbloat the fuck up
809e9e71
more unbloat
0377ebe8
is it simpler?
8d3639d1
one line
2e6066a2
trying to simplify
2431bfeb
simplify ?
adb262b9
nits
5e175d00
will this work?
a4f26d2c
cleanup?
8bc15305
this should make it simpler overall
5243cca0
reject stuffs that need more work for now
035a195f
nits
d5141f8d
renames
647fb5e4
nit
aca01928
super nit
85c8e9e7
doc and rename
efa360a2
space
417f08d1
update
769a6ffb
up
488376e9
Update src/transformers/core_model_loading.py
5564888e
Merge remote-tracking branch 'origin/main' into mixed-nvfp4-fp8-plan
47a6551f
Merge remote-tracking branch 'origin/mixed-nvfp4-fp8-plan' into mixed…
cf8a3e8b
Sort the integrations init (init_isort)
5e4d4fbc
Kimi-K2.5: name the qkv params its converter unfuses
7b1faa50
ai update for most review comments, cleaning up now
94909526
Address review: plan out of Auto, schema readers, per-format experts …
94c9abc2
FP8: drop the MXFP4 paths, MXFP4 experts run on their own interface
ef08fc2e
Rename renamed_names to checkpoint_renames; TODO for MiniMax M3's moe…
3e87b6b6
Shorten the FP8 DeepGEMM dispatch comments (noisy_comments)
baabd050
FP8 dequantize round trip (single format), NVFP4 experts via experts …
4cf548b3
cleanup some shit
04f6f81c
up
b9c16953
Dequantized models save unquantized again (TODO: quantize back)
5a253392
Merge remote-tracking branch 'origin/main' into mixed-nvfp4-fp8-plan
aabbd560
Quantization plan fixes: bitsandbytes loading, NVFP4 parallelism and …
bb5fc01b
Docstrings, spacing and an NVFP4 limitations note; NVFP4 EP test runs…
2baa90ff
add initial tests
78762b20
init draft version
0bab3c2a
Merge branch 'main' into add-inkling2-loop
01147ff1
update tests
a69e0178
more todo
02ee7260
update cache layer handling
39b88b8d
cleanup
aa0cc269
split layers
ef5f4bba
cleanup message passing
3f2a85ff
update gates
f8482977
refactor message passing
aea15837
cleanup
7434736b
update
1c79cce9
cleanup
93542507
remove defensive checks
abe71273
move to correct place
e2abf0f4
dequant step 1
29b216e2
flatten loop config
737f9809
merge message queries
6fa00506
fix
6f3482c6
norm
1aa26ba4
update init weights
5a3dab44
split layer loop
a6a4af27
more splitting
2983e57b
fix tests
c1846e6f
rename
ff6dc648
update inkling2 attentions
d03e8d35
remove unneeded layers
95075700
more cleanup
06661c00
cleanup
ef20ef1b
refactor relative logits
e23ef0ff
merge k/v proj
908b61b9
move things
cb6894c9
fix packed sequences
1a8cad7c
add Inkling2GatedOutputProjection
cc4f3074
drop sdpa
d425f551
refactor scales
6427d4fe
refactor rel pos bias
6620ca61
minor
eb892bfc
fix repo
32049ef7
handle unpadded_vocab_size
ed58c546
merge kv proj
3add8847
fix repo
6e657c3b
fix head loading
9a940316
fix mxfp8 loading
3b759727
use attn interface
b01cd5fc
text_hidden_size is now in the config
48141e7c
use dory caching
1a15086b
move norm to proj
c9ccbdb4
move test
9aed9db2
update docs
e843e8f1
add todo
9ed3671b
update tests
18c8ac85
update docs
3ed39b6e
add blog link
46c36e56
tiny checkpoint
6b1044a9
Merge branch 'main' into add-inkling2-loop
9a1ff825
cleanup
eb4b4a37
make linter happy
6819273b
cleanup
8d628c77
add a10 expectations
09c356e8
skip export tests
f0fd853c
Merge branch 'main' into add-inkling2-mini
e916f57d
update links
c5b376ce
update links
e18dccc2
Merge branch 'add-inkling2-mini' of https://github.com/huggingface/tr…
e3a65e2c
update tests
6173a2f6
fix fp32 loading
bc3f7155
add kernels note
a10ebc3c
Merge the per-layer quantization plan (#50245)
db239281
Native NVFP4 / MXFP8 experts on MPS through the quantization plan
7f80f888
Base automatically changed from
add-inkling2-mini
to
main
2 hours ago
Login to write a write a comment.
Login via GitHub
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub