Quant: OCP FP8 E4M3 support #27512
ORippler
merged
40 commits into
master
from
osimons/ocp_fp8
ORippler
commented
on 2026-10-07
am17an
approved these changes
on 2026-10-15
pwilkin
approved these changes
on 2026-10-23
pwilkin
approved these changes
on 2026-10-23
ggerganov
commented
on 2026-10-28
0cc4m
approved these changes
on 2026-10-29
WIP OCP FP8 E4M3 support
a61fb04e
Fix python type-checker
f225f62d
Materialize input_scales also for FP8 when converting
d5c93499
Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
a551e83a
Allow remaining FP8 tensors to be dequantized during conversion
01f27ffd
Allow tensor-strategy for FP8-only compressed-tensor ckpts
84b69cc6
Fix and test gguf-py behavior for FP8_E4M3
5566de00
Disallow QKV-fusion for FP8 temporarily
39bbcfa9
Fix conversion of fused qkvz & FP8 for Qwen3-Next
5a5d5cd0
Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws
a22a2358
Emit warnings when not fusing FP8 during conversion
e798cfe5
More FP8 conversion fixes
f0c24471
Add supported for strided copies to/from FP8 in CPU
e16593d4
remove unneeded scratch buffer from FP8 cpu cpy/dup path
f0046231
add explanatory comments and TODOs
291f0ac4
Add dedicated encoder and decoder tests for FP8
949e2fbd
Whitelist ggml_op_get_rows in metal instead of blacklisting
c6e02857
Fix more conversion bugs
6ba67406
Defer weight materialization during conversion
43a09ddc
Blacklist fp8 in metal backend instead of white-list
2dfa62ec
Adjust indents
d124cf97
Fix lint + execute tests
a532b97e
Add fast CPU kernel for F8_E4M3
2eb5e76b
Conversion fixes
6095b9a7
Fix remaining gaps
cc572673
Enable tied weights with shared scales
3b8c645a
Make mandatory-F32 rules apply to FP8 during conversion
dd35612e
Fix python type checks
69c13d6a
Fix kaleidai unit-test failures
32dff68e
Relax numeric precision constraints for pre-check
52708293
Leave fast-FP8 CPU on RISC-V for follow-up
f0c6caa9
Allow output scale tensors to be located in any gguf shard
e2ec2a70
Move test_conversion_fp8.py test execution to behavior-check.yaml
1ec1aecf
WIP OCP FP8 E4M3 support
a61fb04e
Fix python type-checker
f225f62d
Materialize input_scales also for FP8 when converting
d5c93499
Fix conversion of https://huggingface.co/RedHatAI/Qwen3.8-27B-NVFP4
a551e83a
Allow tensor-strategy for FP8-only compressed-tensor ckpts
84b69cc6
Fix and test gguf-py behavior for FP8_E4M3
5566de00
Disallow QKV-fusion for FP8 temporarily
39bbcfa9
Fix conversion of fused qkvz & FP8 for Qwen3-Next
5a5d5cd0
Avoid fusing FP8 for --fuse-gate-up-exps as we may have different ws
a22a2358
Emit warnings when not fusing FP8 during conversion
e798cfe5
More FP8 conversion fixes
f0c24471
Add supported for strided copies to/from FP8 in CPU
e16593d4
remove unneeded scratch buffer from FP8 cpu cpy/dup path
f0046231
add explanatory comments and TODOs
291f0ac4
Add dedicated encoder and decoder tests for FP8
949e2fbd
Whitelist ggml_op_get_rows in metal instead of blacklisting
c6e02857
Fix more conversion bugs
6ba67406
Defer weight materialization during conversion
43a09ddc
Blacklist fp8 in metal backend instead of white-list
2dfa62ec
Adjust indents
d124cf97
Fix lint + execute tests
a532b97e
Add fast CPU kernel for F8_E4M3
2eb5e76b
Conversion fixes
6095b9a7
Fix remaining gaps
cc572673
Enable tied weights with shared scales
3b8c645a
Make mandatory-F32 rules apply to FP8 during conversion
dd35612e
Fix python type checks
69c13d6a
Fix kaleidai unit-test failures
32dff68e
Relax numeric precision constraints for pre-check
52708293
Leave fast-FP8 CPU on RISC-V for follow-up
f0c6caa9
Allow output scale tensors to be located in any gguf shard
e2ec2a70
Move mixed compressed-tensor logic into dequant_fp8 + add unit-tests …
52f0c048
Install all requirements also on Windows for behavior-check
31049c8c
LF line endings
2505793e
Extract FP8 tests from llama-arch to its own test-fixture
9acf4473
Update gguf-py/gguf/scripts/gguf_convert_endian.py
6f53ba82
danbev
commented
on 2026-10-29
ggerganov
approved these changes
on 2026-10-29
danbev
approved these changes
on 2026-10-29
Login to write a write a comment.