PR #28413 CUDA: Add OCP FP8 support

ORippler changed the title ~~Initial CUDA WIP version~~ CUDA: Add OCP FP8 support 67 days ago

github-actions added ggml

github-actions added CUDA

0cc4m commented on 2026-09-18

ORippler force pushed from 4bcf38c6 to d4b30c45 42 days ago

ORippler force pushed from d4b30c45 42 days ago

ORippler force pushed to ac9720e7 42 days ago

ORippler force pushed from ac9720e7 to 2de9bb7a 42 days ago

ORippler force pushed from 2de9bb7a to e50eed67 41 days ago

ORippler force pushed from e50eed67 41 days ago

ORippler force pushed to 93f06440 41 days ago

ORippler force pushed from 93f06440 to b7d3ef0d 41 days ago

ORippler force pushed to 98d0007c 40 days ago

github-actions added testing

ORippler force pushed from 021df05a to b757f0d5 32 days ago

ORippler force pushed from b757f0d5 29 days ago

ORippler force pushed to 03a892e2 29 days ago

ORippler force pushed from 03a892e2 to e4595bb0 28 days ago

ORippler force pushed from e4595bb0 to 28413b0b 28 days ago

ORippler force pushed from 28413b0b to 46836d43 28 days ago

ORippler force pushed from 46836d43 27 days ago

ORippler force pushed 27 days ago

ORippler force pushed to 414c1a4a 27 days ago

ORippler force pushed from 414c1a4a to fd3d4118 26 days ago

ORippler force pushed from fd3d4118 to f096911d 26 days ago

ORippler force pushed from f096911d 25 days ago

ORippler marked this pull request as ready for review 25 days ago

ORippler requested a review from

ggerganov 25 days ago

ORippler requested a review 25 days ago

ORippler force pushed to d35a3ede 25 days ago

ORippler force pushed from d35a3ede to 218027a7 21 days ago

ORippler force pushed from 218027a7 to e6c215fa 20 days ago

ORippler force pushed from e6c215fa to 10b241d5 20 days ago

ORippler force pushed from 10b241d5 to ae114bc1 20 days ago

ORippler force pushed from ae114bc1 to c50a3ee7 20 days ago

ORippler force pushed from c50a3ee7 to dd45d715 20 days ago

ORippler force pushed from dd45d715 to 11f98748 19 days ago

ORippler force pushed from 11f98748 to 55889d0c 19 days ago

ORippler force pushed from 55889d0c to 3898c331 19 days ago

ORippler force pushed from 3898c331 to dd5b6152 19 days ago

ORippler force pushed from dd5b6152 to 92df19a0 18 days ago

pwilkin approved these changes on 2026-10-23

ORippler force pushed from 92df19a0 to 57bd0d08 14 days ago

ORippler force pushed from 57bd0d08 to 96e137bf 14 days ago

ORippler force pushed from 96e137bf to bb9a2b4b 14 days ago

ORippler force pushed from bb9a2b4b to 3042ccd0 13 days ago

ORippler force pushed from 3042ccd0 to 7979a169 12 days ago

ORippler force pushed from 7979a169 to 2f3ea89e 12 days ago

ORippler force pushed from 2f3ea89e to 6842be1e 12 days ago

ORippler force pushed from 6842be1e to 42812393 6 days ago

am17an commented on 2026-11-05

ORippler force pushed from 42812393 to a45110d7 5 days ago

ORippler requested a review from

am17an 5 days ago

am17an approved these changes on 2026-11-05

ORippler force pushed from 14ee1619 to 50e9cc80 5 days ago

ORippler requested a review from

gaugarg-nv 5 days ago

ORippler force pushed from 50e9cc80 to 010efd71 1 day ago

ORippler requested a review from

CISC 1 day ago

ORippler requested a review 1 day ago

ORippler force pushed from 010efd71 to 349808bd 1 day ago

ggerganov approved these changes on 2026-11-10

JohannesGaessler approved these changes on 2026-11-10

Base automatically changed from osimons/ocp_fp8_kv to master 22 hours ago

ggerganov requested a review 22 hours ago

ORippler force pushed from 349808bd to df516f78 22 hours ago

ggerganov approved these changes on 2026-11-10

am17an approved these changes on 2026-11-10

github-actions added examples

github-actions added devops

github-actions added conversion

Initial CUDA WIP version

f84cca62

Use fp8 intrinsics

d0f246d1

Do W8A16 for FP8 in GEMV case

cff5bdf2

Optimize GEMV kernel for FP8

45c70602

Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.2

e2c30a0a

Remove trailing whitespace

fa08d58e

Add supported for strided copies to/from FP8 in CUDA

25a05890

Prepare Q_reg for FP8 FlashAttention in fattn-vec.cuh

0a0202c5

Fix odd-rows for convert/getrows, possible with FP8

de870890

Fix FP8 fallback in cuBlas

216b2ecb

Gate mmvf_f8x2_e4m3_to_bf162 on FP8 availability, i.e. CTK >= 11.8

f9a909e5

Restrict (fused) MMVF to 2*ts alignment, and cublas FP8 fo 16-byte

1ca5eb31

Add todo about accumulator/precision honoring

32bb9d99

Avoid races in fp8_fallback

9a246dd2

Allow Hopper and other BW SKUs for FP8

fef6e42a

Use intrinsic over explicit manual conversions

c8962ff3

Resolve TODO

45a0a5a6

Add commend on cublaslt workspaces

f39ed23f

Use convert_unary_cont_cuda over block dequant in convert.cu

889d523b

Use get_rows_cuda_float for fp8 kv cache

3f958c3e

Make FP8 copies work on all IHV's

28a0a541

More cleanup

eeaaa144

Simplify by falling back to shared cuBLAS helper

29937a02

Fuse weight_scale multiplication for FP8 also

769fb62e

More code-cleanup

d4711fc1

Guard __nv_cvt_fp8x2_to_bf162raw on CTK 13.4

4689d127

Use F32 activations in fallback mmvf path

9ffcfd58

Explicitly link against cublaslt

db507a37

Move FP8 conversion intrinsics into ggml_cuda_cast

22380bd2

WIP src1/prec for fp8 in MMVF

d506767f

Remove comment

b8ff6b7c

Keep only f8 x f8 KV cache for now

d2e6b8f3

Just do FP8 x FP32 x FP32 in MMVF as its fast enough

d474e212

Update ggml/src/ggml-cuda/ggml-cuda.cu

9673bea0

Add batched FP8 GEMMS in cublas as suggested by code review

df443f61

Extract fp8x2 -> float2 load into helper

82c5203c

Searapte copy checks to/from fp8

6d5ed5bf

ORippler force pushed from df516f78 to 6d5ed5bf 19 hours ago

ORippler requested a review from

am17an 18 hours ago

Compiler massaging to avoid repeated address offsets

889c7d3f

am17an approved these changes on 2026-11-11

ggerganov approved these changes on 2026-11-11

ggerganov merged 9e9619d0 into master 6 hours ago

ORippler deleted the osimons/ocp_fp8_cuda branch 5 hours ago

llama.cpp
CUDA: Add OCP FP8 support
#28413

Merged

CUDA: Add OCP FP8 support #28413

llama.cpp CUDA: Add OCP FP8 support #28413 Merged

CUDA: Add OCP FP8 support #28413

llama.cpp
CUDA: Add OCP FP8 support
#28413

Merged