onnxruntime
7fe7078f - Cap FP8 GEMV pinned-residency min-blocks hint below SM89 (#32565)

Commit
3 days ago
Cap FP8 GEMV pinned-residency min-blocks hint below SM89 (#32565) ### Description Make the min-blocks-per-SM hint architecture-conditional via `__CUDA_ARCH__`, which nvcc redefines on each per-architecture device-code compilation pass. Below SM89 the hint drops to `1` (a no-op occupancy hint on dead code); at SM89+ it stays `3`, preserving the residency-pinning behavior from #32433 unchanged. No behavior change on SM89+. No currently-selected shape or dispatch path is affected outside of fixing compilation for pre-Ada targets. ### Motivation and Context `MatMulBlockScaledFp8MmaGemvKernelPinned` (added in #32433) carries `__launch_bounds__(32 * KSplit, 3)`. For the only instantiation that's ever launched (`KSplit = 16`), that's `__launch_bounds__(512, 3)`, which requires 512 * 3 = 1536 threads resident per SM. `Fp8MmaGemvPinsResidency` in `matmul_block_scaled_fp8_tiling.h` gates this entry point to SM89+ (Ada), where the max resident threads/SM is exactly 1536 — the intended fit. But nvcc still codegens the template instantiation for every architecture passed to `--generate-code`, and on Turing (`sm_75`, max 1024 threads/SM) 1536 exceeds the hardware limit. `ptxas` rejects it: ``` ptxas error : Value of threads per SM for entry ...MatMulBlockScaledFp8MmaGemvKernelPinned... is out of range. .minnctapersm will be ignored ptxas fatal : Ptx assembly aborted due to errors ``` This breaks any build that includes `sm_75` in its target architectures, even though the kernel is unreachable there at runtime. Note that Nvidia T4 (entry-level GPU available in AWS and GCP) is sm_75. ### Tests Rebuilt `matmul_block_scaled_fp8.cu.o` for `sm_75`, `sm_80`, `sm_89` — compiles clean (previously failed with the `ptxas` error above on `sm_75`). Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Author
Parents
Loading