jax

1a0f79da - [Mosaic GPU] Leverage additional scratch space in cross-warp reductions.

[Mosaic GPU] Leverage additional scratch space in cross-warp reductions. When we have several times more scratch space than necessary for a single cross-warp reduction, we have the opportunity to schedule several cross-warp reductions concurrently, without needing to synchronize across the warpgroup. From benchmarking, we seem to benefit most by issuing 6 to 8 cross-warp reductions simultaneously on H100, and 8 to 10 on Blackwell. ## Benchmarks on H100 | Test index | Dtype | Split | Vector length | Scratch multiplier | Before | After | Diff | Diff % | |------------|---------|------------------|---------------|--------------------|--------|-------|-------|---------| | 0 | float32 | fully_split | 1 | 1 | 2016 | 2012 | -4 | -0.20% | | 1 | float32 | fully_split | 1 | 2 | 2014 | 1570 | -444 | -22.05% | | 2 | float32 | fully_split | 1 | 4 | 2012 | 1502 | -510 | -25.35% | | 3 | float32 | fully_split | 1 | 6 | 2014 | 1443 | -571 | -28.35% | | 4 | float32 | fully_split | 1 | 8 | 2015 | 1445 | -570 | -28.29% | | 5 | float32 | fully_split | 2 | 1 | 3824 | 3840 | 16 | 0.42% | | 6 | float32 | fully_split | 2 | 2 | 3824 | 3712 | -112 | -2.93% | | 7 | float32 | fully_split | 2 | 4 | 3821 | 3624 | -197 | -5.16% | | 8 | float32 | fully_split | 2 | 6 | 3828 | 3605 | -223 | -5.83% | | 9 | float32 | fully_split | 2 | 8 | 3821 | 3667 | -154 | -4.03% | | 10 | float32 | fully_split | 4 | 1 | 22740 | 22837 | 97 | 0.43% | | 11 | float32 | fully_split | 4 | 2 | 23040 | 18697 | -4343 | -18.85% | | 12 | float32 | fully_split | 4 | 4 | 23090 | 19017 | -4073 | -17.64% | | 13 | float32 | fully_split | 4 | 6 | 23060 | 19442 | -3618 | -15.69% | | 14 | float32 | fully_split | 4 | 8 | 23080 | 19466 | -3614 | -15.66% | | 15 | float32 | half_warps_major | 1 | 1 | 2015 | 2023 | 8 | 0.40% | | 16 | float32 | half_warps_major | 1 | 2 | 2015 | 1539 | -476 | -23.62% | | 17 | float32 | half_warps_major | 1 | 4 | 2017 | 1301 | -716 | -35.50% | | 18 | float32 | half_warps_major | 1 | 6 | 2016 | 1225 | -791 | -39.24% | | 19 | float32 | half_warps_major | 1 | 8 | 2014 | 1180 | -834 | -41.41% | | 20 | float32 | half_warps_major | 2 | 1 | 2728 | 2733 | 5 | 0.18% | | 21 | float32 | half_warps_major | 2 | 2 | 2724 | 2351 | -373 | -13.69% | | 22 | float32 | half_warps_major | 2 | 4 | 2721 | 2242 | -479 | -17.60% | | 23 | float32 | half_warps_major | 2 | 6 | 2728 | 2252 | -476 | -17.45% | | 24 | float32 | half_warps_major | 2 | 8 | 2728 | 2283 | -445 | -16.31% | | 25 | float32 | half_warps_major | 4 | 1 | 15363 | 15363 | 0 | 0.00% | | 26 | float32 | half_warps_major | 4 | 2 | 15362 | 6800 | -8562 | -55.73% | | 27 | float32 | half_warps_major | 4 | 4 | 15359 | 6517 | -8842 | -57.57% | | 28 | float32 | half_warps_major | 4 | 6 | 15362 | 5959 | -9403 | -61.21% | | 29 | float32 | half_warps_major | 4 | 8 | 15363 | 6078 | -9285 | -60.44% | | 30 | float32 | half_warps_minor | 1 | 1 | 2032 | 2038 | 6 | 0.30% | | 31 | float32 | half_warps_minor | 1 | 2 | 2026 | 1539 | -487 | -24.04% | | 32 | float32 | half_warps_minor | 1 | 4 | 2031 | 1297 | -734 | -36.14% | | 33 | float32 | half_warps_minor | 1 | 6 | 2036 | 1222 | -814 | -39.98% | | 34 | float32 | half_warps_minor | 1 | 8 | 2027 | 1170 | -857 | -42.28% | | 35 | float32 | half_warps_minor | 2 | 1 | 2759 | 2774 | 15 | 0.54% | | 36 | float32 | half_warps_minor | 2 | 2 | 2762 | 2359 | -403 | -14.59% | | 37 | float32 | half_warps_minor | 2 | 4 | 2759 | 2253 | -506 | -18.34% | | 38 | float32 | half_warps_minor | 2 | 6 | 2761 | 2252 | -509 | -18.44% | | 39 | float32 | half_warps_minor | 2 | 8 | 2760 | 2293 | -467 | -16.92% | | 40 | float32 | half_warps_minor | 4 | 1 | 15278 | 15281 | 3 | 0.02% | | 41 | float32 | half_warps_minor | 4 | 2 | 15280 | 6830 | -8450 | -55.30% | | 42 | float32 | half_warps_minor | 4 | 4 | 15277 | 6265 | -9012 | -58.99% | | 43 | float32 | half_warps_minor | 4 | 6 | 15277 | 5752 | -9525 | -62.35% | | 44 | float32 | half_warps_minor | 4 | 8 | 15277 | 6011 | -9266 | -60.65% | | 45 | float16 | fully_split | 1 | 1 | 2161 | 2188 | 27 | 1.25% | | 46 | float16 | fully_split | 1 | 2 | 2169 | 1691 | -478 | -22.04% | | 47 | float16 | fully_split | 1 | 4 | 2179 | 1492 | -687 | -31.53% | | 48 | float16 | fully_split | 1 | 6 | 2164 | 1462 | -702 | -32.44% | | 49 | float16 | fully_split | 1 | 8 | 2162 | 1429 | -733 | -33.90% | | 50 | float16 | fully_split | 2 | 1 | 2002 | 2013 | 11 | 0.55% | | 51 | float16 | fully_split | 2 | 2 | 2002 | 1553 | -449 | -22.43% | | 52 | float16 | fully_split | 2 | 4 | 2003 | 1469 | -534 | -26.66% | | 53 | float16 | fully_split | 2 | 6 | 2003 | 1419 | -584 | -29.16% | | 54 | float16 | fully_split | 2 | 8 | 2002 | 1414 | -588 | -29.37% | | 55 | float16 | fully_split | 4 | 1 | 3735 | 3743 | 8 | 0.21% | | 56 | float16 | fully_split | 4 | 2 | 3739 | 3809 | 70 | 1.87% | | 57 | float16 | fully_split | 4 | 4 | 3738 | 3615 | -123 | -3.29% | | 58 | float16 | fully_split | 4 | 6 | 3736 | 3587 | -149 | -3.99% | | 59 | float16 | fully_split | 4 | 8 | 3733 | 3619 | -114 | -3.05% | | 60 | float16 | half_warps_major | 1 | 1 | 5812 | 5811 | -1 | -0.02% | | 61 | float16 | half_warps_major | 1 | 2 | 5806 | 1768 | -4038 | -69.55% | | 62 | float16 | half_warps_major | 1 | 4 | 5808 | 1573 | -4235 | -72.92% | | 63 | float16 | half_warps_major | 1 | 6 | 5808 | 1499 | -4309 | -74.19% | | 64 | float16 | half_warps_major | 1 | 8 | 5809 | 1466 | -4343 | -74.76% | | 65 | float16 | half_warps_major | 2 | 1 | 2034 | 2039 | 5 | 0.25% | | 66 | float16 | half_warps_major | 2 | 2 | 2038 | 1562 | -476 | -23.36% | | 67 | float16 | half_warps_major | 2 | 4 | 2033 | 1355 | -678 | -33.35% | | 68 | float16 | half_warps_major | 2 | 6 | 2031 | 1301 | -730 | -35.94% | | 69 | float16 | half_warps_major | 2 | 8 | 2036 | 1269 | -767 | -37.67% | | 70 | float16 | half_warps_major | 4 | 1 | 2764 | 2771 | 7 | 0.25% | | 71 | float16 | half_warps_major | 4 | 2 | 2755 | 2365 | -390 | -14.16% | | 72 | float16 | half_warps_major | 4 | 4 | 2758 | 2260 | -498 | -18.06% | | 73 | float16 | half_warps_major | 4 | 6 | 2752 | 2239 | -513 | -18.64% | | 74 | float16 | half_warps_major | 4 | 8 | 2755 | 2242 | -513 | -18.62% | | 75 | float16 | half_warps_minor | 1 | 1 | 5816 | 5815 | -1 | -0.02% | | 76 | float16 | half_warps_minor | 1 | 2 | 5820 | 1778 | -4042 | -69.45% | | 77 | float16 | half_warps_minor | 1 | 4 | 5827 | 1550 | -4277 | -73.40% | | 78 | float16 | half_warps_minor | 1 | 6 | 5818 | 1485 | -4333 | -74.48% | | 79 | float16 | half_warps_minor | 1 | 8 | 5818 | 1457 | -4361 | -74.96% | | 80 | float16 | half_warps_minor | 2 | 1 | 2031 | 2034 | 3 | 0.15% | | 81 | float16 | half_warps_minor | 2 | 2 | 2029 | 1561 | -468 | -23.07% | | 82 | float16 | half_warps_minor | 2 | 4 | 2029 | 1343 | -686 | -33.81% | | 83 | float16 | half_warps_minor | 2 | 6 | 2029 | 1297 | -732 | -36.08% | | 84 | float16 | half_warps_minor | 2 | 8 | 2028 | 1268 | -760 | -37.48% | | 85 | float16 | half_warps_minor | 4 | 1 | 2758 | 2766 | 8 | 0.29% | | 86 | float16 | half_warps_minor | 4 | 2 | 2754 | 2398 | -356 | -12.93% | | 87 | float16 | half_warps_minor | 4 | 4 | 2763 | 2270 | -493 | -17.84% | | 88 | float16 | half_warps_minor | 4 | 6 | 2759 | 2237 | -522 | -18.92% | | 89 | float16 | half_warps_minor | 4 | 8 | 2756 | 2247 | -509 | -18.47% | | 90 | int8 | fully_split | 1 | 1 | 3503 | 3541 | 38 | 1.08% | | 91 | int8 | fully_split | 1 | 2 | 3490 | 3046 | -444 | -12.72% | | 92 | int8 | fully_split | 1 | 4 | 3488 | 2869 | -619 | -17.75% | | 93 | int8 | fully_split | 1 | 6 | 3500 | 2869 | -631 | -18.03% | | 94 | int8 | fully_split | 1 | 8 | 3498 | 2837 | -661 | -18.90% | | 95 | int8 | fully_split | 2 | 1 | 9133 | 9067 | -66 | -0.72% | | 96 | int8 | fully_split | 2 | 2 | 9086 | 8808 | -278 | -3.06% | | 97 | int8 | fully_split | 2 | 4 | 9116 | 8638 | -478 | -5.24% | | 98 | int8 | fully_split | 2 | 6 | 9073 | 8643 | -430 | -4.74% | | 99 | int8 | fully_split | 2 | 8 | 9132 | 8604 | -528 | -5.78% | | 100 | int8 | fully_split | 4 | 1 | 19516 | 19453 | -63 | -0.32% | | 101 | int8 | fully_split | 4 | 2 | 19525 | 19195 | -330 | -1.69% | | 102 | int8 | fully_split | 4 | 4 | 19513 | 19548 | 35 | 0.18% | | 103 | int8 | fully_split | 4 | 6 | 19498 | 19125 | -373 | -1.91% | | 104 | int8 | fully_split | 4 | 8 | 19420 | 19644 | 224 | 1.15% | | 105 | int8 | half_warps_major | 1 | 1 | 3450 | 3540 | 90 | 2.61% | | 106 | int8 | half_warps_major | 1 | 2 | 3465 | 3057 | -408 | -11.77% | | 107 | int8 | half_warps_major | 1 | 4 | 3460 | 2816 | -644 | -18.61% | | 108 | int8 | half_warps_major | 1 | 6 | 3472 | 2738 | -734 | -21.14% | | 109 | int8 | half_warps_major | 1 | 8 | 3473 | 2683 | -790 | -22.75% | | 110 | int8 | half_warps_major | 2 | 1 | 8356 | 8307 | -49 | -0.59% | | 111 | int8 | half_warps_major | 2 | 2 | 8379 | 7957 | -422 | -5.04% | | 112 | int8 | half_warps_major | 2 | 4 | 8391 | 7727 | -664 | -7.91% | | 113 | int8 | half_warps_major | 2 | 6 | 8316 | 7755 | -561 | -6.75% | | 114 | int8 | half_warps_major | 2 | 8 | 8316 | 7673 | -643 | -7.73% | | 115 | int8 | half_warps_major | 4 | 1 | 20674 | 20518 | -156 | -0.75% | | 116 | int8 | half_warps_major | 4 | 2 | 19993 | 20007 | 14 | 0.07% | | 117 | int8 | half_warps_major | 4 | 4 | 20029 | 20876 | 847 | 4.23% | | 118 | int8 | half_warps_major | 4 | 6 | 19198 | 19382 | 184 | 0.96% | | 119 | int8 | half_warps_major | 4 | 8 | 19129 | 20844 | 1715 | 8.97% | | 120 | int8 | half_warps_minor | 1 | 1 | 3458 | 3502 | 44 | 1.27% | | 121 | int8 | half_warps_minor | 1 | 2 | 3455 | 3027 | -428 | -12.39% | | 122 | int8 | half_warps_minor | 1 | 4 | 3459 | 2807 | -652 | -18.85% | | 123 | int8 | half_warps_minor | 1 | 6 | 3462 | 2720 | -742 | -21.43% | | 124 | int8 | half_warps_minor | 1 | 8 | 3467 | 2681 | -786 | -22.67% | | 125 | int8 | half_warps_minor | 2 | 1 | 8344 | 8277 | -67 | -0.80% | | 126 | int8 | half_warps_minor | 2 | 2 | 8297 | 7981 | -316 | -3.81% | | 127 | int8 | half_warps_minor | 2 | 4 | 8328 | 7785 | -543 | -6.52% | | 128 | int8 | half_warps_minor | 2 | 6 | 8394 | 7879 | -515 | -6.14% | | 129 | int8 | half_warps_minor | 2 | 8 | 8298 | 7667 | -631 | -7.60% | | 130 | int8 | half_warps_minor | 4 | 1 | 19994 | 20088 | 94 | 0.47% | | 131 | int8 | half_warps_minor | 4 | 2 | 19154 | 19290 | 136 | 0.71% | | 132 | int8 | half_warps_minor | 4 | 4 | 19235 | 19813 | 578 | 3.00% | | 133 | int8 | half_warps_minor | 4 | 6 | 19204 | 19355 | 151 | 0.79% | | 134 | int8 | half_warps_minor | 4 | 8 | 19238 | 21120 | 1882 | 9.78% | ## Benchmarks on B200 | Test index | Dtype | Split | Vector length | Scratch multiplier | Before | After | Diff | Diff % | |------------|---------|------------------|---------------|--------------------|--------|-------|-------|---------| | 0 | float32 | fully_split | 1 | 1 | 2327 | 2317 | -10 | -0.43% | | 1 | float32 | fully_split | 1 | 2 | 2329 | 1806 | -523 | -22.46% | | 2 | float32 | fully_split | 1 | 4 | 2327 | 1750 | -577 | -24.80% | | 3 | float32 | fully_split | 1 | 6 | 2322 | 1684 | -638 | -27.48% | | 4 | float32 | fully_split | 1 | 8 | 2320 | 1670 | -650 | -28.02% | | 5 | float32 | fully_split | 1 | 10 | 2322 | 1686 | -636 | -27.39% | | 6 | float32 | fully_split | 2 | 1 | 4517 | 4493 | -24 | -0.53% | | 7 | float32 | fully_split | 2 | 2 | 4519 | 4270 | -249 | -5.51% | | 8 | float32 | fully_split | 2 | 4 | 4524 | 4248 | -276 | -6.10% | | 9 | float32 | fully_split | 2 | 6 | 4509 | 4206 | -303 | -6.72% | | 10 | float32 | fully_split | 2 | 8 | 4508 | 4204 | -304 | -6.74% | | 11 | float32 | fully_split | 2 | 10 | 4509 | 4191 | -318 | -7.05% | | 12 | float32 | fully_split | 4 | 1 | 17450 | 17490 | 40 | 0.23% | | 13 | float32 | fully_split | 4 | 2 | 17435 | 17209 | -226 | -1.30% | | 14 | float32 | fully_split | 4 | 4 | 17496 | 17086 | -410 | -2.34% | | 15 | float32 | fully_split | 4 | 6 | 17396 | 15732 | -1664 | -9.57% | | 16 | float32 | fully_split | 4 | 8 | 17452 | 15057 | -2395 | -13.72% | | 17 | float32 | fully_split | 4 | 10 | 17552 | 15121 | -2431 | -13.85% | | 18 | float32 | half_warps_major | 1 | 1 | 2359 | 2355 | -4 | -0.17% | | 19 | float32 | half_warps_major | 1 | 2 | 2358 | 1808 | -550 | -23.32% | | 20 | float32 | half_warps_major | 1 | 4 | 2359 | 1535 | -824 | -34.93% | | 21 | float32 | half_warps_major | 1 | 6 | 2355 | 1458 | -897 | -38.09% | | 22 | float32 | half_warps_major | 1 | 8 | 2359 | 1402 | -957 | -40.57% | | 23 | float32 | half_warps_major | 1 | 10 | 2352 | 1376 | -976 | -41.50% | | 24 | float32 | half_warps_major | 2 | 1 | 3225 | 3212 | -13 | -0.40% | | 25 | float32 | half_warps_major | 2 | 2 | 3225 | 2800 | -425 | -13.18% | | 26 | float32 | half_warps_major | 2 | 4 | 3230 | 2666 | -564 | -17.46% | | 27 | float32 | half_warps_major | 2 | 6 | 3228 | 2693 | -535 | -16.57% | | 28 | float32 | half_warps_major | 2 | 8 | 3234 | 2750 | -484 | -14.97% | | 29 | float32 | half_warps_major | 2 | 10 | 3224 | 2673 | -551 | -17.09% | | 30 | float32 | half_warps_major | 4 | 1 | 15474 | 15455 | -19 | -0.12% | | 31 | float32 | half_warps_major | 4 | 2 | 15475 | 8070 | -7405 | -47.85% | | 32 | float32 | half_warps_major | 4 | 4 | 15462 | 7389 | -8073 | -52.21% | | 33 | float32 | half_warps_major | 4 | 6 | 15477 | 7269 | -8208 | -53.03% | | 34 | float32 | half_warps_major | 4 | 8 | 15474 | 7119 | -8355 | -53.99% | | 35 | float32 | half_warps_major | 4 | 10 | 15450 | 7606 | -7844 | -50.77% | | 36 | float32 | half_warps_minor | 1 | 1 | 2369 | 2370 | 1 | 0.04% | | 37 | float32 | half_warps_minor | 1 | 2 | 2371 | 1804 | -567 | -23.91% | | 38 | float32 | half_warps_minor | 1 | 4 | 2379 | 1529 | -850 | -35.73% | | 39 | float32 | half_warps_minor | 1 | 6 | 2378 | 1452 | -926 | -38.94% | | 40 | float32 | half_warps_minor | 1 | 8 | 2373 | 1397 | -976 | -41.13% | | 41 | float32 | half_warps_minor | 1 | 10 | 2382 | 1376 | -1006 | -42.23% | | 42 | float32 | half_warps_minor | 2 | 1 | 3263 | 3275 | 12 | 0.37% | | 43 | float32 | half_warps_minor | 2 | 2 | 3270 | 2793 | -477 | -14.59% | | 44 | float32 | half_warps_minor | 2 | 4 | 3273 | 2664 | -609 | -18.61% | | 45 | float32 | half_warps_minor | 2 | 6 | 3280 | 2701 | -579 | -17.65% | | 46 | float32 | half_warps_minor | 2 | 8 | 3266 | 2736 | -530 | -16.23% | | 47 | float32 | half_warps_minor | 2 | 10 | 3260 | 2666 | -594 | -18.22% | | 48 | float32 | half_warps_minor | 4 | 1 | 15335 | 15375 | 40 | 0.26% | | 49 | float32 | half_warps_minor | 4 | 2 | 15367 | 8122 | -7245 | -47.15% | | 50 | float32 | half_warps_minor | 4 | 4 | 15350 | 7212 | -8138 | -53.02% | | 51 | float32 | half_warps_minor | 4 | 6 | 15354 | 7036 | -8318 | -54.17% | | 52 | float32 | half_warps_minor | 4 | 8 | 15378 | 6842 | -8536 | -55.51% | | 53 | float32 | half_warps_minor | 4 | 10 | 15352 | 7455 | -7897 | -51.44% | | 54 | float16 | fully_split | 1 | 1 | 2596 | 2603 | 7 | 0.27% | | 55 | float16 | fully_split | 1 | 2 | 2589 | 2030 | -559 | -21.59% | | 56 | float16 | fully_split | 1 | 4 | 2594 | 1831 | -763 | -29.41% | | 57 | float16 | fully_split | 1 | 6 | 2597 | 1779 | -818 | -31.50% | | 58 | float16 | fully_split | 1 | 8 | 2599 | 1762 | -837 | -32.20% | | 59 | float16 | fully_split | 1 | 10 | 2601 | 1758 | -843 | -32.41% | | 60 | float16 | fully_split | 2 | 1 | 2350 | 2338 | -12 | -0.51% | | 61 | float16 | fully_split | 2 | 2 | 2346 | 1832 | -514 | -21.91% | | 62 | float16 | fully_split | 2 | 4 | 2347 | 1736 | -611 | -26.03% | | 63 | float16 | fully_split | 2 | 6 | 2348 | 1680 | -668 | -28.45% | | 64 | float16 | fully_split | 2 | 8 | 2344 | 1675 | -669 | -28.54% | | 65 | float16 | fully_split | 2 | 10 | 2340 | 1684 | -656 | -28.03% | | 66 | float16 | fully_split | 4 | 1 | 4373 | 4346 | -27 | -0.62% | | 67 | float16 | fully_split | 4 | 2 | 4354 | 4361 | 7 | 0.16% | | 68 | float16 | fully_split | 4 | 4 | 4370 | 4231 | -139 | -3.18% | | 69 | float16 | fully_split | 4 | 6 | 4354 | 4191 | -163 | -3.74% | | 70 | float16 | fully_split | 4 | 8 | 4381 | 4228 | -153 | -3.49% | | 71 | float16 | fully_split | 4 | 10 | 4359 | 4227 | -132 | -3.03% | | 72 | float16 | half_warps_major | 1 | 1 | 6597 | 6608 | 11 | 0.17% | | 73 | float16 | half_warps_major | 1 | 2 | 6595 | 2142 | -4453 | -67.52% | | 74 | float16 | half_warps_major | 1 | 4 | 6606 | 1914 | -4692 | -71.03% | | 75 | float16 | half_warps_major | 1 | 6 | 6601 | 1843 | -4758 | -72.08% | | 76 | float16 | half_warps_major | 1 | 8 | 6585 | 1790 | -4795 | -72.82% | | 77 | float16 | half_warps_major | 1 | 10 | 6603 | 1751 | -4852 | -73.48% | | 78 | float16 | half_warps_major | 2 | 1 | 2399 | 2381 | -18 | -0.75% | | 79 | float16 | half_warps_major | 2 | 2 | 2387 | 1866 | -521 | -21.83% | | 80 | float16 | half_warps_major | 2 | 4 | 2396 | 1617 | -779 | -32.51% | | 81 | float16 | half_warps_major | 2 | 6 | 2384 | 1560 | -824 | -34.56% | | 82 | float16 | half_warps_major | 2 | 8 | 2389 | 1537 | -852 | -35.66% | | 83 | float16 | half_warps_major | 2 | 10 | 2390 | 1536 | -854 | -35.73% | | 84 | float16 | half_warps_major | 4 | 1 | 3292 | 3275 | -17 | -0.52% | | 85 | float16 | half_warps_major | 4 | 2 | 3289 | 2815 | -474 | -14.41% | | 86 | float16 | half_warps_major | 4 | 4 | 3282 | 2707 | -575 | -17.52% | | 87 | float16 | half_warps_major | 4 | 6 | 3277 | 2678 | -599 | -18.28% | | 88 | float16 | half_warps_major | 4 | 8 | 3280 | 2701 | -579 | -17.65% | | 89 | float16 | half_warps_major | 4 | 10 | 3287 | 2696 | -591 | -17.98% | | 90 | float16 | half_warps_minor | 1 | 1 | 6599 | 6601 | 2 | 0.03% | | 91 | float16 | half_warps_minor | 1 | 2 | 6600 | 2150 | -4450 | -67.42% | | 92 | float16 | half_warps_minor | 1 | 4 | 6604 | 1895 | -4709 | -71.31% | | 93 | float16 | half_warps_minor | 1 | 6 | 6604 | 1813 | -4791 | -72.55% | | 94 | float16 | half_warps_minor | 1 | 8 | 6597 | 1780 | -4817 | -73.02% | | 95 | float16 | half_warps_minor | 1 | 10 | 6614 | 1748 | -4866 | -73.57% | | 96 | float16 | half_warps_minor | 2 | 1 | 2392 | 2376 | -16 | -0.67% | | 97 | float16 | half_warps_minor | 2 | 2 | 2389 | 1850 | -539 | -22.56% | | 98 | float16 | half_warps_minor | 2 | 4 | 2395 | 1611 | -784 | -32.73% | | 99 | float16 | half_warps_minor | 2 | 6 | 2388 | 1560 | -828 | -34.67% | | 100 | float16 | half_warps_minor | 2 | 8 | 2386 | 1535 | -851 | -35.67% | | 101 | float16 | half_warps_minor | 2 | 10 | 2385 | 1546 | -839 | -35.18% | | 102 | float16 | half_warps_minor | 4 | 1 | 3279 | 3286 | 7 | 0.21% | | 103 | float16 | half_warps_minor | 4 | 2 | 3297 | 2858 | -439 | -13.32% | | 104 | float16 | half_warps_minor | 4 | 4 | 3289 | 2700 | -589 | -17.91% | | 105 | float16 | half_warps_minor | 4 | 6 | 3291 | 2688 | -603 | -18.32% | | 106 | float16 | half_warps_minor | 4 | 8 | 3272 | 2694 | -578 | -17.67% | | 107 | float16 | half_warps_minor | 4 | 10 | 3285 | 2702 | -583 | -17.75% | | 108 | int8 | fully_split | 1 | 1 | 4274 | 4248 | -26 | -0.61% | | 109 | int8 | fully_split | 1 | 2 | 4272 | 3769 | -503 | -11.77% | | 110 | int8 | fully_split | 1 | 4 | 4288 | 3577 | -711 | -16.58% | | 111 | int8 | fully_split | 1 | 6 | 4272 | 3593 | -679 | -15.89% | | 112 | int8 | fully_split | 1 | 8 | 4268 | 3536 | -732 | -17.15% | | 113 | int8 | fully_split | 1 | 10 | 4265 | 3580 | -685 | -16.06% | | 114 | int8 | fully_split | 2 | 1 | 12203 | 12208 | 5 | 0.04% | | 115 | int8 | fully_split | 2 | 2 | 12215 | 11675 | -540 | -4.42% | | 116 | int8 | fully_split | 2 | 4 | 12300 | 11548 | -752 | -6.11% | | 117 | int8 | fully_split | 2 | 6 | 12231 | 11569 | -662 | -5.41% | | 118 | int8 | fully_split | 2 | 8 | 12298 | 11544 | -754 | -6.13% | | 119 | int8 | fully_split | 2 | 10 | 12198 | 11594 | -604 | -4.95% | | 120 | int8 | fully_split | 4 | 1 | 27148 | 26927 | -221 | -0.81% | | 121 | int8 | fully_split | 4 | 2 | 27171 | 26538 | -633 | -2.33% | | 122 | int8 | fully_split | 4 | 4 | 27125 | 26992 | -133 | -0.49% | | 123 | int8 | fully_split | 4 | 6 | 27142 | 26453 | -689 | -2.54% | | 124 | int8 | fully_split | 4 | 8 | 27136 | 27144 | 8 | 0.03% | | 125 | int8 | fully_split | 4 | 10 | 27256 | 26418 | -838 | -3.07% | | 126 | int8 | half_warps_major | 1 | 1 | 4222 | 4220 | -2 | -0.05% | | 127 | int8 | half_warps_major | 1 | 2 | 4220 | 3725 | -495 | -11.73% | | 128 | int8 | half_warps_major | 1 | 4 | 4229 | 3441 | -788 | -18.63% | | 129 | int8 | half_warps_major | 1 | 6 | 4253 | 3400 | -853 | -20.06% | | 130 | int8 | half_warps_major | 1 | 8 | 4232 | 3280 | -952 | -22.50% | | 131 | int8 | half_warps_major | 1 | 10 | 4231 | 3263 | -968 | -22.88% | | 132 | int8 | half_warps_major | 2 | 1 | 11119 | 11060 | -59 | -0.53% | | 133 | int8 | half_warps_major | 2 | 2 | 11080 | 10582 | -498 | -4.49% | | 134 | int8 | half_warps_major | 2 | 4 | 11119 | 10394 | -725 | -6.52% | | 135 | int8 | half_warps_major | 2 | 6 | 11148 | 10295 | -853 | -7.65% | | 136 | int8 | half_warps_major | 2 | 8 | 11056 | 10170 | -886 | -8.01% | | 137 | int8 | half_warps_major | 2 | 10 | 11060 | 10280 | -780 | -7.05% | | 138 | int8 | half_warps_major | 4 | 1 | 27706 | 27539 | -167 | -0.60% | | 139 | int8 | half_warps_major | 4 | 2 | 26401 | 26802 | 401 | 1.52% | | 140 | int8 | half_warps_major | 4 | 4 | 27392 | 28169 | 777 | 2.84% | | 141 | int8 | half_warps_major | 4 | 6 | 26382 | 26353 | -29 | -0.11% | | 142 | int8 | half_warps_major | 4 | 8 | 26340 | 27798 | 1458 | 5.54% | | 143 | int8 | half_warps_major | 4 | 10 | 26300 | 29385 | 3085 | 11.73% | | 144 | int8 | half_warps_minor | 1 | 1 | 4230 | 4258 | 28 | 0.66% | | 145 | int8 | half_warps_minor | 1 | 2 | 4219 | 3708 | -511 | -12.11% | | 146 | int8 | half_warps_minor | 1 | 4 | 4238 | 3473 | -765 | -18.05% | | 147 | int8 | half_warps_minor | 1 | 6 | 4215 | 3335 | -880 | -20.88% | | 148 | int8 | half_warps_minor | 1 | 8 | 4216 | 3292 | -924 | -21.92% | | 149 | int8 | half_warps_minor | 1 | 10 | 4213 | 3290 | -923 | -21.91% | | 150 | int8 | half_warps_minor | 2 | 1 | 11162 | 11054 | -108 | -0.97% | | 151 | int8 | half_warps_minor | 2 | 2 | 11120 | 10763 | -357 | -3.21% | | 152 | int8 | half_warps_minor | 2 | 4 | 11091 | 10463 | -628 | -5.66% | | 153 | int8 | half_warps_minor | 2 | 6 | 11113 | 10449 | -664 | -5.97% | | 154 | int8 | half_warps_minor | 2 | 8 | 10926 | 10178 | -748 | -6.85% | | 155 | int8 | half_warps_minor | 2 | 10 | 11065 | 10286 | -779 | -7.04% | | 156 | int8 | half_warps_minor | 4 | 1 | 27363 | 27230 | -133 | -0.49% | | 157 | int8 | half_warps_minor | 4 | 2 | 26386 | 26176 | -210 | -0.80% | | 158 | int8 | half_warps_minor | 4 | 4 | 26309 | 27422 | 1113 | 4.23% | | 159 | int8 | half_warps_minor | 4 | 6 | 26406 | 26365 | -41 | -0.16% | | 160 | int8 | half_warps_minor | 4 | 8 | 27353 | 29188 | 1835 | 6.71% | | 161 | int8 | half_warps_minor | 4 | 10 | 26334 | 28987 | 2653 | 10.07% | PiperOrigin-RevId: 859474639