Fix multiple issues in Op Builder and refactor implementation (#9085)
## Problems fixed
- **Pre-built ops lose GPU architectures.**
- Ops that filter their targets (e.g. `fp_quantizer`) wrote the reduced
list back to `TORCH_CUDA_ARCH_LIST`,
and every op `setup.py` built after them inherited it.
- With `7.0;7.5;8.0;9.0`, 10 ops shipped without sm_70/sm_75 code.
- **BF16 kernels dropped by any pre-Ampere target.**
- A single target below sm_80 built the whole op without
`BF16_AVAILABLE`, so `transformer_inference` and
`inference_core_ops` had no bf16 kernels, and 88 of their op tests fail.
- The default arch lists include such targets. A full master prebuild
only got BF16 because the leaked arch
list above dropped them.
- nvcc 12.2+ compiles these kernels for pre-Ampere targets (it computes
bf16 arithmetic in float there), so
BF16 is now enabled whenever nvcc is 12.2 or newer. Older toolkits keep
the old rule.
- Likely related: #6839, where the bf16 cases of these op tests fail on
V100 (since torch 2.3). JIT builds for
V100 now include the bf16 kernels with nvcc 12.2+, but this was not run
on V100.
- **CUDA 13 default arch list.** The default list (`6.0;6.1;7.0`) has no
target CUDA 13 can build, so a default
prebuild failed. The CUDA 13 defaults now start at 7.5. Together with
the version check below, this addresses
#7587 (CUDA 13 support) for op builds.
- **CUDA version check.**
- `assert_no_cuda_mismatch()` accepted a minor-version difference
between torch's CUDA and `nvcc` only when
both versions were in a hand-maintained table
(`cuda_minor_mismatch_ok`).
- Each new CUDA release had to be tested against the others before it
could be added, so the table lagged
behind. CUDA 13 had no entry, so torch built with 13.0 and `nvcc` 13.1
raised `CUDAMismatchException`.
- The table is replaced by PyTorch's own rule: a major mismatch is an
error (`DS_SKIP_CUDA_CHECK=1`
overrides it), and a minor mismatch is a warning. The check now also
covers JIT builds, which PyTorch does
not check.
- **`TORCH_CUDA_ARCH_LIST` parsing.**
- `9.0a` and empty entries (`8.0;8.6;`) crashed with `ValueError`.
- One parser now serves both modes. It supports `+PTX` and the `a`
suffix, and rejects names such as
`Ampere` with a clear error.
- Related: #7589 hits the same `invalid literal for int()` error on
GH200 for `sm_90a`. Its crash is in
PyTorch's own arch parsing, but setting `TORCH_CUDA_ARCH_LIST=9.0a`
there crashed DeepSpeed's parser too.
- **Pre-built and JIT builds used different flags.**
- Flags are now built once and used by both paths. For example, the CPU
accelerator's pre-built
`async_io` was linked without `-laio` and failed to load (`undefined
symbol: io_submit`).
- Pre-built ops now also get PTX for the highest target, as JIT builds
already did.
- **Hard-coded C++ standards.**
- `-std` was chosen in several places and could override the standard
PyTorch requires.
- These flags are removed. PyTorch already adds the standard its headers
need (C++17 before torch 2.12,
C++20 from 2.12).
- **CPU detection.**
- The `lscpu` fallback crashed on CPUs it didn't recognize
(non-Intel/AMD x86, s390x) and missed SVE when
`sve` was the last flag.
- py-cpuinfo results without `flags` crashed on Apple Silicon (#8765).
- **Repeated toolchain probes.** `nvcc -V`, CPU detection and `rocminfo`
now run once per process instead of
once per op or per call.
- **Duplicated compatibility checks.**
- Seven copies of the target filter and GPU check are replaced by a
`MIN_COMPUTE_CAPABILITY` class attribute.
- `cutlass_ops` and `ragged_device_ops` now report pre-Ampere GPUs as
incompatible instead of failing the build.
- **`build_for_cpu`.**
- CUDA builders no longer switch to a CPU-only build when no CUDA
toolkit is found.
- Host-only ops are built by `CPUOpBuilder`: `async_io` moves there, and
`gds` reuses it.
## Design changes
- **`OpBuilder` is accelerator-neutral.**
- The base class no longer checks for CUDA or ROCm.
- It builds and JIT-loads ops through hooks: `extra_compile_args()`,
`jit_load_kwargs()`, `libraries_args()`
and `validate_torch_op_version()`. `builder()` and `jit_load()` both use
these hooks, so pre-built and JIT
builds get the same flags.
- `is_rocm_pytorch()` stays on the base class because `setup.py` and
Megatron-DeepSpeed call it there.
- **All CUDA and ROCm code lives in `CUDAOpBuilder`.**
- It overrides the hooks to add the nvcc flags, arch lists, BF16 and
ROCm settings, and the CUDA toolkit
check. It also checks the CUDA/HIP version of pre-built ops.
- The JIT mode state, the `TORCH_CUDA_ARCH_LIST` save and restore, and
the ROCm helpers moved there too.
- **Ops that don't need CUDA are built by `CPUOpBuilder`.**
- `async_io` moves to the CPU builders, and `gds` reuses it. The CUDA
accelerator now uses the CPU
`AsyncIOBuilder`, as it already did for the CPU optimizers.
- `CUDAOpBuilder` no longer falls back to a CPU-only build
(`build_for_cpu`) when no CUDA toolkit is found,
and `TorchCPUOpBuilder` is removed.
## Behavior and API changes
- **CUDA toolkit check in JIT builds:** JIT builds now fail on a CUDA
major-version mismatch (override with
`DS_SKIP_CUDA_CHECK=1`). Previously only pre-built builds were checked.
- **PTX in pre-built ops:** pre-built ops include PTX for the highest
target without a suffix. Default builds
gain compute_120 PTX.
- **BF16 with nvcc 12.2+:** CUDA ops define `BF16_AVAILABLE` (and
undefine `__CUDA_NO_BFLOAT16_*`) even when
some targets are older than sm_80. JIT builds on pre-Ampere GPUs now
include the bf16 kernels.
- **`-UC10_USE_GLOG` (#7351) is dropped.** Glog-enabled torch builds
define this macro in their generated
`cmake_macros.h`, and a command-line `-U` cannot undefine a header's
`#define`, so it had no effect.
- **Removed:**
- `TorchCPUOpBuilder`, `build_for_cpu`, `cuda_minor_mismatch_ok`,
`command_exists`,
`DEFAULT_TORCH_EXTENSION_PATH`;
- `is_sycl_enabled()` and `sycl_extension()`, which nothing calls;
- `version_dependent_macros` and `csrc/includes/compat.h` (torch 1.x
only);
- `op_builder/async_io.py`;
- the unused `name` argument of `installed_cuda_version()` and
`assert_no_cuda_mismatch()`;
- the `cross_compile_archs` argument of `compute_capability_args()`,
which no caller passes.
- **Moved or changed:**
- the ROCm helpers moved from `OpBuilder` to `CUDAOpBuilder`
(`OpBuilder.is_rocm_pytorch()` stays);
- `filter_ccs()` takes `CudaArch` entries.
---------
Signed-off-by: Hongwei Chen <hongweichenmail@gmail.com>