DeepSpeed
e35282bd - Fix multiple issues in Op Builder and refactor implementation (#9085)

Commit
2 days ago
Fix multiple issues in Op Builder and refactor implementation (#9085) ## Problems fixed - **Pre-built ops lose GPU architectures.** - Ops that filter their targets (e.g. `fp_quantizer`) wrote the reduced list back to `TORCH_CUDA_ARCH_LIST`, and every op `setup.py` built after them inherited it. - With `7.0;7.5;8.0;9.0`, 10 ops shipped without sm_70/sm_75 code. - **BF16 kernels dropped by any pre-Ampere target.** - A single target below sm_80 built the whole op without `BF16_AVAILABLE`, so `transformer_inference` and `inference_core_ops` had no bf16 kernels, and 88 of their op tests fail. - The default arch lists include such targets. A full master prebuild only got BF16 because the leaked arch list above dropped them. - nvcc 12.2+ compiles these kernels for pre-Ampere targets (it computes bf16 arithmetic in float there), so BF16 is now enabled whenever nvcc is 12.2 or newer. Older toolkits keep the old rule. - Likely related: #6839, where the bf16 cases of these op tests fail on V100 (since torch 2.3). JIT builds for V100 now include the bf16 kernels with nvcc 12.2+, but this was not run on V100. - **CUDA 13 default arch list.** The default list (`6.0;6.1;7.0`) has no target CUDA 13 can build, so a default prebuild failed. The CUDA 13 defaults now start at 7.5. Together with the version check below, this addresses #7587 (CUDA 13 support) for op builds. - **CUDA version check.** - `assert_no_cuda_mismatch()` accepted a minor-version difference between torch's CUDA and `nvcc` only when both versions were in a hand-maintained table (`cuda_minor_mismatch_ok`). - Each new CUDA release had to be tested against the others before it could be added, so the table lagged behind. CUDA 13 had no entry, so torch built with 13.0 and `nvcc` 13.1 raised `CUDAMismatchException`. - The table is replaced by PyTorch's own rule: a major mismatch is an error (`DS_SKIP_CUDA_CHECK=1` overrides it), and a minor mismatch is a warning. The check now also covers JIT builds, which PyTorch does not check. - **`TORCH_CUDA_ARCH_LIST` parsing.** - `9.0a` and empty entries (`8.0;8.6;`) crashed with `ValueError`. - One parser now serves both modes. It supports `+PTX` and the `a` suffix, and rejects names such as `Ampere` with a clear error. - Related: #7589 hits the same `invalid literal for int()` error on GH200 for `sm_90a`. Its crash is in PyTorch's own arch parsing, but setting `TORCH_CUDA_ARCH_LIST=9.0a` there crashed DeepSpeed's parser too. - **Pre-built and JIT builds used different flags.** - Flags are now built once and used by both paths. For example, the CPU accelerator's pre-built `async_io` was linked without `-laio` and failed to load (`undefined symbol: io_submit`). - Pre-built ops now also get PTX for the highest target, as JIT builds already did. - **Hard-coded C++ standards.** - `-std` was chosen in several places and could override the standard PyTorch requires. - These flags are removed. PyTorch already adds the standard its headers need (C++17 before torch 2.12, C++20 from 2.12). - **CPU detection.** - The `lscpu` fallback crashed on CPUs it didn't recognize (non-Intel/AMD x86, s390x) and missed SVE when `sve` was the last flag. - py-cpuinfo results without `flags` crashed on Apple Silicon (#8765). - **Repeated toolchain probes.** `nvcc -V`, CPU detection and `rocminfo` now run once per process instead of once per op or per call. - **Duplicated compatibility checks.** - Seven copies of the target filter and GPU check are replaced by a `MIN_COMPUTE_CAPABILITY` class attribute. - `cutlass_ops` and `ragged_device_ops` now report pre-Ampere GPUs as incompatible instead of failing the build. - **`build_for_cpu`.** - CUDA builders no longer switch to a CPU-only build when no CUDA toolkit is found. - Host-only ops are built by `CPUOpBuilder`: `async_io` moves there, and `gds` reuses it. ## Design changes - **`OpBuilder` is accelerator-neutral.** - The base class no longer checks for CUDA or ROCm. - It builds and JIT-loads ops through hooks: `extra_compile_args()`, `jit_load_kwargs()`, `libraries_args()` and `validate_torch_op_version()`. `builder()` and `jit_load()` both use these hooks, so pre-built and JIT builds get the same flags. - `is_rocm_pytorch()` stays on the base class because `setup.py` and Megatron-DeepSpeed call it there. - **All CUDA and ROCm code lives in `CUDAOpBuilder`.** - It overrides the hooks to add the nvcc flags, arch lists, BF16 and ROCm settings, and the CUDA toolkit check. It also checks the CUDA/HIP version of pre-built ops. - The JIT mode state, the `TORCH_CUDA_ARCH_LIST` save and restore, and the ROCm helpers moved there too. - **Ops that don't need CUDA are built by `CPUOpBuilder`.** - `async_io` moves to the CPU builders, and `gds` reuses it. The CUDA accelerator now uses the CPU `AsyncIOBuilder`, as it already did for the CPU optimizers. - `CUDAOpBuilder` no longer falls back to a CPU-only build (`build_for_cpu`) when no CUDA toolkit is found, and `TorchCPUOpBuilder` is removed. ## Behavior and API changes - **CUDA toolkit check in JIT builds:** JIT builds now fail on a CUDA major-version mismatch (override with `DS_SKIP_CUDA_CHECK=1`). Previously only pre-built builds were checked. - **PTX in pre-built ops:** pre-built ops include PTX for the highest target without a suffix. Default builds gain compute_120 PTX. - **BF16 with nvcc 12.2+:** CUDA ops define `BF16_AVAILABLE` (and undefine `__CUDA_NO_BFLOAT16_*`) even when some targets are older than sm_80. JIT builds on pre-Ampere GPUs now include the bf16 kernels. - **`-UC10_USE_GLOG` (#7351) is dropped.** Glog-enabled torch builds define this macro in their generated `cmake_macros.h`, and a command-line `-U` cannot undefine a header's `#define`, so it had no effect. - **Removed:** - `TorchCPUOpBuilder`, `build_for_cpu`, `cuda_minor_mismatch_ok`, `command_exists`, `DEFAULT_TORCH_EXTENSION_PATH`; - `is_sycl_enabled()` and `sycl_extension()`, which nothing calls; - `version_dependent_macros` and `csrc/includes/compat.h` (torch 1.x only); - `op_builder/async_io.py`; - the unused `name` argument of `installed_cuda_version()` and `assert_no_cuda_mismatch()`; - the `cross_compile_archs` argument of `compute_capability_args()`, which no caller passes. - **Moved or changed:** - the ROCm helpers moved from `OpBuilder` to `CUDAOpBuilder` (`OpBuilder.is_rocm_pytorch()` stays); - `filter_ccs()` takes `CudaArch` entries. --------- Signed-off-by: Hongwei Chen <hongweichenmail@gmail.com>
Author
Parents
Loading