Build CPU optimizer ops without CUDA and fix the CPU Lion SIMD update (#8813)
`cpu_adam`, `cpu_adagrad` and `cpu_lion` have had no CUDA code since
#5409, but on CUDA systems they were still built through
`TorchCPUOpBuilder`, which links CUDA libraries and checks the installed
CUDA toolkit version. This caused CPU Adam build failures such as
`cannot find -lcurand` (#5677). As suggested, this PR builds them as
plain CPU (C++) extensions without CUDA dependency.
Fixes #5677. Related: #4768 (its "cuda is missing" warning came from
`get_cuda_compile_flag()`, removed here).
## Changes
- CPU Adam/Adagrad/Lion now use the `CPUOpBuilder` builders in
`op_builder/cpu` on CUDA systems as well. `TorchCPUOpBuilder` and its
top-level builders are removed; `async_io` and `dc` now derive from
`CUDAOpBuilder`. CPU Lion is also registered for the CPU accelerator.
- Fix CPU Lion on AVX2/AVX-512: the sign mask was `-1.0f` instead of
`-0.0f`, so vectorized elements got an almost-zero update instead of
±lr. Add a reference test covering every SIMD path.
- If a CPU optimizer op fails to build/load, raise one `RuntimeError`
naming the op, with the build error as its cause.
`DeepSpeedCPUAdagrad`/`DeepSpeedCPULion` no longer add a misleading
`AttributeError` from `__del__`.
- Skip `TestCPULionGPUError` on the CPU accelerator.
---------
Signed-off-by: Hongwei Chen <hongweichenmail@gmail.com>
Signed-off-by: Hongwei Chen <hongweichen@microsoft.com>
DeepSpeed
ca913e43 - Build CPU optimizer ops without CUDA and fix the CPU Lion SIMD update (#8813)
