Rebuild the DeepSpeed CI image on a plain CUDA 12.6 base
The image was pinned to `PYTORCH=2.8.0` while torchaudio moved on. torchaudio is
in maintenance mode and its final 2.11.0 release no longer pins `torch`, so pip
resolved `torchaudio-2.11.0+cu126` next to `torch-2.8.0+cu126` and the mismatched
extension failed to load:
OSError: .../torchaudio/lib/_torchaudio.abi3.so: undefined symbol: torch_library_impl
That breaks `import transformers` through `audio_utils`, and DeepSpeed imports
transformers while generating its metadata, so the build ended in
`metadata-generation-failed`. The build has been red daily since at least
2026-09-04, and Daily CI reinstalls DeepSpeed inside the GPU VM where it hit the
same error during Setup -- which skipped every downstream DeepSpeed job,
`run_trainer_and_fsdp_gpu` included.
Bumping torch to the version `transformers-all-latest-gpu` uses puts it inside
torchaudio 2.11's supported range and needs no extra pinning. Move off
`nvcr.io/nvidia/pytorch:24.08-py3` onto a plain `nvidia/cuda` base like that image
uses, which drops everything that only existed to undo what the nvcr image
shipped: the `pandas<2`/`numpy<2` pin, uninstalling `transformer-engine`,
`torch-tensorrt`, `nvtx` and `apex`, and repairing `pydantic`. It also drops the
`hypothesis==5.35.1` that image preinstalled, which pytest could no longer load
once torch 2.13 pulled in a setuptools without `pkg_resources`.
Unlike the general test image this one stays on CUDA 12.x: DeepSpeed compiles its
own ops, and `get_default_compute_capabilities()` only knows about CUDA 11 and 12,
so on CUDA 13 it cross-compiles for `6.0;6.1;7.0` and nvcc rejects the lot. The
Dockerfile records this so the base can be moved forward once DeepSpeed catches
up. `libaio-dev`, `WORKDIR /workspace` and a narrowed `TORCH_CUDA_ARCH_LIST` are
kept on top of the general image's setup, all commented in place.
Also pull in the `sklearn` extra: the model zoo tests run the example scripts, which
compute metrics with scikit-learn and scipy. The general test image gets those via
`[dev]`, and the nvcr base used to preinstall them.
Also install `python-is-python3`: the model zoo tests execute DeepSpeed`s
`zero_to_fp32.py` directly, and its `#!/usr/bin/env python` shebang found nothing on
an Ubuntu base that only ships `python3`, so all 54 of those tests exited 127.