transformers
4a7ea01a - Rebuild the DeepSpeed CI image on a plain CUDA 12.6 base

Commit
72 days ago
Rebuild the DeepSpeed CI image on a plain CUDA 12.6 base The image was pinned to `PYTORCH=2.8.0` while torchaudio moved on. torchaudio is in maintenance mode and its final 2.11.0 release no longer pins `torch`, so pip resolved `torchaudio-2.11.0+cu126` next to `torch-2.8.0+cu126` and the mismatched extension failed to load: OSError: .../torchaudio/lib/_torchaudio.abi3.so: undefined symbol: torch_library_impl That breaks `import transformers` through `audio_utils`, and DeepSpeed imports transformers while generating its metadata, so the build ended in `metadata-generation-failed`. The build has been red daily since at least 2026-09-04, and Daily CI reinstalls DeepSpeed inside the GPU VM where it hit the same error during Setup -- which skipped every downstream DeepSpeed job, `run_trainer_and_fsdp_gpu` included. Bumping torch to the version `transformers-all-latest-gpu` uses puts it inside torchaudio 2.11's supported range and needs no extra pinning. Move off `nvcr.io/nvidia/pytorch:24.08-py3` onto a plain `nvidia/cuda` base like that image uses, which drops everything that only existed to undo what the nvcr image shipped: the `pandas<2`/`numpy<2` pin, uninstalling `transformer-engine`, `torch-tensorrt`, `nvtx` and `apex`, and repairing `pydantic`. It also drops the `hypothesis==5.35.1` that image preinstalled, which pytest could no longer load once torch 2.13 pulled in a setuptools without `pkg_resources`. Unlike the general test image this one stays on CUDA 12.x: DeepSpeed compiles its own ops, and `get_default_compute_capabilities()` only knows about CUDA 11 and 12, so on CUDA 13 it cross-compiles for `6.0;6.1;7.0` and nvcc rejects the lot. The Dockerfile records this so the base can be moved forward once DeepSpeed catches up. `libaio-dev`, `WORKDIR /workspace` and a narrowed `TORCH_CUDA_ARCH_LIST` are kept on top of the general image's setup, all commented in place. Also pull in the `sklearn` extra: the model zoo tests run the example scripts, which compute metrics with scikit-learn and scipy. The general test image gets those via `[dev]`, and the nvcr base used to preinstall them. Also install `python-is-python3`: the model zoo tests execute DeepSpeed`s `zero_to_fp32.py` directly, and its `#!/usr/bin/env python` shebang found nothing on an Ubuntu base that only ships `python3`, so all 54 of those tests exited 127.
Author
Parents
Loading