DeepSpeed
05daf059 - [Workflow] Raise modal CI timeouts to absorb slower sandbox provisioning (#8403)

Commit
8 days ago
[Workflow] Raise modal CI timeouts to absorb slower sandbox provisioning (#8403) The `modal-torch-latest` GPU job is killed at its 75-minute cap before the suite finishes. On master this now happens in roughly half the runs (e.g. runs `33622520807`, `33600515110`, `33533630019`, `33526004564`), each annotated by GitHub with `The job has exceeded the maximum execution time of 1h15m0s`. **The test suite is not what grew.** Phase timings extracted from the job logs: | run | provisioning | env setup | pytest | tests | |---|---|---|---|---| | 08-22 12:53 | 0.1m | 6.0m | 37.5m | 1150 passed | | 08-24 05:12 | 0.1m | 6.3m | 41.1m | 1159 passed | | 08-29 00:40 | 0.2m | 6.2m | 39.3m | 1194 passed | | 08-31 08:45 | 7.9m | 6.4m | 48.8m | — | | 08-31 14:11 | 10.8m | 6.8m | 44.5m | — | | 09-01 11:47 | **17.8m** | 6.6m | 48.4m | 1241 passed | | 09-02 04:00 | **22.0m** | 4.8m | 38.5m | 1241 passed | pytest stayed in its usual 38-49 min band for a near-identical test count (1150 -> 1241, +8%). The entire regression is in **Modal sandbox provisioning: ~0.1 min -> 18-22 min since 2026-08-31**. ## What this PR changes This only restores the margin that regression consumed: - outer GitHub job budget: 75 -> 90 min - inner Modal sandbox lifetime: 3600 -> 4200s (60 -> 70 min) The sandbox clock starts when the container starts, so it has to cover env setup plus pytest, which has already been observed at 55 min - only 5 minutes of headroom under the old 3600s value. Both limits are raised together so the inner one does not become the new binding constraint. `ci/test_torch_latest.py` pins both values as a guard against unreviewed changes to the CI configuration, so the two assertions are updated in the same commit. ## Note: the provisioning regression is not addressed here It still needs investigation on the Modal side. Ruled out from the repo side: - image name unchanged: `pytorch/pytorch:2.10.0-cuda12.8-cudnn9-devel`, preset table untouched since #8016 (2026-05-19) - modal client pinned at `modal==1.2.6` - sandbox kwargs unchanged since #8170 The modal client prints no build output during the wait, which points at scheduling/capacity for `l40s:2` rather than an image pull. Concurrency of our own runs does not correlate (2 concurrent -> 0.9 min, 1 concurrent -> 17.8 min). Confirming this requires the Modal dashboard for app `deepspeedai-torch-latest-ci`. If provisioning keeps growing, 90 minutes will be consumed the same way, so the underlying cause is worth tracking separately. Signed-off-by: Ma, Guokai <guokai.ma@intel.com>
Author
Parents
Loading