DeepSpeed

666720b4 - Split the modal CI budget into acquisition and test phases (#8404)

Split the modal CI budget into acquisition and test phases (#8404) Stacked on #8403 — review that one first; this PR's base retargets to `master` automatically once it merges. ## Problem A single wall-clock budget cannot tell **"we never got a GPU"** apart from **"the tests ran long"**. Both surface as the same job timeout today, and a run that spends 22 minutes waiting for capacity takes that time out of the budget the tests still need. Measured on completed master runs, the wait between `Sandbox.create()` and the first sandbox output: | run | acquisition wait | pytest | |---|---|---| | 08-22 12:53 | 0.1m | 37.5m | | 08-29 00:40 | 0.2m | 39.3m | | 08-31 08:45 | 7.9m | 48.8m | | 08-31 14:11 | 10.8m | 44.5m | | 08-31 17:36 | 0.9m | 38.4m | | 09-01 11:47 | 17.8m | 48.4m | | 09-02 04:00 | 22.0m | 38.5m | `Sandbox.create()` returns before the container exists, so the wait for the `l40s:2` reservation surfaces on the first `exec`. ## Change Bound the two phases separately: - **acquisition: 30 min** (`SANDBOX_ACQUIRE_TIMEOUT_SECONDS`). Past that the run aborts with `SandboxStartTimeout`, whose message states that no test ran, instead of holding a runner for the rest of the budget. - **tests: 70 min**, unchanged from #8403. The Sandbox lifetime clock starts when the container starts, so this budget is always fully available once a GPU is reserved, however long acquisition took. - **job timeout: 105 min**, now only a backstop covering 30 + 70 plus runner setup and cleanup. 30 min leaves headroom over the worst observed acquisition (22.0 min) while still failing fast when capacity never arrives. ## Observability The startup duration is printed with `flush=True`. Sandbox output is otherwise block-buffered by Python and lost when the job is killed, which is why the timed-out runs show a silent gap rather than any progress. This makes the acquisition wait directly visible in the log instead of something you have to reconstruct from timestamps. ## Tests `ci/test_torch_latest.py` gains coverage for the abort path: - `test_await_sandbox_start_gives_up_when_the_container_never_runs` — catches a controller that blocks forever on a reservation that is never satisfied. - `test_controller_aborts_without_running_tests_when_sandbox_never_starts` — catches a controller that spends the whole budget waiting, runs commands against a Sandbox that never started, or leaks the Sandbox when startup times out. - `test_await_sandbox_start_reports_startup_duration` Verified by mutation: with the `is_alive()` check removed, the suite hangs instead of failing, which is the exact defect these tests guard against. The acquisition budget is also pinned in `test_sandbox_kwargs_are_fixed_and_secret_free`, so changing a resource limit stays visible in review like the other fixed Sandbox parameters. ## Not addressed here Why acquisition grew from ~0.1 min to 18-22 min. Ruled out from the repo side (image name, preset table, pinned `modal==1.2.6`, sandbox kwargs all unchanged); confirming whether it is GPU queueing or an image pull needs the Modal dashboard for app `deepspeedai-torch-latest-ci`. This PR makes that case fail fast and legibly rather than fixing it. --------- Signed-off-by: Ma, Guokai <guokai.ma@intel.com>