transformers

826edb70 - vLLM CI: fix install and KV-cache memory, add failure report job (#50436)

vLLM CI: fix install and KV-cache memory, add failure report job (#50436) * [vLLM CI] Add failure report job and analysis scripts - analyze_gh_wf_runs.py: refactored to focus on log fetching + parsing only (removed group_failures, suggest_fix, --groups; removed hardcoded RUN_ID) - analyze_vllm_ci.py: new script that produces 5 structured JSON report files per run (failing tests, failing tests with trace, new failures vs previous main-branch run, new failures with trace, error categories by type) Uses urllib directly (no gh CLI dependency, works inside Docker containers) Merges shard matrix jobs under a single normalized group name - vllm-ci-caller.yml: add vllm-ci-report job that runs after all test jobs (if: always()), fetches logs via GITHUB_TOKEN, and uploads reports as an artifact Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [vLLM CI] Add step to print new failing tests in workflow log Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [vLLM CI] Fix log fetch 401: intercept redirect, fetch blob without auth GitHub's job log endpoint returns a 302 redirect to a signed Azure Blob Storage URL. urllib was forwarding the Authorization header to Azure, which rejected it with 401. Fix: use a custom no-redirect handler to intercept the Location URL, then fetch the blob in a second request without any auth. Also fix job filtering: only fetch logs for jobs with conclusion=failure or cancelled (previously fetching skipped and unknown/still-running jobs too, which both fail and produce no useful output). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [vLLM CI] Only fetch logs for failed jobs (drop cancelled) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [vLLM CI] Temporarily disable all jobs except vllm-test-transformers + report Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [vLLM CI] Apply HF memory budget patch to vllm-test-transformers Writes a minimal pytest plugin to /tmp at runtime (no file added to repo) that applies transformers' psutil memory patch before tests run. Prints before/after virtual_memory().total so we can confirm the patch is active. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * [vLLM CI] Keep jobs separate in reports and add job URLs - Drop shard-name normalization: report keys are now the real job names (shards stay separate); tests are deduplicated per job. - Each job entry is now {job_url, num_failing, tests}, sorted by job name. In *_with_trace files, tests are {test, short_error, trace}. - Update the "Show new failing tests" step for the new format and print the job URL for each job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Categorize failures by root-cause error; fix vLLM trace range analyze_vllm_ci.py: - Replace keyword-based categories (which matched e.g. "assert"/"timeout" anywhere in the trace) with the root-cause error "<ExcType>: <message>": - vLLM engine failures (EngineDeadError, "Engine core initialization failed"): use the first exception from the worker subprocess output. - pydantic ValidationError: use the actual "Value error, ..." message. - Strip run-specific noise (memory addresses, c10 log timestamps) so the same failure keeps the same category across runs. - Categories file now follows the other reports' format: category -> {count, jobs: {job_name -> {job_url, num_failing, tests}}}. analyze_gh_wf_runs.py: - VLLMPlugin strategy 2 matched the first log line that merely mentioned the test id, i.e. the "Running N items in this shard: ..." listing, so every test in a shard got the subprocess errors of the first failing test. Only match lines that start with the node id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Document known gaps and categorization TODO - analyze_gh_wf_runs.py: list known log-parsing gaps (ERROR summary entries, multiple pytest summaries, block matching by bare test name, VLLMPlugin process tags, ...). - analyze_vllm_ci.py: document that failed jobs without parsed pytest failures are dropped from the reports (seen on main run 41959891568), and add a TODO for better category grouping (e.g. OOMs with different memory numbers). These are not handled yet; we will monitor real CI runs and improve as needed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Install the torch version required by the vLLM commit under test vLLM pins torch exactly in requirements/cpu.txt (its precompiled kernels are built against that version). Since vLLM bumped to torch 2.14.0 (#52183, 2026-10-14), our hard-coded 'torch<=2.13.0' made the vLLM editable install unresolvable ("no version of torch==2.14.0+cpu": that install line only sees PyPI, which has no +cpu builds), so every main nightly since 2026-10-16 failed in "Install dependencies" for all jobs. Read the torch version from vllm/requirements/cpu.txt at the checked-out commit and install it from the PyTorch CPU index. torchvision / torchaudio pin torch exactly so uv picks matching versions; torchcodec >= 0.12 supports torch >= 2.11, so the old torchcodec cap is dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Document baseline-run gap and real grouping example - Known gaps: a baseline run whose jobs failed without pytest output has 0 parsed failures, so every current failure is reported as new (seen on run 42196234595 vs baseline 42154858517). - Grouping TODO: add the real case from run 42196234595, where 10 identical vLLM KV-cache memory errors ended up in 10 categories. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Set the CPU KV cache size via VLLM_CPU_KVCACHE_SPACE instead of patching vLLM The workflow lowered vLLM's gpu_memory_utilization default (0.92 -> 0.4) by sed-patching vllm/config/cache.py and vllm/entrypoints/llm.py. vLLM #56492 (2026-11-02) rewrote those lines (default is now None, resolved to 0.92 via CacheConfig.DEFAULT_GPU_MEMORY_UTILIZATION), so the sed silently matched nothing and the CPU worker reserved 92% of the NUMA node: 10 transformers backend tests failed with "Available memory on node 0 ... less than requested memory for kv" (run 42196234595). Use the supported VLLM_CPU_KVCACHE_SPACE env var (GiB; read by vllm/platforms/cpu.py, also used by vLLM's own CPU benchmarks). It sets an explicit KV cache size, which bypasses the utilization-based reservation. 32 GiB is close to the previous ~0.4 x 111 GiB minus the process footprint. Keep the test_initialization.py patch (hard-coded gpu_memory_utilization=0.80 in vLLM's test) but fail loudly if the pattern no longer matches. Drop the now-empty patch step of the multimodal job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Document fix for VLLMPlugin process-tag gap Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Move report scripts to utils/ and apply ruff - Move analyze_vllm_ci.py and analyze_gh_wf_runs.py from the repo root to utils/ (covered by the repo's ruff checks); update the report job path and the usage lines. - ruff check --fix + ruff format (ruff 0.14.10, repo config), plus manual fixes for PERF102 / C416. No behavior change: report outputs on saved logs of runs 32835677554 and 42196234595, and the standalone --traces output, are identical before/after. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Revert "[vLLM CI] Temporarily disable all jobs except vllm-test-transformers + report" This reverts commit 1730fcfd39: re-enable the integration, init and multimodal jobs and restore the report job's full `needs` list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Add TODOs from the full run 42207123286 - Workflow: TODO to install kaldi-native-fbank in the multimodal job (needed by the FireRedASR2 processing tests, listed in vLLM's test requirements) and to deselect test_mtp_draft_loads_mtp_weights (GPU-only, no CPU skip). - analyze_vllm_ci.py grouping TODO: the category comes from the last exception of a chain and may embed input data (FireRedASR2: 3 failures -> 2 categories); suggest preferring the first exception of the chain and stripping data reprs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Use the cgroup-aware psutil cap in the HF memory plugin The plugin used get_ci_cpu_memory_budget_gib(), which only reads CI_CPU_MEMORY_LIMIT_GB. That variable isn't set in this workflow, so every run logged "No budget set, psutil reports 495.9 GiB" and nothing was patched. Use get_cpu_ram_total_gib() instead, the same as transformers' root conftest.py (#48290): it measures the cgroup limit / physical RAM and only falls back to the CI budget when neither is available. Also drop the try/except, so a broken import fails the run instead of being skipped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] TEMP: only run transformers backend shard 0 to verify the HF memory plugin To be reverted before the PR is merged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Revert "[vLLM CI] TEMP: only run transformers backend shard 0 to verify the HF memory plugin" This reverts commit 1adeee68ae: re-enable all jobs and shards. The HF memory plugin was verified in run 42228945517 ("psutil total: 123.8 GiB -> 111.3 GiB"). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Shorten VLLMPlugin comment block (noisy-comment check) Fixes NC001 ("comment block has 6 lines (limit: 5)") from `make check-code-quality`, and updates the description of strategy 2 to match the code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [vLLM CI] Avoid ${{ }} expressions in the report step's run script Use the default GITHUB_RUN_ID / GITHUB_REPOSITORY env vars instead of interpolating github.run_id / github.repository into the shell script (hardening; those values are not attacker-controlled). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>