transformers
e84aa52c - [serge] Fix 1 integration tests regressed by commit bd9509355c8a (PR #47493) (#48426)

Commit
80 days ago
[serge] Fix 1 integration tests regressed by commit bd9509355c8a (PR #47493) (#48426) * [serge] Fix 1 integration tests regressed by commit bd9509355c8a (PR #47493) * Say why force_cpu is set here, and point at the precedent The PR body claimed this matched "the existing pattern used for oversized concatenations" without saying where that pattern is. It does exist — exactly once, on qwen4_exp_text's ngram_embedding converter further down this file — but a reader had to grep for it, and that precedent carries a comment explaining itself while these two copies carried none. Names the mechanism, points at both the sibling converter and core_model_loading.convert_and_load_state_dict_in_model where force_cpu is actually applied, and records the caveat the PR body omitted: it is a no-op under tp_plan, which shards before converting. Also notes why the second converter qualifies. Its operations are [MergeModulelist(dim=0)] with no Concatenate, so "oversized concatenations" does not literally describe it — MergeModulelist merges every expert's w2 into one tensor and takes the same temporary buffer. Comments only; no behaviour change. * Reserve device headroom with max_memory instead of patching the loader Reverts the `force_cpu=True` change to `conversion_mapping.py`. That flag is for EXTREME single-parameter cases — it was added for qwen4, whose ngram embedding is ~50B parameters, ~100 GiB in one tensor — and phimoe's parameters are ordinary. Setting it there would also have changed conversion for every user of the model, in production, to make one CI runner pass. The cause is on the test side. `device_map="auto"` plans against each device's full capacity, so the placement consumes the room the loader still needs: merging the per-expert w1/w3 weights into the fused gate_up_proj allocates a temporary buffer on the device the parameter is being placed on, and that is what OOMs inside from_pretrained. Restricting per-device memory is what `max_memory` is for. The previous `cap_psutil_cpu_memory` cap could not do it — it only steers how much goes to CPU/disk — and it scaled *up* with accelerator count, so it was loosest on the multi-GPU configuration where this test actually failed. Reserves ~15% of each accelerator and keeps an explicit CPU budget, so `auto` still offloads what does not fit. * Name the real cause: get_balanced_memory reserves nothing on multi-GPU Replaces the previous max_memory revision with a better-diagnosed one, found by replaying this failure through the new prompt harness. `get_balanced_memory` applies its 10% safety margin only when a SINGLE accelerator is visible — `max_memory[key] *= 0.9` sits inside `if num_devices == 1`. With several devices it hands each its full capacity, so Phi-3.5-MoE (~78 GiB in bf16) lands entirely on 4 x 22.30 GiB with nothing spilled, and GPU 0 is at ~21.5/22.30 when the phimoe WeightConverter merges the experts' w1/w3 into gate_up_proj — a torch.cat needing a ~1.7 GiB temporary per layer. That is also why only the multi-gpu variant fails: single-gpu does get the 0.9 margin and spills to CPU. Also records why the previous mitigation never fired. cap_psutil_cpu_memory(60 GiB x num_accelerators) is a double no-op here: patch_psutil_cpu_memory computes min(mem.total, limit), so a 240 GiB "cap" changes nothing on this runner, and a CPU-side budget cannot create accelerator headroom in any case. Budget 80% of each accelerator via the existing get_accelerator_total_memory_gib helper, preserve #48290's 60 GiB x n CPU budget by passing it as max_memory through the supported API rather than by lying to psutil, and keep offload_folder for the disk tier. Stays None when no accelerator reports memory. * Read each device directly: the tidier budget was a silent no-op The previous revision derived the per-device budget from `get_accelerator_total_memory_gib()` and guarded it with `if num_accelerators > 0 and total_accelerator_memory > 0`, falling back to `max_memory=None`. That helper returns 0.0 whenever `torch_device` is not exactly "cuda"/"xpu", so the guard fell through and the patch set no budget at all. It verified `not_fixed` on GPU (5 runs per tree) with 20.24 GiB allocated on GPU 0 — above the cap it believed it had set — failing on the same `Concatenate` for `layers.15.mlp.experts.gate_up_proj`. The earlier revision, which read `torch.cuda.get_device_properties(i)` per device, verified `fixed`. So: read the devices directly, take the minimum so a heterogeneous box is budgeted by its smallest card, and raise SkipTest when there is no accelerator instead of quietly loading with no budget. A fallback that silently disables the fix is exactly how the original bug survived `cap_psutil_cpu_memory`, whose `min(mem.total, limit)` made a 240 GiB "cap" inert — repeating that shape in the fix would have been the same mistake twice. * Budget 70%, not 85%: at 85% the headroom is the size of the noise Three GPU verifies of near-identical budgets came back fixed, not_fixed, not_fixed — including one that restored the numbers of the run that passed. The v3 traceback says why: the 18 GiB cap WAS honoured (18.67 GiB allocated) and the test still OOMed with 168.69 MiB free of 22.30 GiB, the rest "reserved by PyTorch but unallocated". The failing request is 1.56 GiB, so at 85% the headroom and the allocator's fragmentation are the same size and the outcome flips between runs. 70% leaves ~6.7 GiB against that 1.56 GiB — margin instead of a coin toss. Also asks the allocator not to fragment the reservation away, which the OOM message itself recommends. Best-effort on purpose: the setter is absent on older torch and refused once the allocator is configured, and CUDA may already be initialised by the time a single test runs. The budget is what this rests on. Uses `getattr(torch, torch_device)` rather than `torch.cuda` directly, so the budget is computed on xpu too. * Cut the comment down to what is not evident from the code Sixteen lines of measurement history for six lines of code. The verify verdicts and GiB figures belong in the PR discussion, not in the test file — and the lines they described (a fraction, a dict of budgets) say what they do on their own. Two lines left: why a budget is needed at all, which the code cannot say, and one inline note that the allocator setter is absent on older torch, which is why it is suppressed. * Drop the second, stale copy of the same comment An earlier revision's 15-line block survived alongside the trimmed one, so the file explained the same thing twice — and the older copy still said "budget 80%" after the code moved to 70%, which is worse than verbose. Three lines now, stating only the part the code cannot: that `auto` budgets each device to its full capacity when several are visible, so the merge temporary has nowhere to go. --------- Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com> Co-authored-by: Tarek Ziade <tarek@ziade.org>
Parents
Loading