[serge] Fix 1 integration tests regressed by commit bd9509355c8a (PR #47493) (#48426)
* [serge] Fix 1 integration tests regressed by commit bd9509355c8a (PR #47493)
* Say why force_cpu is set here, and point at the precedent
The PR body claimed this matched "the existing pattern used for oversized
concatenations" without saying where that pattern is. It does exist — exactly
once, on qwen4_exp_text's ngram_embedding converter further down this file — but
a reader had to grep for it, and that precedent carries a comment explaining
itself while these two copies carried none.
Names the mechanism, points at both the sibling converter and
core_model_loading.convert_and_load_state_dict_in_model where force_cpu is
actually applied, and records the caveat the PR body omitted: it is a no-op
under tp_plan, which shards before converting.
Also notes why the second converter qualifies. Its operations are
[MergeModulelist(dim=0)] with no Concatenate, so "oversized concatenations" does
not literally describe it — MergeModulelist merges every expert's w2 into one
tensor and takes the same temporary buffer.
Comments only; no behaviour change.
* Reserve device headroom with max_memory instead of patching the loader
Reverts the `force_cpu=True` change to `conversion_mapping.py`. That flag is
for EXTREME single-parameter cases — it was added for qwen4, whose ngram
embedding is ~50B parameters, ~100 GiB in one tensor — and phimoe's parameters
are ordinary. Setting it there would also have changed conversion for every
user of the model, in production, to make one CI runner pass.
The cause is on the test side. `device_map="auto"` plans against each device's
full capacity, so the placement consumes the room the loader still needs:
merging the per-expert w1/w3 weights into the fused gate_up_proj allocates a
temporary buffer on the device the parameter is being placed on, and that is
what OOMs inside from_pretrained.
Restricting per-device memory is what `max_memory` is for. The previous
`cap_psutil_cpu_memory` cap could not do it — it only steers how much goes to
CPU/disk — and it scaled *up* with accelerator count, so it was loosest on the
multi-GPU configuration where this test actually failed.
Reserves ~15% of each accelerator and keeps an explicit CPU budget, so `auto`
still offloads what does not fit.
* Name the real cause: get_balanced_memory reserves nothing on multi-GPU
Replaces the previous max_memory revision with a better-diagnosed one, found by
replaying this failure through the new prompt harness.
`get_balanced_memory` applies its 10% safety margin only when a SINGLE
accelerator is visible — `max_memory[key] *= 0.9` sits inside
`if num_devices == 1`. With several devices it hands each its full capacity, so
Phi-3.5-MoE (~78 GiB in bf16) lands entirely on 4 x 22.30 GiB with nothing
spilled, and GPU 0 is at ~21.5/22.30 when the phimoe WeightConverter merges the
experts' w1/w3 into gate_up_proj — a torch.cat needing a ~1.7 GiB temporary per
layer. That is also why only the multi-gpu variant fails: single-gpu does get
the 0.9 margin and spills to CPU.
Also records why the previous mitigation never fired. cap_psutil_cpu_memory(60
GiB x num_accelerators) is a double no-op here: patch_psutil_cpu_memory computes
min(mem.total, limit), so a 240 GiB "cap" changes nothing on this runner, and a
CPU-side budget cannot create accelerator headroom in any case.
Budget 80% of each accelerator via the existing get_accelerator_total_memory_gib
helper, preserve #48290's 60 GiB x n CPU budget by passing it as max_memory
through the supported API rather than by lying to psutil, and keep
offload_folder for the disk tier. Stays None when no accelerator reports memory.
* Read each device directly: the tidier budget was a silent no-op
The previous revision derived the per-device budget from
`get_accelerator_total_memory_gib()` and guarded it with
`if num_accelerators > 0 and total_accelerator_memory > 0`, falling back to
`max_memory=None`. That helper returns 0.0 whenever `torch_device` is not
exactly "cuda"/"xpu", so the guard fell through and the patch set no budget at
all.
It verified `not_fixed` on GPU (5 runs per tree) with 20.24 GiB allocated on
GPU 0 — above the cap it believed it had set — failing on the same
`Concatenate` for `layers.15.mlp.experts.gate_up_proj`. The earlier revision,
which read `torch.cuda.get_device_properties(i)` per device, verified `fixed`.
So: read the devices directly, take the minimum so a heterogeneous box is
budgeted by its smallest card, and raise SkipTest when there is no accelerator
instead of quietly loading with no budget. A fallback that silently disables the
fix is exactly how the original bug survived `cap_psutil_cpu_memory`, whose
`min(mem.total, limit)` made a 240 GiB "cap" inert — repeating that shape in the
fix would have been the same mistake twice.
* Budget 70%, not 85%: at 85% the headroom is the size of the noise
Three GPU verifies of near-identical budgets came back fixed, not_fixed,
not_fixed — including one that restored the numbers of the run that passed. The
v3 traceback says why: the 18 GiB cap WAS honoured (18.67 GiB allocated) and the
test still OOMed with 168.69 MiB free of 22.30 GiB, the rest "reserved by
PyTorch but unallocated". The failing request is 1.56 GiB, so at 85% the
headroom and the allocator's fragmentation are the same size and the outcome
flips between runs.
70% leaves ~6.7 GiB against that 1.56 GiB — margin instead of a coin toss.
Also asks the allocator not to fragment the reservation away, which the OOM
message itself recommends. Best-effort on purpose: the setter is absent on older
torch and refused once the allocator is configured, and CUDA may already be
initialised by the time a single test runs. The budget is what this rests on.
Uses `getattr(torch, torch_device)` rather than `torch.cuda` directly, so the
budget is computed on xpu too.
* Cut the comment down to what is not evident from the code
Sixteen lines of measurement history for six lines of code. The verify verdicts
and GiB figures belong in the PR discussion, not in the test file — and the
lines they described (a fraction, a dict of budgets) say what they do on their
own.
Two lines left: why a budget is needed at all, which the code cannot say, and
one inline note that the allocator setter is absent on older torch, which is
why it is suppressed.
* Drop the second, stale copy of the same comment
An earlier revision's 15-line block survived alongside the trimmed one, so the
file explained the same thing twice — and the older copy still said "budget 80%"
after the code moved to 70%, which is worse than verbose.
Three lines now, stating only the part the code cannot: that `auto` budgets each
device to its full capacity when several are visible, so the merge temporary has
nowhere to go.
---------
Co-authored-by: serge[bot] <serge[bot]@users.noreply.github.com>
Co-authored-by: Tarek Ziade <tarek@ziade.org>