[AutoSP] Align cross entropy targets after an in-model causal label shift (#8811)
> **Stacked on #8457.** Only the last commit (`6f7c57b9a`) belongs to
this PR; please review that commit. I will rebase once #8457 is merged.
Addresses
https://github.com/deepspeedai/DeepSpeed/pull/8457#discussion_r4243588078
(in-model causal label shift under AutoSP).
### Problem
AutoSP shards the labels the same way as `input_ids`. A causal LM that
shifts its labels inside the model (Hugging Face `ForCausalLMLoss` pads
and shifts, `labels = F.pad(labels, (0, 1), value=-100)[..., 1:]`;
Megatron-style code slices `logits[..., :-1, :]` / `labels[..., 1:]`)
then shifts each shard on its own. Each rank pairs the last token of its
shard with the wrong target, and the loss is silently wrong. #8457
documents pre-shifted labels as the workaround; this PR makes the
in-model shift itself correct.
### Approach
- Labels are no longer sharded. Any shift the model applies to them runs
on the full sequence, so the label subgraph computes exactly what it
does without SP.
- For each `F.cross_entropy` whose target derives from the labels:
1. Walk from the CE input back to the sequence-sharded logits. Static
sequence slices on the way (e.g. `logits[..., :-1, :]`) are removed,
because each rank must keep its whole shard of logits. Their offsets
give the window `[start, S + stop)` of global positions the CE input
covered.
2. Pad the full target with `ignore_index` back to the global positions
of that window, then hand the CE this rank's shard `[r*S/sp,
(r+1)*S/sp)`. This needs no communication because the labels are
replicated.
3. The loss and gradient aggregation from #8457 is unchanged.
- A static check makes sure the target lines up position by position
with the logits window: the symbolic length of the target's sequence
dimension must equal `S + stop - start`.
- Unsupported patterns that the pass detects raise instead of silently
misaligning (equivalent rewrites such as a `torch.cat` of shifted logits
slices may go undetected; the docs say so):
- Only casts, `contiguous`, `view`/`reshape`/`flatten` (after any
slices) and static sequence slices are allowed between the logits and
the CE.
- The labels may only flow into CE targets. They cannot be combined with
sharded tensors such as `input_ids`, or returned from the module.
- Padding uses `narrow` + `full_like` + `cat` instead of
`F.pad`/`constant_pad_nd`, which specialize the symbolic sequence length
during fake-tensor propagation.
### Validation (8× Intel Arc Pro B70, torch 2.14.0.dev+xpu)
- `TestAutoSPCausalLoss` compares the loss and the parameters after two
optimizer steps against single-device training on the full sequence,
over 4 ranks (sp=2, dp=2). New cases:
- `slice` (`logits[..., :-1, :]` / `labels[..., 1:]`)
- `pad` (HF style)
- each combined with ZeRO-1, `sum`/`mean`, an explicit mesh, and an
empty shard
- `test_rejects_labels_it_cannot_align` covers a strided logits slice
and labels masked by `input_ids`.
- `test_compile_autosp.py`: 45 sequential and 20 non-sequential tests
pass. `TestAutoSPCompile` and `TestAutoSPEngineBackward` were excluded
because they fail on this machine for environmental reasons (dynamo
tracing of `synchronize()`; the XPU C++20 op build, fixed in #8803),
identically on the base branch.
- Tiny HF `LlamaForCausalLM` (transformers 5.17), unshifted `labels`,
sp=2, also checked with `dynamic=True` + `attention_mask`:
| | loss |
|---|---|
| single device | 4.840665 |
| #8457 alone | 4.845766 (wrong) |
| this PR | 4.840665 |
### Notes
- Pre-shifted labels (`shift_labels`) remain supported and are still the
simplest path.
## Contribution checklist
**Related work.** This PR is stacked on #8457 and addresses its review
thread linked above. A search for "AutoSP label shift" found no other
PR.
**Sibling paths checked.** AutoSP exists only as a DeepCompile pass, so
there is no eager variant to fix. #8457 rejects ZeRO-2/3, and the
label-shift handling here inherits that restriction.
**Commands and results.** Hardware and software: 4× Intel Arc Pro B70
(XPU, XCCL), Level Zero driver 1.15.39122 (compute runtime
26.27.39122.11), torch 2.14.0.dev20260804+xpu, transformers 5.17.0.
```
pytest -p no:randomly -m sequential tests/unit/v1/compile/test_compile_autosp.py
pytest -p no:randomly tests/unit/v1/compile/test_compile_autosp.py
```
Results are listed under Validation above.
**Numerics, performance, memory.**
- Numerics: the loss and the parameters after two optimizer steps match
single-device training exactly. The HF Llama table above shows the
end-to-end loss.
- Memory: labels are now replicated instead of sharded. That adds `[B,
S]` integer ids per rank, negligible next to the logits.
- Throughput: the CE target is sliced locally, so no new collectives are
added.
**Compatibility matrix.**
| | Validated | Not validated |
|---|---|---|
| ZeRO | stage 0, stage 1 | stage 2/3 (rejected by #8457); offload |
| Precision | fp32 | bf16/fp16 for the label-shift cases
(`TestAutoSPCompile` was excluded, see above) |
| Parallelism | sp=2 with dp=2 (4 ranks); sp=2 with an explicit mesh |
sp>2 for the label-shift cases |
| Accelerator | XPU | CUDA, HPU |
**AI assistance.** This PR was developed with AI assistance (GitHub
Copilot CLI). I reviewed every changed line and ran the tests above.
---------
Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>