DeepSpeed
adcabcf9 - [AutoSP] Align cross entropy targets after an in-model causal label shift (#8811)

Commit
2 days ago
[AutoSP] Align cross entropy targets after an in-model causal label shift (#8811) > **Stacked on #8457.** Only the last commit (`6f7c57b9a`) belongs to this PR; please review that commit. I will rebase once #8457 is merged. Addresses https://github.com/deepspeedai/DeepSpeed/pull/8457#discussion_r4243588078 (in-model causal label shift under AutoSP). ### Problem AutoSP shards the labels the same way as `input_ids`. A causal LM that shifts its labels inside the model (Hugging Face `ForCausalLMLoss` pads and shifts, `labels = F.pad(labels, (0, 1), value=-100)[..., 1:]`; Megatron-style code slices `logits[..., :-1, :]` / `labels[..., 1:]`) then shifts each shard on its own. Each rank pairs the last token of its shard with the wrong target, and the loss is silently wrong. #8457 documents pre-shifted labels as the workaround; this PR makes the in-model shift itself correct. ### Approach - Labels are no longer sharded. Any shift the model applies to them runs on the full sequence, so the label subgraph computes exactly what it does without SP. - For each `F.cross_entropy` whose target derives from the labels: 1. Walk from the CE input back to the sequence-sharded logits. Static sequence slices on the way (e.g. `logits[..., :-1, :]`) are removed, because each rank must keep its whole shard of logits. Their offsets give the window `[start, S + stop)` of global positions the CE input covered. 2. Pad the full target with `ignore_index` back to the global positions of that window, then hand the CE this rank's shard `[r*S/sp, (r+1)*S/sp)`. This needs no communication because the labels are replicated. 3. The loss and gradient aggregation from #8457 is unchanged. - A static check makes sure the target lines up position by position with the logits window: the symbolic length of the target's sequence dimension must equal `S + stop - start`. - Unsupported patterns that the pass detects raise instead of silently misaligning (equivalent rewrites such as a `torch.cat` of shifted logits slices may go undetected; the docs say so): - Only casts, `contiguous`, `view`/`reshape`/`flatten` (after any slices) and static sequence slices are allowed between the logits and the CE. - The labels may only flow into CE targets. They cannot be combined with sharded tensors such as `input_ids`, or returned from the module. - Padding uses `narrow` + `full_like` + `cat` instead of `F.pad`/`constant_pad_nd`, which specialize the symbolic sequence length during fake-tensor propagation. ### Validation (8× Intel Arc Pro B70, torch 2.14.0.dev+xpu) - `TestAutoSPCausalLoss` compares the loss and the parameters after two optimizer steps against single-device training on the full sequence, over 4 ranks (sp=2, dp=2). New cases: - `slice` (`logits[..., :-1, :]` / `labels[..., 1:]`) - `pad` (HF style) - each combined with ZeRO-1, `sum`/`mean`, an explicit mesh, and an empty shard - `test_rejects_labels_it_cannot_align` covers a strided logits slice and labels masked by `input_ids`. - `test_compile_autosp.py`: 45 sequential and 20 non-sequential tests pass. `TestAutoSPCompile` and `TestAutoSPEngineBackward` were excluded because they fail on this machine for environmental reasons (dynamo tracing of `synchronize()`; the XPU C++20 op build, fixed in #8803), identically on the base branch. - Tiny HF `LlamaForCausalLM` (transformers 5.17), unshifted `labels`, sp=2, also checked with `dynamic=True` + `attention_mask`: | | loss | |---|---| | single device | 4.840665 | | #8457 alone | 4.845766 (wrong) | | this PR | 4.840665 | ### Notes - Pre-shifted labels (`shift_labels`) remain supported and are still the simplest path. ## Contribution checklist **Related work.** This PR is stacked on #8457 and addresses its review thread linked above. A search for "AutoSP label shift" found no other PR. **Sibling paths checked.** AutoSP exists only as a DeepCompile pass, so there is no eager variant to fix. #8457 rejects ZeRO-2/3, and the label-shift handling here inherits that restriction. **Commands and results.** Hardware and software: 4× Intel Arc Pro B70 (XPU, XCCL), Level Zero driver 1.15.39122 (compute runtime 26.27.39122.11), torch 2.14.0.dev20260804+xpu, transformers 5.17.0. ``` pytest -p no:randomly -m sequential tests/unit/v1/compile/test_compile_autosp.py pytest -p no:randomly tests/unit/v1/compile/test_compile_autosp.py ``` Results are listed under Validation above. **Numerics, performance, memory.** - Numerics: the loss and the parameters after two optimizer steps match single-device training exactly. The HF Llama table above shows the end-to-end loss. - Memory: labels are now replicated instead of sharded. That adds `[B, S]` integer ids per rank, negligible next to the logits. - Throughput: the CE target is sliced locally, so no new collectives are added. **Compatibility matrix.** | | Validated | Not validated | |---|---|---| | ZeRO | stage 0, stage 1 | stage 2/3 (rejected by #8457); offload | | Precision | fp32 | bf16/fp16 for the label-shift cases (`TestAutoSPCompile` was excluded, see above) | | Parallelism | sp=2 with dp=2 (4 ranks); sp=2 with an explicit mesh | sp>2 for the label-shift cases | | Accelerator | XPU | CUDA, HPU | **AI assistance.** This PR was developed with AI assistance (GitHub Copilot CLI). I reviewed every changed line and ran the tests above. --------- Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Author
Parents
Loading