DeepSpeed
f483a279 - [checkpoint] Check aliases on the regions a transfer moves (#9093)

Commit
1 day ago
[checkpoint] Check aliases on the regions a transfer moves (#9093) For open question 1 in #9026, as @LiRunGuo proposed in https://github.com/deepspeedai/DeepSpeed/pull/9074#issuecomment-6860067286. In stage two of a relay, a relay rank holds its band as a view of its own destination. It sends the band to its peers and receives the other bands around it, so no element is both read and written. `_validate_aliases` compares whole buffers, though, so `execute_transfers` refuses the batch. **Change.** - The whole-buffer check stays as the fast path. When no target buffer shares storage with a source or another target, nothing changes and nothing extra runs. - Otherwise, once the schedule is built, `_validate_aliases` collects the regions this process reads (sends and local-copy sources) and writes (receives and local-copy targets). - A local copy whose source and target are the same memory at the same strides moves nothing. It is dropped, and its region counts as read, so another write into it is still refused. - No write may overlap a read or another write. Regions are compared by the bytes they span, so strided regions that interleave without sharing an element are still refused. A band that is a view of its holder's destination is one contiguous run of it, so the relay case doesn't hit that. - The check runs before the agreement, as before, so a refusal on any rank stops every rank before data moves. - `schedule_transfers` doesn't see buffers, so it still counts an in-place copy as a local copy, and its docstring says so. `TransferStats.local_copy_bytes` does not count it. - Conjugate and negative views are refused in `_flat`, on the process that holds one, before the agreement. Otherwise a lazy view of a destination would pass as in place and be dropped (raised by the Codex review, @LiRunGuo and @jinyouzhi). Point-to-point transport does not carry the lazy bit, on master either. With gloo on CPU, a conj shard fails inside the send after the agreement while its peer waits, and a neg-view target is never written. The docs say to pass such views resolved. - Region extents use `address_range`, as `_validate_buffers` does. **Tests** (14 new, in `test_affine_transfer_executor.py`): - `TestB2RelayBandInPlace`, at 2 and 4 processes: every rank's band is a view of its own full copy. Each rank completes its copy exactly, copies nothing locally, and sends and receives N − 1 bands. With the view moved to a peer's rows, every rank refuses before data moves, and the cause names the overlap. - In one process: a band kept in place next to a band copied in. Refusals, each matched to its message, for swapped bands, for a band kept in place that another request writes into, and for two requests writing the same rows. - `TestB2LazyViewsRefused`, at 2 processes: a conj or neg view of the destination, and one on either side of a cross-process segment. Every process refuses, the holder's cause names the view, and nothing is written. On master (gloo, CPU) the destination cases fail on the old message, the cross-process conj cases hang, and the neg-target case completes with nothing written. - On master the 3 acceptance cases are refused. If an in-place copy counted as neither read nor written, the kept-band refusal would fail. On CPU (gloo, up to 4 processes), `test_affine_*.py`, `test_autoep_affine*.py`, `test_zero_affine.py` and `v1/moe/test_autoep_checkpoint.py` pass: 970 passed, 4 skipped (956 on master daf175b1). yapf and flake8 at the pre-commit pins are clean. **How it was found.** By reading `_validate_aliases` against the stage-two shape of Runguo's relay on the affine IR (#9026, Q1): each relay rank's band is a view of its own destination, so the whole-buffer comparison refuses the batch although no element is both read and written. The conj/neg part came out of review, and the cross-process behaviour on master from probing each side and view on CPU. **Production impact (estimate).** Today a relay built on `execute_transfers` cannot run its second stage at all; with this change it runs, and copies nothing for the band it already holds. Batches whose buffers do not share storage see no change and pay nothing extra. Conj/neg views need a complex tensor viewed through `.conj()` or an explicit neg view, so we expect them to be rare in checkpoints; where they did occur across processes, master hung or left the target unwritten with gloo on CPU, and they are now refused up front. --------- Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
Author
Parents
Loading