DeepSpeed
1befaa5c - [AffineIR] Check affine holders and source regions without pairwise comparisons (#9011)

Commit
3 days ago
[AffineIR] Check affine holders and source regions without pairwise comparisons (#9011) **Stacked on #9010** (only the last two commits are new here; I will rebase after #9010 merges). `_validate_holders` compared every piece with every other piece, `_sources` compared every source region with every other region, and `_shards_tile` compared each rank's pieces pairwise by shard address. Planning time therefore grew quadratically with the number of pieces, and a map with 1024 pieces already exceeded the default `max_pair_checks`. When no rank repeats a box and distinct boxes do not overlap, a rank covers a box exactly when it holds that very box, so the holders of each box can be read from a table. This PR adds `boxes_disjoint` to `affine_geometry`. It sweeps along the axis whose box origins vary most and compares each box only with the boxes still open where it starts. The same sweep proves that a rank's pieces tile its shard when they are disjoint by address and fill it (suggested by 0z5a in review). Maps that fail the fast checks fall back to the original pairwise walk, which names the offending piece, so error messages are unchanged, and maps that pass no longer hit the per-rank `len(held)**2` budget. The sweep counts its comparisons against `max_pair_checks` too (suggested by Achyuthan-S in review). A disjoint layout can keep every box open for the whole sweep: for example, h full-height one-column boxes plus 2h one-element boxes beside them. When the sweep runs out of budget it reports "not shown disjoint". It compares at most n·(n−1)/2 pairs, so running out implies n² > `max_pair_checks`, and the pairwise fallback refuses at once. With 4,095 such pieces, planning went from 19.2 s to a budget refusal in 0.6 s. Planning time (8 ranks with interleaved rows, re-sharded to 8 row shards, 64 columns; ms, single CPU process): | pieces | before | holders/sources only | + shard addresses (this PR) | |---|---|---|---| | 256 | 211 | 15 | 7 | | 512 | 827 | 46 | 15 | | 1024 | 3278 (refused by default budget) | 156 | 33 | | 2048 | 12926 | 569 | 76 | | 4096 | 50469 | 2227 | 235 | Tests: - Randomized oracle over 200 seeds: plans and errors match the pairwise walk, including replicas split at different points, overlapping pieces and wrong holder lists. - `boxes_disjoint` is checked against an all-pairs reference over 100 seeds. - Mutation check: the tests catch dropped intersections and wrong holder sets. - A rank holding 1024 pieces plans under the default `max_pair_checks` (refused before this change). - The always-open layout above plans correctly under the default budget and is refused with `max_pair_checks` just below its sweep cost. Without the sweep budget, that test plans instead of refusing. - `tests/unit/checkpoint -k affine` passes on 8× NVIDIA L20 (645 passed). At 12663c3 on CPU (`DS_ACCELERATOR=cpu`, 4 ranks): 642 passed, 5 skipped. The remaining near-quadratic steps, the source × target join and `_covers_union`, overlap with #8936 and will follow after it lands; segment merging in `_coalesce` has its own `max_merge_checks` budget and will follow separately. --------- Signed-off-by: Jin, Youzhi <youzhi.jin@intel.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Ma, Guokai <guokai.ma@gmail.com>
Author
Parents
Loading