DeepSpeed
04e0be2b - Fix Ulysses SP registration for string model paths and gloo collectives (#8702)

Commit
48 days ago
Fix Ulysses SP registration for string model paths and gloo collectives (#8702) ## Description Two latent bugs in Ulysses SP, both backend-independent (neither is CPU-specific — the CPU/gloo multi-rank run in #8381 is simply what surfaced them), plus test adaptations so the file can also run on non-CUDA accelerators. ### 1. `register_with_transformers` rejected every string model path `deepspeed/runtime/sequence_parallel/ulysses_sp.py` compared `core_attn_implementation` against the config's `_attn_implementation` even when the caller passed a **model path string**: ```python hf_model_config = AutoConfig.from_pretrained(model_name_or_path) # string branch model_attn_implementation = getattr(hf_model_config, "_attn_implementation", None) if model_attn_implementation is not None and model_attn_implementation != core_attn_implementation: raise ValueError(...) ``` In transformers, `_attn_implementation` is a **property resolved at model load time** — `from_pretrained` resolves which implementation the runtime can actually use and writes it back to the config. A bare `AutoConfig` never goes through that resolution, so the property returns its fallback value for "unresolved": ``` AutoConfig.from_pretrained(path)._attn_implementation -> eager (fallback, not a choice) AutoModelForCausalLM.from_pretrained(path).config._attn... -> sdpa (resolved) ``` So every string-path caller asking for `sdpa`/`flex_attention`/FA2 was rejected with a spurious mismatch against `'eager'`. The check is only meaningful for a config that was resolved, i.e. when the caller handed us a constructed model (or a PEFT wrapper exposing `.config`); scope it to that case. The comparison expression itself is unchanged, and `TestUlyssesSPHFAttnImplMismatch` (which passes a model object) still raises as intended. ### 2. `UlyssesSPDataLoaderAdapter` used a 0-dim tensor in `all_gather` ```python seqlen = torch.tensor(n, ...) # 0-dim seqlens = [torch.zeros(1, ...) for _ in range(sp_world_size)] # shape (1,) dist.all_gather(seqlens, seqlen, group=self.sp_group) ``` `c10d.all_gather` requires each receive tensor to have the same size as the send tensor. NCCL moves flattened bytes and tolerates the mismatch; gloo validates shapes and rejects it (`ProcessGroupGloo::allgather: invalid tensor size at index 0 (expected (), got (1))`). Send a 1-element tensor to match. ### 3. Tests - `TestUlyssesSPHFFlexAttention` needs the CUDA flex kernel -> skip on non-CUDA accelerators (same guard idiom as `tests/unit/util.py`, the nvme/triton CUDA-only tests). - `TestUlyssesSPHFPEFT`: the mock attached a **bare** `AutoConfig`, whose `_attn_implementation` is the unresolved `'eager'` — so this test failed on every backend, not just CPU. Give the mock the load-resolved value a real PEFT wrapper carries (the sibling `TestUlyssesSPHFAttnImplMismatch` in the same file already sets it the same way). - `TestUlyssesSPHFDisableInEval` pinned its tensors/models to `cuda:<rank>` although it runs sdpa and nothing in it is CUDA-specific. Use `get_accelerator().current_device_name()` like the rest of the file (on CUDA that is exactly `cuda:<rank>`, since the test harness sets the device per rank in `tests/unit/common.py`), so the test runs on any backend instead of erroring. ## Validation Hardware: 20-core x86_64 CPU without AVX512-FP16, gloo backend, 2 ranks (`LOCAL_SIZE=2`), transformers 4.51.3, torch 2.13. - `tests/unit/ulysses_alst/test_ulysses_sp_hf.py`: **5 passed, 2 skipped** (flex skipped by the new guard). The sdpa classes pass end to end including their numerical-parity assertions; before the fix they died at registration. - `TestUlyssesSPHFDisableInEval` now actually **executes** on CPU rather than being skipped. - `TestUlyssesSPHFAttnImplMismatch` still raises the expected `ValueError` (the mismatch check is preserved where a resolved config exists). - Full multi-rank CPU CI in #8381: this file went from 5 failures to 0, with the overall job green. (`TestUlyssesSPHFHubKernel` is not exercised by the version pin above — it needs the `lazy_import_flash_attention` API from a newer transformers; it passes in CI, which installs transformers from git main.) Sibling PRs from the same series: #8397, #8398, #8399, #8407, #8409, #8559, #8648, #8684. --------- Signed-off-by: Guokai Ma <guokai.ma@intel.com> Co-authored-by: pengdurice <pengduhit@gmail.com>
Author
Parents
Loading