DeepSpeed
a77aeb67 - feat(rollout): align decode fronts and trim dead zones (#8497)

Commit
47 days ago
feat(rollout): align decode fronts and trim dead zones (#8497) ## Summary This PR adds an opt-in decode-front alignment path for continuous-batching rollouts. When `align_decode_fronts=False`, the existing equal-width padded layout remains the default. Requests may have different effective lengths in their attention masks, but their physical decode fronts are not right-aligned or compacted. When `align_decode_fronts=True`, the rollout: * derives effective prompt lengths from attention masks; * orders requests from longest to shortest; * right-aligns prefill spans; * tracks logical positions and physical write positions separately; * trims dead cache columns with attention metadata and write positions; * restores the original request order in the returned batch. The scheduler no longer exposes `admit_if`. Trimming preserves enough physical cache span for prompts admitted during the current refill. ## Validation ### Test Results - ✅ 14 tests passed - ✅ Changed-file pre-commit checks passed ### Device-Backed Validation - **GPU:** NVIDIA RTX 4000 Ada, 20 GB - **PyTorch:** 2.9.1+cu128 - **CUDA:** 12.8 - **Transformers:** 4.57.1 - **Model:** Qwen/Qwen2.5-3B-Instruct - **Precision:** FP32 > The aligned and equal-width outputs matched the single-request eager oracle token by token. ### Performance #### Workload 1: 32 mixed left-padded requests * **Prompt shape:** `[32, 20]` * **max_new_tokens:** 16 | Case | Time | Throughput | | :--- | :--- | :--- | | **no-CB** | 1183.46 ms | 432.63 tok/s | | **CB equal-width, batch 32** | 1418.99 ms | 360.82 tok/s | | **aligned CB, batch 32** | 2600.53 ms | 196.88 tok/s | | **dynamic CB, continuous_batch_size=8** | 4315.39 ms | 118.65 tok/s | > The optimization is workload-dependent. For uniform short prompts, aligned CB is slower because per-request prefill and cache-management overhead dominate. For long-tail prompt lengths, aligned CB can be much faster by avoiding padded prefill computation. #### Workload 2: Long-tail workload * **Composition:** 1 × 512-token prompt and 31 × 16-token prompts (padded to `[32, 512]`) | Case | Mean time | Mean throughput | | :--- | :--- | :--- | | **no-CB** | 9539.69 ms | 53.67 tok/s | | **CB equal-width, batch 32** | 9668.58 ms | 52.96 tok/s | | **aligned CB, batch 32** | 3711.05 ms | 137.97 tok/s | > The aligned path was **2.57× faster** than no-CB for this long-tail workload by avoiding repeated prefill computation on padded rows. These measurements are workload-dependent and are not presented as a general CB speedup claim. > The observed 2.57x improvement comes from avoiding padded prefill work in the long-tail prompt workload, not from cache trimming. The workload had trim_count=0, so it does not provide real-model trim overhead data. Trimming is covered by the synthetic regression test, while a real trim measurement would require a workload with staggered EOS or per-request token budgets. ### Notes The tested workloads did not trigger real-model trimming: ```text trim_count=0 trimmed_columns=0 trim_latency_ms=0.0 trim_bytes_moved=0 ``` ### Applicability and Limitations | Workload | Expected behavior | Main reason | | --- | --- | --- | | Short prompts with similar lengths | May be slower | Scheduler, cache-management, and per-request prefill overhead dominate | | Highly variable prompt lengths | Can improve throughput | Reduces padded prefill computation | | Dynamic request arrivals | Can improve GPU utilization | Released slots can be refilled with pending requests | | Staggered request completion or EOS retirement | Enables trimming benefits | Dead cache columns can be reclaimed | | Fixed batch with similar-length requests | Standard static batching may be faster | It avoids continuous-batching overhead | | No early request completion | No real trimming benefit | The trimming path is not exercised | *BF16 token-exactness is not claimed; separate BF16 runs showed numerical token divergence.* Related to #8462 and follow-up to #8368. --------- Signed-off-by: nathon-lee <leejianwoo@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading