feat(rollout): align decode fronts and trim dead zones (#8497)
## Summary
This PR adds an opt-in decode-front alignment path for
continuous-batching rollouts.
When `align_decode_fronts=False`, the existing equal-width padded layout
remains the default. Requests may have different effective lengths in
their attention masks, but their physical decode fronts are not
right-aligned or compacted.
When `align_decode_fronts=True`, the rollout:
* derives effective prompt lengths from attention masks;
* orders requests from longest to shortest;
* right-aligns prefill spans;
* tracks logical positions and physical write positions separately;
* trims dead cache columns with attention metadata and write positions;
* restores the original request order in the returned batch.
The scheduler no longer exposes `admit_if`. Trimming preserves enough
physical cache span for prompts admitted during the current refill.
## Validation
### Test Results
- ✅ 14 tests passed
- ✅ Changed-file pre-commit checks passed
### Device-Backed Validation
- **GPU:** NVIDIA RTX 4000 Ada, 20 GB
- **PyTorch:** 2.9.1+cu128
- **CUDA:** 12.8
- **Transformers:** 4.57.1
- **Model:** Qwen/Qwen2.5-3B-Instruct
- **Precision:** FP32
> The aligned and equal-width outputs matched the single-request eager
oracle token by token.
### Performance
#### Workload 1: 32 mixed left-padded requests
* **Prompt shape:** `[32, 20]`
* **max_new_tokens:** 16
| Case | Time | Throughput |
| :--- | :--- | :--- |
| **no-CB** | 1183.46 ms | 432.63 tok/s |
| **CB equal-width, batch 32** | 1418.99 ms | 360.82 tok/s |
| **aligned CB, batch 32** | 2600.53 ms | 196.88 tok/s |
| **dynamic CB, continuous_batch_size=8** | 4315.39 ms | 118.65 tok/s |
> The optimization is workload-dependent. For uniform short prompts,
aligned CB is slower because per-request prefill and cache-management
overhead dominate. For long-tail prompt lengths, aligned CB can be much
faster by avoiding padded prefill computation.
#### Workload 2: Long-tail workload
* **Composition:** 1 × 512-token prompt and 31 × 16-token prompts
(padded to `[32, 512]`)
| Case | Mean time | Mean throughput |
| :--- | :--- | :--- |
| **no-CB** | 9539.69 ms | 53.67 tok/s |
| **CB equal-width, batch 32** | 9668.58 ms | 52.96 tok/s |
| **aligned CB, batch 32** | 3711.05 ms | 137.97 tok/s |
> The aligned path was **2.57× faster** than no-CB for this long-tail
workload by avoiding repeated prefill computation on padded rows. These
measurements are workload-dependent and are not presented as a general
CB speedup claim.
> The observed 2.57x improvement comes from avoiding padded prefill work
in the long-tail prompt workload, not from cache trimming. The workload
had trim_count=0, so it does not provide real-model trim overhead data.
Trimming is covered by the synthetic regression test, while a real trim
measurement would require a workload with staggered EOS or per-request
token budgets.
### Notes
The tested workloads did not trigger real-model trimming:
```text
trim_count=0
trimmed_columns=0
trim_latency_ms=0.0
trim_bytes_moved=0
```
### Applicability and Limitations
| Workload | Expected behavior | Main reason |
| --- | --- | --- |
| Short prompts with similar lengths | May be slower | Scheduler,
cache-management, and per-request prefill overhead dominate |
| Highly variable prompt lengths | Can improve throughput | Reduces
padded prefill computation |
| Dynamic request arrivals | Can improve GPU utilization | Released
slots can be refilled with pending requests |
| Staggered request completion or EOS retirement | Enables trimming
benefits | Dead cache columns can be reclaimed |
| Fixed batch with similar-length requests | Standard static batching
may be faster | It avoids continuous-batching overhead |
| No early request completion | No real trimming benefit | The trimming
path is not exercised |
*BF16 token-exactness is not claimed; separate BF16 runs showed
numerical token divergence.*
Related to #8462 and follow-up to #8368.
---------
Signed-off-by: nathon-lee <leejianwoo@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>