perf(rollout): select generation and prefill buckets with a tunable cost model (#8786)
## November 11 update
Rebased onto master `b9100c5`, which includes #8390 and #9008; new head
`82ac56e`.
Uses #8939's shared sampling kwargs, token selector and FP32
repetition-penalty helper. Manual graph and continuous-batching paths
retain upstream temperature/top-k/top-p sampling. Adaptive prefill
retains its single-sample greedy scope, including fixed-slot graph
decode. #9004's pre-built 4-D graph mask is preserved.
- **#8390 early stopping:** ordinary generation and Auto's batched
fallback share #8390's generation kwargs, so they stop on natural EOS;
ZeRO-3 keeps the group-aware synchronized stop. Auto remains
single-process.
- **Review fixes:** the graph path rejects `align_decode_fronts=True`
only when Auto is disabled, because Auto always uses aligned bucketed
prefill. The accelerator graph test covers both flag values. The
documentation states that Auto takes precedence over
`align_decode_fronts` on the graph path; cache trimming is still
rejected.
- **#9008 lazy KV layers:** the bucketed prefill copy initializes
uninitialized cache layers through the shared
`_continuous_cache_layer()` helper, like the equal-width path.
- **Codex P2 (preparer findings):** after a custom
`_prepare_generation_config()` runs, Auto rejects the prepared config if
it changes any request override (token budget, EOS, PAD or greedy
settings). Otherwise the batched route would honor the change inside
`generate()` while the CB routes keep the request values. A preparer
that replaces or edits the attention mask is rejected for the same
reason.
- **Codex P2 (mask lengths):** effective prompt lengths use
`count_nonzero`, so BF16/FP16 masks longer than their exact-integer
range no longer round the bucket width.
- **Codex P1 (test reliability):** the route-planning profile test uses
a frozen clock advanced by the model's generation-config preparer,
instead of wall-clock bounds or mocked rollout internals.
- The decode cost still uses the configured maximum generation budget.
Natural EOS can shorten decode, so the extra-CB estimate can be too
large; the documentation states this assumption.
Validation on an 8 × H200 node, PyTorch 2.9.1+cu128, CUDA 12.8,
Transformers 4.57.1: **232 passed, 3 skipped, 2 deselected** across the
runtime rollout, scheduler, interface, static-cache and v1 accelerator
suites, including #8390's 4-rank subgroup and 2-GPU ZeRO-3
synchronized-stop tests. Skips require the newer cache query-offset API;
deselections are the native shared-prefill kernel and optional
pretrained-model test. Pre-commit passed on all changed files. A small
natural-EOS check (Qwen2.5-3B-Instruct, four short questions) ended
every response at EOS, with Auto matching ordinary generation. This
update used AI assistance.
The performance tables below retain their original revision attribution;
large-model throughput and peak memory were not remeasured for this
rebase.
## Summary
Closes #8785. Related tracking:
[#8197](https://github.com/deepspeedai/DeepSpeed/issues/8197),
[#8462](https://github.com/deepspeedai/DeepSpeed/issues/8462).
**Performance follow-up to
[#8497](https://github.com/deepspeedai/DeepSpeed/pull/8497).** Aligned
CB avoids padding on long-tail prompts, but one prefill Forward per
request is expensive for short prompts. A padding-ratio threshold cannot
distinguish **128/4** from **512/16**, despite identical ratios and
different fastest baselines.
This change adds **one opt-in automatic policy** that compares ordinary
no-CB generation with aligned CB using **DP-selected prefill buckets**.
For 1×512 + 31×16, the selected buckets are `[1,512]` and `[31,16]`,
reducing prefill Forwards from 32 to 2.
**Auto is disabled by default.** Existing manual modes and positional
configuration arguments are preserved.
```python
cfg = HybridEngineRolloutConfig(adaptive_prefill=True)
rollout = HybridEngineRollout(engine, tokenizer, cfg)
sampling = SamplingConfig(max_new_tokens=16, temperature=0, continuous_batch_size=32)
```
## Selection and configuration
A length-sorted **O(n²)-time / O(n)-space DP** minimizes the additive
estimated cost of contiguous buckets. It explores one batch, singleton
batches and intermediate partitions without running the GPU for
candidate evaluation. Each bucket uses ordinary left padding and a 2D
attention mask; **no varlen kernel is needed**. Model geometry is read
once per plan.
```text
C(B,L) = fixed + token × B×L + attention × estimated_attention_GFLOPs
+ kv × estimated_KV_MiB
attention_GFLOPs = 4 × B × L² × layers × query_heads × head_dim / 1e9
KV_MiB = 2 × B × L × layers × KV_heads × head_dim × element_bytes / 2^20
```
| Parameter | Default | Meaning |
| --- | ---: | --- |
| `prefill_fixed_cost_ms` | 25.0 | Fixed milliseconds per prefill
Forward |
| `prefill_token_cost_ms` | 0.1 | Milliseconds per padded token
position; projection/MLP term |
| `prefill_attention_cost_ms` | 0.01 | Milliseconds per estimated
attention GFLOP |
| `prefill_kv_cost_ms` | 0.04 | Milliseconds per MiB of KV copied into
the CB cache |
| `continuous_decode_cost_ms` | 10.0 | Additional CB milliseconds per
remaining Decode step |
| `prefill_max_tokens` | 65536 | Maximum padded token positions per
prefill Forward; `None` removes this limit |
All cost coefficients must be finite and non-negative. **These are
starting estimates, not universal GPU constants**; calibrate them for
the model, device, precision and attention backend. Head dimension,
query/KV heads, layers and parameter element size are read
automatically. The DP is optimal for the configured estimates, not
guaranteed globally optimal measured latency.
Ordinary generation is eligible when every row fits
`continuous_batch_size`, the physical input fits the prefill/context
limits, and neither explicit static capacity nor trimming is requested.
Its estimated cost omits CB cache-copy traffic; CB adds
`continuous_decode_cost_ms × (max_new_tokens - 1)`. The first output
token comes from Prefill.
For CB, **every admission/refill replans the newly admitted requests**.
Effective KV spans are copied into the existing right-aligned cache;
surviving rows retain their layout. The policy does not change caller
configuration or active-row bounds. Disabling Auto restores manual
selection. Model/OOM errors propagate without trying another path.
## Correctness
- Apply repetition penalty from the first token using **original padded
history**, even when bucket model inputs are trimmed.
- Preserve generated PAD-valued tokens through the first EOS, excluding
structural padding after termination.
- Include route planning and result normalization in end-to-end
profiling; count **productive output tokens** and retain
active/configured capacity fields.
- Resolve HF's effective generation configuration before rejecting
unsupported processors/output modes, including legacy model-config
refresh; preserve explicit generation-config overrides. Auto requires
single-process greedy rollout, one sample per prompt and cache-class
support. With graph capture, Auto uses eager bucketed prefill and
fixed-slot graph decode; cache trimming and shared prefill remain
unsupported.
## Experiments
Related work: [TurboTransformers, §5 (PPoPP
2021)](https://arxiv.org/html/2010.05680) already uses length sorting,
profiled batch costs and an O(n²) DP to partition variable-length
inference requests. [DynaPipe, §4](https://arxiv.org/html/2311.10418)
applies cost-based DP partitioning to training micro-batches. This PR
applies that established scheduling approach to HybridEngine rollout,
with ordinary-generation comparison, refill planning and existing
aligned-cache semantics; it makes no claim of a new DP algorithm.
**Single GPU; Qwen/Qwen2.5-3B-Instruct; FP32 / SDPA; PyTorch
2.9.1+cu128; Transformers 4.57.1.** Each candidate receives **one warmup
and five interleaved timed runs**, 32 requests and 16 output tokens per
request. CB capacity is 32; trimming is disabled, cache capacity unset,
TF32 disabled. The default cost coefficients above are unchanged across
workloads.
| Workload | Effective prompt lengths | Reason |
| --- | --- | --- |
| Short, similar | 20..13 repeated four times | Normal workload shape
from #8497 |
| Long tail | 1×512 + 31×16 | Long-tail workload from #8497 |
| Short tail | 1×20 + 31×1 | Counterexample to ratio-only routing |
| Moderate tail | 1×128 + 31×4 | Same padding ratio as 512/16, different
best legacy path |
| Spread | 16, 32, …, 512 | General distribution with intermediate
buckets |
Inputs repeat non-special token ID 14990. #8497 did not publish complete
short-prompt sequences, so that case reconstructs its documented shape.
Every timed response matches independent single-request eager
generation; each workload produces **512 effective output tokens**.
GPU-synchronized wall-clock timing includes the full `generate()` call,
including Auto planning and normalization. Model loading and oracle work
are outside timing; profiling is disabled during timed runs. All
reported E2E values are **arithmetic means of five runs**.
| Workload | No-CB (ms) | Equal-width CB (ms) | Aligned CB (ms) | Auto
(ms) | Auto / best | Auto choice |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| Short, similar | **505.54** | 673.66 | 1549.74 | **501.77** | 0.9925×
| No-CB |
| 1×512 + 31×16 | 2458.46 | 2596.38 | **1682.10** | **840.06** | 0.4994×
| 2 buckets |
| 1×20 + 31×1 | **501.74** | 667.57 | 1331.82 | **497.57** | 0.9917× |
No-CB |
| 1×128 + 31×4 | **897.17** | 1053.26 | 1468.58 | **634.59** | 0.7073× |
2 buckets |
| 16, 32, …, 512 | 2458.13 | 2595.33 | **2337.75** | **1984.86** |
0.8490× | 6 buckets |
**All five workloads meet the 5% acceptance tolerance.** Auto chooses
ordinary generation for both short cases; their small timing differences
are not claimed as a kernel speedup. Relative to the fastest legacy
baseline, mean latency falls by **50.06%** for 1×512 + 31×16, **29.27%**
for 1×128 + 31×4, **15.10%** for 16, 32, …, 512. These improvements come
from fewer prefill Forwards and reduced padded work.
<details>
<summary>Throughput and peak allocated memory for every path</summary>
Throughput is 512 divided by mean latency. Memory is peak PyTorch
allocated memory during timed calls, including model weights.
| Workload | Path | Output tok/s | Peak allocated memory (MiB) |
| --- | --- | ---: | ---: |
| Short, similar | No-CB | 1012.77 | 11988.96 |
| Short, similar | Equal-width CB | 760.03 | 12036.75 |
| Short, similar | Aligned CB | 330.38 | 11928.77 |
| Short, similar | **Auto** | 1020.40 | 11988.96 |
| 1×512 + 31×16 | No-CB | 208.26 | 15558.69 |
| 1×512 + 31×16 | Equal-width CB | 197.20 | 16747.20 |
| 1×512 + 31×16 | Aligned CB | 304.38 | 13278.51 |
| 1×512 + 31×16 | **Auto** | 609.48 | 13278.49 |
| 1×20 + 31×1 | No-CB | 1020.46 | 11994.64 |
| 1×20 + 31×1 | Equal-width CB | 766.96 | 12033.50 |
| 1×20 + 31×1 | Aligned CB | 384.44 | 11925.07 |
| 1×20 + 31×1 | **Auto** | 1029.00 | 11994.64 |
| 1×128 + 31×4 | No-CB | 570.68 | 12742.62 |
| 1×128 + 31×4 | Equal-width CB | 486.11 | 13066.75 |
| 1×128 + 31×4 | Aligned CB | 348.64 | 12221.97 |
| 1×128 + 31×4 | **Auto** | 806.82 | 12221.95 |
| 16, 32, …, 512 | No-CB | 208.29 | 15558.69 |
| 16, 32, …, 512 | Equal-width CB | 197.28 | 16747.20 |
| 16, 32, …, 512 | Aligned CB | 219.01 | 13278.56 |
| 16, 32, …, 512 | **Auto** | 257.95 | 13682.33 |
</details>
<details>
<summary>Separate profiling breakdown</summary>
One additional profiling call per candidate, outside acceptance timing.
These diagnostics are not averages. Cache management is part of the
generation overhead, not an additional independent total.
| Workload | Path | Prefill (ms) | Decode (ms) | Cache management (ms) |
Prefill / Decode Forwards |
| --- | --- | ---: | ---: | ---: | ---: |
| Short, similar | No-CB | 91.01 | 401.30 | N/A | 1 / 15 |
| Short, similar | Equal-width CB | 91.99 | 499.53 | 26.80 | 1 / 15 |
| Short, similar | Aligned CB | 955.52 | 500.30 | 35.25 | 32 / 15 |
| Short, similar | **Auto** | 90.74 | 396.90 | N/A | 1 / 15 |
| 1×512 + 31×16 | No-CB | 1918.22 | 531.89 | N/A | 1 / 15 |
| 1×512 + 31×16 | Equal-width CB | 1919.76 | 595.14 | 27.09 | 1 / 15 |
| 1×512 + 31×16 | Aligned CB | 981.04 | 594.52 | 35.02 | 32 / 15 |
| 1×512 + 31×16 | **Auto** | 153.66 | 592.66 | 31.43 | 2 / 15 |
| 1×20 + 31×1 | No-CB | 91.06 | 402.45 | N/A | 1 / 15 |
| 1×20 + 31×1 | Equal-width CB | 91.84 | 494.84 | 26.88 | 1 / 15 |
| 1×20 + 31×1 | Aligned CB | 743.30 | 491.93 | 34.48 | 32 / 15 |
| 1×20 + 31×1 | **Auto** | 90.62 | 394.78 | N/A | 1 / 15 |
| 1×128 + 31×4 | No-CB | 490.08 | 399.13 | N/A | 1 / 15 |
| 1×128 + 31×4 | Equal-width CB | 490.85 | 482.08 | 27.09 | 1 / 15 |
| 1×128 + 31×4 | Aligned CB | 889.01 | 483.21 | 34.83 | 32 / 15 |
| 1×128 + 31×4 | **Auto** | 63.80 | 485.00 | 31.43 | 2 / 15 |
| 16, 32, …, 512 | No-CB | 1918.03 | 531.64 | N/A | 1 / 15 |
| 16, 32, …, 512 | Equal-width CB | 1918.95 | 594.68 | 27.32 | 1 / 15 |
| 16, 32, …, 512 | Aligned CB | 1648.16 | 593.25 | 35.91 | 32 / 15 |
| 16, 32, …, 512 | **Auto** | 1301.59 | 593.31 | 32.49 | 6 / 15 |
The 128/4 and 512/16 workloads reduce prefill Forwards from 32 on
aligned CB to **2 on Auto**. Spread uses **6 buckets**. All CB paths in
these performance runs have zero trims and zero moved bytes; ordinary
generation has no static-cache statistics. This makes no new
trimming-speedup claim.
</details>
## Validation and limits
A separate microbenchmark measures **planner overhead**, without model
Prefill: for 128/256 requests of 65,536 tokens, CPU DP takes **0.48/1.82
ms** on average with the default per-bucket token limit. It still checks
all 8,256/32,896 candidate segments; over-limit segments skip cost
evaluation. For 32 requests, DP takes 0.089 ms at length512 and 0.040 ms
at length65,536 (0.078 ms with no token limit). GPU mask
validation/length extraction for 32×65,536 takes 0.83 ms separately.
Measurements use ten repeated timing samples after warmup. **These are
not 64k model Prefill timings or the complete Auto-path overhead.**
**The original validation passed 105 related rollout/cache tests**,
including independent exhaustive partition comparisons for DP,
device-backed tiny-Qwen tests for grouped prefill and refill, both cache
APIs, effective generation-config/legacy refresh, repetition history,
EOS/PAD masks, capacity bounds and external-wall-time checks for
profiling. Existing tests cover retirement, compaction and trimming.
Independent mixed-bucket checks also exercised live-survivor refill and
trimming on/off against padded per-request HF generation.
One pre-existing native shared-prefill extension test is excluded
because the host has no CUDA compiler; **all added tests ran**.
Changed-file pre-commit and `git diff --check` pass. Repeated local
review ended with **no actionable findings**.
```sh
python -m pytest -q \
tests/unit/runtime/rollout/test_hybrid_engine_rollout.py \
tests/unit/runtime/rollout/test_continuous_batching.py \
tests/unit/runtime/rollout/test_rollout_interface.py \
tests/unit/utils/test_static_cache.py \
-k 'not native_repeat_kv_cache_fp16_reverse_copy'
```
The prefill token limit does **not** bound allocated decode KV memory. A
single prompt over that limit raises an explicit error; there is no
chunked prefill. **64k prompts require a model context limit covering
prompt plus output budget, sufficient KV capacity and temporary
memory.** The measured checkpoint is configured for 32k.
Planner/position-limit boundaries are tested; real 64k inference
performance is not claimed.
Performance evidence uses unsharded HF models behind
`HybridEngineRollout`; the additional calibrated and real-question
checks are listed below. Other model/device configurations, FP16, native
kernel injection, ZeRO/TP training integration and production
arrival/EOS distributions remain unmeasured. Controlled performance runs
do not exercise staggered EOS or trimming; those have correctness
coverage.
No new dependency, kernel, online learner or trimming redesign. Related
foundation: [#8368](https://github.com/deepspeedai/DeepSpeed/pull/8368);
profiling: [#8494](https://github.com/deepspeedai/DeepSpeed/pull/8494).
## Reviewer follow-up
[Commit
65a9115](https://github.com/deepspeedai/DeepSpeed/commit/65a9115cdbee1a071d13236615067f404ba65b9e)
propagates the resolved generation configuration through continuous
Prefill, Decode and refill, and omits `output_logits` from legacy manual
generation. **27 targeted tests pass**, including six new cases covering
both Auto routes, both cache APIs and manual equal-width/aligned
compatibility. The new tests reproduced four failures before the fix.
The compatibility test uses a strict generation wrapper that rejects the
unsupported keyword.
Calibration and evaluation tooling moved to [DeepSpeedExamples
#1018](https://github.com/deepspeedai/DeepSpeedExamples/pull/1018), with
its fitting regression test and usage documentation. **Merge the
Examples companion before this PR so the documentation link is
available.** The Examples companion includes MHA-head fallback and
validates a source fingerprint covering rollout, planning and
static-cache code. A seeded tiny-model GPU smoke check completed
calibration and all four saved-config sweep workloads with matching
output IDs on every path. Two independent local reviews returned no
findings.
The calibrated sweep from the preceding routing-overhead fix is separate
from the original 16-token experiments above. It uses **two EOS-disabled
output tokens**, SDPA and one GPU, with independent uniform calibration
shapes and one unchanged coefficient set per configuration:
| Configuration | 128×16 | 1×128 + 31×4 | 16..512 | 1×512 + 31×16 |
Worst regression |
| --- | ---: | ---: | ---: | ---: | ---: |
| Tiny Qwen2 / FP32 | 1.0220× | 0.9685× | 0.9539× | 0.9647× | +2.205% |
| Qwen2.5-3B / FP32 | 1.0005× | 0.2345× | 0.7808× | 0.2265× | +0.054% |
| Qwen3-32B / BF16 | 1.0027× | 0.3881× | 0.7189× | 0.1811× | +0.273% |
Values are **Auto E2E / fastest manual baseline E2E**. The two larger
models select no-CB for 128×16 and bucketed CB for the other three
cases; the tiny model selects no-CB throughout. Auto matches ordinary
output IDs in all twelve cases. Target-model means use three interleaved
repetitions after warmup; the tiny test uses ten. [Full sweep details
and calibration
limits](https://github.com/deepspeedai/DeepSpeed/pull/8786#issuecomment-6096524658).
A separate [Qwen3-32B real-question
check](https://github.com/deepspeedai/DeepSpeed/pull/8786#issuecomment-6094792830)
selected no-CB on both workloads: MATH-500 averaged **31.452 s / 31.393
s** for ordinary/Auto over three runs; ARC-AGI-2 measured **237.560 s /
237.888 s** in one paired run. All eight responses matched. These checks
cover complete generation calls, excluding tokenization, loading,
rewards and training.
A real tiny-OPT calibration additionally exposed a missing HF cache
capability flag. [Commit
ecacbce](https://github.com/deepspeedai/DeepSpeed/commit/ecacbce12c3f6749e4d80f8718a285618e6fd7e9)
adds `is_compileable=True` to the static-cache container, matching its
existing static layers. **Six targeted GPU checks pass**, including new
OPT manual/Auto comparisons with padded prompts and refill against the
HF oracle, plus existing Qwen bucket/cache-path checks. The three
Examples regression tests cover fitting, complete OPT calibration and
saved-source compatibility. Both local reviews found no issues in this
follow-up.
## Final correctness and performance acceptance
[Commit
d4def3d](https://github.com/deepspeedai/DeepSpeed/commit/d4def3db036f721a375d8582cc762d04262712b6)
includes the final fixes: non-default repetition penalties are
calculated in FP32 like HF, and unknown public generation switches are
rejected before routing. The default-penalty path retains its early
return. **Five focused BF16 checks and six unknown-setting CPU cases
pass.** The Examples CLI now requires complete metadata for saved
calibration files. Both independent correctness and performance reviews
returned no findings after these fixes.
A new calibration and sweep were run against measured commit `d4def3d`.
**Qwen3-32B, BF16, SDPA, one GPU; two EOS-disabled output tokens; one
warmup plus three interleaved timed runs per path.** Calibration and
held-out evaluation use separate shapes. The calibration file, sweep and
that measured revision's rollout/planner/static-cache sources have the
same combined SHA256:
`f9d79a43d2e6c0e3fef164ef11e81a26c643820beb1a0d3f1a86f41a46dbdd0c`.
| Workload | No-CB ms | Equal-width CB ms | Aligned CB ms | Auto ms |
Auto / best | Auto path |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| 128×16 | **269.83** | 439.91 | 5717.89 | 270.03 | 1.000743× | No-CB |
| 1×128 + 31×4 | **496.15** | 540.76 | 1501.57 | **191.96** | 0.386886×
| 2 buckets |
| 16,32,…,512 | 1962.87 | 2009.18 | **1864.55** | **1336.57** |
0.716828× | 6 buckets |
| 1×512 + 31×16 | 1962.56 | 2005.33 | **1575.08** | **288.31** |
0.183045× | 2 buckets |
**All four pass the 5% acceptance tolerance; worst measured regression
is +0.07428%. Auto output IDs match ordinary generation in every case**,
and both routes are exercised. Means and ratios were independently
recomputed from raw samples. The legacy aligned spread baseline retains
its previously recorded second-token difference on one row; Auto matches
no-CB there. Fixed output length prevents this numerical-sensitive
baseline difference from changing the amount of timed work.
The fitted coefficients are fixed=28.791679 ms, token=0.093362064 ms,
attention=0.010090454 ms/GFLOP, KV=0.056288540 ms/MiB and additional
decode=17.144541 ms/step. The earlier tables remain historical
measurements on their original versions. This final check does not
establish long-context, natural-EOS, 256-request E2E or dynamic-arrival
performance. Auto remains disabled by default.
[Commit
a0588b7](https://github.com/deepspeedai/DeepSpeed/commit/a0588b7dea72f2d23e8c90ca1daf51675633b4db)
then fixes manual aligned prefill to retain the original padded
repetition history. Its focused GPU regression reproduced the mismatch
and now passes. Auto already passed this same mapping, and the measured
default-penalty paths do not read it; independent correctness and
performance delta reviews returned no findings. The table remains
attributed to `d4def3d`, with no timings or hashes relabeled as a
new-HEAD measurement. The Examples benchmark explicitly requires
Transformers >=4.56; the library's existing compatibility scope is
unchanged.
## Review invitation
cc @nathon-lee @delock — this follows your continuous-batching,
profiling, and aligned-cache work in #8368, #8494, and #8497. Could you
review the automatic route selection, bucket/refill cache semantics, and
cost defaults?
## November refresh
Rebased onto `b62afa8` and squashed to one commit; the PR is now
mergeable. The refresh keeps #8732 graph state, #8839 last-position
prefill logits, #8933 accelerator-neutral graph calls and #8852
`DeepSpeedStaticCache.get_query_offset()`. #8980's explicit accelerator
opt-in is respected: a backend without `is_graph_capture_supported()`
falls back to eager before any routing decision.
*Superseded by 8a9641b:* Auto now combines eager bucketed prefill with
fixed-slot graph decode (see the November 11 update). SegKI (#8602) is
not part of this PR.
Validation, 8×H200-class node with one GPU in use (NVIDIA H20Z, PyTorch
2.9.1+cu128, CUDA 12.8, Transformers 4.57.1):
| Check | Result |
| --- | --- |
| rollout/cache suite | 159 passed, 3 skipped, 1 deselected |
| Skips | Three Transformers 5.20-only query-offset cases added by #8852
|
| Deselected | Native shared-prefill kernel test; the test host has no
CUDA compiler |
| Changed-file pre-commit | Passed |
Graph replay, resolved-config propagation on both routes (FP32/BF16), DP
bucket/refill oracles, Auto/Graph rejection (since replaced by joint
routing) and manual graph dispatch all pass. No new performance numbers
are claimed: the tables above were measured before #8839/#8732, so
refreshed baselines need a separate calibration run if wanted.
AI assistance was used for the rebase, conflict resolution, test
execution and drafting of this description; the submitting human
reviewed the diff and the results above.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: SuperGoodGame <985236470@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>