vllm
[Async][Spec Decoding] Zero-bubble async scheduling + spec decoding
#32951

Merged

[Async][Spec Decoding] Zero-bubble async scheduling + spec decoding #32951

MatthewBonanni merged 112 commits into vllm-project:main from MatthewBonanni:async-eagle-mod

[Perf][Async] Implement zero-bubble async speculative decoding

b1416690

skip compute_slot_mapping for async_spec_zero_bubble_mode

679054f0

remove seq_lens_cpu & num_computed_tokens_cpu for async_spec_zero_bub…

527b55d7

Get rid of async_spec_zero_bubble_mode option

8c2ce4c0

Fully async version

1554ab7a

Increase max_concurrent_batches

315e7e60

Handle reordering

15d8ee9c

mergify added speculative-decoding

mergify added v1

mergify added needs-rebase

Merge branch 'main' into async-eagle-mod

abb8ba7e

gemini-code-assist commented on 2026-01-23

mergify removed needs-rebase

Fix

dd8b7c33

Merge branch 'main' into async-eagle-mod

15730150

MatthewBonanni changed the title ~~[WIP][Async Scheduling][Spec Decoding] Zero-bubble async scheduling + spec decoding~~ [WIP][Async][Spec Decoding] Zero-bubble async scheduling + spec decoding 192 days ago

Cleanup

317b4528

Cleanup

e7e39ce7

Fix

6fb30426

izhuhaoran commented on 2026-01-26

Cleanup

7ff5674f

Fix hybrid

c491421c

izhuhaoran commented on 2026-01-27

Disable mamba cache mode align

a2e861ed

LucasWilkinson commented on 2026-01-27

Treat num_accepted_tokens like num_computed_tokens

0c45e7e0

Treat num_accepted_tokens like num_computed_tokens

1eccc1d2

Treat num_accepted_tokens like num_computed_tokens

7005b527

Cleanup

4e4d8d59

Eliminate compute_slot_mapping

c134792d

Rename compute_slot_mapping_gpu to compute_slot_mapping

ea7f670b

Restore comments

a113c3f5

Restore comments

1153da94

Add TODO comment

e04f1f25

Fix (num_accepted_tokens shouldn't be int64)

a2241449

Cleanup

c1f550b0

Fix

ab2d19ae

Fix

3c732eb4

Fix

843a151b

MatthewBonanni changed the title ~~[WIP][Async][Spec Decoding] Zero-bubble async scheduling + spec decoding~~ [Async][Spec Decoding] Zero-bubble async scheduling + spec decoding 186 days ago

MatthewBonanni marked this pull request as ready for review 186 days ago

MatthewBonanni requested a review from

benchislett 186 days ago

MatthewBonanni requested a review from

luccafong 186 days ago

MatthewBonanni requested a review from

WoosukKwon 186 days ago

MatthewBonanni requested a review from

youkaichao 186 days ago

MatthewBonanni requested a review from

robertgshaw2-redhat 186 days ago

MatthewBonanni requested a review from

mgoin 186 days ago

MatthewBonanni requested a review from

tlrmchlsmth 186 days ago

MatthewBonanni requested a review from

houseroad 186 days ago

MatthewBonanni requested a review from

hmellor 186 days ago

MatthewBonanni requested a review from

yewentao256 186 days ago

MatthewBonanni requested a review from

ProExpertProg 186 days ago

Fix

4c522a29

mergify added needs-rebase

LucasWilkinson commented on 2026-01-30

Cleanup

c329c931

mergify removed needs-rebase

Use CpuGpuBuffer for arange

98c146b4

Make seq_lens GPU-only and introduce optimistic_seq_lens

5f1d06f6

Restructure if block

050959cb

heliqi commented on 2026-02-04

benchislett commented on 2026-02-04

heliqi commented on 2026-02-05

Fix TypeError

718087f6

Fix order of operations error

31c86744

Fix positions and seq_lens calculation

4c6bd9d2

LucasWilkinson commented on 2026-02-09

Improve _get_cumsum_and_arange

03adb2ea

Rename

e76d32ed

Use query_pos

526fea32

Eliminate CPU-side num_computed_tokens from GPUModelRunner. Update op…

48df0ab5

Fix M-RoPE and XD-RoPE

f3a2684f

Factor out _compute_batch_index_mapping

9f207819

Fix optimistic_seq_lens_cpu update

779ee44d

Fix acceptance length

2fab6940

Fix

65f7b3d4

Use CpuGpuBuffer

8cc914cf

Merge branch 'main' into async-eagle-mod

2dacc94b

Fix

fa849f86

Use preallocated buffer

ccc95758

Use triton kernel instead of pytorch ops

bc13824a

Use buffers to avoid sync

3f0be40e

Always use placeholders

664eaf10

Comment

66ae34b8

Bugfix: add arange scratch buffer

c645e05d

Use buffers to prevent allocation on the fly

04e023b4

Re-add indices_match fast path

60ca5ec4

Clean up input batch

82da483e

mergify added needs-rebase

yewentao256 commented on 2026-02-21

Merge branch 'main' into async-eagle-mod

f7d028ea

mergify removed needs-rebase

LucasWilkinson commented on 2026-02-27

Simplify BatchIndexMapping and num_computed_tokens tracking

0cdf7de0

Merge pull request #2 from LucasWilkinson/simplify-batch-index-mapping

d0b43ec6

MatthewBonanni requested a review from

njhill 152 days ago

Fix acceptance length

7fde95ad

Deduplicate

e20565dd

Make positions GPU-only

fa02bb84

Eliminate has_prev_draft_tokens

a74cd155

mergify added needs-rebase

Add CPU correction

6aa50c2e

Merge branch 'main' into async-eagle-mod

3cee2b2f

benchislett commented on 2026-03-05

mergify removed needs-rebase

Skip unnecessary copy

d4304e40

LucasWilkinson commented on 2026-03-05

Rename

56bd8f82

mergify added needs-rebase

Docstring

a694e92d

Use optimistic_seq_lens_cpu and clean up runner

836fed24

Fix arange_size

02e6ee2a

Merge branch 'main' into async-eagle-mod

39807a8b

mergify removed needs-rebase

Undo disable

09ffa74b

mergify added needs-rebase

benchislett commented on 2026-03-06

Rename

10238954

Rename

168ecc21

benchislett commented on 2026-03-09

Docstring

79a2127d

Add comment

d84e371e

Move to utils

504a7e18

Comment

83c52fd3

Clean up unnecessary change

a1cd3c82

Add comment

f0f9978e

Restore fast path

1c7d2a76

Remove unrelated fast path

74fb29a5

Clean up

5f22dce1

Merge branch 'main' into async-eagle-mod

65f18f1f

mergify removed needs-rebase

Update comment

741f1819

mergify added needs-rebase

Merge branch 'main' into async-eagle-mod

1a318108

mergify removed needs-rebase

Clean up commit_slot_mapping (dead code)

865f2c1c

Accumulate rejections and only issue copies when previous has been co…

70c12ede

Fix

279b867d

Revert "Accumulate rejections and only issue copies when previous has…

790a8ab8

Update CPU side with _finalize_async_spec_cpu_state

abc93ec7

Clean up

e2d5dbcd

Clean up

8980abbb

Comments

7a3412ab

Clean up

1cac6260

Comment

4192b659

Use deferred_spec_decode_corrections per Lucas's suggestion

1c2499d7

MatthewBonanni force pushed to 1c2499d7 138 days ago

Merge branch 'main' into async-eagle-mod

30dae23a

MatthewBonanni added ready

LucasWilkinson approved these changes on 2026-03-20

Fix type

f852fcc5

Clean up

5152276d

Merge branch 'main' into async-eagle-mod

39b8b38f

Fix mamba align

377d6095

Fix ngram during correction

51eb871c

Fix mamba

68111115

Fix ngram

26741ad9

mergify added needs-rebase

LucasWilkinson commented on 2026-03-20

Clean up

dfd0896f

Merge branch 'main' into async-eagle-mod

ed3fb142

mergify removed needs-rebase

mergify added needs-rebase

Fix

6bed7ace

Merge branch 'main' into async-eagle-mod

7e29a712

mergify removed needs-rebase

MatthewBonanni merged fafe76b4 into main 133 days ago

MatthewBonanni deleted the async-eagle-mod branch 133 days ago

Reviewers

LucasWilkinson

benchislett

yewentao256

heliqi

izhuhaoran

gemini-code-assist

luccafong

WoosukKwon

youkaichao

robertgshaw2-redhat

mgoin

tlrmchlsmth

houseroad

hmellor

ProExpertProg

njhill

Assignees

No one assigned

Labels

speculative-decoding ready v1

Milestone

No milestone

vllm [Async][Spec Decoding] Zero-bubble async scheduling + spec decoding #32951 Merged

[Async][Spec Decoding] Zero-bubble async scheduling + spec decoding #32951

vllm
[Async][Spec Decoding] Zero-bubble async scheduling + spec decoding
#32951

Merged