PR #28508 Fix `_speculative_sampling` implementation

Fix `_speculative_sampling` implementation #28508

gante merged 7 commits into huggingface:main from ofirzaf:fix-speculative-decoding-algo

Fix _speculative_sampling implementation

0c5a1461

Fix wrong clamping in rejection sampling

d5224cf7

gante approved these changes on 2024-01-15

gante requested a review from

asartran 2 years ago

asartran commented on 2024-01-15

Improve readability of _speculative_sampling

2725e160

@ofirzaf

190a63a0

ofirzaf force pushed to 190a63a0 2 years ago

asartran commented on 2024-01-17

Clarify speculative behavior on acceptance of EOS

48b26852

Avert candidate generation if max_new_tokens == 0

3b9e4063

Add test for _speculative_sampling

a55181f0

gante approved these changes on 2024-01-19

asartran approved these changes on 2024-01-19

gante merged 9efec114 into main 2 years ago

ofirzaf deleted the fix-speculative-decoding-algo branch 2 years ago

Reviewers

asartran

gante

Assignees

No one assigned

Labels

None yet

Milestone

No milestone