PR #3006 Add heuristic algorithm for speculative

Add heuristic algorithm for speculative #3006

ggerganov merged 6 commits into ggml-org:master from leng-yue:add-speculative-heuristic

Add heuristic algo for speculative

98230ef6

bobqianic commented on 2023-09-04

ggerganov requested changes on 2023-09-04

Constrain minimum n_draft to 2

9248528d

speculative : improve heuristic impl

dddd784c

Merge branch 'master' into HEAD

4e6e9513

speculative : be more rewarding upon guessing max drafted tokens

d9559b78

speculative : fix typos

b5efa625

ggerganov commented on 2023-09-05

ggerganov approved these changes on 2023-09-14

ggerganov merged 35f73049 into master 2 years ago

leng-yue deleted the add-speculative-heuristic branch 2 years ago

Reviewers

ggerganov

bobqianic

Assignees

No one assigned

Labels

None yet

Milestone

No milestone