Implement stochastic speculative sampling #5625
ggerganov
merged
17 commits into
ggml-org:master
from
mscheong01:stochastic-speculative-decoding
(WIP) Implement stochastic speculative decoding
c1bad4a5
sample from residual distribution on draft accept failure
a9335a5c
fix #5657: force greedy sampling with probs when temp is 0
4694edde
remove p_accept parameter
fb18827b
mscheong01
changed the title
fix style
34b942a4
remove unused variables
875319b3
add srand() in speculative.cpp
6afc1f60
replace use of rand() with mt19937 sampling
94f6256f
fixes based on review (@JohannesGaessler)
e4896e71
cebtenzzre
commented
on 2024-02-28
fix r random generation
6b35c8b3
randomly select next sequence to verify + fix bug in memory freeing
2ad3f7c2
fix bug in active_seqs sync
c2cd2923
cebtenzzre
commented
on 2024-02-29
fix uniform int distribution initialization
7463569c
remove warnings from comparison between int and size_t
c7613540
check grammar in `llama_sample_probability_distribution_impl`
45465b21
remove malloc code by utilizing vectors
67ad517e
add PR link to README
056bdb30
ggerganov
approved these changes
on 2024-03-04
Reviewers
Assignees
No one assigned
Labels
need feedback
Milestone
No milestone
Login to write a write a comment.
Login via GitHub