PR #5625 Implement stochastic speculative sampling

(WIP) Implement stochastic speculative decoding

mscheong01 committed 2 years ago

sample from residual distribution on draft accept failure

mscheong01 committed 2 years ago

fix #5657: force greedy sampling with probs when temp is 0

mscheong01 committed 2 years ago

remove p_accept parameter

mscheong01 committed 2 years ago

fix style

mscheong01 committed 2 years ago

remove unused variables

mscheong01 committed 2 years ago

add srand() in speculative.cpp

mscheong01 committed 2 years ago

replace use of rand() with mt19937 sampling

mscheong01 committed 2 years ago

fixes based on review (@JohannesGaessler)

mscheong01 committed 2 years ago

fix r random generation

mscheong01 committed 2 years ago

randomly select next sequence to verify + fix bug in memory freeing

mscheong01 committed 2 years ago

fix bug in active_seqs sync

mscheong01 committed 2 years ago

fix uniform int distribution initialization

mscheong01 committed 2 years ago

remove warnings from comparison between int and size_t

mscheong01 committed 2 years ago

check grammar in `llama_sample_probability_distribution_impl`

mscheong01 committed 2 years ago

remove malloc code by utilizing vectors

mscheong01 committed 2 years ago

add PR link to README

mscheong01 committed 2 years ago

llama.cpp Implement stochastic speculative sampling #5625 Merged