fixes infill incorrect tokenization #3508
ggerganov
merged
15 commits into
ggml-org:master
from
vvhg1:master
infill tokens correction
dfeda32a
Merge branch 'ggerganov:master' into master
8bd24b2e
serverinfill tokens correction
6796e745
vvhg1
changed the title
removing any leading whitespace from infill suffix and removing leead…
377be2f3
ggerganov
commented
on 2023-10-06
removing any leading whitespace from infill suffix and removing leead…
b4046aab
only rm when params.escape, rm space if possible which is added back …
05265607
ggerganov
commented
on 2023-10-07
only rm when params.escape, rm space if possible which is added back …
63ba0b62
Revert "only rm when params.escape, rm space if possible which is add…
003c15bf
Merge branch 'master' of github.com:ggerganov/llama.cpp
fc01dc0c
fix interactive prompt escaping and fix server infill leading space h…
c3a7f848
rm unnecessary bool check
b1b6beff
Merge branch 'master' of github.com:ggerganov/llama.cpp
4a214689
Merge branch 'master' of github.com:ggerganov/llama.cpp
d9dae931
Merge branch 'master' of github.com:ggerganov/llama.cpp
141329f8
ggerganov
approved these changes
on 2023-10-09
Merge branch 'master' of github.com:ggerganov/llama.cpp
35177291
ggerganov
approved these changes
on 2023-10-10
Login to write a write a comment.
Login via GitHub