llama.cpp
server : Add option to return token pieces in /tokenize endpoint
#9108
Merged
Merged
server : Add option to return token pieces in /tokenize endpoint #9108
ngxson
merged
9 commits into
ggml-org:master
from
mathijshenquet:feature/tokenize-with-pieces
server : added with_pieces functionality to /tokenize endpoint
a2d4d191
ggerganov
approved these changes
on 2024-08-21
ngxson
requested changes
on 2024-08-21
server : Add tokenize with pieces tests to server.feature
198daa4e
Handle case if tokenizer splits along utf8 continuation bytes
b11e63ce
Add example of token splitting
42fb6707
Remove trailing ws
0c5baa1c
Fix trailing ws
0d198bbf
Maybe fix ci
0fbed972
ngxson
approved these changes
on 2024-08-27
Merge branch 'master' into feature/tokenize-with-pieces
ad971140
maybe this fix windows ci?
661a740d
Login to write a write a comment.
Login via GitHub