llama.cpp
vocab: refactor tokenizer to reduce the overhead of creating multi times tokenizer
#9449
Merged
Merged
vocab: refactor tokenizer to reduce the overhead of creating multi times tokenizer #9449
ggerganov
merged
12 commits into
ggml-org:master
from
kylo5aby:refactor-tokenizer
kylo5aby
changed the title 
ngxson
commented
on 2024-09-12
ngxson
commented
on 2024-09-18
refactor tokenizer
d949c584
ggerganov
commented
on 2024-09-20
llama : make llm_tokenizer more private
6e873e56
refactor tokenizer
403758f9
Merge branch 'gg/tokenizer-cleanup' of https://github.com/ggerganov/l…
d653d251
refactor tokenizer
2ec25dbf
llama : make llm_tokenizer more private
02629d98
Merge branch 'ggerganov-gg/tokenizer-cleanup' into refactor-tokenizer
8be5d11c
remove unused files
25d4599e
remove unused fileds to avoid unused filed build error
768c43f8
avoid symbol link error
21ee3806
Update src/llama.cpp
95e433ce
Update src/llama.cpp
cd145b11
Login to write a write a comment.
Login via GitHub