Update test to use new tokenizer type API #2692
ggerganov
merged
22 commits into
ggml-org:gguf
from
gguf
Merge tokenizer fixes into the gguf branch.
bb892663
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
98f5b1fa
Add test vocabularies
45665332
Merge branch 'gguf' into gguf
712b614a
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
fb591e1f
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
71084488
Adapt convert-new.py (and fix a clang-cl compiler error on windows)
cfb0e6ff
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
c9c3b87a
Improved tokenizer test
99e0e907
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
c545d85f
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
d864596e
Merge branch 'ggerganov:gguf' into gguf
c16ea8e1
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
21d88645
Improve token type support
370a95f5
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
dc65fb30
Exclude platform dependent tests
da837401
More sentencepiece compatibility by eliminating magic numbers
aea173f5
Merge branch 'gguf' of https://github.com/ggerganov/llama.cpp into gguf
dea1e4c0
Restored accidentally removed comment
6586487e
Improve commentary
44dd9ed2
Merge branch 'gguf' of https://github.com/goerch/llama.cpp into gguf
a8566856
Use token type API in test-tokenizer-1.cpp
11e3806b
ggerganov
approved these changes
on 2023-08-21
Login to write a write a comment.
Login via GitHub