feat: add Phi-1.5/Phi-2 tokenizer #9361
ggerganov
merged
3 commits into
ggml-org:master
from
daminho:master
add phi2 tokenizer
04c9c9d8
add phi name to convert_hf_to_gguf_update.py
87d8636e
make tokenizer_pre consistent; llama.cpp work
195a0629
ggerganov
approved these changes
on 2024-09-09
Login to write a write a comment.
Login via GitHub