llama.cpp
1b67731e
- BERT tokenizer fixes (#6498)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
1 year ago
BERT tokenizer fixes (#6498) Key changes: * BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS * Nomic Embed conversion: pad vocab instead of slicing embedding tensor * llama_tokenize: handle added special tokens like HF does
References
#6498 - BERT tokenizer fixes
Author
cebtenzzre
Parents
c4a3a4ff
Loading