llama.cpp
1b67731e - BERT tokenizer fixes (#6498)

Commit
1 year ago
BERT tokenizer fixes (#6498) Key changes: * BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS * Nomic Embed conversion: pad vocab instead of slicing embedding tensor * llama_tokenize: handle added special tokens like HF does
Author
Parents
Loading