llama.cpp
88035827 - llama : handle added special tokens like HF does

Commit

1 year ago

llama : handle added special tokens like HF does Now the BERT tokenizer actually uses the SEP and CLS tokens from SpecialVocab.

References

Author

cebtenzzre

Committer

cebtenzzre

Parents