PR #6498 BERT tokenizer fixes

BERT tokenizer fixes #6498

cebtenzzre merged 9 commits into master from ceb/bert-tokenizer-fixes

convert-hf-to-gguf : fix BERT abuse of LlamaHfVocab

748fc8ba

llama : handle added special tokens like HF does

88035827

Merge branch 'master' into ceb/bert-tokenizer-fixes

0d052cbe

convert : fix Tensor type annotations

6a9d3c09

convert scripts : fix python 3.8 compatibility

909f6be2

convert : remove now-unused ignore_nonllama parameter

45983e3a

spm : fix special_add_bos default

d1a1b614

examples : rely on new behavior of add_special

92591c12

speculative : more robust tokenizer comparison

a37696d4

cebtenzzre requested a review from

iamlemec 1 year ago

cebtenzzre requested a review from

ggerganov 1 year ago

iamlemec commented on 2024-04-05

iamlemec approved these changes on 2024-04-05

ggerganov approved these changes on 2024-04-08

cebtenzzre merged 1b67731e into master 1 year ago

Reviewers

ggerganov

iamlemec

Assignees

No one assigned

Labels

None yet

Milestone

No milestone