llama.cpp
Tokenizer SPM fixes for phi-3 and llama-spm (bugfix)
#7425

Merged

Tokenizer SPM fixes for phi-3 and llama-spm (bugfix) #7425

jaime-m-p merged 5 commits into ggml-org:master from jaime-m-p:tokenizer-spm-fixes

Style

c78a53ad

Update brute force test: add_special

3d490e85

Enable rtrim when pre-inserting BOS

9b21dc3a

Default values for add_bos_token and add_eos_token

f2340b43

github-actions added testing

github-actions added python

Revert "server : fix test regexes"

0594bfdd

github-actions added examples

github-actions added server

ggerganov approved these changes on 2024-05-21

jaime-m-p merged d7e852c1 into master 1 year ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

testing examples python server

Milestone

No milestone