PR #7375 Tokenizer SPM fixes for phi-3 and llama-spm

Tokenizer SPM fixes for phi-3 and llama-spm #7375

jaime-m-p merged 6 commits into ggml-org:master from jaime-m-p:tokenizer-spm-fixes

Update brute force test: special tokens

04aad94a

Fix added tokens

5b61c042

Fix special tokens rtrim

dd0d1590

github-actions added testing

github-actions added python

Type fix

a46dfcfd

ggerganov approved these changes on 2024-05-19

Apply suggestions from code review

0ae2860f

mofosyne added Review Complexity : Medium

github-actions added examples

github-actions added server

server : fix test regexes

7fb66eb5

ggerganov force pushed to 7fb66eb5 1 year ago

jaime-m-p merged 917dc8cf into master 1 year ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

testing examples python Review Complexity : Medium server

Milestone

No milestone