llama.cpp
llama : Added support for Tekken pre-tokenizer (#8577)
#8579

Merged

llama : Added support for Tekken pre-tokenizer (#8577) #8579

ggerganov merged 4 commits into ggml-org:master from m18coppola:master

m18coppola changed the title ~~llama : Added support for Viking pre-tokenizer (#8577)~~ llama : Added support for Tekken pre-tokenizer (#8577) 1 year ago

m18coppola force pushed 1 year ago

github-actions added python

HanClinto commented on 2024-07-19

llama : Added support for Tekken pre-tokenizer (#8577)

7fc85054

m18coppola force pushed to 7fc85054 1 year ago

compilade approved these changes on 2024-07-19

llama : fix order of pre-tokenizers

447c0809

ggerganov approved these changes on 2024-07-19

Merge branch 'ggerganov:master' into master

8506e139

mofosyne added Review Complexity : Low

atelepov approved these changes on 2024-07-19

* Tekken pre-tokenizer no longer uses clean_up_tokenization_spaces

dd5a0bff

HanClinto commented on 2024-07-19

ggerganov merged 94036222 into master 1 year ago

Reviewers

ggerganov

compilade

atelepov

HanClinto

Assignees

No one assigned

Labels

Review Complexity : Low python

Milestone

No milestone

llama.cpp llama : Added support for Tekken pre-tokenizer (#8577) #8579 Merged

llama : Added support for Tekken pre-tokenizer (#8577) #8579

llama.cpp
llama : Added support for Tekken pre-tokenizer (#8577)
#8579

Merged