llama.cpp
Update special token handling in conversion scripts for gpt2 derived tokenizers
#3746

Merged

Update special token handling in conversion scripts for gpt2 derived tokenizers #3746

goerch merged 2 commits into ggml-org:master from Galunid:update-conversion-scripts

Update special token handling

217f82e7

Add mpt

c04ddb69

goerch approved these changes on 2023-10-23

goerch merged 69a67350 into master 2 years ago

Galunid deleted the update-conversion-scripts branch 2 years ago

Reviewers

goerch

Assignees

No one assigned

Labels

None yet

Milestone

No milestone