PR #3567 Minor improvements in GPT2 tokenizer

Minor improvements in GPT2 tokenizer #3567

goerch merged 2 commits into ggml-org:master from goerch:fix-3502

Fixing minor bugs in bpe_gpt2_preprocess

8d0c5758

goerch requested a review from

ggerganov 2 years ago

goerch changed the title ~~Minor improvements for #3502~~ Minor improvements in GPT2 tokenizer 2 years ago

Don't add bos token in test

d0c8d14e

goerch requested a review from

cebtenzzre 2 years ago

ggerganov approved these changes on 2023-10-10

cebtenzzre removed review request from

cebtenzzre 2 years ago

goerch merged 233fc1c6 into master 2 years ago

goerch deleted the fix-3502 branch 2 years ago

Reviewers

ggerganov

Assignees

No one assigned

Labels

None yet

Milestone

No milestone