transformers
fix tiktoken convert to pass AddedToken to Tokenizer
#36566

Merged

fix tiktoken convert to pass AddedToken to Tokenizer #36566

ArthurZucker merged 9 commits into main from 36438_tiktoken_convert

github-actions marked this pull request as draft 303 days ago

itazap changed the title ~~36438 tiktoken convert~~ fix tiktoken convert to pass AddedToken to Tokenizer 303 days ago

itazap requested a review from

ArthurZucker 303 days ago

ArthurZucker approved these changes on 2025-03-06

shethaadit approved these changes on 2025-03-07

itazap force pushed from 78fc6c50 to 84b892d4 300 days ago

pass AddedToken to Tokenizer

0e4b9c85

ruff

1b32bb8d

handle dict for special tokens

445cd2dc

option: test tokenizer from tiktoken same as fast

d7f52c57

itazap force pushed from 7cfaf746 to d7f52c57 300 days ago

itazap requested a review from

ArthurZucker 300 days ago

ruff

bbcdf8e1

ruff

d7c4d985

itazap marked this pull request as ready for review 297 days ago

Merge branch 'main' into 36438_tiktoken_convert

06cea98c

Merge branch 'main' into 36438_tiktoken_convert

291127fb

Merge branch 'main' into 36438_tiktoken_convert

28da5209

ArthurZucker approved these changes on 2025-03-20

ArthurZucker merged 3f03c379 into main 288 days ago

ArthurZucker deleted the 36438_tiktoken_convert branch 288 days ago

Reviewers

ArthurZucker

shethaadit

Assignees

No one assigned

Labels

None yet

Milestone

No milestone