PR #30289 Upgrading to tokenizers 0.19.0

Upgrading to tokenizers 0.19.0 #30289

ArthurZucker merged 4 commits into main from upgrade_tokenizers

[DO NOT MERGE] Testing tokenizers 0.19.0rc0

5f770d4c

Accounting for the breaking change.

17daeefa

Ruff.

9a72749b

Upgrading to tokenizers `0.19` (new release with preprend_scheme fixed

0dac6038

Narsil requested a review from

ArthurZucker 1 year ago

Narsil removed review request from

ArthurZucker 1 year ago

Narsil requested a review from

amyeroberts 1 year ago

Narsil requested a review from

ArthurZucker 1 year ago

amyeroberts approved these changes on 2024-04-17

ArthurZucker approved these changes on 2024-04-17

ArthurZucker merged 8e5f76f5 into main 1 year ago

ArthurZucker deleted the upgrade_tokenizers branch 1 year ago

Reviewers

ArthurZucker

amyeroberts

Assignees

No one assigned

Labels

None yet

Milestone

No milestone