PR #39587 fix(tokenization): check token.content for trie

fix(tokenization): check token.content for trie #39587

ArthurZucker merged 1 commit into huggingface:main from phiott:fix-token-update

fix: check token.content for trie

c86a9d5b

ArthurZucker approved these changes on 2025-07-28

ArthurZucker merged 5c15eb55 into main 332 days ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone