transformers
Fix Qwen2Tokenizer
#29929
Merged
Overview
Commits
1
Changes
View On GitHub

Fix Qwen2Tokenizer #29929

ArthurZucker merged 1 commit into huggingface:main from jklj077:fix-qwen2tokenizer
jklj077
jklj077 qwen2: fixed tokens starting with # in slow tokenizer; add tests
deef05a7
ArthurZucker
ArthurZucker approved these changes on 2024-04-03
ArthurZucker ArthurZucker merged 851f253f into main 1 year ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum