transformers
851f253f
- Fix Qwen2Tokenizer (#29929)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
1 year ago
Fix Qwen2Tokenizer (#29929) qwen2: fixed tokens starting with # in slow tokenizer; add tests Co-authored-by: jklj077 <17811943+jklj077@users.noreply.github.com>
References
#29929 - Fix Qwen2Tokenizer
Author
jklj077
Parents
17b06e2c
Loading