Update tokenizer `apply_chat_template` functionality #647
xenova
merged
17 commits into
main
from
tokenizer-apply-chat-template-kwargs
Allow custom kwargs in `tokenizer.apply_chat_template`
8d5580a1
Update jinja dependency version
8d617b2a
Add `tokenizer_kwargs` options
4524fe10
Add support for dictionaries of chat templates in the tokenizer config
d10141f3
Add `CohereTokenizer`
6f5bc6d3
`apply_chat_template` is no longer async
d366121a
Add unit test for multiple chat templates
ae794200
Update tokenizers.js
1ec55b3b
Also update when `chat_template` is undefined
926fbf56
Support setting tokenizer and text from URL
761dba9e
Update Claude tokenizer display name
c1b3b858
Add Cohere Command-R tokenizer to playground
265769a7
Add `Grok1Tokenizer`
f54ea64f
Throw error if chat template object is malformed
a65ecfaa
Improved error checking
20ff0fa3
Remove redundant error check
a6aa7264
`template_dict` can be a null-prototype object
d26f9861
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub