transformers.js
f0ef2e8e - Update tokenizer `apply_chat_template` functionality (#647)

Commit
1 year ago
Update tokenizer `apply_chat_template` functionality (#647) * Allow custom kwargs in `tokenizer.apply_chat_template` * Update jinja dependency version * Add `tokenizer_kwargs` options * Add support for dictionaries of chat templates in the tokenizer config * Add `CohereTokenizer` * `apply_chat_template` is no longer async * Add unit test for multiple chat templates * Update tokenizers.js * Also update when `chat_template` is undefined * Support setting tokenizer and text from URL * Update Claude tokenizer display name * Add Cohere Command-R tokenizer to playground * Add `Grok1Tokenizer` * Throw error if chat template object is malformed * Improved error checking * Remove redundant error check * `template_dict` can be a null-prototype object
Author
Parents
Loading