transformers.js
68ed7f6c - Add Gemma Tokenizer (#598)

Commit
1 year ago
Add Gemma Tokenizer (#598) * Fix styling for whitespace tokens * Add `GemmaTokenizer` * Update minimum `@huggingface/jinja` version * Add Gemma to tokenizer playground * Add Gemma tokenizer unit test * Update tokenizer names in playground * Update Gemma tokenizer test
Author
Parents
Loading