transformers.js
68ed7f6c - Add Gemma Tokenizer (#598)

Commit

1 year ago

Add Gemma Tokenizer (#598) * Fix styling for whitespace tokens * Add `GemmaTokenizer` * Update minimum `@huggingface/jinja` version * Add Gemma to tokenizer playground * Add Gemma tokenizer unit test * Update tokenizer names in playground * Update Gemma tokenizer test

References

#598 - Add Gemma Tokenizer

Author

xenova

Parents

6d2808b5

Files6

examples/tokenizer-playground/src
- App.jsx
- components
  - Token.jsx
package-lock.json
package.json
src
- tokenizers.js
tests
- generate_tests.py

transformers.js 68ed7f6c - Add Gemma Tokenizer (#598)

transformers.js
68ed7f6c - Add Gemma Tokenizer (#598)