transformers.js
5e7e82b9 - Improve tokenizer unit tests

Commit

262 days ago

Improve tokenizer unit tests

References

#545 - 🚀🚀🚀 Transformers.js V3 🚀🚀🚀

Author

xenova

xenova

Parents

Files10

tests
- models
  - albert
    - tokenization.js
  - all_tokenization_tests.js
  - bert
    - tokenization.js
  - bloom
    - tokenization.js
  - llama
    - tokenization.js
  - t5
    - tokenization.js
  - test_strings.js
  - vits
    - tokenization.js
  - wav2vec2
    - tokenization.js
- tokenizers.test.js