transformers.js
060ac830 - Add M2M100 tokenizer (Closes #235) (#250)

Commit
2 years ago
Add M2M100 tokenizer (Closes #235) (#250) * Add `M2M100Tokenizer` * Allow `added_tokens` list to be empty * Apply hot-fix for issue in HF's `M2M100Tokenizer` * Skip M2M100 tokenizer tests for now TODO: Remove when https://github.com/huggingface/transformers/pull/25478 is merged * Fix `_build_translation_inputs` for `M2M100Tokenizer` * Add example code in JSDoc for `TranslationPipeline` * Update supported_models.py
Author
Parents
Loading