PR #250 Add M2M100 tokenizer (Closes #235)

Add M2M100 tokenizer (Closes #235) #250

xenova merged 7 commits into main from m2m100-tokenizer

Add `M2M100Tokenizer`

474a5bf7

Allow `added_tokens` list to be empty

8a96319f

Apply hot-fix for issue in HF's `M2M100Tokenizer`

58425088

Skip M2M100 tokenizer tests for now

acd3dccc

Fix `_build_translation_inputs` for `M2M100Tokenizer`

2ed962a3

Add example code in JSDoc for `TranslationPipeline`

813ac15c

Update supported_models.py

ee57167f

xenova merged 060ac830 into main 2 years ago

xenova deleted the m2m100-tokenizer branch 150 days ago

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone