transformers
Fix `MarianTokenizer` to remove metaspace character in `decode`
#26091

Merged

Fix `MarianTokenizer` to remove metaspace character in `decode` #26091

ArthurZucker merged 5 commits into huggingface:main from tanaymeh:fix_marian_tokenizer

add: check to remove metaspace from marian tokenizer

4a07c8ae

fix: metaspace character being removed from everywhere

fdb2fcb0

xenova commented on 2023-09-11

fix: remove redundant check at top

11add1d6

ArthurZucker commented on 2023-09-11

add: test for marian tokenizer decode fix

132cf855

ArthurZucker approved these changes on 2023-09-11

fix: simplified the test

d2916e5d

ArthurZucker approved these changes on 2023-09-12

tanaymeh requested a review from

xenova 2 years ago

xenova approved these changes on 2023-09-12

ArthurZucker merged 12f043ea into main 2 years ago

Reviewers

xenova

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone