transformers
Add clean_up_tokenization_spaces to config
#22341
Merged
Overview
Commits
12
Changes
View On GitHub

Add clean_up_tokenization_spaces to config #22341

ArthurZucker merged 12 commits into huggingface:main from ArthurZucker:fix-cleanup-tokenization
ArthurZucker
ArthurZucker add draft changes
9490b471
HuggingFaceDocBuilderDev
ArthurZucker
ArthurZucker fix failing wav2vec
88cba31b
ArthurZucker style
b35c02ea
ArthurZucker
ArthurZucker ArthurZucker marked this pull request as ready for review 3 years ago
ArthurZucker ArthurZucker requested a review from Narsil Narsil 3 years ago
ArthurZucker ArthurZucker requested a review from sgugger sgugger 3 years ago
ArthurZucker
Narsil
Narsil commented on 2023-03-27
sgugger
sgugger approved these changes on 2023-03-27
ArthurZucker make sure that the argument is saved + add tests
0a599a2a
ArthurZucker style
0b86e1da
ArthurZucker fixup
9ebf001c
ArthurZucker update test
8066fee7
ArthurZucker default clean_up_tokenization_spaces to False for Bloom and Llama
d2854e75
Narsil
Narsil commented on 2023-03-28
ArthurZucker Update code based on review
47f845c8
ArthurZucker style
b90547f7
ArthurZucker Merge branch 'main' of https://github.com/huggingface/transformers in…
cc395f89
ArthurZucker quality
a2f94fc7
Narsil
Narsil approved these changes on 2023-03-28
ArthurZucker ArthurZucker merged 8d9c3836 into main 3 years ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum