transformers
fix: decode() batch path respects self.clean_up_tokenization_spaces
#47793

Open

fix: decode() batch path respects self.clean_up_tokenization_spaces #47793

lorenzozanee wants to merge 1 commit into huggingface:main from lorenzozanee:candidate/tokenizer-processor

lorenzozanee force pushed from 5eee30a7 to c046f254 10 days ago

lorenzozanee force pushed from c046f254 to 71256f19 10 days ago

lorenzozanee force pushed from 71256f19 to bd2e9181 10 days ago

lorenzozanee force pushed from e8bbbda8 to c0d15412 9 days ago

lorenzozanee force pushed from c0d15412 to d6e8ca92 9 days ago

Fix decode() batch path hardcoding clean_up_tokenization_spaces=False…

06fea321

lorenzozanee force pushed from d6e8ca92 to 06fea321 9 days ago

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone