transformers
4ff98bf2 - fix: decode() batch path respects self.clean_up_tokenization_spaces (#47793)

Commit
82 days ago
fix: decode() batch path respects self.clean_up_tokenization_spaces (#47793) Fix decode() batch path hardcoding clean_up_tokenization_spaces=False instead of None The batch path of decode() was hardcoding clean_up_tokenization_spaces=False when the kwarg was not explicitly provided, overriding self.clean_up_tokenization_spaces. The single-sequence path correctly leaves it unspecified (None), allowing _decode to fall back to the tokenizer's configured value. Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>
Author
Parents
Loading