PR #34569 Fix convert_tokens_to_string when decoder is None

Fix convert_tokens_to_string when decoder is None #34569

ArthurZucker merged 2 commits into huggingface:main from dszeto:convert-tokens-to-string-wordlevel

dszeto force pushed 1 year ago

Fix convert_tokens_to_string when decoder is None

07ec57f7

dszeto force pushed to 07ec57f7 1 year ago

ArthurZucker approved these changes on 2024-11-25

revert unrelated changs

50c4ea09

ArthurZucker merged 74db22f9 into main 1 year ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone