transformers
Fix `skip_special_tokens` for `Wav2Vec2CTCTokenizer._decode`
#29311

Merged

Fix `skip_special_tokens` for `Wav2Vec2CTCTokenizer._decode` #29311

ArthurZucker merged 5 commits into huggingface:main from msublee:fix_wav2vec2ctctokenizer_skip_special_tokens

msublee changed the title ~~Fix skip_special_tokens process for Wav2Vec2CTCTokenizer._decode~~ Fix `skip_special_tokens` for `Wav2Vec2CTCTokenizer._decode` 1 year ago

ArthurZucker commented on 2024-02-29

ArthurZucker commented on 2024-03-04

Fix skip_special_tokens process for Wav2Vec2CTCTokenizer._decode

faa316a3

msublee force pushed to faa316a3 1 year ago

Fix skip_special_tokens for Wav2Vec2CTCTokenizer._decode

796b7314

Exclude pad_token filtering since it is used as CTC-blank token

ea24cda3

Add small test for skip_special_tokens

204c29b0

ArthurZucker commented on 2024-03-07

ArthurZucker requested a review from

sanchit-gandhi 1 year ago

ArthurZucker removed review request from

sanchit-gandhi 1 year ago

Update decoding test for added new token

d01e421d

ArthurZucker approved these changes on 2024-04-02

ArthurZucker merged 15cd6871 into main 1 year ago

msublee deleted the fix_wav2vec2ctctokenizer_skip_special_tokens branch 1 year ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers Fix `skip_special_tokens` for `Wav2Vec2CTCTokenizer._decode` #29311 Merged

Fix `skip_special_tokens` for `Wav2Vec2CTCTokenizer._decode` #29311

transformers
Fix `skip_special_tokens` for `Wav2Vec2CTCTokenizer._decode`
#29311

Merged