PR #22870 Fix to removing ESM special tokens

Fix to removing ESM special tokens #22870

Rocketknight1 merged 1 commit into main from more_esm_tokenizer_tweaks

Fix to make sure the EOS token doesn't come back

13b5107d

Rocketknight1 requested a review from

asartran 3 years ago

Rocketknight1 requested a review from

sgugger 3 years ago

sgugger approved these changes on 2023-04-19

Rocketknight1 merged 898efca7 into main 3 years ago

Rocketknight1 deleted the more_esm_tokenizer_tweaks branch 3 years ago

Reviewers

sgugger

asartran

Assignees

No one assigned

Labels

None yet

Milestone

No milestone