PR #28535 Allow add_tokens for ESM

Allow add_tokens for ESM #28535

Rocketknight1 merged 5 commits into main from allow_esm_add_tokens

Allow non-special tokens to be added

8d4cfeb1

ArthurZucker commented on 2024-01-16

Add test, fix token adding code

6acba9bf

ArthurZucker dismissed these changes on 2024-01-17

ArthurZucker dismissed their stale review 2 years ago

Revert changes to id_to_token and token_to_id

b3fa7e12

Rocketknight1 requested a review from

ArthurZucker 2 years ago

Update the ESM tokenizer to be a bit more standardized

7fdb3f20

ArthurZucker approved these changes on 2024-01-18

Update src/transformers/models/esm/tokenization_esm.py

2dca2707

Rocketknight1 merged d1578159 into main 2 years ago

Rocketknight1 deleted the allow_esm_add_tokens branch 2 years ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone