transformers
Accumulate tokens into batches in `PreTrainedTokenizerBase.add_tokens()`
#17119

Merged

Accumulate tokens into batches in `PreTrainedTokenizerBase.add_tokens()` #17119

SaulLu merged 2 commits into huggingface:main from Witiko:fix/slow-tokenizer-with-added-tokens

Witiko force pushed 4 years ago

Witiko changed the title ~~[WIP] Accumulate tokens into batches in `PreTrainedTokenizerBase.add_tokens()`~~ Accumulate tokens into batches in `PreTrainedTokenizerBase.add_tokens()` 4 years ago

Narsil commented on 2022-05-16

Witiko force pushed 4 years ago

Witiko requested a review from

Narsil 4 years ago

SaulLu approved these changes on 2022-05-20

Witiko force pushed 4 years ago

Accumulate tokens into batches in PreTrainedTokenizerBase.add_tokens()

cc3c7b11

Extend explanation of batching added tokens

2b1b81a6

Witiko force pushed to 2b1b81a6 4 years ago

SaulLu merged 6ee1474b into main 4 years ago

Reviewers

SaulLu

Narsil

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers Accumulate tokens into batches in `PreTrainedTokenizerBase.add_tokens()` #17119 Merged

Accumulate tokens into batches in `PreTrainedTokenizerBase.add_tokens()` #17119

transformers
Accumulate tokens into batches in `PreTrainedTokenizerBase.add_tokens()`
#17119

Merged