transformers
fix `train_new_from_iterator` in the case of byte-level tokenizers
#17549
Merged
Merged
fix `train_new_from_iterator` in the case of byte-level tokenizers #17549
SaulLu
merged
15 commits into
huggingface:main
from
SaulLu:ls/train_new_with_bytelevel
fix for ByteLevel tokenizers
e3d5c53b
SaulLu
changed the title 
Merge branch 'main' into ls/train_new_with_bytelevel
cf712e6d
change vocab size in conversational pipeline tests
432a204a
Revert "change vocab size in conversational pipeline tests"
17175504
change vocab size in conversational pipeline tests
5b961b9f
SaulLu
commented
on 2022-06-06
SaulLu
changed the title 
sgugger
approved these changes
on 2022-06-06
Narsil
commented
on 2022-06-07
Revert "change vocab size in conversational pipeline tests"
875ef434
fix bart
eba3e65a
fix blendertbot
9e7fcde0
fix yoso
b72cf898
fix roberta
1ac949ba
fix longformer
f426ae7d
fix led
f9cd33d6
fix gpt + deberta
50d7d58d
format
6d3ce0da
LysandreJik
approved these changes
on 2022-06-08
Narsil
approved these changes
on 2022-06-08
Merge remote-tracking branch 'upstream/main' into ls/train_new_with_b…
98285988
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub