transformers
Explicitly check if token ID's are None in TFBertTokenizer constructor
#28824
Merged
Merged
Explicitly check if token ID's are None in TFBertTokenizer constructor #28824
ArthurZucker
merged
1 commit into
huggingface:main
from
skumar951:skumar/fix_tokenizer_id_none_check
Add an explicit none-check, since token ids can be 0
fc901d51
ArthurZucker
approved these changes
on 2024-02-02
Login to write a write a comment.
Login via GitHub