transformers
3d7c2f9d - #32184 save total_vocab_size (#32240)

Commit
1 year ago
#32184 save total_vocab_size (#32240) * save total_vocab_size = vocab_size + user added tokens to speed up operation * updating length when added_tokens_decoder is set * add test len(tokenizer)
Author
Parents
Loading