transformers
3d7c2f9d
- #32184 save total_vocab_size (#32240)
Go
Login via GitHub
Home
Pricing
FAQ
Install
Login
via GitHub
Commit
View On
GitHub
Commit
1 year ago
#32184 save total_vocab_size (#32240) * save total_vocab_size = vocab_size + user added tokens to speed up operation * updating length when added_tokens_decoder is set * add test len(tokenizer)
References
#32240 - #32184 save total_vocab_size
Author
itazap
Parents
3bb646a5
Loading