transformers
b262577d - add special tokens to unique_added_tokens_encoder

Commit

6 years ago

add special tokens to unique_added_tokens_encoder

References

#2312 - Correct tokenization for special and added tokens

Author

vitaliyradchenko

vitaliyradchenko

Parents

Loading