transformers
2c2bbd15 - Move max_len_ properties to PreTrainedTokenizerFast and override in specific subclasses.

Commit

6 years ago

Move max_len_ properties to PreTrainedTokenizerFast and override in specific subclasses. Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Author

mfuntowicz

Parents