Doc tokenizer (#6110)
* Start doc tokenizers
* Tokenizer documentation
* Start doc tokenizers
* Tokenizer documentation
* Formatting after rebase
* Formatting after merge
* Update docs/source/main_classes/tokenizer.rst
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
* Address comment
* Update src/transformers/tokenization_utils_base.py
Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
* Address Thom's comments
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>
