transformers
35c15505 - Fix PretrainedTokenizerFast check => Fix PretrainedTokenizerFast Save (#35835)

Commit
356 days ago
Fix PretrainedTokenizerFast check => Fix PretrainedTokenizerFast Save (#35835) * Fix the bug in tokenizer.save_pretrained when saving tokenizer_class to tokenizer_config.json * Update tokenization_utils_base.py * Update tokenization_utils_base.py * Update tokenization_utils_base.py * add tokenizer class type test * code review * code opt * fix bug * Update test_tokenization_fast.py * ruff check * make style * code opt * Update test_tokenization_fast.py --------- Co-authored-by: Qubitium-ModelCloud <qubitium@modelcloud.ai> Co-authored-by: LRL-ModelCloud <165116337+LRL-ModelCloud@users.noreply.github.com>
Author
Parents
Loading