transformers
c4d78f01 - Fix tokenizer saving during training with `Trainer` (#12806)

Comment changes are shownComment changes are hidden
Commit
3 years ago
Fix tokenizer saving during training with `Trainer` (#12806) * add test in trainer and test tokenizer saving wi th trainer * quality * reverse trainer changes * replace test in test_trainer by a test for all the tokenizers * format * add can_save_slow_tokenizer attribute to all tokenizers * fix Herbert * format * Change comment in error * add comments and a new assert * Update src/transformers/models/albert/tokenization_albert_fast.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * change ValueError barthez * change ValueError BigBird * change ValueError Camembert * change ValueError Mbart50 * change ValueError Pegasus * change ValueError ReFormer * change ValueError T5 * change ValueError RoBERTa * XLNET fast * Update tests/test_tokenization_common.py Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com> * change `assert` into `self.assertIn` * format Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Author
Parents
  • src/transformers
    • models
      • albert
        • File
          tokenization_albert_fast.py
      • barthez
        • File
          tokenization_barthez_fast.py
      • big_bird
        • File
          tokenization_big_bird_fast.py
      • camembert
        • File
          tokenization_camembert_fast.py
      • herbert
        • File
          tokenization_herbert_fast.py
      • mbart50
        • File
          tokenization_mbart50_fast.py
      • pegasus
        • File
          tokenization_pegasus_fast.py
      • reformer
        • File
          tokenization_reformer_fast.py
      • t5
        • File
          tokenization_t5_fast.py
      • xlm_roberta
        • File
          tokenization_xlm_roberta_fast.py
      • xlnet
        • File
          tokenization_xlnet_fast.py
    • File
      tokenization_utils_fast.py
  • tests
    • File
      test_tokenization_common.py
Loading