transformers
Saving `Trainer.collator.tokenizer` in when `Trainer.processing_class` is `None`
#36552
Merged
Overview
Commits
24
Changes
View On GitHub

Saving `Trainer.collator.tokenizer` in when `Trainer.processing_class` is `None` #36552

ArthurZucker merged 24 commits into huggingface:main from innerNULL:feat/improve_tokenizer_saving_strategy
innerNULL
innerNULL feat: Saving tokenizer in collator when processing_class is None
ea5a55a9
github-actions github-actions marked this pull request as draft 1 year ago
github-actions
innerNULL innerNULL marked this pull request as ready for review 1 year ago
innerNULL chore: Style issue
491c71f8
innerNULL chore: Typo
9d9ce407
innerNULL dbg: Check why test failed
b951cb9b
innerNULL dbg: Remove logics and another test failed which successed before, so…
04176066
innerNULL innerNULL changed the title Saving tokenizer in collator when `processing_class` is None Saving Trainer.collator.tokenizer` in when `Trainer.processing_class` is `None` 1 year ago
innerNULL innerNULL changed the title Saving Trainer.collator.tokenizer` in when `Trainer.processing_class` is `None` Saving `Trainer.collator.tokenizer` in when `Trainer.processing_class` is `None` 1 year ago
Rocketknight1
SunMarc
SunMarc commented on 2025-03-06
innerNULL test: Init unit-test
7185cc40
innerNULL chore: Style
b397df66
innerNULL innerNULL force pushed from 0712a906 to b397df66 1 year ago
innerNULL chore: Add err log
6381e6cb
innerNULL fix: Case
8b37d68d
SunMarc
SunMarc commented on 2025-03-06
Update tests/trainer/test_trainer.py
745caf74
innerNULL chore: Try to use get_regression_trainer
c372315b
innerNULL innerNULL force pushed from bcd8f219 to c372315b 1 year ago
innerNULL fix: Impl and style
7582ed6e
innerNULL fix: Style
a8ce4ef2
innerNULL fix: Case
0a801160
innerNULL fix: Import err
9e4cbae6
innerNULL fix: Missed import
9f82a0f7
innerNULL fix: Import block un-sorted problem
c236f00d
innerNULL innerNULL force pushed from 382f4100 to c236f00d 1 year ago
Merge branch 'main' into feat/improve_tokenizer_saving_strategy
73db7527
innerNULL fix: Try another tokenizer
913aa9e6
innerNULL fix: Test logic
8938e4ce
innerNULL innerNULL force pushed from 776db7cf to 8938e4ce 1 year ago
innerNULL innerNULL force pushed from d2006d42 to 153b8c28 1 year ago
innerNULL chore: Light updates
963bd5e4
innerNULL innerNULL force pushed from 153b8c28 to 963bd5e4 1 year ago
innerNULL chore: Reformat
6d1347c9
Merge branch 'main' into feat/improve_tokenizer_saving_strategy
fdcccc92
innerNULL
SunMarc
SunMarc commented on 2025-03-07
SunMarc Merge branch 'main' into feat/improve_tokenizer_saving_strategy
d92d1b07
SunMarc
SunMarc approved these changes on 2025-03-07
SunMarc SunMarc requested a review from ArthurZucker ArthurZucker 1 year ago
ArthurZucker
ArthurZucker approved these changes on 2025-03-20
ArthurZucker ArthurZucker merged 8b479e39 into main 1 year ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum