transformers
[trainer] ensure special tokens in model configs are aligned with tokenizer at train time
#38441

Merged

[trainer] ensure special tokens in model configs are aligned with tokenizer at train time #38441

gante merged 4 commits into huggingface:main from gante:align_eos_token_trainer

tmp commit

1fd03d62

add test

e94955fc

gante requested a review from

lewtun 359 days ago

gante requested a review from

SunMarc 359 days ago

gante commented on 2025-05-28

make fixup

10cab115

gante changed the title ~~[trainer] ensure special tokens model configs are aligned with tokenizer~~ [trainer] ensure special tokens in model configs are aligned with tokenizer at train time 359 days ago

reset warns/info in test

54d1dc6a

SunMarc approved these changes on 2025-06-03

gante merged 83dbebc4 into main 283 days ago

gante deleted the align_eos_token_trainer branch 283 days ago

Reviewers

SunMarc

lewtun

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

transformers [trainer] ensure special tokens in model configs are aligned with tokenizer at train time #38441 Merged

[trainer] ensure special tokens in model configs are aligned with tokenizer at train time #38441

transformers
[trainer] ensure special tokens in model configs are aligned with tokenizer at train time
#38441

Merged