PR #20401 Use updated attributes when saving tokenizers

Use updated attributes when saving tokenizers #20401

ydshieh merged 3 commits into main from better_tokenizer_save

Use updated values

9e6a3ac3

Use updated values

9d92156d

Use updated values

2b0b3c50

ydshieh changed the title ~~Use updated values~~ Use updated attributes when saving tokenizers 3 years ago

ydshieh commented on 2022-11-23

ydshieh requested a review from

sgugger 3 years ago

ydshieh requested a review from

SaulLu 3 years ago

sgugger approved these changes on 2022-11-23

ydshieh merged 9a5b84a0 into main 3 years ago

ydshieh deleted the better_tokenizer_save branch 3 years ago

Reviewers

sgugger

SaulLu

Assignees

No one assigned

Labels

None yet

Milestone

No milestone