PR #128 Save tokenizer in conversion script

Save tokenizer in conversion script #128

jaketae merged 4 commits into main from feat-save-tokenizer

feature: save tokenizer based on script args

2dbe65c6

chore: use none instead of empty str for consistency

32bc91de

jaketae requested a review from

stas00 4 years ago

fix: rm duplicate args, save `tokenizer_class` key

5a5e1e9c

stas00 approved these changes on 2021-10-07

Update tools/convert_checkpoint/deepspeed_to_transformers.py

2e83beca

jaketae merged 23dded08 into main 4 years ago

jaketae deleted the feat-save-tokenizer branch 4 years ago

Reviewers

stas00

Assignees

No one assigned

Labels

None yet

Milestone

No milestone