PR #128 Save tokenizer in conversion script

feature: save tokenizer based on script args

Jake Tae committed 4 years ago

chore: use none instead of empty str for consistency

Jake Tae committed 4 years ago

fix: rm duplicate args, save `tokenizer_class` key

Jake Tae committed 4 years ago

Update tools/convert_checkpoint/deepspeed_to_transformers.py

jaketae committed 4 years ago

Megatron-DeepSpeed Save tokenizer in conversion script #128 Merged