Save tokenizer in conversion script (#128)
* feature: save tokenizer based on script args
* chore: use none instead of empty str for consistency
* fix: rm duplicate args, save `tokenizer_class` key
* Update tools/convert_checkpoint/deepspeed_to_transformers.py
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
Co-authored-by: Jake Tae <>
Co-authored-by: Stas Bekman <stas00@users.noreply.github.com>
