transformers
25a91051 - Fix KeyError in convert_to_native_format for dict vocab (#44452)

Commit
25 days ago
Fix KeyError in convert_to_native_format for dict vocab (#44452) When loading tokenizers like vesteinn/ScandiBERT whose tokenizer_config specifies XLMRobertaTokenizer (model=Unigram) but whose tokenizer.json contains a dict-type vocab, the expression vocab[0] raises KeyError because dict keys are strings, not integers. Add an isinstance(vocab, list) guard so the list-to-tuple conversion is only attempted on list vocabs.
Author
Parents
Loading