transformers.js

fc1c1763 - Do not remove duplicate keys for `Unigram` models

Do not remove duplicate keys for `Unigram` models See https://huggingface.co/camembert-base for an example of a Unigram tokenizer that has two tokens with the same value (`<unk>`)