transformers
9d0bb61f - Follow the SentencePiece model when converting a bare `tokenizer.model` (#50555)

Commit
10 hours ago
Follow the SentencePiece model when converting a bare `tokenizer.model` (#50555) * Follow the SentencePiece model when converting a bare tokenizer.model `TokenizersBackend` built from a `tokenizer.model` without a `tokenizer.json` ignored the model's normalizer spec, merge order and user-defined pieces, so it didn't encode like SentencePiece. Follow them, let `added_tokens_decoder` swap pieces, and check the replacement offsets against the text the processor tokenized rather than its decoded ids, which SentencePiece doesn't round-trip around added tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Normalize user-defined pieces without a dummy prefix Gemma's config lists its whitespace-only user-defined pieces ("▁▁", "▁▁▁", ...) with `normalized=False`, so they were matched against raw spaces and never matched, and merges can't build them. SentencePiece matches user-defined pieces on the normalized text, so do the same. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * skip generating merges that are rebuilt from piece scores * Keep special user-defined pieces matched on the raw text Normalizing every user-defined piece without a dummy prefix broke InternLM3, where almost the whole vocab is user-defined: `<|im_start|>` lost its raw-text priority and the overlapping piece `'<` split `'<|im_start|>'`. Only normalize non-special pieces, which still covers Gemma's whitespace pieces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Use Metaspace and isolate user-defined pieces instead of Prepend Drops the added-token normalization workarounds; user-defined pieces are split in the pre-tokenizer like SentencePiece. * Drop the generate_merges flag, simplify renames, use the Metaspace decoder * Prefix each chunk between added tokens, like the remote SentencePiece tokenizers * Keep the dummy prefix before a leading metaspace * Restore the Strip decoder: Metaspace's strips every ▁ of the first token * Follow legacy for the prefix after added tokens, like _get_prepend_scheme --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: itazap <ita.zaporozhets@huggingface.co> Co-authored-by: Arthur <arthur.zucker@gmail.com>
Author
Parents
Loading