Follow the SentencePiece model when converting a bare `tokenizer.model` (#50555)
* Follow the SentencePiece model when converting a bare tokenizer.model
`TokenizersBackend` built from a `tokenizer.model` without a `tokenizer.json` ignored the model's normalizer
spec, merge order and user-defined pieces, so it didn't encode like SentencePiece. Follow them, let
`added_tokens_decoder` swap pieces, and check the replacement offsets against the text the processor
tokenized rather than its decoded ids, which SentencePiece doesn't round-trip around added tokens.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Normalize user-defined pieces without a dummy prefix
Gemma's config lists its whitespace-only user-defined pieces ("▁▁", "▁▁▁", ...) with `normalized=False`, so they were matched against raw spaces and never matched, and merges can't build them. SentencePiece matches user-defined pieces on the normalized text, so do the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* skip generating merges that are rebuilt from piece scores
* Keep special user-defined pieces matched on the raw text
Normalizing every user-defined piece without a dummy prefix broke InternLM3, where almost the whole vocab is user-defined: `<|im_start|>` lost its raw-text priority and the overlapping piece `'<` split `'<|im_start|>'`. Only normalize non-special pieces, which still covers Gemma's whitespace pieces.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Use Metaspace and isolate user-defined pieces instead of Prepend
Drops the added-token normalization workarounds; user-defined pieces are split in the pre-tokenizer like SentencePiece.
* Drop the generate_merges flag, simplify renames, use the Metaspace decoder
* Prefix each chunk between added tokens, like the remote SentencePiece tokenizers
* Keep the dummy prefix before a leading metaspace
* Restore the Strip decoder: Metaspace's strips every ▁ of the first token
* Follow legacy for the prefix after added tokens, like _get_prepend_scheme
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: itazap <ita.zaporozhets@huggingface.co>
Co-authored-by: Arthur <arthur.zucker@gmail.com>