transformers

b7e0c20b - Revert "Fix CodeLlama tokenizer dropping leading whitespace on decode (#47488)" (#47861)

Revert "Fix CodeLlama tokenizer dropping leading whitespace on decode (#47488)" (#47861) This reverts commit 1b20eb168c63e7e21f01f1431def2fa69c8e6167. #47488 replaced the `Metaspace` pre-tokenizer with a `Prepend("▁") + Replace(" ", "▁")` normalizer. That changes tokenization in two ways that break existing checkpoints. 1. Added-token contents are themselves normalized when the added-vocabulary matcher (daachorse) is rebuilt, so `<s>` is stored as `▁<s>` and stops matching `<s>` in the text. Any checkpoint shipping its specials with `normalized=True` loses them: tok.tokenize("Hey how are<s>you?<unk> <unk>") ['▁Hey', '▁how', '▁are', '<', 's', '>', 'you', '?', '<', 'unk', '>', '▁<unk>'] 2. The prepend runs on every chunk produced by the added-token split, so a metaspace is inserted after every special token. `[INST] How are you doing?<s>[/INST]` moves `[` from 29961 to 518, i.e. every multi-turn Llama-2 chat prompt tokenizes differently. #47488 wrote this into its own expected outputs rather than treating it as a bug. The leading-whitespace round-trip reported in #47487 is real, but it is decode-only and does not need a pipeline change.