transformers

1ed9eaec - Fix: use `TokenizersBackend` for Olmo3 to preserve custom `pre_tokenizer` (#44294)

Fix: use `TokenizersBackend` for Olmo3 to preserve custom `pre_tokenizer` (#44294) fix: use TokenizersBackend for olmo3 to preserve custom pre_tokenizer GPT2Tokenizer.__init__ unconditionally overwrites the pre_tokenizer with a generic ByteLevel, discarding the custom Sequence pre_tokenizer defined in the model's tokenizer.json. This causes incorrect tokenization of consecutive newlines (e.g. '\n\n' is split into separate tokens instead of being kept together). Switching to TokenizersBackend loads tokenizer.json directly, preserving the original Split + ByteLevel(use_regex=False) sequence. Fixes allenai/Olmo-3-* tokenization regression introduced in v5.0.0rc1.