transformers

b8eea415 - fix: route Granite models to TokenizersBackend to preserve tokenizer.json pre-tokenizer (#45813)

fix: route Granite models to TokenizersBackend to preserve tokenizer.json pre-tokenizer (#45813) * fix: add Granite models to MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS Fixes #45812 * fix: route Granite models to TokenizersBackend to preserve tokenizer.json pre-tokenizer previous fix was not triggering Fixes #45812 * test: add tokenizer encoding test for Granite 4+ Verify that AutoTokenizer produces correct token IDs for digit strings, punctuation, and mixed alphanumeric inputs. --------- Co-authored-by: Khoi-Nguyen Tran <kndtran@ibm.com> Co-authored-by: Ita Zaporozhets <31893021+itazap@users.noreply.github.com>