transformers

a1b89d7f - fix(serving): resolve rust tokenizer from ProcessorMixin in streaming generation (#45368)

fix(serving): resolve rust tokenizer from ProcessorMixin in streaming generation (#45368) ProcessorMixin subclasses (e.g. Qwen3VLProcessor) expose the fast tokenizer at .tokenizer, not ._tokenizer. Use getattr() to handle both ProcessorMixin and PreTrainedTokenizerFast when extracting the rust tokenizer backend for DirectStreamer and CBStreamer. Fixes #45362 Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>