transformers

001c526a - processor tests: use tiny Hub repos to reduce CI memory (#47115)

processor tests: use tiny Hub repos to reduce CI memory (#47115) * trocr * llava * llava * ernie4_5_vl_moe: use tiny processor Hub repo to reduce memory usage Replace the custom setUpClass that loaded the full 100k-vocab tokenizer (~342 MB) with tiny_model_id pointing to the pre-built hf-internal-testing/tiny-processor-ernie4_5_vl_moe (509 tokens). processor_config.json in the Hub repo has patch_size=4 and size={shortest_edge:784, longest_edge:3136} baked in so no kwargs needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * qwen3_vl: use tiny processor Hub repo to reduce memory usage Replace model_id pointing to the full 151k-vocab Qwen3-VL-235B tokenizer (~327 MB) with tiny_model_id = hf-internal-testing/tiny-processor-qwen3_vl (500 tokens). Settings patch_size=4, max_pixels=3136, min_pixels=784 are baked into preprocessor_config.json and tokenizer_config.json so the _setup_from_pretrained override is no longer needed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * glm4v: use tiny processor repo to reduce test memory (~309 MB) - Create tiny Hub repo hf-internal-testing/tiny-processor-glm4v with trimmed 500-token BPE tokenizer (from 151k) and baked-in processor config (patch_size=4, size={shortest_edge:144, longest_edge:324}) - Remove _setup_from_pretrained override that passed patch_size/size kwargs - Fix make_tiny_processor.py validation to handle config-only additional special tokens (e.g. <|image|>, <|video|>) that HuggingFace auto-adds from tokenizer_config.json rather than tokenizer.json added_tokens Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * minicpmv4_6: use tiny processor repo to reduce test memory (~308 MB) - Create tiny Hub repo hf-internal-testing/tiny-processor-minicpmv4_6 with trimmed 500-token BPE tokenizer (from 248k vocab) - No processor config changes needed (no _setup_from_pretrained override) - Fix pre-existing: add image_unstructured_max_length=100 override (MiniCPM expands image tokens to ~70 tokens, exceeding default of 76) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * llava_next_video: use tiny processor repo to reduce test memory (~290 MB) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * qwen2_vl: use tiny processor repo to reduce test memory (~273 MB) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * qwen3_omni_moe: use tiny processor repo to reduce test memory (~261 MB) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * qwen2_5_omni: use tiny processor repo to reduce test memory (~260 MB) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * altclip: use tiny processor repo to reduce test memory (~254 MB) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * smolvlm: use tiny processor repo to reduce test memory (~226 MB) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * internvl: use tiny processor repo to reduce test memory (~206 MB) Also add padding=True to test_frames_binding to fix fragile assertion that relied on both texts tokenizing to the same length. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * qwen2_5_vl, mistral3, video_llama_3: use tiny processor Hub repos to reduce test memory - qwen2_5_vl: add tiny_model_id (uses existing qwen2_vl checkpoint, separate Hub repo) - mistral3: add tiny_model_id; processor-only repo (no config.json); custom tests that check exact token IDs updated to use full_tmpdirname since they depend on full vocab - video_llama_3: add tiny_model_id with patch_size/pixel overrides Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * colqwen2, csm: use tiny processor repo to reduce test memory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * pixtral: use tiny processor repo to reduce test memory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * janus: use tiny processor repo to reduce test memory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * clip: use tiny processor repo to reduce test memory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * gemma3n, lighton_ocr: use tiny processor repo to reduce test memory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * glm46v: use tiny processor repo (shared with glm4v) to reduce test memory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * idefics3: use tiny processor repo to reduce test memory Override image_unstructured_max_length to 100 because the tiny tokenizer (500 vocab) produces slightly longer sequences than the default max_length of 76. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * donut, grounding_dino, owlv2, parakeet, qianfan_ocr: use tiny processor repo to reduce test memory Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * kosmos2_5: use tiny processor repo to reduce test memory test_image_procesor_load_save_reload and test_can_load_various_tokenizers use full_tmpdirname since they require the original vocab. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * qwen2_audio: use tiny processor repo to reduce test memory test_can_load_various_tokenizers, test_tokenizer_integration, and test_chat_template use full_tmpdirname since they reference model_id directly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * mllama: use tiny processor repo to reduce test memory test_apply_chat_template and test_process_interleaved_images_prompts_image_splitting use get_processor(use_tiny_ckpt=False) since they contain hardcoded full-tokenizer token IDs. Token IDs within those tests are now read from the local processor variable rather than class attributes (which come from the tiny processor). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * aria, aya_vision, glm_image, idefics2: use tiny processor repo to reduce test memory - aria: add tiny_model_id; all custom tests compute IDs dynamically, no changes needed - aya_vision: add tiny_model_id; fix _setup_tokenizer to use tiny_model_id (not full); skip test_get_num_multimodal_tokens_matches_processor_call (tiny max_patches mismatch) - glm_image: add tiny_model_id; update _setup_from_pretrained/_setup_tokenizer/ _setup_image_processor hooks to detect tiny vs full via full_tmpdirname - idefics2: add tiny_model_id; skip test_process_interleaved_images_prompts_no_image_splitting (batch padding order depends on tokenization lengths which differ with tiny tokenizer) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * dia: use tiny DacModel repo to reduce test memory DacModel from descript/dac_44khz was ~95 MB; the new hf-internal-testing/tiny-dac-44khz has encoder_hidden_size=4, decoder_hidden_size=16 (~0.54 MB) while preserving downsampling_ratios, n_codebooks=9, codebook_size=1024, and sampling_rate=44100 so all encode/decode tests still pass. DiaTokenizer is byte-based (no large vocab file) so only the audio tokenizer checkpoint changes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * seamless_m4t: use tiny processor repo to reduce test memory SeamlessM4TTokenizer is a fast tokenizer (TokenizersBackend) that loads from tokenizer.json. The original facebook/hf-seamless-m4t-medium has a 256k-token BPE vocab from sentencepiece.bpe.model (~200 MB in RAM). hf-internal-testing/tiny-processor-seamless_m4t provides a tokenizer.json with 504 base tokens + 202 language tokens (~negligible), cutting the test_save_load_pretrained_default delta from 199 MB to ~2 MB. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: use tiny repos for model_id in llava, llava_next_video, internvl, pixtral No test in these files uses full_tmpdirname, so pointing model_id at the full checkpoint would download large tokenizers unnecessarily at test setup. Set model_id = tiny_model_id for all four. Also revert pixtral model_id from mistral-labs to correct mistral-community org. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: model_id=None when no full checkpoint needed; skip full_tmpdirname setup When tiny_model_id is set but no test needs a full checkpoint, setting model_id=tiny_model_id was semantically misleading. Instead, leave model_id unset (None) and update the base class to skip creating full_tmpdirname entirely in that case. Affects ernie4_5_vl_moe, glm4v, internvl, llava, llava_next_video, minicpmv4_6, qwen3_vl. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: fix formatting and unused import in aya_vision and qwen2_audio processor tests Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * smolvlm: reduce prepare_video_inputs frames 8→2 to stay under max_length thresholds 8 frames produced ~201 tokens, exceeding the common test max_length values (162/167/172/176), causing 6 video kwargs tests to fail. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * aya_vision: un-skip test_get_num_multimodal_tokens_matches_processor_call The tiny Cohere tokenizer encodes '<image>' as 7 subword fragments instead of the 3 tokens the full tokenizer produces, so the processor can't locate image token positions in input_ids. Fix: build a one-off processor with the full tokenizer (from model_id) and our custom max_patches=2 image processor. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * aya_vision: remove test_get_num_multimodal_tokens_matches_processor_call override The tiny processor repo now includes TILE as a proper added_token (ID 537), so the base class test runs directly with the tiny tokenizer without needing to swap in the full Cohere tokenizer. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * aya_vision: remove _setup_tokenizer hook and model_id The tiny Hub repo now has padding_side=left baked into tokenizer_config.json, so the _setup_tokenizer hook (which forced padding_side="left") is redundant. With no test needing the full tokenizer (no get_processor(use_tiny_ckpt=False) calls), model_id is also removed to avoid creating an unused full_tmpdirname. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * review * clean up * clean up * clean up --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>