Commit
1 day ago
fixing bugs (#50877) * feat(hy_v4_vl): add open-source HY4-VL model Add the HY4-VL (Hunyuan V4 Vision-Language) model to Transformers: modular/modeling code, configuration, image & video processors (torch and PIL variants), processor, auto mappings and conversion mapping, plus model documentation and tests. - inherit the Qwen2-VL top level and reuse the framework RoPE layout and tensor-parallel sinks, dropping most model-specific overrides - match the released checkpoint numerics - register the model in the auto classes, models __init__ and toctree * refactor(hy_v4_vl): keep TP sinks model-local, support cached encoder outputs - drop the framework-level `colwise_local` TP style; shard attention sinks with plain `colwise` and convert them to a local tensor inside the eager attention path instead - rename the attention layer type to `indexed_attention` - accept `mm_encoder_outputs` in `forward` and expand cached visual features per sample (instead of per image) when generation expands the batch - image/video processors: copy the default `size` instead of mutating it, deprecate call-time `min_pixels`/`max_pixels` in favor of `size` - tests: sharded-sinks forward/backward, eager attention without sinks, RGB conversion before preprocess, size override handling * refactor(hy_v4_vl): share the vision patch merger with hunyuan_vl - split HunYuanVLVisionPatchMerger.forward into `_reshape_features` and `_postprocess_features` hooks, keeping the shared projection and the newline/begin/end tokens in the parent - make HYV4VLVisionPatchMerger inherit it and override the two hooks for its per-frame merge; alias `before_rms` to the existing `norm` so the released checkpoints keep loading - rename `act` to `proj_act` to match the parent's naming - tests: projection and boundary tokens, spatial checks, conv reference, and unused parameters stay gradient-free * refactor(hy_v4_vl): apply community review on the PR base (#34) Port the community review changes without bringing unrelated upstream history into the PR. Retain model-local packed generation and test adaptations required by the existing base, and expose router outputs through local recording hooks. Simplify configuration and processor APIs, use checkpoint key conversion for before_rms, and regenerate the models against the pinned base. Validate CUDA regression tests, real vision feature parity, tensor-parallel gradients, and complete text, image, and video generation outputs. Co-authored-by: manayang <manayang@tencent.com> * refactor(hy_v4_vl): address follow-up review on rebased main (#40) * refactor(hy_v4_vl): address follow-up review on rebased main Use declarative subconfig specs and regenerate Auto mappings. Remove deprecated rotary device handling, duplicate packed-generation overrides, and redundant generation tests while preserving model-specific positions and TP sink gradients. Move router-logit defaults to the conditional generation wrapper and unify video patch and grid estimates. Verify CUDA model regression, real image/video preprocessing and vision parity, two-rank gradients, and text/image/video generation across 32 tensor-parallel ranks. * test(hy_v4_vl): leave shared RoPE regression unchanged Restore the shared modeling test to its upstream version rather than changing all models' RoPE checks for HY4-VL. The inherited GPU frequency check still compares CPU and CUDA buffers after deprecated implicit device placement is removed; keep the failure visible without adding a skip or changing model behavior. --------- Co-authored-by: manayang <manayang@tencent.com> * refactor(hy_v4_vl): address final review on the 1013 main - Inherit the text TP plan and output recorders from HYV4; drop the lm_head TP plan. - Reuse HunYuanVL's merger and vision attention, commenting only the differences. - Follow Qwen2-VL for vision positions and the rope index, and only compute 3D positions when none are passed. - Keep MiMo-V2-Flash's eager attention, with a local view of TP-sharded sinks. - Simplify the mRoPE config check and list the fp32 modules explicitly. - Use TP+EP in the docs, drop `.eval()` and `inference_mode`, run `make fix-repo`. - Reduce the model tests to the significant multimodal cases. * docs(hy_v4_vl): add spacing to the usage examples * refactor(hy_v4_vl): address the 1014 review - Comment out the attention entries of HYV4's TP plan until sinks support a local colwise layout, and import MiMo-V2-Flash's eager attention directly. - Restore the docstrings of `get_vision_position_ids` / `get_rope_index`. - Compute video features through `get_image_features`. - Align `get_num_of_video_patches` with the Qwen3-VL parent. - Use real sample media in the docs. - Drop the torch guard and the redundant cache test; add an integration test on a tiny random checkpoint. * test(hy_v4_vl): point the integration test to hf-tiny-v2 * test(hy_v4_vl): clean up the tests - Use the memory cleanup mixin in the integration test and resolve the sample urls through `url_to_local_path`, pointing the image at the copy CI caches. - Pass the integration test arguments by keyword; keep `Expectations` only for the image's greedy tokens, as the logits match on cpu and cuda. - Drop the redundant `require_vision` / `require_torch` decorators, which `require_torchvision` already implies. - Remove the overrides the common mixins handle on their own; keep the ones needed because the video output is a flattened patch sequence or because the processors follow `torch_device`, and say why. * test(hy_v4_vl): drop redundant processor tests - Build the word-level test tokenizer with TokenizersBackend, which is the class behind the legacy PreTrainedTokenizerFast alias. - Drop test_auto_image_registration_and_canonical_metadata and test_no_default_padding: both assert generic auto-mapping and padding behaviour that the shared mixins and the parent model already cover. - Drop test_processor_attributes_and_model_input_names. Its attribute and model_input_names assertions are covered elsewhere; the only HY4-VL specific part, fps-derived frame counts clamped by min_frames and max_frames, moved into test_call_sample_frames. Verified against a wheel build of this tree (all 3074 src/transformers Python files hash-identical to the commit): tests/models/hy_v4_vl gives 203 passed / 0 failed, and the full tests/models/{hy_v4_vl,hunyuan_vl, hy_v4} run under RUN_SLOW=1 shows the same eight pre-existing failures, one hub-download timeout that passes in isolation, and one tiny-model greedy-text drift in hunyuan_vl, which this change does not touch. * Hy4vl video resize port (#43) * feat(hy_v4_vl): give the image processor its own name Expose HYV4VLImageProcessor and HYV4VLImageProcessorPil as model-named classes whose implementation is the HunYuanVL one. The two classes are declared as empty subclasses in the modular file, so the generated files hold a full, in-sync copy of the parent implementation, the same way hunyuan_vl relates to qwen2_vl. This keeps checkpoints that record image_processor_type as HYV4VLImageProcessor loadable again, while checkpoints that record HunYuanVLImageProcessor keep working. Registration is regenerated rather than hand-written: check_inits builds the module exports and check_auto emits the image processor mapping entry, which also removes the manual parent-model mapping that was only needed while the classes were absent. Rename the video resize helper to video_smart_resize. It shares its name with the image-side smart_resize but takes an extra num_frames and max_image_size, and the converter would otherwise bind the image classes to the video signature. Verified with a wheel build of the base commit: both classes produce bit-identical pixel_values and image_grid_thw against the parent on the torchvision and pil backends, both checkpoint metadata variants load through AutoImageProcessor and AutoProcessor, and the single-GPU suite reports 591 passed with the same pre-existing failures as before. * feat(hy_v4_vl): port the adaptive video resize Replace the area-budget resize with the token-aware adaptive resize. A frame of rows x cols merged patches costs rows * (cols + 1) tokens, one per row, so the newline tokens are now part of the budget instead of being ignored, and frames are never upscaled beyond their input resolution. The helper backs both resize and get_num_of_video_patches so the serving-side estimate and the produced grid cannot drift apart, and re-running it on its own output returns the same size. Keep the helper named video_adaptive_resize: this modular file also holds the image processor classes, and their smart_resize has a different signature, so reusing that name would bind the image classes to the video one. Replace test_rejects_infeasible_minimum_and_cap, which only described the old minimum-vs-cap errors, with idempotency, token-budget and no-upscale coverage. Measured impact across 42 clip/frame combinations: output is unchanged for typical inputs, while 6 combinations exceeded the token budget before this change and none do after it (for example 128 frames of 720p used 33792 tokens against a 32768 budget). --------- Co-authored-by: manayang <manayang@tencent.com> * refactor(hy_v4_vl): return (height, width) from the video resize helper Move the helper onto the video processor as a private static method and make it return (height, width), matching the argument order used by the other video resizers, so call sites read the same way as the image path. It cannot be called smart_resize: the modular file shares one namespace with the image processors, whose own smart_resize(height, width, factor, min_pixels, max_pixels) would then be shadowed by the video version in the generated image processor file and called with the wrong arity. It also cannot be a method of that name, because the generated video file would then contain both the unused module-level smart_resize copied from Qwen3-VL and the method. Document that size.shortest_edge is part of the configuration contract but is never a lower bound, since frames are no longer upscaled. No numerical change: the same four deterministic videos produce identical video_grid_thw and byte-identical pixel_values_videos before and after. * refactor(hy_v4_vl): name the video resize helper smart_resize Keep the familiar name from the other video processors, but as a static method on the video processor so it does not occupy a module-level symbol. The modular file shares one namespace with the image processors, and a module-level video smart_resize would be picked up by the generated image processor file, where the inherited resize calls it with five arguments. The generated video file now also carries Qwen3-VL's unused module-level smart_resize copy next to the method; it is a converter artifact and does not affect behaviour. No numerical change: the same four deterministic videos produce identical video_grid_thw and byte-identical pixel_values_videos, and the image processors still bind the five-argument parent helper. * fix(hy_v4_vl): only require longest_edge for the video resize size.shortest_edge is not a lower bound any more since frames are never upscaled, so the resize entry point no longer demands it. Only longest_edge, the clip pixel budget, is required now, and the error message names that key. Configs that still carry shortest_edge keep working: the field is accepted and ignored, which the init kwargs docstring states explicitly. Adjust the size validation test to match the new error message. * nit after main * fix tests and lets merge * refactor(hy_v4_vl): follow upstream API changes after the main merge - Use the attention tensor-parallel plan inherited from HYV4 again: sinks are handled by the `local_params` style upstream, so the attention is sharded. - Drop HYV4VLProcessorKwargs; `return_mm_token_type_ids = True` is a class default of the parent processor now. - Drop the HYV4VLModel.forward override: the base forward returns the text model outputs directly (with router logits) since rope deltas left the model output. - Rename `past_key_values` to `cache`, use `slice_logits_to_keep`, drop the no-grad decorator from `_init_weights` and FloatTensor annotations. - Tests build sub-configs from `vision_*` tester attributes; docs list HYV4VLVisionModel. * fix * test fixes * repo fixes * skip --------- Co-authored-by: manayang <manayang@tencent.com> Co-authored-by: manayang <jackmanayang@gmail.com> Co-authored-by: raushan <raushan@huggingface.co>
Author
Parents
Loading