fixing bugs (#50877)
* feat(hy_v4_vl): add open-source HY4-VL model
Add the HY4-VL (Hunyuan V4 Vision-Language) model to Transformers:
modular/modeling code, configuration, image & video processors (torch and
PIL variants), processor, auto mappings and conversion mapping, plus model
documentation and tests.
- inherit the Qwen2-VL top level and reuse the framework RoPE layout and
tensor-parallel sinks, dropping most model-specific overrides
- match the released checkpoint numerics
- register the model in the auto classes, models __init__ and toctree
* refactor(hy_v4_vl): keep TP sinks model-local, support cached encoder outputs
- drop the framework-level `colwise_local` TP style; shard attention sinks
with plain `colwise` and convert them to a local tensor inside the eager
attention path instead
- rename the attention layer type to `indexed_attention`
- accept `mm_encoder_outputs` in `forward` and expand cached visual features
per sample (instead of per image) when generation expands the batch
- image/video processors: copy the default `size` instead of mutating it,
deprecate call-time `min_pixels`/`max_pixels` in favor of `size`
- tests: sharded-sinks forward/backward, eager attention without sinks,
RGB conversion before preprocess, size override handling
* refactor(hy_v4_vl): share the vision patch merger with hunyuan_vl
- split HunYuanVLVisionPatchMerger.forward into `_reshape_features` and
`_postprocess_features` hooks, keeping the shared projection and the
newline/begin/end tokens in the parent
- make HYV4VLVisionPatchMerger inherit it and override the two hooks for
its per-frame merge; alias `before_rms` to the existing `norm` so the
released checkpoints keep loading
- rename `act` to `proj_act` to match the parent's naming
- tests: projection and boundary tokens, spatial checks, conv reference,
and unused parameters stay gradient-free
* refactor(hy_v4_vl): apply community review on the PR base (#34)
Port the community review changes without bringing unrelated upstream history into the PR. Retain model-local packed generation and test adaptations required by the existing base, and expose router outputs through local recording hooks.
Simplify configuration and processor APIs, use checkpoint key conversion for before_rms, and regenerate the models against the pinned base. Validate CUDA regression tests, real vision feature parity, tensor-parallel gradients, and complete text, image, and video generation outputs.
Co-authored-by: manayang <manayang@tencent.com>
* refactor(hy_v4_vl): address follow-up review on rebased main (#40)
* refactor(hy_v4_vl): address follow-up review on rebased main
Use declarative subconfig specs and regenerate Auto mappings. Remove deprecated rotary device handling, duplicate packed-generation overrides, and redundant generation tests while preserving model-specific positions and TP sink gradients.
Move router-logit defaults to the conditional generation wrapper and unify video patch and grid estimates. Verify CUDA model regression, real image/video preprocessing and vision parity, two-rank gradients, and text/image/video generation across 32 tensor-parallel ranks.
* test(hy_v4_vl): leave shared RoPE regression unchanged
Restore the shared modeling test to its upstream version rather than changing all models' RoPE checks for HY4-VL. The inherited GPU frequency check still compares CPU and CUDA buffers after deprecated implicit device placement is removed; keep the failure visible without adding a skip or changing model behavior.
---------
Co-authored-by: manayang <manayang@tencent.com>
* refactor(hy_v4_vl): address final review on the 1013 main
- Inherit the text TP plan and output recorders from HYV4; drop the lm_head TP plan.
- Reuse HunYuanVL's merger and vision attention, commenting only the differences.
- Follow Qwen2-VL for vision positions and the rope index, and only compute
3D positions when none are passed.
- Keep MiMo-V2-Flash's eager attention, with a local view of TP-sharded sinks.
- Simplify the mRoPE config check and list the fp32 modules explicitly.
- Use TP+EP in the docs, drop `.eval()` and `inference_mode`, run `make fix-repo`.
- Reduce the model tests to the significant multimodal cases.
* docs(hy_v4_vl): add spacing to the usage examples
* refactor(hy_v4_vl): address the 1014 review
- Comment out the attention entries of HYV4's TP plan until sinks support a local colwise
layout, and import MiMo-V2-Flash's eager attention directly.
- Restore the docstrings of `get_vision_position_ids` / `get_rope_index`.
- Compute video features through `get_image_features`.
- Align `get_num_of_video_patches` with the Qwen3-VL parent.
- Use real sample media in the docs.
- Drop the torch guard and the redundant cache test; add an integration test
on a tiny random checkpoint.
* test(hy_v4_vl): point the integration test to hf-tiny-v2
* test(hy_v4_vl): clean up the tests
- Use the memory cleanup mixin in the integration test and resolve the sample
urls through `url_to_local_path`, pointing the image at the copy CI caches.
- Pass the integration test arguments by keyword; keep `Expectations` only for
the image's greedy tokens, as the logits match on cpu and cuda.
- Drop the redundant `require_vision` / `require_torch` decorators, which
`require_torchvision` already implies.
- Remove the overrides the common mixins handle on their own; keep the ones
needed because the video output is a flattened patch sequence or because the
processors follow `torch_device`, and say why.
* test(hy_v4_vl): drop redundant processor tests
- Build the word-level test tokenizer with TokenizersBackend, which is the
class behind the legacy PreTrainedTokenizerFast alias.
- Drop test_auto_image_registration_and_canonical_metadata and
test_no_default_padding: both assert generic auto-mapping and padding
behaviour that the shared mixins and the parent model already cover.
- Drop test_processor_attributes_and_model_input_names. Its attribute and
model_input_names assertions are covered elsewhere; the only HY4-VL
specific part, fps-derived frame counts clamped by min_frames and
max_frames, moved into test_call_sample_frames.
Verified against a wheel build of this tree (all 3074 src/transformers
Python files hash-identical to the commit): tests/models/hy_v4_vl gives
203 passed / 0 failed, and the full tests/models/{hy_v4_vl,hunyuan_vl,
hy_v4} run under RUN_SLOW=1 shows the same eight pre-existing failures,
one hub-download timeout that passes in isolation, and one tiny-model
greedy-text drift in hunyuan_vl, which this change does not touch.
* Hy4vl video resize port (#43)
* feat(hy_v4_vl): give the image processor its own name
Expose HYV4VLImageProcessor and HYV4VLImageProcessorPil as model-named
classes whose implementation is the HunYuanVL one. The two classes are
declared as empty subclasses in the modular file, so the generated files
hold a full, in-sync copy of the parent implementation, the same way
hunyuan_vl relates to qwen2_vl. This keeps checkpoints that record
image_processor_type as HYV4VLImageProcessor loadable again, while
checkpoints that record HunYuanVLImageProcessor keep working.
Registration is regenerated rather than hand-written: check_inits builds
the module exports and check_auto emits the image processor mapping entry,
which also removes the manual parent-model mapping that was only needed
while the classes were absent.
Rename the video resize helper to video_smart_resize. It shares its name
with the image-side smart_resize but takes an extra num_frames and
max_image_size, and the converter would otherwise bind the image classes
to the video signature.
Verified with a wheel build of the base commit: both classes produce
bit-identical pixel_values and image_grid_thw against the parent on the
torchvision and pil backends, both checkpoint metadata variants load
through AutoImageProcessor and AutoProcessor, and the single-GPU suite
reports 591 passed with the same pre-existing failures as before.
* feat(hy_v4_vl): port the adaptive video resize
Replace the area-budget resize with the token-aware adaptive resize. A
frame of rows x cols merged patches costs rows * (cols + 1) tokens, one
per row, so the newline tokens are now part of the budget instead of
being ignored, and frames are never upscaled beyond their input
resolution. The helper backs both resize and get_num_of_video_patches so
the serving-side estimate and the produced grid cannot drift apart, and
re-running it on its own output returns the same size.
Keep the helper named video_adaptive_resize: this modular file also holds
the image processor classes, and their smart_resize has a different
signature, so reusing that name would bind the image classes to the video
one.
Replace test_rejects_infeasible_minimum_and_cap, which only described the
old minimum-vs-cap errors, with idempotency, token-budget and no-upscale
coverage.
Measured impact across 42 clip/frame combinations: output is unchanged
for typical inputs, while 6 combinations exceeded the token budget before
this change and none do after it (for example 128 frames of 720p used
33792 tokens against a 32768 budget).
---------
Co-authored-by: manayang <manayang@tencent.com>
* refactor(hy_v4_vl): return (height, width) from the video resize helper
Move the helper onto the video processor as a private static method and
make it return (height, width), matching the argument order used by the
other video resizers, so call sites read the same way as the image path.
It cannot be called smart_resize: the modular file shares one namespace
with the image processors, whose own smart_resize(height, width, factor,
min_pixels, max_pixels) would then be shadowed by the video version in
the generated image processor file and called with the wrong arity. It
also cannot be a method of that name, because the generated video file
would then contain both the unused module-level smart_resize copied from
Qwen3-VL and the method.
Document that size.shortest_edge is part of the configuration contract
but is never a lower bound, since frames are no longer upscaled.
No numerical change: the same four deterministic videos produce
identical video_grid_thw and byte-identical pixel_values_videos before
and after.
* refactor(hy_v4_vl): name the video resize helper smart_resize
Keep the familiar name from the other video processors, but as a static
method on the video processor so it does not occupy a module-level
symbol. The modular file shares one namespace with the image processors,
and a module-level video smart_resize would be picked up by the generated
image processor file, where the inherited resize calls it with five
arguments.
The generated video file now also carries Qwen3-VL's unused module-level
smart_resize copy next to the method; it is a converter artifact and does
not affect behaviour.
No numerical change: the same four deterministic videos produce
identical video_grid_thw and byte-identical pixel_values_videos, and the
image processors still bind the five-argument parent helper.
* fix(hy_v4_vl): only require longest_edge for the video resize
size.shortest_edge is not a lower bound any more since frames are never
upscaled, so the resize entry point no longer demands it. Only
longest_edge, the clip pixel budget, is required now, and the error
message names that key. Configs that still carry shortest_edge keep
working: the field is accepted and ignored, which the init kwargs
docstring states explicitly.
Adjust the size validation test to match the new error message.
* nit after main
* fix tests and lets merge
* refactor(hy_v4_vl): follow upstream API changes after the main merge
- Use the attention tensor-parallel plan inherited from HYV4 again: sinks are
handled by the `local_params` style upstream, so the attention is sharded.
- Drop HYV4VLProcessorKwargs; `return_mm_token_type_ids = True` is a class
default of the parent processor now.
- Drop the HYV4VLModel.forward override: the base forward returns the text
model outputs directly (with router logits) since rope deltas left the
model output.
- Rename `past_key_values` to `cache`, use `slice_logits_to_keep`, drop the
no-grad decorator from `_init_weights` and FloatTensor annotations.
- Tests build sub-configs from `vision_*` tester attributes; docs list
HYV4VLVisionModel.
* fix
* test fixes
* repo fixes
* skip
---------
Co-authored-by: manayang <manayang@tencent.com>
Co-authored-by: manayang <jackmanayang@gmail.com>
Co-authored-by: raushan <raushan@huggingface.co>