Fix more stuff (#48979)
* fix(minicpmv4_6): correct get_sliced_grid semantics and tie-break for training parity
* feat: add M-RoPE canvas
* feat: add minicpmv4_7 with canvas mRoPE; revert minicpmv4_6
* refactor(minicpmv4_7): thin-inherit from minicpmv4_6
* refactor(minicpmv4_7): inherit processor from minicpmv4_6
* Fix canvas M-RoPE: resolve structural token ids from config, reject malformed visual spans, add regression tests
* Order target_sizes_mrope by text placeholder order, not modality processing order
* Drop stale canvas helper comment that cites a removed function
* refactor(minicpmv4_7): drop redundant tp/ep plan rewriting from config __post_init__
* feat(minicpmv4_7): add packed-sequence training support with cu_seqlens
* Reuse the shared get_max_seqlen utility in _prepare_packed_attention_kwargs instead of recomputing the packed max length inline
* Remove packed-sequence handling from modeling since FA varlen kwargs are expected to come from a user data collator
* Drop the unused CONFIG_MAPPING import and reformat so the code-quality gate stops skipping every CI test job
* Move all processor classes into the modular file so no cross-model imports remain outside it
* Stop numbering video placeholders with local image ids since a video is one temporal sequence and was trained without them
* Remove low-value comments in the config class and the dangling video local-id note
* Add the MiniCPM-V 4.7 model card and register the model in the doc toc and repo checks
* Collapse per-axis canvas position writes into leading-axis slices and drop the input sanity checks the processor already guarantees
* Merge vision windows per image so slicing batches with mixed patch grids no longer crash
The vectorised merge lifted one (h, w) from target_sizes[0] and reshaped the
whole batch against it. MiniCPM-V slicing emits a global thumbnail grid next to
the slice grids, and the two differ whenever the image is not square-ish, e.g.
896x448 at max_slice_nums=9 gives [[24, 44], [32, 32], [32, 32]] and the view()
raises. Merge each image against its own grid instead, matching the pre-#46738
behaviour.
* Cover the vision window merger with mixed per-image patch grids
Pins the token count a slicing batch produces when the thumbnail grid disagrees
with the slice grids, and checks that an image merges the same way regardless of
what else is packed next to it.
* Drop the standalone mrope test module
* fix(minicpmv4_7): regenerate modeling from modular
* Cover canvas M-RoPE position ids with get_rope_index unit tests
* review comments - rope tests are green
* Add auto_docstring to the 4.7 model classes and drop the unreachable token-id branch gates (#3)
* feat: reimpl mrope (#4)
* Pin the canvas M-RoPE layouts with golden position ids
Snapshot of the exact (3, batch, seq) coordinates for every layout the processor
can emit, taken before refactoring the canvas internals. Verified identical to the
pre-review implementation, so the numbers describe intended behaviour rather than
the current code.
* Return the cached-delta positions while decoding and drop a debug print
* Derive the LLM grid from config.downsample_mode instead of counting tokens
* Build the canvas from `mm_token_type_ids` and fold the video path into the image one
* Let the LLM grid follow the per-call downsample_mode
* Group adjacent frames whatever their modality
* Drop the unused special_token_ids argument and fix the return-shape docstring
* Drop the unreachable grid fallbacks and inline the one-line grid helper
* test with bugfix (#5)
* Keep the per-sample mrope grids out of tensor stacking and run the merger test on the accelerator
* Pad the per-sample mrope grids into one tensor and refresh the video slice-mode token count
* Regenerate the processing file so the new property sits where the converter puts it
* Point the integration tests at MiniCPM-V-4.7 and refresh the expected generations
* first clean-up on processors
* modualr, will upstream in a sec on main
* fix mrope tests and use shared fn for meshgrid
* run fix repo
* the missing test
* be gone, now we have a test to check it matches
* same here, remove - let config set attr to False
* trf
* delete tests as well
* Anton won't like it here...
* unnecesary revert
* Update docs/source/en/model_doc/minicpmv4_7.md
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update src/transformers/models/minicpmv4_7/modular_minicpmv4_7.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* Update src/transformers/models/minicpmv4_7/modular_minicpmv4_7.py
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
* address comments (#7)
* Guard window merger divisibility, rename patch_residual, and fix processor kwargs unpacking
* Address review comments
* address review comments: validate mm_token_type_ids, forward **kwargs to loss_function, align naming with Qwen VL, and regenerate modular files with pinned ruff (#8)
* move _frame_end_idx and _group_visual_frames into MiniCPMV4_7Model (#9)
* address review (#10)
* unskip test_generate_from_inputs_embeds and test_left_padding_compatibility, which pass as-is
* export MiniCPMV4_7VisionModel and register it in the auto model mapping, following Qwen3_5
* return metadata tuples instead of dicts from _group_visual_frames
* last commit
* CI checks
* again, repo check
* export
* comment
---------
Co-authored-by: airlsyn <airlsyn.ai@gmail.com>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
