transformers

92cd495f - Smthn smthn (#49364)

Smthn smthn (#49364) * Add EmbeddingGemma 2 as a standalone model EmbeddingGemma 2 is a multimodal embedding model built on the Gemma 4 architecture. This adds it as its own model folder rather than extending Gemma 4 in place, so embedding-specific behaviour does not leak into an already-released architecture. The model is defined in modular_embedding_gemma2.py, which inherits from Gemma 4 and reuses its vision and audio towers through auto; the remaining modeling, configuration and processing files are generated from it. Outside the new folder the change is 12 added lines across six files, all of them auto-mapping, __init__ and doc-TOC registrations. gemma4 and gemma4_unified are untouched. Highlights: - Projection-only per-layer embeddings; there is no per-layer token lookup table, so vocab_size_per_layer_input is removed. - embedding_projection lives on the text model and is applied at the end of its forward. - Self-contained conversion script, preserving the task prompts, chat template, extra special tokens, and the fusion of the EOA and EOI embeddings into the main embedding table. - Per-layer attention shapes come from the checkpoint: an explicit per_layer_config is used as given, otherwise every layer falls back to the model-level head_dim and num_key_value_heads. Verified against the reference checkpoint: 148 external end-to-end, preprocessing and numerical-accuracy tests pass, alongside 278 in-repo unit tests. TAG=agy CONV=3ab06090-bf3a-4fb1-a7aa-e1567fc209ff * Update max_position_embeddings config. Sentence Transfomers uses this value to set the max_sequence length if a value isn't provided. Current config is too low, aligning with Gemma 4. * Trim EmbeddingGemma 2 down to a standalone encoder The first pass leaned on Gemma 4 for most of the text stack, which dragged in decoder-only machinery this model never uses. Reimplement the parts that diverge and keep modular inheritance only where the parent is reused as-is. Config: - `EmbeddingGemma2TextConfig` and `EmbeddingGemma2Config` derive from `PreTrainedConfig` instead of their Gemma 4 counterparts, are `@strict`, and spell out every field. Drop the knobs the released checkpoints never set: MoE (`enable_moe_block`, `top_k_experts`, `moe_intermediate_size`), KV sharing (`num_kv_shared_layers`, `attention_k_eq_v`), `use_double_wide_mlp`, `use_bidirectional_attention` and `tie_word_embeddings`. - Per-layer attention shapes are derived from the new `global_head_dim` and `num_global_key_value_heads` fields into `per_layer_config`, and the 5:1 sliding/full alternation is a typed `sliding_window_pattern` field rather than a hardcoded local. Model: - Own `EmbeddingGemma2TextAttention` and `EmbeddingGemma2TextDecoderLayer` instead of subclassing Gemma 4's, with no cache, no shared-KV plumbing and `is_causal = False`. - Bidirectional masks throughout (`create_bidirectional_mask` and `create_bidirectional_sliding_window_mask`), so the text model returns a plain `BaseModelOutput` instead of a `...WithPast` variant. - Projection-only PLE: `project_per_layer_inputs` takes `inputs_embeds` alone, and the `embed_tokens_per_layer` accessors are removed rather than stubbed. - The processor no longer emits `mm_token_type_ids`, which nothing consumes. Conversion script: remove checkpoint layouts the converter never reaches, identified with branch coverage over two real conversions -- the unstacked `transformer/layer_N/...` format, vision quantization/clipping, the vision `exit/scale` and `standardize` paths, the alternate `attn/k_einsum` layout, the `rlx_networks/policy_network/` prefix strip, and four constants left orphaned by those removals. Statement coverage rises from 76.4% to 94.9%; every remaining miss is a `raise ValueError` guard. Also fix `--text_dtype float32`, which crashed in `save_pretrained` with a shared tensor error: the audio clip path intentionally emits the same numpy array for q/k/v, and without a dtype cast `torch.from_numpy` handed safetensors several views over one storage. Clone in the no-cast branch to match what the cast already does. Housekeeping: allowlist the model for TRF041 and its extra config attributes in `check_config_attributes`, update the docs and tests for the removed fields and for `mm_token_type_ids`, and shorten the over-long explanatory comments flagged in review. * Bump the actions group across 1 directory with 13 updates (#1) Updates the requirements on [actions/checkout](https://github.com/actions/checkout), [actions/cache](https://github.com/actions/cache), [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action), [docker/login-action](https://github.com/docker/login-action), [docker/build-push-action](https://github.com/docker/build-push-action), [huggingface/doc-builder/.github/workflows/build_main_documentation.yml](https://github.com/huggingface/doc-builder), [actions/setup-python](https://github.com/actions/setup-python), [huggingface/doc-builder/.github/workflows/build_pr_documentation.yml](https://github.com/huggingface/doc-builder), [actions/download-artifact](https://github.com/actions/download-artifact), [slackapi/slack-github-action](https://github.com/slackapi/slack-github-action), [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish), [trufflesecurity/trufflehog](https://github.com/trufflesecurity/trufflehog) and [huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml](https://github.com/huggingface/doc-builder) to permit the latest version. Updates `actions/checkout` from 4.4.0 to 7.0.1 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4.4.0...3d3c42e5aac5ba805825da76410c181273ba90b1) Updates `actions/cache` from 5.0.5 to 6.1.0 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/27d5ce7f107fe9357f9df03efb73ab90386fccae...55cc8345863c7cc4c66a329aec7e433d2d1c52a9) Updates `docker/setup-buildx-action` from 4.1.0 to 4.3.0 - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](https://github.com/docker/setup-buildx-action/compare/d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5...37fe631027851001ddb9b187196cc803df7f5f0e) Updates `docker/login-action` from 4.2.0 to 4.6.0 - [Release notes](https://github.com/docker/login-action/releases) - [Commits](https://github.com/docker/login-action/compare/650006c6eb7dba73a995cc03b0b2d7f5ca915bee...dbcb813823bdd20940b903addbd779551569679f) Updates `docker/build-push-action` from 7.2.0 to 7.3.0 - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](https://github.com/docker/build-push-action/compare/f9f3042f7e2789586610d6e8b85c8f03e5195baf...53b7df96c91f9c12dcc8a07bcb9ccacbed38856a) Updates `huggingface/doc-builder/.github/workflows/build_main_documentation.yml` from e60a538eea9817ab312196d0d233604b01697265 to 17ccdf17e0f5e5f88b1ce9f20b3b73fb19017169 - [Release notes](https://github.com/huggingface/doc-builder/releases) - [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md) - [Commits](https://github.com/huggingface/doc-builder/compare/e60a538eea9817ab312196d0d233604b01697265...17ccdf17e0f5e5f88b1ce9f20b3b73fb19017169) Updates `actions/setup-python` from 5.6.0 to 7.0.0 - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](https://github.com/actions/setup-python/compare/v5.6.0...5fda3b95a4ea91299a34e894583c3862153e4b97) Updates `huggingface/doc-builder/.github/workflows/build_pr_documentation.yml` to 17ccdf17e0f5e5f88b1ce9f20b3b73fb19017169 - [Release notes](https://github.com/huggingface/doc-builder/releases) - [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md) - [Commits](https://github.com/huggingface/doc-builder/commits/17ccdf17e0f5e5f88b1ce9f20b3b73fb19017169) Updates `actions/download-artifact` from 4 to 8 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/v4...v8) Updates `slackapi/slack-github-action` from 1.25.0 to 4.0.0 - [Release notes](https://github.com/slackapi/slack-github-action/releases) - [Changelog](https://github.com/slackapi/slack-github-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/slackapi/slack-github-action/compare/v1.25...dcb1066f776dd043e64d0e8ba94ca15cc7e1875d) Updates `pypa/gh-action-pypi-publish` from 1.14.0 to 1.14.2 - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/cef221092ed1bacb1cc03d23a2d87d1d172e277b...dc37677b2e1c63e2034f94d8a5b11f265b73ba33) Updates `trufflesecurity/trufflehog` from 3.95.5 to 3.97.4 - [Release notes](https://github.com/trufflesecurity/trufflehog/releases) - [Commits](https://github.com/trufflesecurity/trufflehog/compare/d411fff7b8879a62509f3fa98c07f247ac089a51...363923b901c911a9164f50b6c423f47c15372b1c) Updates `huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml` from bcff59fca682130d2e7271ca8589911b7ac0b8bf to 17ccdf17e0f5e5f88b1ce9f20b3b73fb19017169 - [Release notes](https://github.com/huggingface/doc-builder/releases) - [Changelog](https://github.com/huggingface/doc-builder/blob/main/RELEASE.md) - [Commits](https://github.com/huggingface/doc-builder/compare/bcff59fca682130d2e7271ca8589911b7ac0b8bf...17ccdf17e0f5e5f88b1ce9f20b3b73fb19017169) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 6.1.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/checkout dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/download-artifact dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: actions/setup-python dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: docker/build-push-action dependency-version: 7.3.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: docker/login-action dependency-version: 4.6.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: docker/setup-buildx-action dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions - dependency-name: huggingface/doc-builder/.github/workflows/build_main_documentation.yml dependency-version: 9978a4134f9e245a927f8d85f73c026609b38b47 dependency-type: direct:production dependency-group: actions - dependency-name: huggingface/doc-builder/.github/workflows/build_pr_documentation.yml dependency-version: 9978a4134f9e245a927f8d85f73c026609b38b47 dependency-type: direct:production dependency-group: actions - dependency-name: huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml dependency-version: 9978a4134f9e245a927f8d85f73c026609b38b47 dependency-type: direct:production dependency-group: actions - dependency-name: pypa/gh-action-pypi-publish dependency-version: 1.14.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: actions - dependency-name: slackapi/slack-github-action dependency-version: 4.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: actions - dependency-name: trufflesecurity/trufflehog dependency-version: 3.97.2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: actions ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Fix CI after the main merge - `test_vision_axial_rope` arrived with main and builds `position_ids` without a batch dim, but the Gemma 4 vision RoPE we reuse splits the H/W axis at dim 2 and needs one, so `cos` came out `4 * seq_len` wide instead of `4 * inv_freq_len` (`AssertionError: 8 != 64`). Override it the same way `Gemma4Vision2TextModelTest` does. - Refresh the model card date: `add_dates` falls back to today's date while the model is absent from `main`, and tolerates a one-day drift. * Address the second review pass on EmbeddingGemma 2 Reworks the model in response to the inline review comments of 2026-09-18, plus two fixes found while verifying them. Configuration - `sliding_window` is stored as the plain inclusive radius (512) that the bidirectional mask consumes. The `__post_init__` width-to-radius conversion and the `to_dict` override that undid it are both gone; the flash-attention off-by-one is compensated only on the flash branch of the attention `__init__`, so every backend realises the same radius. - `sliding_window_pattern`, `global_head_dim` and `num_global_key_value_heads` become builder-only kwargs popped in `__post_init__` rather than config fields, as in Gemma 4. They still shape `layer_types` and `per_layer_config`, but leave no attribute and no `to_dict` key, which makes the `EmbeddingGemma2TextConfig` entry in `utils/check_config_attributes.py` obsolete. Modeling - `EmbeddingGemma2TextModel` inherits from `Gemma3TextModel` and the layer from `Gemma3DecoderLayer`, so both `__init__` bodies and the full `forward` decorator stack, including `merge_with_config_defaults`, come from the parents. Four classes drop their `Text` infix so the converter's auto-rename resolves. - The layer is now `EmbeddingGemma2EncoderLayer`: attention is bidirectional everywhere, so `DecoderLayer` was a misnomer. - PLE is split in two. `EmbeddingGemma2TextPLE` (`self.ple`) produces the per-layer tensor at model level; `EmbeddingGemma2TextPLEBlock` (`self.ple_block`) consumes one slice of it inside each layer. Inner weight names are unchanged, so the state dict gains only the `ple.` and `ple_block.` prefixes, across 74 keys. - The composite `__init__` is `super().__init__(config)` plus a `del self.vocab_size_per_layer_input` converter directive, down from 20 lines to 7. - RoPE is applied llama-style. `apply_rotary_pos_emb(q, k, cos, sin)` is sourced from Gemma 3 so that its `use_kernel_forward_from_hub` registration carries over, with `@use_kernelized_func` on the attention class. - Attention and layer `forward` now take `hidden_states`, `attention_mask`, `position_embeddings` in that order, with `position_embeddings` typed as a tuple, and the layer takes `**kwargs: Unpack[TransformersKwargs]`. - `v_norm` is built with an explicit `dim=` keyword. Conversion - The embedding head moved from `hybrid_transformer/...` to `transformer/default_projection/linear` in the checkpoint; only the new spelling is accepted. - Round the two fused boundary-token scale constants to the precision they are applied in at runtime. The reference embedder scales the extra table in bfloat16, so `sqrt(1536)` is 39.25 rather than 39.1918, and `EmbeddingGemma2TextScaledWordEmbedding` casts `embed_scale` to the weight dtype before multiplying. With full-precision constants the two fused rows came out roughly 0.15% short of the reference; relative error is now 7.1e-5 and 3.7e-5, down from 1.2e-3 and 2.0e-3. Tests and docs - Drop seven tests that asserted framework behaviour or restated an `__init__`, and set the tester's `sliding_window` to 4 so the window still binds on a 7-token sequence now that the field is a radius. - Rewrite the model page around `<hfoptions>` blocks: Sentence Transformers first, then `AutoModel` with explicit mask-weighted mean pooling, plus task prompts, Matryoshka truncation and per-modality examples. Still marked WIP pending the processing review. * squash and split video sampling * oops, typo and missing default * default max is 32 * update as per slack * comments * ci * upstream `main` has new fixes in modular * tests * Video sampling follow-ups: converter defaults, `add_timestamps`, metadata-less arrays Follow-up to #11, which landed the `fps` / `max_frames` / `overflow_strategy` sampling API. Three changes. 1. The conversion script exports the sampling contract. `_build_processor` still passed the removed `num_frames` and `use_1fps_linear_sampling`, so every exported checkpoint carried a stale `num_frames: 32`. `preprocess` forwards that to `sample_frames`, which rejects it, so video input raised on every call. It now writes `fps=1`, `max_frames=32`, `overflow_strategy="uniform"` and no `num_frames`. 2. `exclude_timestamps` -> `add_timestamps`, default `False`. The old name was framed against Gemma 4, which always emits timestamps, so the EG2 override read as an exclusion. As a standalone model there is no baseline to subtract from, and a negative boolean defaulting to `True` reads backwards at the call site. The positive opt-in follows `add_special_tokens` / `add_generation_prompt`. 3. A metadata-less array no longer fails. `sample_frames` raised when `fps` was requested but the metadata had no `fps` or `duration`. A decoded array never has either, so this broke video input through Sentence Transformers, which has no way to pass `VideoMetadata`. It now warns, skips rate-based sampling, and lets the `max_frames` budget do the work. No frame rate is guessed. Gemma 4's `fps = 24` fallback has no analogue here: Gemma 4 samples a fixed `num_frames=32` and never enters the fps branch. A guess would also have to synthesize `duration = total_num_frames / 24`, which collapses any clip of <= 32 frames to a single frame. `replace_video_token` still raises when timestamps are requested without an `fps`. Timestamps are prompt content, so a guessed rate writes wrong `mm:ss` labels into the input; it previously defaulted to 24 and mutated the caller's `VideoMetadata` in place. Net contract: raw arrays are supported without timestamps, and everything else is supported when proper metadata is provided. Also fixes two typos and three stale documentation passages: both removed flag names, a flat "32 frames sampled per video" claim where 32 is a cap, and a hardcoded 140 tokens per frame where the value is `max_soft_tokens`. New tests cover the under-budget-is-not-upsampled regression (directly and through `preprocess`), `truncate`, `num_frames` rejection, invalid arguments, the `fps=None` cap-only recipe, incomplete metadata falling back to the cap, and the timestamp raise. The serialization round-trip now pins all three sampling knobs and asserts that no `num_frames` key is written. * Batch videos of different lengths Answers tomaarsen's review comment on #4: a batch of two videos with different frame counts raised RuntimeError: stack expects each tensor to be equal size This is the normal case, not an edge case. `fps` sampling under a `max_frames` cap returns as many frames as each clip's duration warrants, so two clips of different lengths produce different frame counts. Sentence Transformers batches whatever the user passes, and sorts by length before doing so, which makes a ragged batch the likely shape rather than an unlucky one. 1. The video processor concatenates instead of stacking. `_preprocess` built `(num_videos, num_frames, max_patches, ...)` by stacking per-video tensors on a new leading axis, which requires a uniform frame count. It now concatenates along the frame axis to `(total_num_frames, max_patches, ...)` and reports `num_frames_per_video` alongside, so the flat sequence can be split back. Padding to the longest video was rejected: it would inflate the vision tower's batch with frames that exist only to be discarded, and the cost is paid by the whole batch, not by the outlier. `num_frames_per_video` is not a new concept in the library. `minicpmv4_6` and `minicpmv4_7` already return a key of that name with that meaning. 2. `get_video_features` takes the frame counts. It previously recovered the per-video split by summing the non-padding mask over both the frame and patch axes, which only works while the frame axis is still nested under a video axis. It now sums per frame and regroups with `num_frames_per_video`. The argument is required, not optional: with the frames concatenated there is no way to recover video boundaries from the tensors alone, and silently treating the batch as one long video would produce a single wrong embedding rather than an error. A 4-D input is no longer accepted. The neighbouring `k_squared` computation is also simplified. `int((max_patches // (max_patches // k**2)) ** 0.5) ** 2` reduces to `k**2`, because `max_patches = max_soft_tokens * pooling_kernel_size**2` is divisible by construction. `gemma4_unified` overrides both `_preprocess` and `get_video_features` wholesale, so it inherits neither fix and carries its own copy of each. It also inherits `Gemma4VideoProcessor.model_input_names`, which `test_model_input_names` compares for exact set equality against the processor's output keys, so declaring the new key in Gemma 4 alone would break it. `embedding_gemma2` inherits both methods and picks them up through the modular converter, but overrides `forward`, whose signature, docstring and call site are updated by hand. Two shared test hooks: - `test_processing_common` learns to read `num_frames_per_video`, next to the existing Qwen `grid_thw` branch, when counting video placeholder tokens. This un-skips `test_apply_chat_template_video` for the Gemma 4 family and for the two MiniCPM-V models, which emit the same key. - `test_video_processing_common` grows an overridable `assert_expected_videos_shape`, used by the four `test_call_*` tests that previously inlined the same assertion. Models whose output is not `(num_videos, num_frames, ...)` override the one method instead of the four tests. Both replace per-model skips and overrides with coverage that actually runs. One numerical note for anyone verifying this by hand. A batched encode and a single-video encode agree to bf16 precision, not exactly. Batching pads the shorter row, SDPA tiles its reduction over the padded key length, and the accumulation order shifts; with 8 significand bits that is visible. It is not specific to video or to this change: a ragged text batch drifts by the same amount, eager attention is padding-invariant and shows none of it, and float32 brings it down to ~1e-7. It is also not a masking bug. Filling the padded slots with pad tokens, zeros, random ids or eos under the same attention mask gives bit-identical outputs at the real positions, and every bf16 configuration sits the same distance from the float32 answer whether padded or not, so padding costs no accuracy. The part this commit actually rewrote is exact: `get_video_features` on a ragged batch reproduces the per-video runs bit for bit. * Preserve chat-template modality order, validate multimodal placeholders, and pin ST >= 6.1.0 - Update `_EMBEDDING_CHAT_TEMPLATE` to emit system prompts first right after `<bos>`, preserve caller-provided content ordering (`{image, text}` vs `{text, image}`), and disable automatic placeholder insertion when manual `<|image|>`, `<|video|>`, or `<|audio|>` markers are present in the text. - Declare `sentence-transformers >= 6.1.0` in `config_sentence_transformers.json` (`requirements`). - Extend `EmbeddingGemma2Processor.validate_inputs` to enforce placeholder-to-input count parity for `<|video|>` and `<|audio|>` alongside `<|image|>`. - Add unit tests in `test_processing_embedding_gemma2.py` covering placeholder count validation (`test_validate_inputs_multimodal_placeholder_counts`) and chat-template ordering / manual placeholders (`test_chat_template_ordering_and_manual_placeholders`). - Update `docs/source/en/model_doc/embedding_gemma2.md` with `<hfoptions>` tabs for multimodal ordering and manual placeholders. * Tighten EmbeddingGemma 2 overview and task prompt docs * Support warning-free partial tower loading and enable audio/video feature tests - Set `base_model_prefix = "language_model"` on `EmbeddingGemma2PreTrainedModel` and ignore `vision_tower.*`, `embed_vision.*`, `audio_tower.*`, and `embed_audio.*` on `EmbeddingGemma2TextModel` so loading the text model directly from the multimodal checkpoint succeeds with zero missing or unexpected keys. - Populate `self._keys_to_ignore_on_load_unexpected` in `EmbeddingGemma2Model.__init__` when `config.vision_config is None` and/or `config.audio_config is None` to suppress unexpected-key load warnings when disabling individual towers. - Guard `EmbeddingGemma2Model.get_image_features` and `get_video_features` with a `ValueError` when `self.vision_tower is None`, matching `get_audio_features`. - Add `test_dynamic_tower_loading_and_guards` in `test_modeling_embedding_gemma2.py` covering `audio_config=None`, `vision_config=None`, both `None`, and direct `EmbeddingGemma2TextModel.from_pretrained`. - Implement `_audio_features_prepare_config_and_inputs` and `_video_features_prepare_config_and_inputs` on `EmbeddingGemma2ModelTest` and remove the 6 `@unittest.skip` overrides for `test_get_audio_features_*` and `test_get_video_features_*`. * Address review nits: add v_norm TP plan and shorten inline comments - Add `"layers.*.self_attn.v_norm": "replicated_with_grad_allreduce"` to `EmbeddingGemma2TextConfig.base_model_tp_plan`. - Shorten inline comments for `self.sliding_window` and `del self.vocab_size_per_layer_input` in `modular_embedding_gemma2.py`. * Consolidate video preprocessing controls into Processor doc section * Declare _keys_to_ignore_on_load_unexpected statically on EmbeddingGemma2Model and revert base_model_prefix - Move `_keys_to_ignore_on_load_unexpected` on `EmbeddingGemma2Model` from `__init__` to a static class attribute. - Revert `base_model_prefix = "language_model"` on `EmbeddingGemma2PreTrainedModel` and remove `_keys_to_ignore_on_load_unexpected` from `EmbeddingGemma2TextModel`. * Support nested and combined multimodal inputs when text is None in EmbeddingGemma2Processor - Unpack nested per-sample `audio` and `videos` lists (`[[a1, a2], [a3]]`) in `EmbeddingGemma2Processor.prepare_inputs_layout`. - Synthesize `<|image|>`, `<|video|>`, and `<|audio|>` placeholders per batch sample across all provided modalities when `text` is omitted, raising `ValueError` on mismatched outer batch sizes. - Add `test_multimodal_and_nested_inputs_without_text` in `test_processing_embedding_gemma2.py`. - Update `embedding_gemma2.md` Processor docs to recommend Sentence Transformers dictionaries for batched multimodal inputs and manual placeholders for exact ordering control. * add EmbeddingGemma2IntegrationTest * Use CONFIG_MAPPING and PreTrainedConfig to avoid Gemma 4 cross-imports in EmbeddingGemma 2 - Replace Gemma4VisionConfig and Gemma4AudioConfig in EmbeddingGemma2Config (sub_configs, type annotations, and __post_init__) with AutoConfig, PreTrainedConfig, and CONFIG_MAPPING lookups ('gemma4_vision' and 'gemma4_audio'). - Annotate multimodal_config as PreTrainedConfig in EmbeddingGemma2MultimodalEmbedder.__init__ so generated configuration_embedding_gemma2.py and modeling_embedding_gemma2.py no longer import from ..gemma4. - Shorten inline comments in EmbeddingGemma2TextModel.__init__. - Remove the EmbeddingGemma2TextModel.from_pretrained sub-test in test_dynamic_tower_loading_and_guards that was accidentally re-introduced in da45411f0a. * Support nested audio lists in make_list_of_audio and simplify EmbeddingGemma2Processor.prepare_inputs_layout - Update make_list_of_audio in audio_utils.py (and fetch_audio in feature_extraction_sequence_utils.py) to flatten nested audio lists (list[list[AudioInput]]) into a flat list of audio inputs. - Call make_batched_videos(videos) and ProcessorMixin.prepare_inputs_layout directly in EmbeddingGemma2Processor.prepare_inputs_layout, recording audio_per_sample and videos_per_sample prior to flattening only when text is None. - Add test coverage in test_processing_embedding_gemma2.py for nested audio and video lists with explicit text placeholders. * Restructure the EmbeddingGemma 2 model doc around multimodal usage - Reorganize "Multimodal embeddings" into five numbered subsections (single modalities and cross-modal retrieval, composed multimodal embeddings, automatic ordering vs. manual placeholders, heterogeneous batching, and disabling unused modality towers), mirroring the scenarios covered by EmbeddingGemma2IntegrationTest. - Document capabilities that had no example before: cross-modal retrieval scored with model.similarity, purely non-text combinations such as image + audio, several items of the same modality in one embedding, and skipping unused towers at load time via vision_config=None / audio_config=None. - Use processor.apply_chat_template in the AutoModel multimodal examples, and upcast pooled embeddings with .float() before F.normalize so the AutoModel tabs match the Sentence Transformers similarity computation. - Make every code example self-contained (imports, model/processor loading, and inputs) so no snippet depends on variables defined in another section, and rewrite the Matryoshka examples to rank the same query/document pair at 256 dimensions instead of only producing a truncated tensor. - Split the Processor section into max_soft_tokens, video frame sampling controls, and direct processor(...) calls, documenting fps, max_frames, overflow_strategy, and add_timestamps individually. * Address final review comments: simplify make_list_of_audio, clarify layout comment, and trim redundant tests * Use full RoPE on EmbeddingGemma2 global attention layers The internal model applies RoPE to all global-layer head dims (global_rope_proportion=1.0, theta 1e6), not proportional RoPE with partial_rotary_factor=0.25. The partial variant caused position-dependent drift on long inputs (e.g. text x100 cos 0.997 -> 0.99994, two images 0.987 -> 0.99988 vs the JAX reference). * repo * fix * revert these * fix docstring --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Jovan <jandonov@google.com> Co-authored-by: Sara Smoot <sarasmoot@google.com> Co-authored-by: raushan <raushan@huggingface.co> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: eustlb <94853470+eustlb@users.noreply.github.com>