transformers
856b32a8 - Align image feature dtype in kosmos2 and kosmos2_5 embedding merge (#47691)

Commit
111 days ago
Align image feature dtype in kosmos2 and kosmos2_5 embedding merge (#47691) Both models move the vision features to the text device but not to the text dtype before merging them into `inputs_embeds` with an advanced index assignment. Under `torch.autocast`, `inputs_embeds` comes from `nn.Embedding`, which is not on the autocast cast list and stays float32, while `image_embeds` comes from a projection ending in `nn.Linear` and is cast to bfloat16. `index_put_` does not type promote, so the forward pass raises: RuntimeError: Index put requires the source and destination dtypes match, got Float for the destination and BFloat16 for the source. Passing the dtype alongside the device matches what idefics2, idefics3 and modernvbert already do at the same merge point. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Author
Parents
Loading