Honor a concrete dtype in AutoModel for composite checkpoints (#46514)
* Honor a concrete dtype in AutoModel for composite checkpoints
AutoModelForCausalLM.from_pretrained(..., dtype=X) silently ignored a concrete dtype X
for composite/multimodal checkpoints (e.g. Qwen3.5). Only dtype="auto" was popped before
building the config, so a concrete value was absorbed onto the composite config by AutoConfig
and dropped when the factory swapped to the text sub-config, leaving the model on the saved
dtype.
Pop a concrete dtype before building the config and re-inject it afterwards, mirroring the
adjacent quantization_config handling. The saved dtype is kept by default and overridden only
when a dtype is passed.
Add test_automodelforcausallm_dtype to the Qwen3.5 and Qwen3.5-MoE testers.
* Assert the VLM's weights, not just its config, load in the requested dtype
* Keep the user-provided dtype on the composite config for BC and drop the redundant default-dtype guard in tests
* Keep the resolved dtype on the top-level config in `_from_config`
For leaf configs without sub-configs, `_from_config` built the weights in the
requested dtype but left `config.dtype` stale (None), so weights and config
disagreed. Set it on the top-level config too, matching `from_pretrained`.
Fixes #46512
---------
Co-authored-by: Raushan Turganbay <raushan@huggingface.co>