transformers
5fd67cf1 - Fix CLVP `c_proj` residual scaled-init silently never applied (#47478)

Commit
6 days ago
Fix CLVP `c_proj` residual scaled-init silently never applied (#47478) * Fix CLVP c_proj residual scaled-init never being applied The GPT-2 depth-scaling of residual-projection weights was gated in _init_weights on `isinstance(module, ClvpForCausalLM)` and then an exact parameter-name check `name == "c_proj.weight"`. Because named_parameters() yields fully-qualified dotted names (e.g. model.decoder.layers.0.mlp.c_proj.weight), that check never matched, so c_proj kept the default 0.02 init instead of initializer_range / sqrt(2 * num_hidden_layers). Apply the scaling on the concrete ClvpDecoderMLP module that owns c_proj, mirroring the GPT-2 fix. Resolve the decoder sub-config since self.config is the composite ClvpConfig when the full model is initialized (referencing num_hidden_layers/initializer_range on it directly would otherwise raise). Same root cause as the GPT-2 bug in #47458. * Address review: drop flaky init test and trim comment Per maintainer feedback on #47478: remove the expected-value-of-random-init test (can become flaky in CI) and trim the c_proj scaled-init comment to mirror the GPT-2 fix in #47459. --------- Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com> Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
Author
Parents
Loading