transformers
5909a467 - Always tie embeddings for LongT5 and Pop2Piano (#47620)

Commit
50 days ago
Always tie embeddings for LongT5 and Pop2Piano (#47620) * Always tie structural weights, regardless of `tie_word_embeddings` `tie_word_embeddings` only controls whether the output embeddings are tied to the input embeddings, but `get_expanded_tied_weights_keys` was returning an empty mapping whenever it was False, dropping the structural entries as well. For encoder-decoder models, `encoder.embed_tokens` and `decoder.embed_tokens` are aliases of `shared` and are absent from the checkpoints, so they ended up randomly initialized (e.g. longt5, which has `tie_word_embeddings=False`). Only drop the entries pointing at `get_output_embeddings()` instead. * Restore pre-#41580 tying expectations for fsmt/mbart `tie_word_embeddings=False` only unties the output head; the encoder/decoder embedding aliases are structural and stay tied. These two counts were 2 from #26292/#26422 until #41580 bumped them to 3/4 to match the all-or-nothing gate. * Add T5 tying test and document regex-form key handling * Always tie embeddings for LongT5 and Pop2Piano Reverts the framework-level change: `_tied_weights_keys` is an all-or-nothing mapping gated by `tie_word_embeddings`, so `get_expanded_tied_weights_keys` should not split it into "structural" and "head" entries. `modeling_utils` and the fsmt/mbart/t5 test expectations are back to their state on main. The actual bug is per-model. LongT5 and Pop2Piano are derived from T5 and inherit its `tie_word_embeddings` quirk: the flag historically encoded "do not scale the decoder outputs" (T5 1.1), not "do not share the embeddings". Every official checkpoint sets it to `False` (`google/long-t5-local-base`, `google/long-t5-tglobal-base`, `Stancld/longt5-tglobal-large-16384-pubmed-3k_steps`, `sweetcocoa/pop2piano`) while only storing `shared.weight`, so the aliased embeddings end up randomly initialized. Apply the same fix T5/MT5/UMT5 already have: force `tie_word_embeddings=True` and move the scaling to its own `scale_decoder_outputs` flag. Before, on `google/long-t5-local-base`: encoder.embed_tokens.weight | MISSING decoder.embed_tokens.weight | MISSING lm_head.weight | MISSING shared std 10.09 vs embed std 1.00 Pop2Piano is affected too, though only latently: the hub ships a `.bin`, which preserves shared storage, but a safetensors round-trip stores `shared.weight` alone and both `embed_tokens` come back randomly initialized. * Shorten comments
Author
Parents
Loading