transformers
e872ca25 - Fix CpmAnt loading: size lm_head to vocab_size (#48012)

Commit
78 days ago
Fix CpmAnt loading: size lm_head to vocab_size (#48012) * Fix CpmAnt loading: size lm_head to vocab_size The released openbmb/cpm-ant-10b checkpoint ships lm_head.weight with shape (vocab_size, hidden) = (30720, 4096), bit-identical to the first vocab_size rows of input_embedding.weight (31744, 4096). The trailing prompt_types * prompt_length rows are soft-prompt embeddings and are never decoding targets. The model built the head at vocab_size + prompt_types * prompt_length and declared a whole-tensor tie to the embedding, so from_pretrained hit a shape mismatch and then crashed in tie_weights comparing meta tensors. Size the head to vocab_size, which matches the checkpoint, and drop the whole-tensor tie mapping that could never have been valid at these shapes. * Shorten lm_head comment
Author
Parents
Loading