Fix safetensors failing tests (#27231)
* Fix Kosmos2
* Fix ProphetNet
* Fix MarianMT
* Fix M4T
* XLM ProphetNet
* ProphetNet fix
* XLM ProphetNet
* Final M4T fixes
* Tied weights keys
* Revert M4T changes
* Apply suggestions from code review
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>
---------
Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com>