Move VLM conversions to the main mapping (#44627)
* oke, let's see
* revert core and check
* ooops
* nope revert it back
* forgot revert
* simplify loading changes
* make sure classes will be reset
* fix modular
* these don't recurse so copy
* a few typos in model type and regex
* another fix: model type has underscores unlike class names
* rebase brought an extra closing bracket, delete
* how did it end up here?
* fix repo
* t5 gemma is back
* revert not related
---------
Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com>
