🚨 Allow `check_model_inputs` in core VLMs (#40342)
* allow `check_model_inputs` in core VLMs
* address comments
* fix style
* why this didnt fail prev?
* chec for Noneness instead
* batch update vlms
* fix some tests
* fix copies
* oops delete
* fix efficientloftr
* fix copies
* i am stupid, fix idefics
* fix GC
* return type and other comments
* we shouldn't manually change attention anymore
* fix style
* fix copies
* fix the test
