[cache] Simplify cache dispatch based on layer_types (#47118)
* simplify
* unify
* fix comment
* fix
* fix
* Update src/transformers/cache_utils.py
Co-authored-by: Rémi Ouazan <83456801+remi-or@users.noreply.github.com>
* Update src/transformers/cache_utils.py
Co-authored-by: Rémi Ouazan <83456801+remi-or@users.noreply.github.com>
---------
Co-authored-by: Rémi Ouazan <83456801+remi-or@users.noreply.github.com>