Remove getattr overhead in spda (#934)
* remove getattr
* nit
* profiling script
* fix
* remove getattr overhead
* remove getattr overhead
* simplification
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix
* fix issues
* fix test
* fix on transformers version
* fix
