Remove more unused attributes in config classes (#21543)
* Remove unused decoder_layerdrop
* Update SPECIAL_CASES_TO_ALLOW for MT5Config
* Remove unused position_embedding_init_scale
* Remove unused decoder_max_relative_position
* Use unused decoder_max_relative_position
* Remove unused init_std
* Remove unused forgotten attributes
* Remove unused patch_norm
* Remove unused max_seq_len
* Update SPECIAL_CASES_TO_ALLOW for OneFormerConfig
---------
Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>