Stabilize low-precision custom autoencoder RMS normalization (#13316)
* Stabilize low-precision custom autoencoder RMS normalization
* Add fp8/4
* Apply style fixes
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>