Unify flags_for_effects handling (#51238)
There was two places where we were turning Effects into flags, one in
the optimizer, one in inference. Unify these two both use the same
function, but we need to refactor a bit to make sure all the effects get
passed to the correct place.
---------
Co-authored-by: Shuhei Kadowaki <aviatesk@gmail.com>
