[di] avoid copying optional input for get_real_inputs_from_optional_inputs_v2 when possible (#81137)
Summary: avoid copies and casting by moving out of the inputs list
Differential Revision: D37572556
Pull Request resolved: https://github.com/pytorch/pytorch/pull/81137
Approved by: https://github.com/mikeiovine
