Only allow numbers as tensors if operator was explicitly allowlisted so (#80587)
Fixes https://github.com/pytorch/pytorch/issues/80508
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/80587
Approved by: https://github.com/ngimel
