Properly avoid wrapping numbers as tensors before backend (#96193)
This partially reverts https://github.com/pytorch/pytorch/pull/96051 with a proper fix.
Signed-off-by: Edward Z. Yang <ezyang@meta.com>
Pull Request resolved: https://github.com/pytorch/pytorch/pull/96193
Approved by: https://github.com/jansel
