Revert "[functorch] test: try using reference_inputs in vmap tests (#91355)"
This reverts commit a51090d4b14610d72a8e22209a7d69b5a90bf45d.
Reverted https://github.com/pytorch/pytorch/pull/91355 on behalf of https://github.com/kshitij12345 due to Broke trunk