Don't build nvfuser benchmarks by default (#67857)
None of the other benchmarks are built by default, so this seems unnecessary.
cc @jjsjann123
Pull Request resolved: https://github.com/pytorch/pytorch/pull/67857
Approved by: https://github.com/jjsjann123, https://github.com/janeyx99
