pytorch

d312aeb6 - Implement faster gradcheck but not enabled for most things (#54480)

Implement faster gradcheck but not enabled for most things (#54480) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/54480 This PR shouldn't really change the behavior of gradcheck for most ops. However, the changes in test_autograd allow us to run basic checks for both fast and slow (instead of previously just slow). All it should be doing is wrapping the preexisting tests we introduced in prior PRs in a function which takes `fast_mode` as a param. We then call this function twice, once with `fast_mode=True` and once with `fast_mode=False`. Plan for rollout: - This PR should only land the code (and runs some basic checks as described above). - This should help us verify that a) slow is still working as expected b) basic functionality of fast works - After we land this, but before we run the next PR in the stack, we should land https://github.com/pytorch/pytorch/pull/55182. This is to ensure that there is no gap where the slow tests aren't running. - The next PR is responsible for enabling the fast_mode=True flag on all tests (where the function has real inputs/outputs), and selectively disabling for the cases the fail. - Finally in a later PR, we reenable fast-gradcheck for functions w/ complex inputs/outputs TODOs and open questions (not necessarily blocking this PR): - ~How do we think about atol/rtol~ (scale atol, keep rtol as-is) - ~reenable fast-gradcheck for complex numbers~ - ~when inputs are uncoalesced we don't truly test this case because we coalesce the inputs before calling function. Revisit this when https://github.com/pytorch/pytorch/pull/52874/files is landed~ ### Developer Experience Sample output when jacobian mismatch occurs: ``` Traceback (most recent call last): File "/home/s/local/pytorch4/test/test_autograd.py", line 4220, in test_gradcheck_jacobian_mismatch check(fast_mode=True) File "/home/s/local/pytorch4/test/test_autograd.py", line 4196, in check gradcheck(fn, (x,), fast_mode=fast_mode) File "/home/s/local/pytorch4/torch/testing/_internal/common_utils.py", line 2067, in gradcheck return torch.autograd.gradcheck(fn, inputs, **kwargs) File "/home/s/local/pytorch4/torch/autograd/gradcheck.py", line 1020, in gradcheck if not fast_gradcheck(fail_test, seeded_func, func_out, tupled_inputs, outputs, eps, rtol, File "/home/s/local/pytorch4/torch/autograd/gradcheck.py", line 915, in fast_gradcheck return fail_test(get_notallclose_msg(a, n, i, j, prefix) + jacobians_str) File "/home/s/local/pytorch4/torch/autograd/gradcheck.py", line 996, in fail_test raise RuntimeError(msg) RuntimeError: Jacobian mismatch for output 0 with respect to input 0, numerical:tensor(0.9195) analytical:tensor(0.9389) The above quantities relating the numerical and analytical jacobians are computed in fast mode. See: https://github.com/pytorch/pytorch/issues/53876 for more background about fast mode. Below, we recompute numerical and analytical jacobians in slow mode: Numerical: tensor([[1.0000, 0.0000, 0.0000, 0.0000], [0.0000, 1.0000, 0.0000, 0.0000], [0.0000, 0.0000, 1.0000, 0.0000], [0.0000, 0.0000, 0.0000, 1.0000]]) Analytical: tensor([[1.0100, 0.0100, 0.0100, 0.0100], [0.0100, 1.0100, 0.0100, 0.0100], [0.0100, 0.0100, 1.0100, 0.0100], [0.0100, 0.0100, 0.0100, 1.0100]]) The max per-element difference (slow mode) is: 0.010000000000054632. ``` Additionally, if the per-element difference is small i.e., `allclose(analytical_slow, numerical_slow, rtol, atol) is True` we follow up with this message: ``` Fast gradcheck failed but element-wise differences are small. This means that the test might've passed in slow_mode! If you are adding a new operator, please file an issue and then use one of the workarounds. The workaround depends on how your test invokes gradcheck/gradgradcheck. If the test - manually invokes gradcheck/gradgradcheck, then call gradcheck/gradgradcheck with `fast_mode=False` as a keyword argument. - is OpInfo-based (e.g., in test_ops.py), then modify the OpInfo for the test to have `gradcheck_fast_mode=False` - is a Module test (e.g., in common_nn.py), then modify the corresponding module_test entry to have `gradcheck_fast_mode=False` ``` Test Plan: Imported from OSS Reviewed By: walterddr, ejguan Differential Revision: D27825160 Pulled By: soulitzer fbshipit-source-id: 1fe60569d8b697c213b0d262a832622a4e9cf0c7