Add check for no grad in transformer encoder nestedtensor conversion (#78832) (#78832) (#79029)
Summary:
Before, we allowed inputs with grad to be converted to NestedTensors. Autograd attempts to find the size of the NestedTensor, but NestedTensor throws an exception for its size function. This causes all calls to nn.TransformerEncoder with grad enabled to fail.
Fix: we add a check for no grad in transformer encoder so we do not convert tensor with grad to nestedtensor.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/78832
Approved by: https://github.com/cpuhrsch, https://github.com/jbschlosser
Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/1f819ee965894b8332cb364a67c91855c91c9dcc
Reviewed By: frank-wei, mikekgfb
Differential Revision: D36907614
Pulled By: erichan1
fbshipit-source-id: 576be36530da81c1eff59ac427ae860bfb402106
