Revert "Add error checking for padding modules (#106147)"
This reverts commit 0547b6279d6f7249c0e588508c2561589514d3aa.
Reverted https://github.com/pytorch/pytorch/pull/106147 on behalf of https://github.com/jeanschmidt due to sadly it is breaking internal builds, and I can't coordinate a FF due to timezone differences ([comment](https://github.com/pytorch/pytorch/pull/106147#issuecomment-1661870970))
