Revert "Add -Werror=non-virtual-dtor (#80584)"
This reverts commit 76700358624053b2267b1a3c8697cefb9ee3c018.
Reverted https://github.com/pytorch/pytorch/pull/80584 on behalf of https://github.com/malfet due to Broke nighly builds, see https://github.com/pytorch/pytorch/runs/7209779559?check_suite_focus=true
