Don't hold ThreadPool lock when destructing task (#56817)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/56817
Fix https://github.com/pytorch/pytorch/issues/56701 and https://github.com/pytorch/pytorch/issues/56786
Signed-off-by: Edward Z. Yang <ezyang@fb.com>
Test Plan: Imported from OSS
Reviewed By: pritamdamania87
Differential Revision: D27975642
Pulled By: ezyang
fbshipit-source-id: b7f4a6c18a4fa65c38bacc7c46246f0865c95f86
