pytorch

8ed20b3f - Leak Caffe2 threadpool in child processes right after fork to prevent segfault (#54895)

Leak Caffe2 threadpool in child processes right after fork to prevent segfault (#54895) Summary: ## Problem summary Fixes https://github.com/pytorch/pytorch/issues/54752 - when the number of threads is more than 3 and at least one `set_num_threads` invocation has taken place before forking child processes by the dataloader, `set_num_threads(1)` in the child process causes a segfault, as during its invocation, the child process is made to handle the data structures of the Caffe2 thread-pool of the parent process, whose data structures it inherits from the parent process (these threads don't exist in the child process, but some of its data structures do, due to the copy-on-write technique used by `fork`). ## Solution malfet [advised](https://github.com/pytorch/pytorch/issues/54752#issuecomment-810315302) & [authored code](https://github.com/pytorch/pytorch/pull/54895#pullrequestreview-625670122) for adding a `pthread_atfork` handler in `pytorch/caffe2/utils/threadpool/pthreadpool-cpp.cc`, that's invoked in the child process right after fork, to leak the Caffe2 thread-pool (the child inherits the thread-pool's data structures from its parent process, but doesn't actually have those threads, since after `fork` , a child process only has one thread). ## Additional changes Added unittest `test_no_segfault` to test for this issue in `test_dataloader.py` Also enabled `test_segfault` (which actually makes sure that segfaults happen in worker processes in a particular case). Pull Request resolved: https://github.com/pytorch/pytorch/pull/54895 Reviewed By: zhangguanheng66 Differential Revision: D27542253 Pulled By: malfet fbshipit-source-id: 10f9c67ce1ff1aa37d3efebf405bd93f7f9d2489