pytorch

69e3ee2d - DataLoader: properly diagnose exceeding file descriptor limit (#34768)

DataLoader: properly diagnose exceeding file descriptor limit (#34768) Summary: Fixes https://github.com/pytorch/pytorch/issues/973 Common failure scenario: * DataLoader creates workers and communicates with them through SHMs * Workers send back through an AF_UNIX socket file descriptors to SHMs containing data * The limit of open files gets fully used * A FD gets stripped from a socket message coming back from a worker, without the worker knowing this. * This causes a `RuntimeError: received 0 items of ancdata` in the standard `multiprocessing` package * The exception is not handled by PyTorch and so is presented to the users. After this change the user will see ``` Traceback (most recent call last): File "/home/wbaranowski/git/Quansight/pytorch/torch/utils/data/dataloader.py", line 761, in _try_get_data data = self._data_queue.get(timeout=timeout) File "/home/wbaranowski/miniconda3/envs/pytorch-cuda-dev/lib/python3.6/multiprocessing/queues.py", line 113, in get return _ForkingPickler.loads(res) File "/home/wbaranowski/git/Quansight/pytorch/torch/multiprocessing/reductions.py", line 294, in rebuild_storage_fd fd = df.detach() File "/home/wbaranowski/miniconda3/envs/pytorch-cuda-dev/lib/python3.6/multiprocessing/resource_sharer.py", line 58, in detach return reduction.recv_handle(conn) File "/home/wbaranowski/miniconda3/envs/pytorch-cuda-dev/lib/python3.6/multiprocessing/reduction.py", line 184, in recv_handle return recvfds(s, 1)[0] File "/home/wbaranowski/miniconda3/envs/pytorch-cuda-dev/lib/python3.6/multiprocessing/reduction.py", line 162, in recvfds len(ancdata)) RuntimeError: received 0 items of ancdata During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/wbaranowski/git/Quansight/pytorch/torch/utils/data/dataloader.py", line 787, in _try_get_data fs = [tempfile.NamedTemporaryFile() for i in range(10)] File "/home/wbaranowski/git/Quansight/pytorch/torch/utils/data/dataloader.py", line 787, in <listcomp> fs = [tempfile.NamedTemporaryFile() for i in range(10)] File "/home/wbaranowski/miniconda3/envs/pytorch-cuda-dev/lib/python3.6/tempfile.py", line 551, in NamedTemporaryFile (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type) File "/home/wbaranowski/miniconda3/envs/pytorch-cuda-dev/lib/python3.6/tempfile.py", line 262, in _mkstemp_inner fd = _os.open(file, flags, 0o600) OSError: [Errno 24] Too many open files: '/tmp/tmpnx_f6v_f' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "test_shm_leak.py", line 56, in <module> worker_init_fn=worker_init_fn File "/home/wbaranowski/git/Quansight/pytorch/torch/utils/data/dataloader.py", line 345, in __next__ data = self._next_data() File "/home/wbaranowski/git/Quansight/pytorch/torch/utils/data/dataloader.py", line 861, in _next_data idx, data = self._get_data() File "/home/wbaranowski/git/Quansight/pytorch/torch/utils/data/dataloader.py", line 828, in _get_data success, data = self._try_get_data() File "/home/wbaranowski/git/Quansight/pytorch/torch/utils/data/dataloader.py", line 791, in _try_get_data "Too many open files. Communication with the" RuntimeError: Too many open files. Communication with the workers is no longer possible. Please increase the limit using `ulimit -n` in the shell or change the sharing strategy by calling `torch.multiprocessing.set_sharing_strategy('file_system')` at the beginning of your code ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/34768 Differential Revision: D20538053 Pulled By: ezyang fbshipit-source-id: be4425cf2fa02aff61619b2b829c153cb1a867cb