pytorch

e9c6a511 - [torchelastic] Make sure torchelastic mp wait for queue to be drained before finishing the process

[torchelastic] Make sure torchelastic mp wait for queue to be drained before finishing the process Summary: The diff resolves bug where worker processes could exit before torchelastic process would read the return values. This is a rare event, but still can happen, e.g. https://fb.workplace.com/groups/319878845696681/permalink/512409069776990/ When users want to return torch.Tensor object from worker process, the torchelastic multiprocessing will fail. Currently worker process finishes its job after it writes output to the IPC queue without receiver process confirmation. When this happens, the underlying channel between worker and torchelastic process could be closed (in case of mp.SimpleQueue it is file descriptors, that is why we see FileNotFoundException: since worker process finished execution, the file descriptor just got deleted, and torchelastic process cannot find it). Test Plan: buck test mode/dev-nosan //caffe2/test/distributed/elastic/agent/server/test:local_agent_test User workflow: f263531643 Reviewed By: cbalioglu, wilson100hong Differential Revision: D27572158 fbshipit-source-id: 9a360468acc98d85d587ebf223e7e96d4b43fe4b