pytorch

a1c67b07 - Silence harmless error logs of TensorPipe agent during shutdown (#51785)

Silence harmless error logs of TensorPipe agent during shutdown (#51785) Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/51785 The TensorPipe pipes do not really support a "graceful" shutdown: if one side is expecting data (i.e., it has scheduled a readDescriptor call) and the other side closes, the former will receive an error. Such an error will not even be predictable, as it depends on the backend: some may detect this and report it "well" (through an EOFError), others may not be able to tell this apart from a failure and report it as such. This meant that during shutdown some of these errors would fire and thus the agent would log them as warning. We did add a note that these were expected under some conditions, so that users wouldn't be alarmed, but it was still a far-from-ideal experience. In principle we could build a "protocol" on top of these pipes to "agree" on a graceful shutdown, and this was the plan to solve this. However, it was rather complicated to implement. Here I am proposing a quicker, but perhaps hackier, solution, which re-uses the already existing graceful shutdown "protocol" of the agent (i.e., the `join` method) to put the agent in a special state in which it will silence all errors due to a remote shutting down. Such a check cannot happen in the `shutdown` method, because that's also used in case of ungraceful shutdown (in which case I believe we'd still want to display errors). Since it needs to make sure that all participants have transitioned to this new state before any of them can continue (as otherwise one of them may close its pipes before another one has realized that this is now expected), we need to perform a barrier. Hence the ideal place for it is the `join` method, where we're already doing a lot of gang-wide synchronization. Since the `join` method isn't only called during shutdown, we need to make sure we only switch the agent to this state when it's the last call to join, and we do so by adding a new optional argument to it (which will be ignored by all agents except the TensorPipe one). I realize this isn't the prettiest solution, and since it changes the agent's API it's worth discussing it carefully. Let me know what you think! ghstack-source-id: 121131940 Test Plan: Run on CircleCI, where this occurred quite a bit, and check the logs. Reviewed By: mrshenli Differential Revision: D26276137 fbshipit-source-id: 69ef14fe10908e80e627d9b4505352e482089cc8