[TensorPipe] Acquire lock when adding message to timeout map (#39398)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/39398
The `timeoutMapMutex_` was only used to guard accesses in the timeout thread, but it should have been used also to guard accesses in the `send` method.
The way I found this bug is rather odd. A test was failing because a timeout of 0.5 seconds was firing when it wasn't supposed to. The test was built with TSAN enabled and the point where we were wasting those 500ms was precisely when accessing the `timeoutMap_` in the `send` method. There is of course no reason it would take so long, so I suspect that either such an access triggered a whole lot of lengthy checks in TSAN or, perhaps, that TSAN was delaying it on purpose because it thought it was smelly and wanted to see whether it could cause a race.
ghstack-source-id: 105088618
Test Plan: The test started passing.
Differential Revision: D21838465
fbshipit-source-id: 02cf2bf1fef2e97da99b9c4e77070fe35d2bcbb0
