Fix persistent worker exits before pin_memory thread (#72269)
* release 1.11 Install torch from test channel, Pin builder and xla repo (#72217)
* Fix persistent worker exits before pin_memory thread
ghstack-source-id: 2d15b14df2e2d84b309081dffbedc4836495ae95
Pull Request resolved: https://github.com/pytorch/pytorch/pull/71579
Co-authored-by: Andrey Talman <atalman@fb.com>
