Clean up usage of torch._six partially (#49785)
Summary:
See https://github.com/pytorch/pytorch/issues/42919
Pull Request resolved: https://github.com/pytorch/pytorch/pull/49785
Reviewed By: mruberry
Differential Revision: D25963833
Pulled By: bugra
fbshipit-source-id: 11c90d6b8d3f206c9d0a4d8621b773beb10c6ba2
