Save some little memory in `default_collate` (#61424)
Summary:
It can be a non-little save if there are many workers and a large batch size.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/61424
Reviewed By: soulitzer
Differential Revision: D29635477
Pulled By: ejguan
fbshipit-source-id: 1fc48b5964e873bd8833ad81bed9d51b0b6d137e
