Allow recency weight pooling for fp16 (#20506)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/20506
as titled
Reviewed By: alex1o1o7cloud
Differential Revision: D15342758
fbshipit-source-id: 89e7cb6d7b9511ef6c70611359736328571d7fc0