remove dependency to fp32 batch permutation op (#15723)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/15723
As title says.
Reviewed By: jianyuh
Differential Revision: D13578604
fbshipit-source-id: 0da0ac31ae83c1e0daa9077e878feb4deffed6a3
