[RFC][FSDP] Don't move ignored params / buffers to device (#108033)
Since these are ignored by FSDP, don't move them.
Differential Revision: [D48727044](https://our.internmc.facebook.com/intern/diff/D48727044/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/108033
Approved by: https://github.com/awgu
ghstack dependencies: #108032
