[RFC] Don't materialize ignored modules for FSDP (#108032)
Per title. This seems needed for cases where I have a large embedding
I want to separately manage, but FSDP would initialize it and thus consume the
memory.
Currently the interaction with torchdistX materialize_module is not tested,
this can be done as follow up work.
Differential Revision: [D48722046](https://our.internmc.facebook.com/intern/diff/D48722046/)
Pull Request resolved: https://github.com/pytorch/pytorch/pull/108032
Approved by: https://github.com/awgu
