Revert "[mobile] Fix lightweight dispatch OOM error by introducing selective build"
This reverts commit 272bdb1442ee3750861d9f2f10690cc3f1521b92.
Reverted https://github.com/pytorch/pytorch/pull/78983 on behalf of https://github.com/osalpekar due to broke internal mobile tests
