[SPMD] Change assertions to raising value errors in distributed.py (#54825)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/54825
These assertions are tested in test_c10d.py
Context: https://github.com/pytorch/pytorch/pull/54454#discussion_r602657818
ghstack-source-id: 125602462
Test Plan: buck test mode/dev-nosan caffe2/test/distributed:c10d -- test_ddp_multi_device_module_config
Reviewed By: rohan-varma
Differential Revision: D27381649
fbshipit-source-id: 9b994e9c2acf796770c2f2af2cebdd5561834d14
