Inference changes for incorporating meta loading checkpoint (#4692)
1. In both files, the same logic was done that if when it is meta no
need to move the tensors to the device.
2. Deletion of an unused member of the class
---------
Co-authored-by: Olatunji Ruwase <olruwase@microsoft.com>