fix double copying of constants (#24412)
Summary:
Fix for https://github.com/pytorch/pytorch/issues/24369
Pull Request resolved: https://github.com/pytorch/pytorch/pull/24412
Test Plan: Imported from GitHub, without a `Test Plan:` line.
Differential Revision: D16843311
Pulled By: eellison
fbshipit-source-id: b25552c49b963c031c98749bcda31f65cd82f19d
