remove multisets from work set (#21222)
Summary:
Pull Request resolved: https://github.com/pytorch/pytorch/pull/21222
ghimport-source-id: 0eb6daa92bef68a35bef918c3f3a791b401812aa
Reviewed By: jamesr66a
Differential Revision: D15592319
Pulled By: suo
fbshipit-source-id: 895d26538ba1edcd73b83147a68b7e4069084230
