Update DALLE2_pytorch (#1490)
Summary:
This PR does the following updates to DALLE2_pytorch model.
- remove memory copies
- update dalle2 to the latest version with upstream
Also fix the stableness issue of DALLE2_pytorch https://github.com/pytorch/benchmark/issues/1409
DALLE2_pytorch obtains about 13X speedup for inference and trivial speedup for training on A100.
Related discussion https://github.com/lucidrains/DALLE2-pytorch/issues/281
Pull Request resolved: https://github.com/pytorch/benchmark/pull/1490
Reviewed By: nmacchioni
Differential Revision: D44259970
Pulled By: xuzhao9
fbshipit-source-id: eae2443f90d4e56c7b6973887645ab7ebad74dbf
