Simplify data structures, add uniform approximation, fix mem leak (#63162)
Summary: Pull Request resolved: https://github.com/pytorch/pytorch/pull/63162
Test Plan: Imported from OSS
Reviewed By: mrshenli
Differential Revision: D30284617
Pulled By: andwgu
fbshipit-source-id: 9bd9e5f89abcc0d3dac56b85d55cc88e843baa9f
