pytorch
26046b91 - [caffe2][publish] Optimize metanetdef load

Commit
3 years ago
[caffe2][publish] Optimize metanetdef load Summary: When loading optional blobs from a large file to workspace, for instance: https://fburl.com/diffusion/l0mcnofg, we are currently loading the file in multiple times. https://fburl.com/diffusion/qhbpyq0e This diff optimized the load time by loading in the large model file only once, and using the allow_incomplete arg into LoadOp. The implementation of the LoadOp with this arg previously did not delete the blobs that were not found, which is also fixed in this diff. Test Plan: Existing unit tests: ``` buck test //caffe2/caffe2/fb/distribute/tests:meta_net_def_storage_utils_test ``` Many sandcastle integration tests. scuba logs: https://fburl.com/scuba/dai_modelstore/txdf3pjt Reviewed By: TailofJune Differential Revision: D27575622 fbshipit-source-id: 7c2b25ef603a378e87ebdbe349c94c2f1952493c
Author
Georgia Hong
Parents
Loading