pytorch
1903b32c - Directly Return when Numel == 0 for WeightedSum and ScatterWeightedSum

Commit
3 years ago
Directly Return when Numel == 0 for WeightedSum and ScatterWeightedSum Summary: Current Caffe2 operators WeightedSum and ScatterWeightedSum will enforce that the first input is not empty; otherwise they will raise error. However, in some cases we will have 0 batch size in training and eval. For example, when training and eval current AF and AI OC models, we will filter out the search ads in data pipeline, which might cause 0 batch size in some iterations. As a result, if the models are using Dper3 modules that contains WeightedSum or ScatterWeightedSum (e.g., HistogramBinningCalibration module), they will occasionally fail in training or eval. To address this issue, we revise the implementation of WeightedSum and ScatterWeightedSum so that we will directly return when their first inputs are empty without failing the operators. Test Plan: We tested the code change by building a Dper3 backend canary package. All the jobs for AF and AI OC succeeded with the modified Caffe2 operators: f251058001 f251058142 f251058332 To compare, all the jobs with identical model configs but with the canary package built from master failed: f250993908 f250994106 f250994174 Reviewed By: chenshouyuan, huayuli00 Differential Revision: D26444645 fbshipit-source-id: 1c2f81a078810e3ef3c17c133a715090dee2c0ff
Author
Yangxin Zhong
Parents
Loading