pytorch

058d7228 - Expose the interface of nesterov of SGD Optimizer from caffe2 to dper

Expose the interface of nesterov of SGD Optimizer from caffe2 to dper Summary: Expose the interface of `nesterov` of SGD Optimizer from caffe2 to dper. dper sgd optimizer (https://fburl.com/diffusion/chpobg0h) has referred to NAG sgdoptimizer in caffe2: https://fburl.com/diffusion/uat2lnan. So just need to add the parameter 'nesterov' in dper sgd optimizer. Analysis of run resutls: N345540. - train_ne increases as momentum (m) decreases. - for m=0.95, 0.9: eval_ne is lower with NAG than production (no NAG, m = 0.95). - for m=0.99: eval_ne with or without NAG is higher than production. It indicates larger variance in validation and overfit in training (lower train_ne). Test Plan: 1. unit tests: `buck test caffe2/caffe2/fb/dper/layer_models/tests/split_1:sparse_nn_test -- test_sgd_without_nesterov` `buck test caffe2/caffe2/fb/dper/layer_models/tests/split_1:sparse_nn_test -- test_sgd_with_nesterov` . 1. build dper front end package: `flow-cli canary ads.dper3.workflows.sparse_nn.train --mode opt --entitlement ads_global --run-as-secure-group team_ads_ml_ranking`. The build result (refreshed) is here https://www.internalfb.com/intern/buck/build/2a368b55-d94b-45c1-8617-2753fbce994b. Flow package version is ads_dper3.canary:856b545cc6b249c0bd328f845adeb0d2. . 2. To build dper back end package: `flow-cli canary dper.workflows.dper3.train --mode opt --entitlement ads_global --run-as-secure-group team_ads_ml_ranking`. The build result (refreshed) is here: https://www.internalfb.com/intern/buck/build/70fa91cd-bf6e-4a08-8a4d-41e41a77fb52. Flow package version is aml.dper2.canary:84123a34be914dfe86b1ffd9925869de. . 3. Compare prod with NAG-enabled runs: a) refreshed prod run (m=0.95): f213877098 NAG enabled run (m=0.95): f213887113 . b) prod run (m=0.9): f214065288 NAG enabled run (m=0.9): f214066319 . c) prod run (m=0.99): f214065804 NAG enabled run (m=0.99): f214066725 . d) change date type of nestrov to `bool` and launched a validation run NAG enabled (m=0.95): f214500597 Reviewed By: ustctf Differential Revision: D23152229 fbshipit-source-id: 61703ef6b4e72277f4c73171640fb8afc6d31f3c