pytorch

289e9a07 - Move Tanh backward to Aten(CPU+CUDA) (#30224)

Move Tanh backward to Aten(CPU+CUDA) (#30224) Summary: VitalyFedyunin, This PR is about port Tanh backward to Aten: **Test script:** ``` import torch import torch.nn as nn import time torch.manual_seed(0) def _time(): if torch.cuda.is_available(): torch.cuda.synchronize() return time.time() device = "cpu" m = nn.Tanh() if torch.cuda.is_available(): device = "cuda" m = m.cuda() #warm up for n in [100, 10000]: input = torch.randn(128, n, requires_grad=True, device=device) grad_output = torch.ones(128, n, device=device) for i in range(1000): output = m(input) output.backward(grad_output) for n in [100, 10000]: input = torch.randn(128, n, requires_grad=True, device=device) grad_output = torch.ones(128, n, device=device) bwd_t = 0 for i in range(10000): output = m(input) t1 = _time() output.backward(grad_output) t2 = _time() bwd_t = bwd_t + (t2 - t1) bwd_avg = bwd_t / 10000 * 1000 print("input size(128, %d) backwad avg time is %.2f (ms)." % (n, bwd_avg)) ``` Test Device: CPU: skx-8180, GPU: Tesla P40. Perfromance: Before: ``` GPU: input size(128, 100) backwad avg time is 0.12 (ms). input size(128, 10000) backwad avg time is 0.17 (ms). CPU input size(128, 100) backwad avg time is 0.05 (ms). input size(128, 10000) backwad avg time is 0.35 (ms). ``` After: ``` GPU: input size(128, 100) backwad avg time is 0.12 (ms). input size(128, 10000) backwad avg time is 0.17 (ms). CPU input size(128, 100) backwad avg time is 0.04 (ms). input size(128, 10000) backwad avg time is 0.25 (ms). ``` `OMP_NUM_THREADS=1:` ``` Before: input size(128, 100) backwad avg time is 0.03 (ms). input size(128, 10000) backwad avg time is 1.85 (ms). After: input size(128, 100) backwad avg time is 0.02 (ms). input size(128, 10000) backwad avg time is 1.16 (ms). ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/30224 Differential Revision: D18810045 Pulled By: VitalyFedyunin fbshipit-source-id: ab37948ab8f76bdaf9f3d1388562eaf29dacc0ea