pytorch

a9583c1f - Vectorize softplus and its backward function on CPU (#32944)

Vectorize softplus and its backward function on CPU (#32944) Summary: The benchmarking shows a huge performance gain (2-7x faster). Also note that I removed Half support because it isn't generally supported on CPU. Benchmark: (Debian 10, Release build, gcc 8.3, no turbo, Intel(R) Xeon(R) E-2136 CPU @ 3.30GHz) ```python import timeit for op in ('Softplus',): print('Forward') for dtype in ('torch.double', 'torch.float'): for n, t in [(10_000, 10000), (100_000, 1000)]: print(f'torch.nn.{op}()(a), numel() == {n} for {t} times, dtype={dtype}') print(timeit.timeit('m(a)', setup=f'import torch; m = torch.nn.{op}(); a = torch.randn({n}, dtype={dtype})', number=t)) print('Backward') for dtype in ('torch.double', 'torch.float'): for n, t in [(10_000, 40000), (100_000, 4000)]: print(f'torch.nn.{op}()(a), numel() == {n} for {t} times, dtype={dtype}') print(timeit.timeit('y.backward(retain_graph=True)', setup=f'import torch; m = torch.nn.{op}(); a = torch.randn({n}, dtype={dtype}, requires_grad=True); x = m(a); y = x.sum()', number=t)) ``` Before: ``` Forward torch.nn.Softplus()(a), numel() == 10000 for 10000 times, dtype=torch.double 3.73130346799735 torch.nn.Softplus()(a), numel() == 100000 for 1000 times, dtype=torch.double 3.6790116359916283 torch.nn.Softplus()(a), numel() == 10000 for 10000 times, dtype=torch.float 2.7477027159911813 torch.nn.Softplus()(a), numel() == 100000 for 1000 times, dtype=torch.float 2.7382752639969112 Backward torch.nn.Softplus()(a), numel() == 10000 for 40000 times, dtype=torch.double 7.037510035006562 torch.nn.Softplus()(a), numel() == 100000 for 4000 times, dtype=torch.double 5.855093962003593 torch.nn.Softplus()(a), numel() == 10000 for 40000 times, dtype=torch.float 3.413616877005552 torch.nn.Softplus()(a), numel() == 100000 for 4000 times, dtype=torch.float 2.5485514330066508 ``` After: ``` Forward torch.nn.Softplus()(a), numel() == 10000 for 10000 times, dtype=torch.double 0.9465823079954134 torch.nn.Softplus()(a), numel() == 100000 for 1000 times, dtype=torch.double 0.8799468770012027 torch.nn.Softplus()(a), numel() == 10000 for 10000 times, dtype=torch.float 0.39715987400268205 torch.nn.Softplus()(a), numel() == 100000 for 1000 times, dtype=torch.float 0.3563060039887205 Backward torch.nn.Softplus()(a), numel() == 10000 for 40000 times, dtype=torch.double 2.400547721001203 torch.nn.Softplus()(a), numel() == 100000 for 4000 times, dtype=torch.double 1.4740848699875642 torch.nn.Softplus()(a), numel() == 10000 for 40000 times, dtype=torch.float 1.6684603010071442 torch.nn.Softplus()(a), numel() == 100000 for 4000 times, dtype=torch.float 0.6815649690106511 ``` Pull Request resolved: https://github.com/pytorch/pytorch/pull/32944 Differential Revision: D19725407 Pulled By: VitalyFedyunin fbshipit-source-id: 7430de838df731bd17617eff63f10107d5ad6b8b