pytorch
9e137ee5 - more numerically stable cosine_similarity

Commit
3 years ago
more numerically stable cosine_similarity **Previous behavior**: compute inner product, then normalize. **This patch**: first normalize, then compute inner product. This should be more numerically stable because it avoids losing precision in inner product for inputs with large norms. By design ensures that cosine similarity is within `[-1.0, +1.0]`, so it should fix [#29442](https://github.com/pytorch/pytorch/issues/29442). P.S. I had to change tests because this implementation handles division by 0 differently. This PR computes cosine similarity as follows: <x/max(eps, ||x||), y/max(eps, ||y||)>. Let f(x,y) = <x,y>/(||x|| * ||y||), then df/dx = y/(||x|| * ||y||) - (||y||/||x|| * <x,y> * x)/(||x|| * ||y||)^2. The changed test checks division by zero in backward when x=0 and y != 0. For this case the non-zero part of the gradient is just y / (||x|| * ||y||). The previous test evaluates y/(||x|| * ||y||) to y / eps, and this PR to 1/eps * y/||y||. Pull Request resolved: https://github.com/pytorch/pytorch/pull/31378 Approved by: https://github.com/ezyang, https://github.com/albanD
Author
Committer
Parents
Loading