pytorch

af346df6 - [PyTorch][Vulkan]fix the issue of log 0 after softmax (#119898)

[PyTorch][Vulkan]fix the issue of log 0 after softmax (#119898) Summary: In some cases the output of `softmax` are so small that they are below the float16 precision. These values are represented as 0 in float16 and result in `-inf` when log is applied. According to [Wikipedia](https://en.wikipedia.org/wiki/Half-precision_floating-point_format#Exponent_encoding), the minimum strictly positive (subnormal) value is 2^−24 ≈ 5.9605 × 10^−8. Therefore, we add 6 x 10^-8 to the output of softmax to avoid the numerical issue. Test Plan: We add two tests: - `log_softmax_underflow_exception` tests the log_softmax without adding epsilon to the output of softmax, so we expect to get nan or -inf. (**NOTE**: this test has passed on both devserver and on Android device, but failed on the ` fbsource//xplat/caffe2:vulkan_ops_testAndroid` test on CI. In this test, `log` of small numbers [even `log 0` shows output -88 instead of `-inf`](https://interncache-cco.fbcdn.net/v/t49.3276-7/379414752_342395058779076_6447867753374424757_n.txt?ccb=1-7&_nc_sid=ce8ad4&efg=eyJ1cmxnZW4iOiJwaHBfdXJsZ2VuX2NsaWVudC9pbnRlcm4vc2l0ZS94L3Rlc3RpbmZyYSJ9&_nc_ht=interncache-cco&oh=00_AfApTdId1WOHUqdoSTc66s6adnrQt1YS0NDT-LDppIvX0g&oe=65D0CC99). We cannot reproduce this error on device now, so we **DISABLE** this test for now to integrate into CI.) - `log_softmax_underflow` tests the updated implementation of log_softmax, nan and -inf have been removed ## test on devserver ``` luwei@devbig984.prn1 /data/users/luwei/fbsource (9f6b78894)]$ LD_LIBRARY_PATH=third-party/swiftshader/lib/linux-x64/ buck2 run fbcode/mode/dev-nosan //xplat/caffe2:pt_vulkan_api_test_bin -- --gtest_filter="*log_softmax_underflow*" File changed: fbcode//caffe2/aten/src/ATen/test/vulkan_api_test.cpp File changed: fbsource//xplat/caffe2/aten/src/ATen/test/vulkan_api_test.cpp Buck UI: https://www.internalfb.com/buck2/baaaa683-60da-4dd8-95b9-6848fe1d7d74 Network: Up: 53KiB Down: 1.4MiB (reSessionID-9580ce4f-7e1e-4c65-8497-52443329b796) Jobs completed: 6. Time elapsed: 24.2s. Cache hits: 0%. Commands: 2 (cached: 0, remote: 1, local: 1) BUILD SUCCEEDED Running main() from third-party/googletest/1.14.0/googletest/googletest/src/gtest_main.cc Note: Google Test filter = *log_softmax_underflow* [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from VulkanAPITest [ DISABLED ] VulkanAPITest.DISABLED_log_softmax_underflow_exception [ RUN ] VulkanAPITest.log_softmax_underflow [ OK ] VulkanAPITest.log_softmax_underflow (169 ms) [----------] 1 test from VulkanAPITest (169 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (169 ms total) [ PASSED ] 1 test. YOU HAVE 1 DISABLED TEST ``` full test results: P1184164670 ``` [----------] 428 tests from VulkanAPITest (21974 ms total) [----------] Global test environment tear-down [==========] 428 tests from 1 test suite ran. (21974 ms total) [ PASSED ] 427 tests. [ SKIPPED ] 1 test, listed below: [ SKIPPED ] VulkanAPITest.querypool_flushed_shader_log YOU HAVE 11 DISABLED TESTS ``` ## test on device: - build ``` [luwei@devbig984.prn1 /data/users/luwei/fbsource (82c91e8da)]$ buck2 build -c ndk.static_linking=true -c pt.enable_qpl=0 --target-platforms=ovr_config//platform/android:arm32-fbsource //xplat/caffe2:pt_vulkan_api_test_binAndroid --show-output ``` - push to device and run ``` [luwei@devbig984.prn1 /data/users/luwei/fbsource (82c91e8da)]$ adb shell /data/local/tmp/pt_vulkan_api_test_binAndroid --gtest_filter="*log_softmax_underflow*" Running main() from third-party/googletest/1.14.0/googletest/googletest/src/gtest_main.cc Note: Google Test filter = *log_softmax_underflow* [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from VulkanAPITest [ DISABLED ] VulkanAPITest.DISABLED_log_softmax_underflow_exception [ RUN ] VulkanAPITest.log_softmax_underflow [ OK ] VulkanAPITest.log_softmax_underflow (292 ms) [----------] 1 test from VulkanAPITest (293 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (294 ms total) [ PASSED ] 1 test. YOU HAVE 1 DISABLED TEST ``` Reviewed By: yipjustin Differential Revision: D53694989 Pull Request resolved: https://github.com/pytorch/pytorch/pull/119898 Approved by: https://github.com/jorgep31415