pytorch

6e50f8e3 - Allow External Scripts (e.g. vscode) To Discover and Execute unittest Tests (#85584)

Allow External Scripts (e.g. vscode) To Discover and Execute unittest Tests (#85584) Fixes #85578 Currently, many test modules customize test loading and discovery via the [load_tests protocol](https://docs.python.org/3/library/unittest.html#load-tests-protocol). The salient custom behavior included (introduced with https://github.com/pytorch/pytorch/pull/13250) is to verify that the script discovering or executing the test is the same script in which the test is defined. I believe this unnecessarily precludes the use of external tools to discover and execute tests (e.g. the vscode testing extension is widely used and IMHO quite convenient). This simple PR retains the current restriction by default while offering users the option to disable the aforementioned check if desired by setting an environmental variable. For example: 1. Setup a test env: ```bash ./tools/nightly.py checkout -b some_test_branch conda activate pytorch-deps conda install -c pytorch-nightly numpy expecttest mypy pytest hypothesis astunparse ninja pyyaml cmake cffi typing_extensions future six requests dataclasses -y ``` 2. The default test collection behavior discovers 5 matching tests (only tests within `test/jit/test_cuda.py` because it doesn't alter the default `load_test` behavior: ```bash python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py \ ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/testing_tools/unittest_discovery.py \ ./test test_cuda.py | grep test_cuda | wc -l 5 ``` 3. Set the new env variable (in vscode, you would put it in the .env file) ```bash export PYTORCH_DISABLE_RUNNING_SCRIPT_CHK=1 ``` 4. All of the desired tests are now discovered and can be executed successfully! ```bash python ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/get_output_via_markers.py \ ~/.vscode-server/extensions/ms-python.python-2022.14.0/pythonFiles/testing_tools/unittest_discovery.py \ ./test test_cuda.py | grep test_cuda | wc -l 175 ```  A potentially relevant note, the previous behavior of the custom `load_tests` flattened all the `TestSuite`s in each test module: https://github.com/pytorch/pytorch/blob/4c01c51266afae57c6d6952c84fff2802d9b2bb9/torch/testing/_internal/common_utils.py#L3260-L3262 I haven't been able to find any code that depends upon this behavior but I think retaining the `TestSuite` structure is preferable from a user perspective and likely safe (`TestSuite`s [can be executed](https://docs.python.org/3/library/unittest.html#load-tests-protocol:~:text=test%20runner%20to-,allow%20it%20to%20be%20run,-as%20any%20other) just like `TestCase`s and this is the structure [recommended](https://docs.python.org/3/library/unittest.html#load-tests-protocol:~:text=provides%20a%20mechanism%20for%20this%3A%20the%20test%20suite) by the standard python documentation). If necessary, I can change this PR to continue flattening each test module's `TestSuite`s. Since I expect external tools using the `unittest` `discover` API will usually assume discovered `TestSuite`s to retain their structure (e.g. like [vscode](https://github.com/microsoft/vscode-python/blob/192c3eabd8a065492f237196b052145364e68cb4/pythonFiles/visualstudio_py_testlauncher.py#L336-L349)) retaining the `testsuite` flattening behavior would likely require customization of those external tools for PyTorch though. Thanks to everyone in the community for the continued contributions to this incredibly valuable framework! Pull Request resolved: https://github.com/pytorch/pytorch/pull/85584 Approved by: https://github.com/huydhn