pytorch-lightning
Skip test with compile error on torch=2.2.2 on Windows
#19734
Merged

Skip test with compile error on torch=2.2.2 on Windows #19734

awaelchli merged 3 commits into master from tests/skip-windows-dynamo-test
awaelchli
awaelchli1 year ago (edited 1 year ago)

What does this PR do?

The following tests started failing when CI updated to the recent release of PyTorch 2.2.2:

______________________ test_trainer_compiled_model_test _______________________

tmp_path = WindowsPath('C:/Users/runneradmin/AppData/Local/Temp/pytest-of-runneradmin/pytest-0/test_trainer_compiled_model_te0')

    @pytest.mark.skipif(sys.platform == "darwin", reason="fatal error: 'omp.h' file not found")
    @RunIf(dynamo=True)
    def test_trainer_compiled_model_test(tmp_path):
        model = BoringModel()
        compiled_model = torch.compile(model)
    
        trainer = Trainer(
            default_root_dir=tmp_path,
            fast_dev_run=True,
            enable_checkpointing=False,
            enable_model_summary=False,
            enable_progress_bar=False,
            accelerator="cpu",
        )
>       trainer.test(compiled_model)

utilities\test_compile.py:156: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\trainer\trainer.py:754: in test
    return call._call_and_handle_interrupt(
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\trainer\call.py:44: in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\trainer\trainer.py:794: in _test_impl
    results = self._run(model, ckpt_path=ckpt_path)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\trainer\trainer.py:987: in _run
    results = self._run_stage()
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\trainer\trainer.py:1026: in _run_stage
    return self._evaluation_loop.run()
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\loops\utilities.py:182: in _decorator
    return loop_run(self, *args, **kwargs)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\loops\evaluation_loop.py:135: in run
    self._evaluation_step(batch, batch_idx, dataloader_idx, dataloader_iter)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\loops\evaluation_loop.py:396: in _evaluation_step
    output = call._call_strategy_hook(trainer, hook_name, *step_args)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\trainer\call.py:311: in _call_strategy_hook
    output = fn(*args, **kwargs)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\lightning\pytorch\strategies\strategy.py:425: in test_step
    return self.lightning_module.test_step(*args, **kwargs)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\eval_frame.py:489: in _fn
    return fn(*args, **kwargs)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\eval_frame.py:655: in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\convert_frame.py:727: in _convert_frame
    result = inner_convert(frame, cache_entry, hooks, frame_state)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\convert_frame.py:383: in _convert_frame_assert
    compiled_product = _compile(
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\convert_frame.py:646: in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\utils.py:244: in time_wrapper
    r = func(*args, **kwargs)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\convert_frame.py:562: in compile_inner
    out_code = transform_code_object(code, transform)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\bytecode_transformation.py:1033: in transform_code_object
    transformations(instructions, code_options)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\convert_frame.py:151: in _fn
    return fn(*args, **kwargs)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\convert_frame.py:527: in transform
    tracer.run()
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\symbolic_convert.py:2128: in run
    super().run()
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\symbolic_convert.py:818: in run
    and self.step()
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\symbolic_convert.py:781: in step
    getattr(self, inst.opname)(inst)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\symbolic_convert.py:2243: in RETURN_VALUE
    self.output.compile_subgraph(
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\output_graph.py:945: in compile_subgraph
    self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\contextlib.py:79: in inner
    return func(*args, **kwds)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\output_graph.py:1087: in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\utils.py:244: in time_wrapper
    r = func(*args, **kwargs)
C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_dynamo\output_graph.py:1159: in call_user_compiler

    @classmethod
    def load_by_key_path(
        cls,
        key: str,
        path: str,
        linemap: Optional[List[Tuple[int, str]]] = None,
        attrs: Optional[Dict[str, Any]] = None,
    ) -> ModuleType:
        if linemap is None:
            linemap = []
        if key not in cls.cache:
            with open(path) as f:
                try:
                    code = compile(f.read(), path, "exec")
                except Exception as e:
>                   raise RuntimeError(
                        f"Failed to import {path}\n{type(e).__name__}: {e}"
                    ) from None
E                   torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
E                   RuntimeError: Failed to import C:\Users\RUNNER~1\AppData\Local\Temp\torchinductor_runneradmin\3e\c3echiribpf5lk3o3wovc4ap55h4jvdbyvra356dynpzti6bfrvh.py
E                   SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 13-14: truncated \UXXXXXXXX escape (c3echiribpf5lk3o3wovc4ap55h4jvdbyvra356dynpzti6bfrvh.py, line 52)
E                   
E                   Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
E                   
E                   
E                   You can suppress this exception and fall back to eager by setting:
E                       import torch._dynamo
E                       torch._dynamo.config.suppress_errors = True

C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\site-packages\torch\_inductor\codecache.py:1886: BackendCompilerFailed

This is currently blocking us from merging other PRs.

cc @carmocca @Borda

awaelchli skip test
ce6fd57c
awaelchli awaelchli requested a review from carmocca carmocca 1 year ago
awaelchli awaelchli requested a review from justusschock justusschock 1 year ago
awaelchli awaelchli requested a review from Borda Borda 1 year ago
awaelchli awaelchli added bug
awaelchli awaelchli added ci
awaelchli awaelchli added tests
github-actions github-actions added pl
awaelchli awaelchli removed pl
awaelchli awaelchli added fun
awaelchli awaelchli added this to the 2.2.x milestone 1 year ago
github-actions
github-actions1 year ago (edited 1 year ago)

⚡ Required checks status: All passing 🟢

Groups summary

🟢 pytorch_lightning: Tests workflow
Check ID Status
pl-cpu (macOS-11, lightning, 3.8, 1.13, oldest) success
pl-cpu (macOS-11, lightning, 3.10, 1.13) success
pl-cpu (macOS-11, lightning, 3.10, 2.1) success
pl-cpu (macOS-11, lightning, 3.10, 2.2) success
pl-cpu (ubuntu-20.04, lightning, 3.8, 1.13, oldest) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 1.13) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.1) success
pl-cpu (ubuntu-20.04, lightning, 3.10, 2.2) success
pl-cpu (windows-2022, lightning, 3.8, 1.13, oldest) success
pl-cpu (windows-2022, lightning, 3.10, 1.13) success
pl-cpu (windows-2022, lightning, 3.10, 2.1) success
pl-cpu (windows-2022, lightning, 3.10, 2.2) success
pl-cpu (macOS-11, pytorch, 3.8, 1.13) success
pl-cpu (ubuntu-20.04, pytorch, 3.8, 1.13) success
pl-cpu (windows-2022, pytorch, 3.8, 1.13) success
pl-cpu (macOS-12, pytorch, 3.11, 2.0) success
pl-cpu (macOS-12, pytorch, 3.11, 2.1) success
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.0) success
pl-cpu (ubuntu-22.04, pytorch, 3.11, 2.1) success
pl-cpu (windows-2022, pytorch, 3.11, 2.0) success
pl-cpu (windows-2022, pytorch, 3.11, 2.1) success

These checks are required after the changes to tests/tests_pytorch/utilities/test_compile.py.

🟢 pytorch_lightning: Azure GPU
Check ID Status
pytorch-lightning (GPUs) (testing Lightning | latest) success
pytorch-lightning (GPUs) (testing PyTorch | latest) success

These checks are required after the changes to tests/tests_pytorch/utilities/test_compile.py.


Thank you for your contribution! 💜

Note
This comment is automatically generated and updates for 60 minutes every 180 seconds. If you have any other questions, contact carmocca for help.

awaelchli awaelchli added 3rd party
pre-commit-ci[bot] [pre-commit.ci] auto fixes from pre-commit.com hooks
64160855
github-actions github-actions added pl
carmocca
carmocca approved these changes on 2024-04-03
Borda
Borda approved these changes on 2024-04-03
awaelchli skip one more
f4f71ba3
awaelchli awaelchli force pushed from 6ca3c529 to f4f71ba3 1 year ago
mergify mergify added ready
codecov
codecov1 year ago

Codecov Report

Merging #19734 (f4f71ba) into master (d25014d) will decrease coverage by 31%.
The diff coverage is n/a.

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #19734      +/-   ##
==========================================
- Coverage      84%      53%     -31%     
==========================================
  Files         424      416       -8     
  Lines       34915    34762     -153     
==========================================
- Hits        29357    18441   -10916     
- Misses       5558    16321   +10763     
awaelchli awaelchli merged 8947d135 into master 1 year ago
awaelchli awaelchli deleted the tests/skip-windows-dynamo-test branch 1 year ago

Login to write a write a comment.

Login via GitHub

Assignees
No one assigned
Labels
Milestone