xla
Truncate python stack when outputting frame that cause the graph executation
#5933
Merged
Overview
Commits
3
Changes
View On GitHub

Truncate python stack when outputting frame that cause the graph executation #5933

JackCaoG merged 3 commits into master from JackCaoG/truncate_python_stack
JackCaoG
JackCaoG1 year ago (edited 1 year ago)

motivation of this pr is simple, currently frame analysis looks like

Execution Analysis: ================================================================================
Execution Analysis: Execution Cause
Execution Analysis:   user mark_step
Execution Analysis: Python Frame Triggered Execution: 
Execution Analysis:   mark_step (/src/pytorch/xla/torch_xla/core/xla_model.py:826)
Execution Analysis:   broadcast_master_param (/src/pytorch/xla/torch_xla/core/xla_model.py:1216)
Execution Analysis:   train_imagenet (/src/pytorch/xla/test/test_train_mp_imagenet.py:261)
Execution Analysis:   _mp_fn (/src/pytorch/xla/test/test_train_mp_imagenet.py:365)
Execution Analysis:   __call__ (/src/pytorch/xla/torch_xla/_internal/pjrt.py:176)
Execution Analysis:   _thread_fn (/src/pytorch/xla/torch_xla/_internal/pjrt.py:70)
Execution Analysis:   run (/usr/local/lib/python3.8/concurrent/futures/thread.py:57)
Execution Analysis:   _worker (/usr/local/lib/python3.8/concurrent/futures/thread.py:80)
Execution Analysis:   run (/usr/local/lib/python3.8/threading.py:870)
Execution Analysis:   _bootstrap_inner (/usr/local/lib/python3.8/threading.py:932)
Execution Analysis:   _bootstrap (/usr/local/lib/python3.8/threading.py:890)
Execution Analysis: --------------------------------------------------------------------------------
Execution Analysis: ================================================================================

but the stack below _mp_fn are not very useful so I want to truncate them, after this pr it becomes.

Execution Analysis: ================================================================================
Execution Analysis: Execution Cause
Execution Analysis:   user mark_step
Execution Analysis: Python Frame Triggered Execution: 
Execution Analysis:   mark_step (/workspaces/dk2/pytorch/xla/torch_xla/core/xla_model.py:826)
Execution Analysis:   broadcast_master_param (/workspaces/dk2/pytorch/xla/torch_xla/core/xla_model.py:1216)
Execution Analysis:   train_imagenet (/workspaces/dk2/pytorch/xla/test/test_train_mp_imagenet.py:261)
Execution Analysis:   _mp_fn (/workspaces/dk2/pytorch/xla/test/test_train_mp_imagenet.py:365)
Execution Analysis:   ..........
Execution Analysis: --------------------------------------------------------------------------------
Execution Analysis: ================================================================================
JackCaoG Truncate python stack when outputting frame that cause the graph exec…
4fdf3ede
add mp tests
afa0b9d4
JackCaoG move tests to a new dir
ad5d3136
JackCaoG JackCaoG changed the title [WIP]Truncate python stack when outputting frame that cause the graph exec… Truncate python stack when outputting frame that cause the graph executation 1 year ago
JackCaoG JackCaoG requested a review from yeounoh yeounoh 1 year ago
JackCaoG JackCaoG requested a review from will-cromar will-cromar 1 year ago
JackCaoG JackCaoG added usability
will-cromar
will-cromar approved these changes on 2023-11-29
will-cromar1 year ago

Your original TODO mentioned configurability. Would it make sense to have a flag to print the entire stack?

JackCaoG
JackCaoG1 year ago

yea.. I am thinking about something less fine grain like verbose level for pt xla debug, when it is high enough it will print all sorts of things.

JackCaoG JackCaoG merged a5e07389 into master 1 year ago

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
No one assigned
Labels
usability
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum