[torch.compile] add logging for compilation time #10941
compute compilation time
fix one graph
๐ Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.
Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.
To run CI, PR reviewers can do one of these:
- Add
readylabel to the PR - Enable auto-merge.
๐
update name
example output (for the toy tests):
Compiling a graph for general shape takes 1.55 s
see https://buildkite.com/vllm/fastcheck/builds/9379#01939a08-6359-4659-9b3c-c07576a7efee/6465-8353
add context manager
dirty impl
use api
fi
use timing at engine level
engine level logging
remove the call in the worker
revert
fix logging
fix symbolic only
fix format
this pr aims to add three logging information:
- engine-level initialization, e.g.
INFO 12-05 22:37:22 llm_engine.py:493] init engine (profile, create kv cache, warmup model) took 15.08 seconds
INFO 12-05 22:39:24 llm_engine.py:493] init engine (profile, create kv cache, warmup model) took 40.77 seconds
INFO 12-05 22:41:51 llm_engine.py:493] init engine (profile, create kv cache, warmup model) took 50.18 seconds
- graph compilation for every shape (including the symbolic shape compilation), e.g.
INFO 12-05 22:39:02 backends.py:55] Compiling a graph for general shape takes 14.73 s
INFO 12-05 22:41:51 backends.py:58] Compiling a graph for shape 1 takes 9.99 s
- aggregation of the numbers in 2, e.g.
INFO 12-05 22:39:04 monitor.py:13] graph compilation takes 14.73 s in total
INFO 12-05 22:41:51 monitor.py:13] graph compilation takes 24.758146286010742 s in total
how to read it:
the increase in 1, when using or not using torch.compile , is the total cost of torch.compile .
2 shows the cost of every shape compilation, so that users can select it according to their budget. Note that some compilation like Dynamo bytecode compilation and triton compilation are not considered here.
3 just aggregates 2.
Honestly, I don't have enough familiarity to the code for proper review. However, the code apparently looks OK to me. Please feel free to merge.
| 9 | |||
| 10 | |||
| 11 | def end_monitoring_torch_compile(compilation_config: CompilationConfig): | ||
| 12 | if compilation_config.level == CompilationLevel.PIECEWISE: | ||
| 13 | logger.info("graph compilation takes %.2f s in total", | ||
| 14 | compilation_config.compilation_time) |
WoosukKwon148 days agoDumb question: Why does it print only for pw CUDA graphs?
youkaichao148 days agoCompilationLevel.PIECEWISE is piecewise compile, not piecewise cudagraph. this is orthogonal to cudagraph.
| 108 | 120 | # we share the global graph pool among all the backends | |
| 109 | 121 | global_graph_pool = None | |
| 110 | 122 | ||
| 123 | compilation_start_time = 0.0 |
WoosukKwon148 days agoMaybe None instead of 0.0?
WoosukKwon148 days agoJust wondering: can we somehow make this more robust? Since the code touching this var is scattered into different places, I feel it's error prone...
youkaichao148 days agothat's why I want to merge some functions in the worker/executor. however, given the current code status, i don't have bandwidth to refactor those files.
INFO 12-05 22:41:51 monitor.py:13] graph compilation takes 24.758146286010742 s in total
Please make sure to use %2f for this log (if it hasn't been fixed yet).
Update vllm/compilation/backends.py
merge code
@WoosukKwon thanks for the review!
Login to write a write a comment.