benchmark
10b51a20 - Fix skipped compiled model is Dynamo benchmarks (#195703)

Commit
76 days ago
Fix skipped compiled model is Dynamo benchmarks (#195703) Summary: FIXES #195696 Fixes a bug in the PT2 HF LLM benchmarks where the compiled model isn't used as intended. This effectively means we were ignoring our compile settings such as backend, and only using the HF default to compile for generation and not prefill. The compiled model was being ignored in generate. The reason we see speedup is because of another bug where we forget to disable Transformers default compile settings in the generation config, so the decode is compiled, but the prefill is still fully eager. This can be seen in running with `--backend=eager`, which actually still resulted in a speedup. Aditionally to reuse the graphs compiled during warmup properly and make sure the compiled forward is used both prefill and decode, this PR directly uses `model.forward` in a couple of callsites. Without this I found that the compiled model wasn't being invoked. Good new is the ground truth compile figures are better than were being reported (at least on qwen 0.6B) | State | Backend | Speedup | Events | Report | |---|---|---|---|---| | Before | Inductor | 1.683× | 1,293 | [tlparse](https://manifold.edge.x2p.facebook.net/v0/read/tree/logs/arshzahed/4a2b107a-c6c6-478d-93e0-ddd4a40d49b0/custom/index.html?bucketName=tlparse_reports&apiKey=tlparse_reports-key&withPayload=1&timeoutMsec=10000) | | Before | Eager | 1.770× | 1,293 | [tlparse](https://manifold.edge.x2p.facebook.net/v0/read/tree/logs/arshzahed/171a5642-1b08-4eb1-bcf4-db6260116132/custom/index.html?bucketName=tlparse_reports&apiKey=tlparse_reports-key&withPayload=1&timeoutMsec=10000) | | After | Inductor | 3.171× | 3,698 | [tlparse](https://manifold.edge.x2p.facebook.net/v0/read/tree/logs/arshzahed/984810af-c93b-4121-9746-fe6a058c555d/custom/index.html?bucketName=tlparse_reports&apiKey=tlparse_reports-key&withPayload=1&timeoutMsec=10000) | | After | Eager | 1.108× | 3,527 | [tlparse](https://manifold.edge.x2p.facebook.net/v0/read/tree/logs/arshzahed/e952ece6-3f8d-4406-b491-7b6442e99bf2/custom/index.html?bucketName=tlparse_reports&apiKey=tlparse_reports-key&withPayload=1&timeoutMsec=10000) | We may want to backfill benchmark results across nightlies or releases so we have a better sense of trajectory. X-link: https://github.com/pytorch/pytorch/pull/195703 Approved by: https://github.com/anijain2305 Reviewed By: huydhn Differential Revision: D118560089 fbshipit-source-id: 1e150694c227a3cc9a757b8a97910c1c17305431
Author
Committer
Parents
Loading