benchmark
cc2b9a40 - Add Qwen3.5-0.8B to Hugging Face benchmarks (#198261)

Commit
42 days ago
Add Qwen3.5-0.8B to Hugging Face benchmarks (#198261) Summary: ### Summary Adds `Qwen/Qwen3.5-0.8B` to the Dynamo text-generation benchmarks using the existing `TextGenerationBenchmark` adapter. The pinned Transformers 5.13.0 supports the model's text component and hybrid attention cache, allowing it to reuse the existing loading, generation, and timing paths. - Registers the model with an effective batch size of 1 (batch-size hint 8, divisor 8) and follows the existing CPU skip policy. - Adds ten expected-result rows across CUDA and ROCm for static/dynamic inference, static/dynamic training, and AOTInductor. Existing training and AOTInductor failures are recorded as `eager_fail_to_run` and `fail_to_run`, respectively. - Adds Qwen3.5 to the existing ROCm LLM graph-metric flaky list. ### Test plan **GPU** | Check | Result | | --- | --- | | Inference accuracy with static and dynamic shapes | Runs and passes | | Performance with `--disable-cudagraphs` | Runs successfully; **3.86× speedup** | **Local CPU development environment** | Check | Result | | --- | --- | | BF16 Inductor accuracy with a 1,000-token prompt | Pass: 1 graph, 21,933 captured calls, 0 graph breaks, 0 eager fallbacks | | Eager static-cache generation with a 1,000-token prompt and 4 forced output tokens | Pass | | Shortened compiled-generation performance run | Completed | | 4 suite-selection checks | Pass | | 10 new expected-result rows checked with both CSV validators | Pass; recorded local results used as fixtures | | `python test/dynamo/test_benchmark_runner.py -v` | Both tests pass | | `spin quicklint` | Pass | X-link: https://github.com/pytorch/pytorch/pull/198261 Approved by: https://github.com/azahed98, https://github.com/anijain2305 Reviewed By: georgehong Differential Revision: D122422646 fbshipit-source-id: 24eaf6009b1ef34b58ca6c94fc6cfec793221030
Author
Committer
Parents
Loading