benchmark

4d38aa80 - track ops captured and eager fallbacks in accuracy benchmarks (#191357)

track ops captured and eager fallbacks in accuracy benchmarks (#191357) Summary: The dynamo accuracy benchmarks only hard-track `graph_breaks`, which is a misleading signal for features like nested graph breaks: NGB can *reduce* graph breaks by resuming, but it can also *increase* the raw break count while capturing dramatically more of the model (e.g. a QAT model went from 2 ops captured in 1 frame to 84 ops across 19 frames -- more breaks, far more coverage). Break count alone flags that as a regression. Track two more metrics so regressions in what Dynamo actually captures are caught: - ops captured (`calls_captured`) -- already collected in get_dynamo_stats and written to the per-model CSV; higher is better (capturing fewer is the regression). - fallbacks to eager (`fallbacks_to_eager`, new) -- frames Dynamo saw but could not convert (`counters["frames"]["total"] - ["ok"]`), so they ran eagerly; lower is better. `unique_graphs` is deliberately NOT tracked: a change in the graph count is ambiguous (fewer graphs can mean graphs were merged / fewer breaks -- an improvement -- not less capture), and it is covered indirectly by graph_breaks and calls_captured. `check_graph_breaks.py` compares each metric per its own mode and skips any metric whose column is absent from a baseline, so existing `ci_expected_accuracy` CSVs (which have only `graph_breaks`) are unaffected until regenerated. `graph_breaks` and `fallbacks_to_eager` are exact-matched (lower is better); `calls_captured` is higher-is-better and gated by a drop threshold (COVERAGE_DROP_TOL, 5%) -- only a meaningful drop is a regression while increases and small drift are absorbed, so that large fine-grained count does not churn the baselines on routine decomp/lowering changes. `update_expected.py`, which regenerates the baselines from CI artifacts, is extended to persist the new metric columns when a run produced them (entries carried over from an older CSV are backfilled to 0), so a follow-up regeneration activates the new checks; the committed baselines are intentionally left unchanged in this PR. The same metric list is kept (small, duplicated) in `check_graph_breaks.TRACKED_METRICS` and `update_expected.METRIC_COLUMNS`, each with a comment to keep them in sync. X-link: https://github.com/pytorch/pytorch/pull/191357 Approved by: https://github.com/rtimpe, https://github.com/mlazos, https://github.com/anijain2305 ghstack dependencies: #189600 Reviewed By: huydhn Differential Revision: D114416165 fbshipit-source-id: de01b78d2ba6e0b91c0ec7627a1900380df7f3d6