unstructured

f6eea758 - fix(ci): stabilize partition benchmark with a rolling-median baseline (#4375)

fix(ci): stabilize partition benchmark with a rolling-median baseline (#4375) ## Problem The Partition Benchmark gates every PR against a **single all-time-minimum** runtime stored in S3 (`benchmark_best.json`), updated by any run that comes in faster. On GitHub's shared `ubuntu-latest` runners the whole benchmark scales **~1.6× with runner speed** (partition work *and* fixed model/import overhead alike), so one lucky-fast run set an unbeatable floor — a frozen **81 s "best" vs a real ~130 s fleet** — and every PR since failed by ~60%. Evidence: across the last 25 runs, **0 passed** (12 failures on multiple unrelated branches, all `current ≈ 130 s` vs `best 81.13 s`, limit `97.36 s`). It was blocking unrelated PRs. ## Fix Replace the min-ratchet with a **rolling median of the last N `main` runs**: - **`push: main` records** one immutable per-run JSON object under a `history/` prefix. **PRs read** the window and gate against its **median** — read-only, so PRs can no longer write (and poison) the baseline. - A single fast/slow outlier can't move a median, so the gate tracks the real fleet instead of an unrepeatable minimum. - **Warm-up:** below `--min-samples` records, the gate records-and-passes — it self-bootstraps, and degrades gracefully to pass when S3 creds are absent (fork PRs). - **Recording happens regardless of pass/fail**, so the baseline can't get stuck the way the min could. - Threshold `0.20 → 0.30` to absorb runner variance; window/min-samples are env-tunable. Records keep per-file timings + runner info for later tuning. The job name (`Measure and compare partition() runtime`) is **unchanged**, so required-check config keeps matching. ## Validation Ran the new comparator locally against real recent run data (median 130.2 s): - typical run (131 s) → **PASS** (within 30%) - regressed run (200 s) → **FAIL** (exit 1) - < min-samples history → **WARM-UP** pass - `--record` writes a record and dedupes by sha (re-run replaces) `ruff check` + `ruff format --check` clean. ## Deploy steps (outside this PR — need the metrics-bucket creds) 1. Seed the `history/` prefix with recent real runs (so the baseline is live on day one rather than warming up). 2. Remove the stale `benchmark_best.json`. 3. Optional: add a lifecycle rule to expire `history/` objects older than ~90 days. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/Unstructured-IO/unstructured/pull/4375?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>