gradio

a1eb5b53 - Add Hugging Face bucket storage to run history (#13763)

Add Hugging Face bucket storage to run history (#13763) * add BucketHistory primitive + bucket sync UI * cleanup * add changeset * tweak * fixes * format * revert * format * load outputs * trim tests * format * clean up * Merge branch 'main' into workflow-buckets # Conflicts: # js/workflowcanvas/workflow/WorkflowCanvas.svelte * address copilot comments * format * revert * fix: use builtins.list in BucketHistory annotations The class defines a .list() method that shadows the builtin under PEP 563 deferred annotations, tripping the type checker on the method signatures. * ui tweak * fix generation render * format * output + fix auth * format * refresh * clean up * clean up, add push rate limit * format and lint * remove spa - move to separate pr * add changeset * format * add changeset * undo oatuh tweak * rm comments * - refactor server - frontend tweaks * ruff/format * tweak * refactor * lint * fix * type * handle object results * use created_at for history gen order * rm comments * fornat * rm dead css * rm dead css * add changeset * history: extract run-history router, fix asset externalization Move the /run-history/* handlers out of the create_app closure into a module-level APIRouter in gradio/history_routes.py. routes.py drops 247 lines for two. The repetition is gone rather than the routes: a get_store dependency folds the token check and session lookup into one parameter, Path/Query patterns replace the hand-rolled validate/try/except blocks (and get the 422s into OpenAPI), pydantic models replace manual body.get() digging, and a single offload() helper maps store errors to HTTP once. Fixes found along the way: - POST /run-history/records never called externalize_assets and passed local_assets=None, so no media was ever uploaded and no __asset__ marker was ever written. Records were persisted with raw temp paths, which cannot be replayed after a restart, and the __asset__ branches in WorkflowCanvas/WorkflowHistoryPanel were dead code. - extract_local_file_path only matched root-relative /gradio_api/file= URLs, but the workflow executor emits absolute ones, so even once wired the walk would have found nothing. Now matches the marker anywhere in the URL and unquotes the path. - Inputs and outputs are externalized with a shared counter so their asset ids cannot collide. - limit was validated after the store lookup, so an out-of-range limit could return 409/502 instead of 422. - app_key read `hash` off the /app_id route function and so was always the literal "app"; it now uses blocks.app_id. bounded_json_body becomes enforce_body_limit, a router-level dependency, since pydantic handles the parsing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: fix record ordering, owner isolation, and asset handling Addresses the findings from a full review pass on the bucket-history work. Backend: - Records were sorted with `getattr(f, "last_modified", "")`, a field BucketFile does not have (it exposes mtime/uploaded_at). The key always fell back to the random record id, so "newest first" was arbitrary and the 200-file scan cap picked an arbitrary subset — a new run could be permanently invisible. Sort on mtime/uploaded_at, tolerate naive and missing timestamps, and re-sort parsed records by created_at. - list_bucket_tree defaults to recursive=False, so listing `assets/` returned BucketFolder entries and never the files inside them. clear_records therefore left every uploaded image/audio/video in the bucket and could 502 by handing a directory path to a delete. Pass recursive=True and filter to files. - owner_id was written into every record but never checked on read or delete. In a shared org bucket — a supported feature — anyone could list, download, and permanently delete anyone else's runs. list/get/ asset/delete/clear are now owner-scoped, and a foreign record 404s rather than 403s. app_key is deliberately not part of that check: blocks.app_id is random.getrandbits(64), so filtering on it would hide a user's own history after every restart. - The asset route served attacker-supplied bytes inline under a guessed mimetype, bypassing the XSS_SAFE_MIMETYPES rule gradio's own file route enforces; an .html or .svg on a file node rendered on the app's origin. Now inline only for safe types, attachment otherwise, plus an immutable Cache-Control so 50 thumbnails stop re-hitting the Hub. - get_asset_bytes re-downloaded the whole record JSON per request; added a bounded record cache invalidated alongside the list cache. - The /run-history/* routes ignored the documented run_history=False kill switch and stayed live, pushing to the Hub with recording disabled. - enforce_body_limit measured len(await request.body()), which buffers the whole stream first. A chunked request has no Content-Length, so the cap was bypassable pre-auth. Stream and abort at the limit instead. - created_at is validated and now_utc_iso matches the canvas's millisecond precision, so mixed-source records sort consistently. Frontend: - url() emitted a root-absolute path when root was "", 404ing every history call under mount_gradio_app(path=...). Resolve against document.baseURI instead. - Push failures were swallowed by `catch {}` and the returned status was discarded, so 401/403/409/502 all looked like a successful save. - list_bucket_records mapped every error to [], so a 409 rendered as "No generations yet" for a bucket holding hundreds of records. - The bucket lived only in the server session while the client kept it in localStorage, and the two were never reconciled: a restart or a second tab silently listed the wrong bucket. Re-assert the binding before listing and before pushing. - Input media is a blob: object URL that dies with the tab, and the server only recognises paths and file URLs, so inputs were never persisted and Load restored revoked blobs. Upload them first, reusing the executor's upload path. - The connect modal's prefill effect read the field it wrote, making the bucket input impossible to clear. - The panel's refresh effect tracked `loading` and double-fetched on reopen. - fromGradioOutput returned null for any unexpected shape outside text/html/json, blanking the node with no explanation, and relied on JSON.stringify(undefined) being a string. - Fixed the type errors this PR introduced (unknown-typed record values, optional subgraph). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * testing changes * testing changes * history: one shared history per bucket, reuse gradio's fetch and traversal A bucket now holds one shared history: everyone who can write to it sees the same runs. That is the point of pointing a team at an org bucket, and it makes the Hub's own access control the only permission model here. Removes the per-owner prefix built to keep users apart inside a bucket: `owner_segment`, `owner_key`, the `runs/<owner>/<app>/...` layout (now `runs/<app>/...`), and the owner component of the store cache key. Each record still carries the `owner_id` of whoever ran it, as metadata to display rather than a filter — a shared timeline still wants to show who ran what. Remote asset capture goes through `processing_utils.async_ssrf_protected_get` instead of a hand-rolled guard. Deletes `_is_public_address` and the bespoke streaming client, and closes a gap they had: the shared helper pins the validated IP and re-validates every redirect, so a DNS rebind between the check and the connect cannot slip through. `externalize_assets` runs on `client_utils.async_traverse` rather than a custom walker. `is_file_obj` requires a local `path`, which a canvas value produced by a remote Space does not have, so `_is_asset_node` extends the gradio predicate with the url-only case instead of replacing it. Capturing assets is now awaited on the event loop and only the blocking Hub upload is offloaded. Drops surface nothing reaches: the `/endpoints`, single-record GET, `DELETE /records` and `/orphans` routes (9 -> 5), their client helpers, and `clear_records`, `list_endpoints` and the orphan sweep. The owner-isolation tests are replaced by ones that pin the new contract: a bucket is one shared timeline, records still say who ran them, and two apps sharing a bucket stay separate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: key runs by app_id, and drop what that makes unnecessary Runs are now filed under `runs/<app_id>/<endpoint>/`, where app_id is `blocks.app_id`. That is minted per process, so a new commit (which restarts the Space) or a local restart starts a fresh folder. Deliberate: an app's endpoints — and a workflow's entire graph — can change between deploys, and runs recorded against the old shape are not replayable against the new one. Older folders stay in the bucket, they just stop being this app's history. It is also the key the browser-local history already uses, so both backends partition runs the same way. The bucket name already identifies the app (`<user>/<space>_history`), so nothing needed a separate stable app identity: - `stable_app_key` and its GRADIO_HISTORY_APP_KEY / SPACE_ID / hashed-title fallback chain is replaced by a one-line `app_id_of`. - `gr.Workflow` no longer sets `history_app_key`; it is filed by the same rule as any other app. Also removes surface that turned out to be unused: - `owner_id` is off the store entirely. It was pure pass-through — the recorder resolved the identity, handed it to `store_for`, then read it straight back off the store to stamp the record. The store now knows where to read and write and which credential to use, and nothing about who is asking; who ran a run is a property of the record. That also drops the `_owner_id` route helper, so a read no longer 401s for a session that has a token but no OAuth subject. - `MAX_ASSET_BYTES` was declared and never referenced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: replace the store object with functions, drop delete, trim tests `BucketRunHistoryStore` was 317 lines, of which ~236 were Hub calls that work identically as free functions. The rest existed only to hold three pieces of state, and once `owner_id` moved onto the record it was down to two identity fields — not enough to be a class. Of that state, only one piece earned its place: the memo that a bucket has already been created and checked private, without which every recorded run pays `create_bucket` + `bucket_info` before it can write. That is now a module-level `_ensured` set keyed by `(token, bucket)`. Both read caches are gone: `_record_cache` never actually hit (it was populated by `get_record` but never by `list_records`, so a panel of N thumbnails missed N times), and `_path_cache` only deduplicated a listing the user had explicitly asked for. What replaces the class is `HistoryTarget`, a three-field NamedTuple of bucket, token and app_id. It holds no connection, no cache and no lifecycle — the bucket paths are the model, and those three values are all it takes to address them. `store_for` and its LRU go with it. Delete is out for v1: the panel already links to the bucket, so records can be managed there. That removes a route, a client helper, the store function and the confirm-then-delete UI. Routes are down to four — connect, buckets, records, asset — from nine before this pass. Tests: 59 -> 29. Removed regex-level checks, restatements of behaviour already covered elsewhere, and tests for the functions that no longer exist. Everything that pins a decision or a security property stays: untrusted paths are not uploaded, stored asset paths are prefix-checked, remote fetches go through gradio's SSRF-protected client, a public bucket is refused, the body cap survives chunked encoding, there is no client-push route, and a client cannot choose a record id or owner. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: fold into one module, move routes to routes.py, drop comments `history_recorder.py` and `history_routes.py` are gone. Everything that is not a route now lives in `gradio/history.py`; the four route handlers moved into `routes.py` alongside the rest of the app's routes, under the existing `run_history` guard. `init_history_state` no longer allocates `bucket_history_cache` and its lock, which have been dead since `store_for` was removed. Inline comments are gone and docstrings are one line each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: simplify record IDs and request handling * history: one exception type, fewer validators, imports at the top Five exception classes existed only so a mapper could turn each back into an HTTP status. `HistoryError` carries the status instead, and `offload` raises `HTTPException(exc.status, ...)` directly, which removes the four subclasses and the mapper. `validate_record_id` is gone: a valid record id is already a valid path segment, so `validate_segment` covers the same ground. `validate_bucket_id` folds its two checks into one. `tempfile`, `processing_utils` and `HfApi` are imported at the top rather than inside functions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: simplify storage and workflow recording * history: allow public buckets `ensure_private_bucket` refused to store history in a bucket that already existed and was public. Whether a bucket is public is the owner's choice, and writing to one is not an error, so the check and its `bucket_info` call are gone and the function is now `ensure_bucket`. Buckets it creates are still private by default. Dropping the check also removes a Hub round-trip from the first write to each bucket, and lets `FakeHub` lose its `private` flag and `bucket_info`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: fold away single-use helpers, trim tests Path helpers: `app_prefix` and `endpoint_prefix` were an f-string and a one-line composition, and `endpoint_prefix` had a single caller. Both are gone in favour of `RUNS_PREFIX`/`ASSETS_PREFIX` constants, leaving `record_path` and `asset_prefix` to compose and validate. Listing: `_safe_list_tree`, `_is_file` and `_list_paths` were three functions, each with one caller, feeding one place. They are now one `_list_paths`. `_try_delete` had a single caller and is inlined. `resolve_token` and `resolve_bucket_id` lose their step-by-step None guards for a try/except around the one thing that can fail. `AssetName` and `EndpointSeg` were the same annotation twice, now `Segment`. `record_run` no longer assigns its own result to a second name. Tests: 28 -> 19. Removed the heavily-mocked and the marginal — schema version rejection, orphan cleanup after a failed commit, the ensure memo, id formatting, cross-endpoint ordering, session-state resolution, and two route checks the framework already enforces. What is left either drives a real app over HTTP or pins a security property. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: accept the host's local HF token, not just an OAuth session Running locally there is no OAuth session, so `require_oauth_token` 401'd every history route even though the canvas was signed in with the host's `huggingface-cli login` token — the connect modal reported it as "sign in first". `resolve_token` now follows the same order `gradio.workflow._resolve_token` already uses for inference and ZeroGPU: the caller's OAuth token, else the host's local token when the request carries the write token. The routes depend on `require_token` rather than `require_oauth_token`. Also fixes a bug from the previous commit: `list_bucket_tree` returns a generator, so folding `_safe_list_tree` into `_list_paths` left the HTTP call outside the `try` and a Hub error escaped as an unhandled `HfHubHTTPError`. The iteration is inside the guard again, a missing prefix returns nothing, and anything else surfaces with the Hub's status. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * workflow: bound the bucket list in the connect modal The list rendered every bucket the account has, so an account with a lot of them pushed the modal past the viewport in both directions. The list is capped at 180px with its own scrollbar, and the panel at 85vh. Both use `overscroll-behavior: contain` — without it, scrolling past either end chains to the canvas underneath, which zooms on wheel. At most 50 buckets are rendered; if there are more, the modal says how many are not shown rather than silently dropping them, and any bucket can still be reached by typing its id. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: fix ty diagnostics `_fastapi_request` returns `Any | None`, so reading `.session` and `.headers` off it inside a try was flagged as a possibly-missing attribute. The explicit None check is back — clearer than relying on the except to cover it. The test used `api_name=False` to make an event non-public, which the signature types as `str | None`; `api_visibility="private"` says the same thing directly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: give the recording tests room under load `test_a_normal_prediction_is_recorded` failed in CI with "no record was written". Recording is fire-and-forget, so the tests poll for the side effect rather than awaiting anything, and the 5s ceiling was reachable on a parallel runner — locally the same suite under xdist swings between 2.5s and 7s. The ceiling is now 30s, which costs a passing run nothing since the poll returns as soon as the condition holds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * history: expose bucket storage in Gradio apps * history: show new runs without a refresh, and quiet the sign-in states The history panel only fetched on mount and on the refresh button, so a run you had just made did not appear until you asked for it again. Polling would be the obvious fix and the wrong one: a refresh is a recursive bucket listing plus a download of up to 50 record files, so a timer would scale Hub traffic with viewers rather than with activity. The client already knows when its own run finished, so nothing needs discovering. `record_run` returns the stored record instead of just its id, the canvas hands it to the panel, and the panel prepends it. No Hub call at all for the case that prompted this. Repeat opens were also paying the full listing every time. A small per (token, bucket, app) cache, invalidated by `save_record` and expiring after 30s, makes reopening the panel free while still picking up writes from elsewhere. Separately, sign-in was styled as a failure in both history UIs. It is the next step, not an error: - The workflow panel's single red `.history-error` splits into an invitation (heading, bucket, and the sign-in control inline) and a real failure notice that carries red as one edge and offers a retry the panel never had. - `HistoryStorageControl` hardcoded `#991b1b`/`#fef2f2`/`#fca5a5` while every other rule in the file used theme tokens, so it ignored a custom theme. Now on `--border-color-primary`/`--body-text-color`, with red reserved for the failure variant. The toolbar's sign-in button and token field move into `HfAuthControl` so the panel mounts the same control rather than a second copy of it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * workflow: keep zoom controls clear of the history panel, hide Load on stale runs Two things the history panel got wrong once it was actually in use. The panel is a fixed 380px overlay, so it covered the zoom controls sitting at `bottom/right: 12px`. They now slide left by the panel's width while it is open, staying in the bottom-right of the visible canvas rather than jumping to the opposite corner — the same thing Figma and tldraw do when a right drawer opens. The shift tracks `min(380px, 95vw)` because the panel is also capped at 95vw, and a flat 380px would push the controls off a narrow window. Loading a run only ever calls `updateNodeData(nodeId, portId, value)`, so a record whose nodes are gone silently did nothing at all. A record is loadable exactly when every node it references still exists with a matching port, which is cheap to check against the current graph. Incompatible runs keep their card and thumbnail — the generation is still the user's, and the image is most of its value — but show "Different workflow" in place of the button. The check is deliberately all-or-nothing: a partial restore would populate some nodes and leave others stale, which looks like a bug rather than a limitation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: gradio-pr-bot <gradio-pr-bot@users.noreply.github.com> Co-authored-by: Abubakar Abid <abubakar@huggingface.co> Co-authored-by: Abubakar Abid <islamrealm@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>