Buckets API (#3673)
* [draft] Basic bucket API - subject to evolve
* xet download not working because local
* batch upload
* download resolved
* style
* slight update
* download files in parallel
* add hf sync and hf bucket ls CLI
* CLI first implem
* move to hf bucket sync
* alias hf sync
* --delete instead of --mirror
* apply instead of execute
* add --ignore-times --ignore-sizes --existing --ignore-existing / remove --force-upload --force-download
* cli ref
* more examples
* get_bucket_file_metadata
* fix typing
* create bucket revisited
* better delete repo
* repo url parsed
* create empty files
* pass self.endpoint to RepoUrl
* no need for size in batch upload
* expose bucket api
* Start adding tests
* better batch API ?
* docstrings
* first tests
* remove /latest prefix in tree and resolve endpoints
* Remove /latest from xet-write-token route
* list buckets lists only under namespace
* update test list_buckets
* list_bucket_tree returns BucketFile dataclass
* adapt to BucketFile
* more optimized hf sync
* fix get_bucket_paths_info
* batched /paths-info
* BucketUrl instead of RepoUrl
* BucketUrl in CLI
* Warn on missing files in download_bucket_files
* chunked upload
* global add-progress in batch_bucket_files
* better progress in CLI sync
* print nb files in list buckets and bucket info
* dedicated BucketNotFoundError error
* add backoff in main bucket methods
* quiet mode in CLI
* stop passing token if not needed
* alignment in table
* cannot hf bucket list files
* hf bucket cp
* fix bucket head /resolve
* fix mypy
* fix tests
* fix test
* remove some stuff
* review HfApi methods
* more tests
* more tests
* some CLI tests
* migrate to 'hf buckets' CLI
* little refacto cli tests
* add tests for hf buckets cp
* Add hf buckets sync tests
* test hf sync alias
* tree recursive
* test hf buckets tree
* draft bucket guide
* CLI guide
* export as root
* better documentation
* CLI guide
* fix mtime comparison subsecnd
* add dry run
* fix format
* hf buckets list
* not satisfying alias
* better alias
* better docs
* Fix two Bugbot-reported issues in bucket download and sync
1. Fix download_bucket_files using first file for metadata: After skipping
missing files, the code used files[0][0] to fetch Xet connection metadata,
which could be a missing file. Now tracks the first valid BucketFile during
the loop and uses that instead.
2. Fix broad exception catch in _compute_sync_plan: Changed from catching all
exceptions to only catching RepositoryNotFoundError when listing remote
files. This prevents silently swallowing auth errors, network timeouts, and
other transient failures that should be surfaced to the user.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Add tests for download_bucket_files with missing first file
This is a regression test for the fix in hf_api.py where download_bucket_files
used files[0][0] to fetch Xet connection metadata. If the first file was
missing (and skipped), this would cause a 404 error. The fix now tracks the
first valid BucketFile and uses that for metadata.
Tests added:
- test_download_bucket_files_skips_missing_first_file: Verifies download
succeeds when first file is missing
- test_download_bucket_files_raises_on_missing_when_requested: Verifies
raise_on_missing_files=True behavior
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Fix: Use BucketNotFoundError instead of RepositoryNotFoundError in sync
The bucket API raises BucketNotFoundError when a bucket doesn't exist,
not RepositoryNotFoundError. This fixes the exception handling in
_compute_sync_plan so that syncing to a non-existent bucket correctly
treats the remote as empty instead of crashing with an unhandled exception.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Add test for sync to non-existent bucket
This is a regression test for the fix that catches BucketNotFoundError
instead of RepositoryNotFoundError when listing remote files during sync.
The test verifies that syncing to a non-existent bucket correctly treats
the remote as empty (planning to upload all local files) instead of
crashing with an unhandled exception.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Fix: Validate file path in cp download path
Add validation that the source path for download includes a file name,
not just a bucket root or directory path. Without this validation,
paths like 'hf://buckets/user/my-bucket' would result in empty filename,
leading to empty local paths and incorrect behavior.
This matches the validation already done for stdin upload paths.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Fix: Strip trailing slashes from prefix to avoid double slashes in remote paths
In _execute_plan, remote paths were constructed as f"{prefix}/{op.path}",
but prefix could have a trailing slash when the user specifies a path like
'hf://buckets/user/bucket/sub/'. This produced double-slash paths like
'sub//file.txt' which may create files at incorrect paths.
Fix by stripping trailing slashes from prefix after parsing in both
upload and download paths.
Added regression test to verify trailing-slash prefixes work correctly.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Fix: Filter zero-size files before calling xet download
In download_bucket_files, zero-size files were pre-created as empty files
but still included in xet_download_infos when passed to download_files.
This could cause redundant downloads or potential errors in the xet library.
Now zero-size files are filtered out after pre-creation, and only non-zero
files are passed to the xet download function. Also added an early return
if all files are zero-size (nothing left to download).
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* BucketFolder instead of BucketDirectory
* move buckets tests to xet only
* fix hot reload reference
* finally fix tests
* explain
* Fix: Unsafe prefix stripping that could corrupt sync relative paths
The _list_remote_files function's prefix-stripping logic had a fallback
branch 'elif path.startswith(prefix)' that could match files sharing a
prefix string without a directory boundary. For example, if prefix='sub'
and the API returned 'submarine.txt', the code would strip 'sub' yielding
'marine.txt' as the relative path.
Fix by changing the second branch to only match exact path==prefix cases
(for single file downloads), not partial string matches.
Added regression test that uploads both 'sub/file.txt' and 'submarine.txt',
then syncs with prefix 'sub' to verify only the file under sub/ is
downloaded and not incorrectly processed.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Fix: Preserve progress bar state when downloading to stdout
The cp command's download-to-stdout path now checks if progress bars
were already disabled before disabling them, and only re-enables if
they weren't already disabled by the caller. This prevents undoing
a caller's intentional disabling of progress bars via global state.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Fix: Sync skip reason accurately reports dest newer instead of same mtime
When using --ignore-sizes, the skip reason now correctly distinguishes
between 'same mtime' (timestamps within 1s window) and 'remote newer' or
'local newer' (destination actually has a newer timestamp than source).
This provides more accurate information in verbose output and sync plan
files, helping users understand why files are being skipped.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Refactor: Extract shared sync comparison logic into helper function
The _compute_sync_plan function had nearly identical comparison logic
(~100 lines) duplicated between the upload and download branches.
Introduced _compare_files_for_sync helper that:
- Handles size/mtime comparisons for both directions
- Manages ignore_times/ignore_sizes/ignore_existing flags
- Returns the appropriate SyncOperation
This reduces code duplication and ensures both upload and download
paths stay in sync with identical comparison behavior.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Fix: Use Literal type for action parameter in _compare_files_for_sync
The type checker requires the action parameter to be Literal['upload', 'download']
to match the SyncOperation dataclass field type.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* Fix: Skip files that don't properly match prefix in _list_remote_files
When listing remote files with a prefix (e.g., 'sub'), the API may return
files that share the prefix string but don't belong to the directory
(e.g., 'submarine.txt'). Previously, we kept these files with their full
path, but this caused them to be incorrectly included in sync operations.
Now we skip such files entirely, ensuring that prefix-based sync only
includes files that:
1. Start with prefix + '/' (are in the directory)
2. Exactly match the prefix (single file sync)
This fixes the failing test test_sync_download_prefix_does_not_match_similar_names.
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
* make style
* Update src/huggingface_hub/cli/buckets.py
Co-authored-by: célina <hanouticelina@gmail.com>
* Update src/huggingface_hub/hf_api.py
Co-authored-by: célina <hanouticelina@gmail.com>
* Update docs/source/en/guides/buckets.md
Co-authored-by: célina <hanouticelina@gmail.com>
* Update src/huggingface_hub/hf_api.py
Co-authored-by: célina <hanouticelina@gmail.com>
* Update src/huggingface_hub/hf_api.py
Co-authored-by: célina <hanouticelina@gmail.com>
* docstring
* constant
* regenerate hf --help in cli.md
* Add test for content_type resolution
* refacto _split_bucket_id_and_prefix + add test$
* export BucketFolder
* fix bucket_files_by_path not defined
* fix hf spaces examples
* fix mypy
* Update docs/source/en/guides/buckets.md
Co-authored-by: célina <hanouticelina@gmail.com>
* Update src/huggingface_hub/cli/_cli_utils.py
Co-authored-by: célina <hanouticelina@gmail.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Lucain <Wauplin@users.noreply.github.com>
Co-authored-by: célina <hanouticelina@gmail.com>