Xet integration (#2958)
* add hf_xet as an optional dependency
* update installed packages at runtime
* split xet testing in CI
* fix workflow
* fix windows
* Xet download workflow (#2875)
* first draft
* remove comment
* hf_xet instead of xet
* update docstring
* fix
* update docstring
* simplify typing
* quality
* add logging
* fix tests
* add unit tests for xet utilities
* first draft of download testing
* more tests
* address some comments
* fix tests
* check if hf_xet is available or not
* remove unnecessary dest dir creation
* keep comment
Co-authored-by: Lucain <lucain@huggingface.co>
* post-review improvements
* Update tests/test_xet_download.py
---------
Co-authored-by: Lucain <lucain@huggingface.co>
* Add ability to enable/disable xet storage on a repo (#2893)
* add ability to enable/disable xet storage
* add test
* better way to check if all settings are none
* don't strip authorization header with downloading with xet
* update comment
* Xet upload workflow (#2887)
* add upload workflow
* fixes and tests
* use helper for prgress bar
* use tmp repo in tests
* some fixes
* update tests
* mock HF_XET_CACHE
* fix tests
* fix utils tests
* debug CI
* fix
* check if xet is enabled
* debug CI
* debug CI again
* revert
* debugging
* don't rerun xet tests
* revert
* remove pytest timeout
* don't run tests in parallel
* add comment
* revert and rename variable
* don't skip tests
* remove warning
* fix tests
* Apply suggestions from code review
* fixes
* fix syntax error with python 3.8
* catch Invalid credentials
* fix
* record Space API VCR test
* use raise instead of raise e
Co-authored-by: Lucain <lucain@huggingface.co>
* disable xet storage for the other tests
* reverting
* isolate xet tests for windows
* fix windows
* install hf_xet for xet testing
---------
Co-authored-by: Lucain <lucain@huggingface.co>
Co-authored-by: Lucain Pouget <lucainp@gmail.com>
* Xet Docs for huggingface_hub (#2899)
* Xet docs
* PR feedback, added waitlist links
* Added HF_XET_CACHE env variable docs
* PR feedback
* Doc feedback
* Added two lines about flow of upload/download
* Updating links to Hub doc location
* Reformat headings, less levels in TOC
---------
Co-authored-by: Julien Chaumond <julien@huggingface.co>
Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>
Co-authored-by: Célina <hanouticelina@gmail.com>
Co-authored-by: Lucain <lucainp@gmail.com>
* Adding Token Refresh Xet Test (#2932)
Directly calling hfxet.download_files() with token_refresher callback
to ensure that hfxet calls the token refresher as expected.
---------
Co-authored-by: Celina Hanouti <hanouticelina@gmail.com>
* Using a two stage download path for xet files. (#2920)
* Adding request header on resolve endpoint indicating that we can receive xet info.
* Adding test to ensure that the header is always sent on metdata request
* Using a two stage download path for xet files.
* Using the GET call's JSON
* Using xet_backed for the whether the file is a xet file or not to disambiguate from whether xet is enabled
* Adding and fixing tests
* Testing fix WIP
* Rewriting xet download to use the refresh route to resolve the xetmetadata
* Parameter type check
* Docs
* Removing extraneous constant
* Fixing file_download tests
* Readding the refresh route into the file metadata
* Refactoring the XetMetadata object into two objects to reflect the Hub changes.
* Fixing broken tests
* Code cleanup from self review
* Fixing types
* Quality & Lint
* Handling when hub returns the entire refresh route in its headers.
* Update tests/test_xet_utils.py
* Fixing merge conflicts in the new tests
* Extracting the refresh route from the link header (#2953)
* Getting the refresh route from the links header
* refactor xet_file_data func signature & tests
Co-authored-by: Lucain <lucain@huggingface.co>
Co-authored-by: Rajat Arya <rajat@huggingface.co>
* Update src/huggingface_hub/constants.py
Co-authored-by: Célina <hanouticelina@gmail.com>
---------
Co-authored-by: Celina Hanouti <hanouticelina@gmail.com>
Co-authored-by: Rajat Arya <rajatarya@users.noreply.github.com>
Co-authored-by: Julien Chaumond <julien@huggingface.co>
Co-authored-by: Pierric Cistac <Pierrci@users.noreply.github.com>
Co-authored-by: Brian Ronan <brian.ronan@huggingface.co>
Co-authored-by: Rajat Arya <rajat@huggingface.co>
