CLI to upload arbitrary huge folder (#2254)
* still an early draft
* this is better
* fix
* revampt/refactor download process
* resume download by default + do not upload .huggingface folder
* compute sha256 if necessary
* fix hash
* add tests + fix some stuff
* fix snapshot download tests
* fix test
* lots of docs
* add secu
* as constant
* dix
* fix tests
* remove unused code
* don't use jsons
* style
* Apply suggestions from code review
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Apply suggestions from code review
Co-authored-by: Lysandre Debut <hi@lysand.re>
* Warn more about resume_download
* fix test
* Add tests specific to .huggingface folder
* remove advice to use hf_transfer when downloading from cli
* fix torhc test
* more test fix
* feedback
* First draft for large upload CLI
* Fixes + CLI
* verbose by default
* ask for report
* line
* suggested changes
* more robust
* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* comment
* commen
* robust tests
* fix CI
* ez
* rules update
* more ribust?
* allow for 1s diff
* don't raise on unlink
* style
* robustenss
* tqdm while recovering
* make sure upload paths are correct on windows
* test get_local_upload_paths
* only 1 preupload LFS at a time if hf_transfer enabled
* upload one at a time if hf_transfer
* Add waiting workers in report
* better reporting
* raise on KeyboardInterrupt + can disable bars
* fix type annotation on Python3.8
* make repo_type required
* dcostring
* style
* fix circular import
* docs
* docstring
* init
* guide
* dedup
* instructions
* add test
* styl
* tips
* Apply suggestions from code review
Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
* typo
* remove comment
* comments
* move determine_task to its own method
* rename to upload_large_folder
* fix md
* update
* dont wait on exit
* Fix typo in docs
* Apply suggestions from code review
Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
* add PR tips
* add comment
* add comment about --no-bars
---------
Co-authored-by: Lysandre Debut <hi@lysand.re>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>
