pytorch
d4045e9a - initial commit to refactor all s3 access codes to s3_stats_parser (#54681)

Commit
3 years ago
initial commit to refactor all s3 access codes to s3_stats_parser (#54681) Summary: First step to move all S3 related operations into S3 parser utils. in the end we provide APIs from s3_stats_parser: 1. downloading data as reports and uploading data as reports 2. filter by job name and handle all compression, formatting inside. TODO - [ ] Refactor out upload into s3_stats_parser - [ ] Remove all S3/BOTO related checkers and try/catch blocks outside of s3_stats_parser Pull Request resolved: https://github.com/pytorch/pytorch/pull/54681 Test Plan: 1. Running tools/test/* covers the refactoring logic (test_test_history.py and test_stats.py as entrypoint and both using the 2 new APIs in s3_stats_parser after the refactoring. 2. print_test_stats.py's main argparse entrypoint is covered by CI step Report Test Result step. 3. run `python test/run_test.py --export-past-test-times` before and after this PR should result in the same file content in .pytorch-test-times Reviewed By: ailzhang Differential Revision: D27346742 Pulled By: walterddr fbshipit-source-id: fb40162e631e007fed9d5821fe4f190bda2cb52e
Author
Rong Rong (AI Infra)
Parents
Loading