[benchmarks] overhaul benchmarks (#11565)
* start overhauling the benchmarking suite.
* fixes
* fixes
* checking.
* checking
* fixes.
* error handling and logging.
* add flops and params.
* add more models.
* utility to fire execution of all benchmarking scripts.
* utility to push to the hub.
* push utility improvement
* seems to be working.
* okay
* add torchprofile dep.
* remove total gpu memory
* fixes
* fix
* need a big gpu
* better
* what's happening.
* okay
* separate requirements and make it nightly.
* add db population script.
* update secret name
* update secret.
* population db update
* disable db population for now.
* change to every monday
* Update .github/workflows/benchmark.yml
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
* quality improvements.
* reparate hub upload step.
* repository
* remove csv
* check
* update
* update
* threading.
* update
* update
* updaye
* update
* update
* update
* remove peft dep
* upgrade runner.
* fix
* fixes
* fix merging csvs.
* push dataset to the Space repo for analysis.
* warm up.
* add a readme
* Apply suggestions from code review
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
* address feedback
* Apply suggestions from code review
* disable db workflow.
* update to bi weekly.
* enable population
* enable
* updaye
* update
* metadata
* fix
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: Luc Georges <McPatate@users.noreply.github.com>
