llm-foundry

148c0793 - Add runtime error in eval.py if yaml config is improperly formatted with extraneous or missing values (#521)

Add runtime error in eval.py if yaml config is improperly formatted with extraneous or missing values (#521) ## Description This PR enables us to sanity check our eval yaml configuration files before we run the full eval pipeline. This enables us to catch errors in the YAML config prior to an eval run starting. If a yaml config is improperly formatted with extraneous or missing values, a runtime error will be thrown. ## Unit Test: Added unit tests to make sure we raise `omegaconf.errors` and/or warn users if the yaml is incorrectly formatted. Warnings are used if the parameter has an optional default value. ## Integration Test: **Test 1**: Model reaches 0.56 COPA accuracy before and after refactor (1 GPU) ``` python composer eval/eval.py \ eval/yamls/hf_eval.yaml \ icl_tasks=eval/yamls/copa.yaml \ model_name_or_path=mpt-125m-hf ``` **Test 2**: Model gauntlet runs: `mcli run -f mcli/mcli-hf-eval.yaml --follow` Main branch: `mcli logs -f all-eval-main-nphii5` ``` Ran mosaicml/mpt-7b-instruct eval in: 5852.91842341423 seconds Printing gauntlet results for all models | model_name | average | world_knowledge | commonsense_reasoning | language_understanding | symbolic_problem_solving | reading_comprehension | |:-------------------------|----------:|------------------:|------------------------:|-------------------------:|---------------------------:|------------------------:| | mosaicml/mpt-7b-instruct | 0.354255 | 0.398764 | 0.415097 | 0.371509 | 0.171216 | 0.414691 | Printing complete results for all models | Category | Benchmark ``` chuck/add_yaml_check_eval branch: `mcli logs -f all-eval-chuck-branch-UqSCPR` ``` Ran mosaicml/mpt-7b-instruct eval in: 5858.392446279526 seconds Printing gauntlet results for all models | model_name | average | world_knowledge | commonsense_reasoning | language_understanding | symbolic_problem_solving | reading_comprehension | |:-------------------------|----------:|------------------:|------------------------:|-------------------------:|---------------------------:|------------------------:| | mosaicml/mpt-7b-instruct | 0.354255 | 0.398764 | 0.415097 | 0.371509 | 0.171216 | 0.414691 | Printing complete results for all models | Category | Benchmark ``` ## Issues Addressed This PR chips away at this debt [issue](https://mosaicml.atlassian.net/browse/RESEARCH-717).