Initial TorchBench Summary Metadata Validation (#526)
Summary:
This is a duplicate of PR https://github.com/pytorch/benchmark/issues/489. Somehow I cannot update the previous forked branch.
# TorchBench Summary Metadata Validation V0
To improve model quality, TorchBench will introduce a summary metadata feature to ensure clear system-level configurations, allow easy generation with defaults, and enforce validation against expected metadata.
*Requirements:*
* Enable the user to see the expected configurations for each model
* Enable TorchBench runs to check against expected system-level configurations
*Goals:*
* Clearly show the default configurations used by each model (also used in the CI)
* Be easy to extract, generate, re-generate and verify
* Allow new metadata to be added without negative impact
* Guarantee well-formed structure (through yaml)
## List of Metadata
PyTorch Name | Details | Attribute | Default
-- | -- | -- | --
backends.cudnn.benchmark | enables benchmark mode in PyTorch | train_benchmark | TRUE
backends.cudnn.deterministic | | train_deterministic | FALSE
backends.cudnn.benchmark | enables benchmark mode in PyTorch | eval_benchmark | TRUE
backends.cudnn.deterministic | deterministic is not always required in eval | eval_deterministic | FALSE
set_grad_enabled() | grad is not required at inference for most models | eval_nograd | TRUE
jit.optimize_for_inference | model is optimized for inference | jit_optimized_for_inference | FALSE
### Future Considerations
* What are the shapes of the input tensors?
* Where does the model come from?
* What are the tests supported (eg. is cuda/jit supported)?
* What are the operators covered by this model?
* Is this model memory or compute intensive?
* yaml may not be enough to capture complex details - look to adapting in the future
## Implementation
This section will outline the tools to enable (1) extraction and generation, (2) subprocess reading and checking against expected, and (3) updating or modifying existing summaries.
### 1. Extractor and Generator of Metadata (Standalone script)
A python script (ex. gen_summary_metadata.py) to handle extracting and generating *metadata.yaml* files.
*Goal:* Completely mechanical (Attributes are deduced in the Python class, the yaml file simply records them and provides a way to catch changes.)
#### Extractor
* Detect the facts about models programmatically through module imports and attribute extractions
* Models should specify certain system level configurations explicitly
#### Generator
* Set defaults for missing or unknown attributes
* Push metadata files directly inside of repo structure
### 2. Subprocess Validation (Launched by test scripts)
* Benchmark (test_bench.py): ModelTask will perform checks against static config for security and verification
* Model runs will import metadata and extract model details
* Model runs will compare relevant model attributes against what is expected in metadata
* Unit Tests (test.py): Perform checks against static metadata in 3 key phases
* After init, after train and after eval to verify that system settings are valid
### 3. Metadata Updates (Users) - **TODO** currently users manually update model __init__.py files.
## Location of Metadata Files
* Config files live beside each model’s __init__.py file
* torchbenchmark/models/
* BERT_pytorch/
* \<model files\>
* __init__.py
* *metadata.yaml*
* Background_Matting/
* \<model files\>
* __init__.py
* *metadata.yaml*
* ...
## Example of a Summary Metadata File (metadata.yaml)
```
eval_benchmark: false
eval_deterministic: false
eval_nograd: true
optimized_for_inference: true
train_benchmark: false
train_deterministic: false
```
Pull Request resolved: https://github.com/pytorch/benchmark/pull/526
Reviewed By: xuzhao9
Differential Revision: D31938494
Pulled By: aaronenyeshi
fbshipit-source-id: 4d8be9658c396c4b55725e297eb520f69f0a015e
