onnxruntime
68831ecd - Update eval_squad to use API of latest optimum (#17918)

Commit
2 years ago
Update eval_squad to use API of latest optimum (#17918) Update eval_squad with latest optimum. Tested with: * optimum 1.13.1 * transformers 4.31.0 * onnxruntime-gpu 1.16.0 * onnx 1.14.1 * datasets 2.14.5 * evaluate 0.4.0 * torch version 2.2.0.dev20230920+cu121 Example output in A100: {'exact': 86.66035950804162, 'f1': 92.99622739711005, 'total': 10570, 'HasAns_exact': 86.66035950804162, 'HasAns_f1': 92.99622739711005, 'HasAns_total': 10570, 'best_exact': 86.66035950804162, 'best_exact_thresh': 0.9998456239700317, 'best_f1': 92.9962273971104, 'best_f1_thresh': 0.9998456239700317, 'total_time_in_seconds': 84.74025378189981, 'samples_per_second': 124.73410838731417, 'latency_in_seconds': 0.008017053337928081, 'provider': 'CUDAExecutionProvider', 'disable_fused_attention': False, 'pretrained_model_name': 'bert-large-uncased-whole-word-masking-finetuned-squad', 'onnx_path': './bert-large-uncased-whole-word-masking-finetuned-squad/optimized_model.onnx', 'batch_size': 1, 'sequence_length': 384, 'use_io_binding': True}
Author
Committer
Parents
Loading