onnxruntime
Add flash attention v2 and INT4 CUDA for LLaMA E2E benchmarking
#20149
Merged
Merged
Add flash attention v2 and INT4 CUDA for LLaMA E2E benchmarking #20149
kunal-vaishnavi
merged
5 commits into
microsoft:main
from
kunal-vaishnavi:kvaishnavi/llama-add-flash-attn
Enable flash attention v2 for PyTorch models when benchmarking
0fce15e0
Add instructions for installing flash attention v2
701d5f3b
Add INT4 CUDA benchmarking for PyTorch eager
15f0ab6a
Add instructions for installing PyTorch quantization
3232e42d
hanbitmyths
commented
on 2024-03-29
Use flash attention v2 for CUDA and SDPA for CPU
3e7b79e6
hanbitmyths
commented
on 2024-03-29
hanbitmyths
approved these changes
on 2024-03-29
Login to write a write a comment.
Login via GitHub