Adding support for alibi when using flash attention (#820)
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* ..
* Update llmfoundry/models/layers/attention.py
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
* ..
* ..
* ..
* ..
* ..
* Update tests/models/layers/test_flash_attn.py
Co-authored-by: Irene Dea <deaairene@gmail.com>
* ..
* ..
* Update tests/models/layers/test_flash_attn.py
Co-authored-by: Irene Dea <deaairene@gmail.com>
* ..
---------
Co-authored-by: Shashank Rajput <shasshank.rajput@databricks.com>
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
Co-authored-by: Daniel King <43149077+dakinggg@users.noreply.github.com>
Co-authored-by: Irene Dea <deaairene@gmail.com>
