onnxruntime
Add support for custom position ids and attention bias to GQA CPU operator
#23944

Merged

Add support for custom position ids and attention bias to GQA CPU operator #23944

derdeljan-msft merged 25 commits into main from derdeljan/gqa-tree-decoding

Scalar support for custom position ids and mask in GQA

d7f5aa1a

Vectorized attention mask application for fp32

15172c3b

Vectorized attention mask application for fp16

d7eae786

Add mask upscale to fp32 if the platform doesn't support fp16

9d244dd7

Fix typo in fp16 eltwise kernels

8faee661

Add validation for custom attention parameters

147d19b6

Add mlas unit test for eltwise kernels

4b1262eb

Refactor python unit GQA tests

f7a07881

Cleanup comments

9dec0564

derdeljan-msft requested a review from

tianleiwu 1 year ago

derdeljan-msft requested a review from

jywu-mysoft 1 year ago

derdeljan-msft requested a review from

fajin-corp 1 year ago

derdeljan-msft requested a review from

aciddelgado 1 year ago

derdeljan-msft assigned

derdeljan-msft 1 year ago

derdeljan-msft requested a review 1 year ago

derdeljan-msft changed the title ~~Add support custom position ids and attention mask to GQA CPU operator~~ Add support for custom position ids and attention mask to GQA CPU operator 1 year ago

github-advanced-security commented on 2025-03-07

jywu-mysoft requested a review from

liqunfu 1 year ago

kunal-vaishnavi commented on 2025-03-07

aciddelgado commented on 2025-03-07

Fix CI pipeline errors

5d23817e

github-actions commented on 2025-03-07

Apply suggestions from code review

42e83d68

Fix docs pipeline build

bc0d69b9

Fix docs pipeline build

ab60cbc6

fajin-corp commented on 2025-03-10

Fix first batch of PR comments

4e0ca5c2

Fix PR comments

949118f5

github-actions commented on 2025-03-13

Linter fix

62d39a5a

tianleiwu commented on 2025-03-13

Update attention_mask input description

0349678c

Fix build break

0865ddbf

Fix docs gen CI pipeline

55e09c9a

tianleiwu commented on 2025-03-13

Apply attention mask after softcap

e3bc338c

Cleanup mlas eltwise module

757af32c

tianleiwu commented on 2025-03-13

Fix PR comments

0c268c94

github-actions commented on 2025-03-13

Fix position_ids handling for the first prompt

c36a9cfd

Fix build break

86a7737f

derdeljan-msft requested a review from

tianleiwu 1 year ago

derdeljan-msft requested a review from

fajin-corp 1 year ago

derdeljan-msft changed the title ~~Add support for custom position ids and attention mask to GQA CPU operator~~ Add support for custom position ids and attention bias to GQA CPU operator 1 year ago

tianleiwu commented on 2025-03-14

Fix PR comments and fix docs gen CI pipeline

56fe7683

tianleiwu approved these changes on 2025-03-14

fajin-corp approved these changes on 2025-03-14

derdeljan-msft merged 5a694bcb into main 1 year ago

derdeljan-msft deleted the derdeljan/gqa-tree-decoding branch 1 year ago

Reviewers

tianleiwu

fajin-corp

github-actions

aciddelgado

kunal-vaishnavi

github-advanced-security

jywu-mysoft

liqunfu

Assignees

derdeljan-msft

Labels

None yet

Milestone

No milestone

onnxruntime Add support for custom position ids and attention bias to GQA CPU operator #23944 Merged

Add support for custom position ids and attention bias to GQA CPU operator #23944

onnxruntime
Add support for custom position ids and attention bias to GQA CPU operator
#23944

Merged