onnxruntime
Attention kernel update to handle different Q,K,V hidden sizes
#8039
Merged
Overview
Commits
54
Changes
View On GitHub

Attention kernel update to handle different Q,K,V hidden sizes #8039

viboga merged 54 commits into master from Vish/tnlrv4_opt
viboga
viboga changes working to convert akv nodes
4c8d7b61
viboga changes to replace nodes
56b917ce
viboga changes to accomodate qkv hidden sizes as attributes
27ca20e5
viboga kernel to accept qkv_hidden_size attributes
f14c03f4
viboga viboga requested a review 4 years ago
viboga viboga changed the title Vish/tnlrv4 opt [Not ready for review yet] Vish/tnlrv4 opt 4 years ago
viboga viboga marked this pull request as draft 4 years ago
viboga Merge remote-tracking branch 'origin/master' into Vish/tnlrv4_opt
4f281dd6
viboga Merge remote-tracking branch 'origin/master' into Vish/tnlrv4_opt
498bdd5c
viboga Working till compute for varied dimension, todo applyattention()
7d23451b
viboga changes to make all regression tests work
6c52522f
viboga inference running successfully without prepack
23efce57
viboga success inference with pre-pack weights
d06981e9
viboga add test for diff sizes
614fdbef
viboga bias shape need not be a mul of 3
4ed2be15
viboga get the output_hidden_size from input
65895565
viboga infer output shape from input
c8d34026
viboga merge with master
43109b59
viboga merge with master
31917f38
viboga Merge remote-tracking branch 'origin/master' into Vish/tnlrv4_opt
5404524f
viboga cleaning up files that got merged wrong
1918bfcb
viboga accurancy at accepted level
b75740f0
viboga added unit test case for different dimensions
c5d38328
viboga all unit tests passing
e2895644
viboga packed weights working for attention
debf9511
viboga prepacked weights working
51f45cb4
viboga added test case for newly added extra qk input
acaa5f79
viboga updated unit test to test only extra add qk
07268ef6
viboga fixing build error
eb2c07dd
viboga removing few debugs
637ac36a
viboga reverting test changes
fe58b8e6
viboga all python test passing
f4d276ab
viboga cleaning up
48920b66
viboga new unit test added, major clean up of code
f6b7258e
viboga removed extra code
2e4958a7
viboga minor
af217379
viboga minor fix to tests
aaf9bbbe
viboga prepack weights code cleaned up
86b64975
viboga compacted compute() in attention.cc
690a62a7
viboga reformat compute()
e64d01d6
viboga making a parameter T
5abf7a4f
viboga adding 3 q,k,v buffers in all cases
986801db
viboga viboga assigned yufenglee yufenglee 4 years ago
viboga viboga assigned tianleiwu tianleiwu 4 years ago
viboga viboga assigned wangyems wangyems 4 years ago
viboga viboga marked this pull request as ready for review 4 years ago
viboga Merge remote-tracking branch 'origin/master' into Vish/tnlrv4_opt
9f66e606
viboga
viboga commented on 2021-07-09
viboga viboga changed the title [Not ready for review yet] Vish/tnlrv4 opt Attention kernel update to handle different Q,K,V hidden sizes 4 years ago
viboga fixing build
d0f79bbb
viboga running tests only on cpu
17b901e5
viboga Merge remote-tracking branch 'origin/master' into Vish/tnlrv4_opt
f2f1f375
viboga Updating docs
8dc7824b
viboga Merge remote-tracking branch 'origin/master' into Vish/tnlrv4_opt
86aa84bc
viboga trigger ci builds
b8f05faf
viboga Merge remote-tracking branch 'origin/master' into Vish/tnlrv4_opt
e6bb52a9
wangyems
wangyems commented on 2021-07-14
yufenglee
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
tianleiwu
tianleiwu commented on 2021-07-15
viboga
viboga Addressing comments in PR
ae7a256f
viboga addressing some more comments
aabfa46b
viboga get add_qk_str from add_qk node directly
491f600c
viboga updating docs, added extra check to verify attn inputs
84b3ec39
tianleiwu
tianleiwu requested changes on 2021-07-16
viboga Optimized the extra add by parallelizing
324350e6
tianleiwu
tianleiwu commented on 2021-07-16
tianleiwu
tianleiwu commented on 2021-07-16
tianleiwu
tianleiwu commented on 2021-07-16
viboga added attention_shape to symbolic_shape_infer.py
02044e31
viboga minor refactoring to address comments
35fa8088
tianleiwu
tianleiwu approved these changes on 2021-07-19
tianleiwu
tianleiwu commented on 2021-07-19
viboga viboga merged afce0e25 into master 4 years ago
viboga viboga deleted the Vish/tnlrv4_opt branch 4 years ago
hariharans29
hariharans29 commented on 2021-07-19
tianleiwu
tianleiwu commented on 2021-07-19

Login to write a write a comment.

Login via GitHub

Reviewers
Assignees
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum