onnxruntime
8c98fa14 - Fix CPU MHA shared cache length handling (#32723)

Commit
54 days ago
Fix CPU MHA shared cache length handling (#32723) This pull request introduces several improvements and bug fixes to the CPU implementation of the `MultiHeadAttention` operator, particularly focusing on the handling of shared cache buffers (`past_present_share_buffer`). The main changes ensure correct memory operations, add input validation, and improve test coverage for shared cache scenarios. **Shared cache buffer handling:** - Updated memory copy operations in `attention_cpu_base.h` to use `kv_input_chunk_length` instead of `head_size` or `v_head_size` when copying key and value tensors, ensuring correct buffer sizes are used when `past_present_share_buffer` is enabled. [[1]](diffhunk://#diff-74032c915b0807c974428ab3ef12b84bb7bd16775f852a15c479837f039a2eeaL328-R329) [[2]](diffhunk://#diff-74032c915b0807c974428ab3ef12b84bb7bd16775f852a15c479837f039a2eeaL394-R396) [[3]](diffhunk://#diff-74032c915b0807c974428ab3ef12b84bb7bd16775f852a15c479837f039a2eeaL548-R551) - Modified present tensor allocation in `GetPresent` to only occur when both `past_sequence_length == 0` and `past_present_share_buffer` is false, preventing unnecessary allocations. **Input validation:** - Added a check in `multihead_attention_helper.h` to ensure that the sum of `past_sequence_length` and `kv_sequence_length` does not exceed `max_sequence_length` when using a shared cache buffer, preventing out-of-bounds memory access. **Operator logic and test improvements:** - Changed logic in `multihead_attention.cc` to always perform buffer copying when `past_present_share_buffer` is enabled, and updated function calls to propagate new parameters. [[1]](diffhunk://#diff-577d5da8f5509ea08b0ad079b5ac38f7e3667b3be937ed4f9ff975e737c9593aL269-R289) [[2]](diffhunk://#diff-577d5da8f5509ea08b0ad079b5ac38f7e3667b3be937ed4f9ff975e737c9593aL306-R308) - Added new unit tests to verify correct behavior for shared cache initialization and to ensure the operator rejects sequences that exceed the cache capacity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Author
Parents
Loading