PR #4965 Enhance query APIs for text generation

Enhance query APIs for text generation #4965

mrwyattii merged 9 commits into master from tohtana/api_remaining_capacity

add api to get remaining capacity in block

f6727e1a

add option to skip check in put()

a72f3f07

use int values for kv cache blocks instead of torch tensors

cc9ad227

fix condition to skip checking schedulability

d894f100

Merge branch 'master' into tohtana/api_remaining_capacity

e7a470d4

tohtana marked this pull request as ready for review 1 year ago

tohtana requested a review from

mrwyattii 1 year ago

tohtana requested a review from

awan-10 1 year ago

tohtana requested a review from

arashb 1 year ago

arashb approved these changes on 2024-01-16

Merge branch 'master' into tohtana/api_remaining_capacity

b2b0b81b

mrwyattii commented on 2024-01-18

add docstring and comment

e1c8d9d2

Merge branch 'tohtana/api_remaining_capacity' of github.com:microsoft…

6aa1737d

mrwyattii approved these changes on 2024-01-18

Merge branch 'master' into tohtana/api_remaining_capacity

1353c86c

mrwyattii merged 5dea776a into master 1 year ago

mrwyattii deleted the tohtana/api_remaining_capacity branch 1 year ago

Reviewers

mrwyattii

arashb

awan-10

Assignees

No one assigned

Labels

None yet

Milestone

No milestone