text-generation-inference
Kvrouter that will increase the kv-cache hits in case of multiple routing strategy
#2965
Open
Overview
Commits
9
Changes
View On GitHub

Kvrouter that will increase the kv-cache hits in case of multiple routing strategy #2965

Narsil wants to merge 9 commits into main from kvrouter
Narsil
Narsil More logs in the allocator.
7ef8b89e
Narsil Adding Dummy kvrouter.
6a88063c
Narsil Updating the kvrouter to support roundrobin
0a495ad1
Narsil Remove kvrouter from default members.
914b1637
Narsil Adding kvrouter to the workspace.
1932c5b9
Narsil Cleaner version.
57fa04ad
Narsil Narsil force pushed from ded50855 to 57fa04ad 1 year ago
Narsil Narsil changed the title [WIP] Kvrouter that will increase the kv-cache hits in case of multiple routing strategy Kvrouter that will increase the kv-cache hits in case of multiple routing strategy 1 year ago
Narsil CI must be green.
50c8ebde
HuggingFaceDocBuilderDev
Narsil Fixing the Trie in case of exact prefix match split.
3aa85646
Narsil Tiny fix.
338cdc2e

Login to write a write a comment.

Login via GitHub

Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Terms Of Use Privacy Policy Refund Policy Impressum