PR #31325 Offloaded KV Cache

Offloaded KV Cache #31325

ArthurZucker merged 8 commits into huggingface:main from n17s:offloaded-cache

gante commented on 2024-06-14

gante approved these changes on 2024-06-27

gante requested a review from

ArthurZucker 1 year ago

n17s force pushed 1 year ago

ArthurZucker commented on 2024-07-22

ArthurZucker commented on 2024-07-23

Initial implementation of OffloadedCache

8e57b081

enable usage via cache_implementation

d0e86661

Address feedback, add tests, remove legacy methods.

2e63564f

Remove flash-attn, discover synchronization bugs, fix bugs

cf31e0ea

n17s force pushed to cf31e0ea 1 year ago

Prevent usage in CPU only mode

daf8702e

ArthurZucker commented on 2024-07-24

Add a section about offloaded KV cache to the docs

47950328

Fix typos in docs

1a76762f

ArthurZucker commented on 2024-07-26

Clarifications and better explanation of streams

667811a6

ArthurZucker approved these changes on 2024-08-01

ArthurZucker merged ca59d6f7 into main 1 year ago

Reviewers

ArthurZucker

gante

Assignees

No one assigned

Labels

None yet

Milestone

No milestone