transformers
Generate: Mistral/Mixtral FA2 cache fix when going beyond the context window
#28037

Merged

Generate: Mistral/Mixtral FA2 cache fix when going beyond the context window #28037

gante merged 8 commits into huggingface:main from gante:mixtral_cache

add breakpoint

aacffa85

only slice if the cache has contents

70cc5023

get the right slice

564750c0

f0380663

3f24fb01

tmp commit

d4867f70

fix FA2 + mis(x)tral

cf7cc6dd

gante commented on 2023-12-14

gante requested a review from

amyeroberts 2 years ago

tomaarsen commented on 2023-12-14

add self.layer_idx exception (as suggested by @tomaarsen)

fbfb543a

amyeroberts approved these changes on 2023-12-14

gante merged 388fd314 into main 2 years ago

gante deleted the mixtral_cache branch 2 years ago

Reviewers

amyeroberts

tomaarsen

Assignees

No one assigned

Labels

None yet

Milestone

No milestone