PR #31775 Fixes to alternating SWA layers in Gemma2

Fixes to alternating SWA layers in Gemma2 #31775

ArthurZucker merged 4 commits into huggingface:main from turboderp:gemma2-fix

HybridCache: Flip order of alternating global-attn/sliding-attn layers

a1a3ccd6

HybridCache: Read sliding_window argument from cache_kwargs

5d9679d3

Gemma2Model: Flip order of alternating global-attn/sliding-attn layers

e45fb6e2

Code formatting

5bde080e

ArthurZucker approved these changes on 2024-07-10

ArthurZucker commented on 2024-07-10

ArthurZucker merged a695c186 into main 1 year ago

Reviewers

ArthurZucker

Assignees

No one assigned

Labels

None yet

Milestone

No milestone