transformers
RoPE loses precision for Llama / Gemma + Gemma logits.float()
#29285
Merged
Merged
RoPE loses precision for Llama / Gemma + Gemma logits.float() #29285
ArthurZucker
merged
20 commits into
huggingface:main
from
unslothai:main
Update modeling_llama.py
7a257201
Update modeling_llama.py
db8237f4
Update modeling_gemma.py
3de95c42
ArthurZucker
commented
on 2024-02-26
ArthurZucker
commented
on 2024-02-27
ArthurZucker
commented
on 2024-02-27
Merge branch 'huggingface:main' into main
9e5cbb06
@torch.no_grad()
99d564e7
@torch.no_grad()
d0c08bf6
ArthurZucker
approved these changes
on 2024-02-27
Merge branch 'huggingface:main' into main
bd3a2142
Cos, Sin to float32
abffebb6
cos, sin to float32
c2e31bf4
Update src/transformers/models/gemma/modeling_gemma.py
f487800f
Update src/transformers/models/llama/modeling_llama.py
c8526756
Resolve PR conflicts
1a50a4bc
Fix RoPE for llama
b860a22d
Revert "Fix RoPE for llama"
790e4a3a
Merge remote-tracking branch 'upstream/main'
06c76346
Fix RoPE for llama
aa03a433
gante
approved these changes
on 2024-02-28
RoPE device
5730a503
ArthurZucker
commented
on 2024-02-28
Autocast device type
31cea3b3
ArthurZucker
commented
on 2024-02-28
RoPE
ae9957f3
RoPE isinstance
ec9ef17f
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub