Llama: RoPE refactor #32135
gante
merged
30 commits into
huggingface:main
from
gante:llama_rope_refactor
gante
commented
on 2024-07-22
gante
commented
on 2024-07-22
amyeroberts
approved these changes
on 2024-07-22
Add YaRN and Dynamic-YaRN RoPE Scaling Methods
1e1d6b46
Refactor YaRN implementation for LLaMA
7efbbc3e
Refactor Tensor Building Logic for YaRN
f2122cde
remove unwanted file
b8df7a2b
all diff except the llama folder
0166869e
add updated config
7e4e4d82
add updated rope class (and break related copies)
65641950
related classes
95304b55
llama attention
3904d32e
fa2 (and break a few more copies)
c7837ebd
sdpa (and break a few more copies)
e5e1cde1
up to the model class
f68b9cd5
up to ForSequenceClassification
2f5ace35
last set?
5d352872
missing this one
4c56e43d
make fixup
f36ec3a7
Update src/transformers/modeling_rope_utils.py
35699b36
Update src/transformers/modeling_rope_utils.py
b095ebbd
rename 'type' and 'scaling_type' to a clearer 'rope_type'
3f6458b9
abstract out key validation
6d315cae
safety getattr; explicit docstring
5809b5eb
docstring nit
a7502ed8
add tests
3bc7c520
remove external position_embeddings interface
39e216a2
test nit
000aeba7
Update src/transformers/models/llama/modeling_llama.py
80a04229
Update src/transformers/models/llama/modeling_llama.py
48ed2510
make fixu
c824be06
Merge branch 'main' into llama_rope_refactor
fc1255ef
make fixup and make fix-copies
75b23915
Login to write a write a comment.
Login via GitHub