Add JetMoE model #30005
ArthurZucker
merged
122 commits into
huggingface:main
from
main
init jetmoe code
b63bcafb
Merge branch 'huggingface:main' into main
03f646e8
update archive maps
ed52b578
remove flax import
150cd931
fix import error
436a44cd
update README
bcf597f6
ruff fix
5c0400ec
update readme
e61d131f
fix
57b13ebc
update config
1f27ad43
fix issue
2ea5542f
merge files
109a8c2f
fix model bug
21a4c2d7
fix test
9d542ac2
auto fix
c5092b46
model size
41f24364
add comments
3052ce8d
fix form
539cfb90
add flash attention support
0f6af1d5
fix attention head number
165e20d4
fix init
68633f9b
fix support list
d39a0e9e
sort auto mapping
ef62bf3f
fix test
c0a3076a
fix docs
4d79ce65
update test
e5336b50
fix test
67aedd15
fix test
c87de945
change variable name
2f02e7e9
fix config
fc39dcc2
fix init
b4b57381
update format
c370377d
clean code
f443c293
fix config
852ef61b
fix config
9517a2b8
change default config
30b826db
update config
a18a67a6
fix issues
88991b59
Merge branch 'main' into main
7d23d95a
update formate
28ed7c43
update config argument
913dc9e6
update format
5611f096
Update src/transformers/models/jetmoe/modeling_jetmoe.py
430ea6c6
Update src/transformers/models/jetmoe/modeling_jetmoe.py
3aff0697
Merge branch 'main' into main
5528995b
change to mixtral aux loss
c51e9870
change to cache_position
4927e607
debug
cc89ea38
fix bugs
4dcbd268
Merge branch 'main' into main
7a379707
debug
5cdc9c74
fix format
535c24c3
fix format
192202e9
fix copy
ddb11d07
fix format
f9877f28
fix format
7ec96dcf
fix sort
2a0e123e
fix sort
cecb26fc
fix sort
5f2cffb4
add copy comment
797a89b8
add copy from
a7a6e2d5
remove debug code
b37bb87a
Merge branch 'main' into main
675867d7
revert readme update
ae19e2cd
add copy
d4787969
debug
b8cdc4a6
remove debug code
15170eef
fix flash attention
6e464179
add comments
be88983f
Merge branch 'huggingface:main' into main
06a5d624
clean code
8633fc55
Merge branch 'main' of https://github.com/yikangshen/transformers
97125e3e
clean format
d3002f3a
fix format
bcad4fa4
fix format
16f6fd88
Update src/transformers/models/jetmoe/modeling_jetmoe.py
71f6431f
Update src/transformers/models/jetmoe/modeling_jetmoe.py
fb26a0ef
Update src/transformers/models/jetmoe/modeling_jetmoe.py
6cb4df00
Update src/transformers/models/jetmoe/modeling_jetmoe.py
330a89ba
Update src/transformers/models/jetmoe/modeling_jetmoe.py
0ff62a91
Update src/transformers/models/jetmoe/modeling_jetmoe.py
f6ffe33d
change variable name
e40e1702
add copied from
7db942f6
Merge branch 'huggingface:main' into main
8a2b5934
fix variable name
05635bfd
Merge branch 'main' of https://github.com/yikangshen/transformers
89142638
remove deprecated functinos
ea7daa16
sync to llama implementation
22a03f04
fix format
303942af
fix copy
30031242
fix format
5b7101a3
update format
5f335c31
younesbelkada
approved these changes
on 2024-05-02
remove repr
d5a66045
add comment for moe weight
82069a1b
Merge branch 'huggingface:main' into main
01733bd2
fix copy
c6e5e8b5
Update src/transformers/models/jetmoe/configuration_jetmoe.py
5cfc652a
Update src/transformers/models/jetmoe/modeling_jetmoe.py
42c02b04
Update src/transformers/models/jetmoe/modeling_jetmoe.py
976b4cf4
Update src/transformers/models/jetmoe/modeling_jetmoe.py
6588db55
Update src/transformers/models/jetmoe/modeling_jetmoe.py
dea51cba
Update src/transformers/models/jetmoe/modeling_jetmoe.py
410882aa
Update src/transformers/models/jetmoe/modeling_jetmoe.py
077e46a6
Update src/transformers/models/jetmoe/modeling_jetmoe.py
193a9efb
Update src/transformers/models/jetmoe/modeling_jetmoe.py
14512fc4
Update src/transformers/models/jetmoe/modeling_jetmoe.py
0bbfc87d
Update src/transformers/models/jetmoe/modeling_jetmoe.py
ecb03378
Update src/transformers/models/jetmoe/modeling_jetmoe.py
7f6d5296
add comments and reformat config
4b327ba3
fix format
7f447519
fix format
6c0ea955
Merge branch 'main' into main
a9e2c22c
fix format
9c8081de
update test
cf172045
update doc string in config
41d1a703
Update src/transformers/models/jetmoe/modeling_jetmoe.py
58e56274
update config doc
8341eeab
Merge branch 'main' of https://github.com/yikangshen/transformers
71a29392
update attention cache
9e8b7594
Merge branch 'huggingface:main' into main
5c21dfee
fix format
1b8ed083
fix copy
060af341
Reviewers
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Login to write a write a comment.
Login via GitHub