Add Qwen4Exp model #48337
Cyrilvallez
merged
90 commits into
main
from
qwen4-exp
support Qwen4-Exp.
30ea9842
Fix placement of concatenated checkpoint tensors
a514ed86
Align Qwen4-Exp checkpoint conversion
2dbfa108
Simplify Qwen4-Exp model integration
5c92d61b
Add Qwen4-Exp parallel coverage and documentation
afed628f
revert some equivalent operation
11284d0a
update qwen4_exp doc.
23ea7559
fix
2b8d53b0
Merge remote-tracking branch 'origin/main' into qwen4-exp-clean
e676759a
no need to change qwen3.5
6d09b103
sets activation directly inside norm
63fe9993
remove specific head models
12bb7259
native rotary
77937b37
simplify
120d28cd
remove overwrite
2bf8ff4c
remove non compile criteria
13eb28a2
move outside the class
84c52bee
remove constants
e7d0d009
Merge remote-tracking branch 'origin/main' into qwen4-exp-clean
1a62d729
simplify config
e9ac87df
indexer is always present
33d97e0a
indexer needs to train
901a6e13
fix and simplify the selection of valid indices in indexer
a2f6038d
remove external functions
3abe2c5e
simplify indexer further
aaf467ac
replace einsum with matmul
412231b7
start vectorizing
5642e57d
simplify
80f6918f
always use a mask for simplicity
f52cbc48
create and combine mask much more easily in indexer
49e31770
simplify hyper connections
18d9aad7
fix all prefixes
ba91022b
fix ple input by reversing embeddings instead
891e5443
capture hidden states
d18f17e2
follow signature order
6c17711b
always forward both masks
01377e76
fix and simplify past context in ngram embedding
ca261705
fix
1c189717
make do_natural_key more readable
a8114b29
combine norm and grouped norm
075d0aed
fix shapes
a454b4ac
fix qsa check
e6db072e
fix mask
e9810b4f
style
164eabc9
add a few decorators
b199d7f3
update minimal test
75e9095e
adapt to newest config
1d30f205
remove unnecessary test
e42de784
combine related test
ebea6955
fix index mask broadcast error
c837cc41
fix gdn previous state check when ple is enable
05bc9504
avoid update conv state in place
47a975c2
confirmed hybrid_indexed was not used
6daa4a70
fix rope test
f6936677
use vlm tester to pass ci
e3bca939
rewrite layer type in config
6163b28e
no need to reslice
afc9c2d8
short conv up to latest standards
878a16d4
dilution not supported in kernels ðŸ˜
50aefd63
remove unused attr
884d32d3
vectorize qsa indexer!!!
931ee4ea
qsa can use a separate rotary for simply
89fe6529
add qwen_sparse_attention to support block key store!!!
19c8b624
update test
4888fd1c
Merge remote-tracking branch 'origin/main' into qwen4-exp-clean
eedb5c60
modular based on latest main
a6d31667
make quality happy
6f22d1a7
keep simple for now, generalize all library indexers later
807cb38f
do not keep cos/sin in the cache for memory
5cd919bd
a bit cleaner
1dda1695
extend 2d position_ids
b7db0596
ignore rule
99250c57
simplify conversion Ops
b3bd1cff
fix loading and tests
02bcc13f
do not use attn in tp_plan cause num_kv_head=2
369c6cee
remove created file
1ddbe200
oupsi
68559baa
fix chunked prefill
4d4d8f25
fix positions
b8135c9c
fix reverse loading
04305920
improve conversion
92227c0a
slightly clearer key ordering
d807e71a
shard the embedding on dim1
59cb4883
revert test
72de5567
force conversion on cpu for device_map
83b4888b
make things work on simple hardware
0b696f40
small fix
0682e8ec
oupsi
2ff8a4b2
Merge branch 'main' into qwen4-exp
0cdf3e47
latest upstream modular
b61b98be
Reviewers
No reviews
Assignees
No one assigned
Labels
New model
Milestone
No milestone
Login to write a write a comment.
Login via GitHub