[fix][wip] GlmMoeDsa: try implement DSA (#43912)
* init
Signed-off-by: JaredforReal <w13431838023@gmail.com>
* format
Signed-off-by: JaredforReal <w13431838023@gmail.com>
* not indexer_rope_interleave
Signed-off-by: JaredforReal <w13431838023@gmail.com>
* set MLA rope interleave to False
Signed-off-by: JaredforReal <w13431838023@gmail.com>
* get rid of interleave in apply_rotary_pos_emb
Signed-off-by: JaredforReal <w13431838023@gmail.com>
* reintroduce attention interface
Signed-off-by: JaredforReal <w13431838023@gmail.com>
* reset _cached_keys
Signed-off-by: JaredforReal <w13431838023@gmail.com>
* remove yarn
Signed-off-by: JaredforReal <w13431838023@gmail.com>
* fix tp plan for multi node runs
* support sdpa?
* style
* fix copies
* style
* updates + skip some tests
* skip some of the tests
* skip some tests
* style and fix copies
* oups
* style
* fml
* generate issues
* fix config
* :)
* :) :) :) :)
* FMLFMLFMLFMLF
---------
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com>
