transformers
0366c779 - Make first_k_dense_replace configurable and fix tiny-config MoE test failures

Commit
129 days ago
Make first_k_dense_replace configurable and fix tiny-config MoE test failures - Add first_k_dense_replace config field (default 3) so the dense/MoE layer split is explicit; mlp_layer_types is derived from it in __post_init__. - Tiny DeepseekV32 tester: first_k_dense_replace=1 gives a sparse MoE layer (so test_keep_in_fp32_modules_exist sees e_score_correction_bias) and n_group=1/topk_group=1 keeps group routing valid for 8 routed experts. - Skip test_training_overfit (MoE on a tiny random model is unstable), matching glm_moe_dsa. - Allowlist first_k_dense_replace in check_config_attributes.
Author
Parents
Loading