transformers
84a2be05 - nemotron_h_sparse: support optional latent MoE projection

Commit
67 days ago
nemotron_h_sparse: support optional latent MoE projection Add `moe_latent_size`: when set, the routed experts run in a latent space (down-proj before, up-proj after; shared expert stays at hidden_size). Only Nemotron-3 Nano (A3B) leaves it unset — Super and Ultra both use it, so it lives on the shared sparse block. `None` is a no-op (Identity).
Author
Parents
Loading