transformers
da5a518f - refactor: Jamba-style decoder layers, inherit NemotronMLP/JambaModel

Commit
113 days ago
refactor: Jamba-style decoder layers, inherit NemotronMLP/JambaModel - Replace the single-class `if layer_type == ...` decoder layer with two dedicated classes per architecture, mirroring Jamba's pattern: `NemotronH{Dense,Sparse}{Mamba,Attention}DecoderLayer`, each containing norm -> mixer -> residual -> norm -> ffn -> residual. - `NemotronHDenseMLP` now inherits `NemotronMLP`; dense/sparse Models and ForCausalLM inherit from the Jamba stack via the modular converter. - Collapse the per-char `hybrid_override_pattern` into a per-decoder-layer `layer_types` list: each `M`/`*` becomes one layer (with mlp or moe FFN tail), `-` / `E` are absorbed. `num_hidden_layers` is now the count of logical decoder layers (not raw pattern characters). - Sparse MoE experts continue to inherit from MixtralExperts (non-gated variant), MoE block from DeepseekV3MoE. - Tests updated for the new layer structure; BC dispatcher still routes `NemotronHConfig(hybrid_override_pattern=...)` to the right subclass.
Author
Parents
Loading