transformers
30e208b1 - Add Dory model support (#50099)

Commit
1 day ago
Add Dory model support (#50099) * Add Dory model support Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Shantanu Acharya <shantanua@nvidia.com> * Reuse shared attention and rotary implementations for Dory Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Shantanu Acharya <shantanua@nvidia.com> * Simplify fixed Dory configuration and clarify normalized updates Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Shantanu Acharya <shantanua@nvidia.com> * Align Dory defaults and simplify recurrent forward loops Replace the nested forward helper with explicit input, recurrent, and output loops. Match released 4B defaults while retaining the 72-block checkpoint layout. Move chunked prefill/decode coverage into DoryModelTest and test configuration defaults. Co-authored-by: OpenAI Codex <noreply@openai.com> Signed-off-by: Shantanu Acharya <shantanua@nvidia.com> * [Dory] collapse (attention, mlp) blocks into standard decoder layers DoryL2Norm in Gemma2 sandwich slots, AfmoeAttention parent, LayerPairCollapse to load released weights; config serializes back to the released 72-block layout. * [Dory] keep the sub-norms before o_proj/down_proj (BitNet names) for exact parity * make fix-repo: mark dory modeling as generated * [Dory] recurrent cache layer types: per-loop histories, last_loop read like KV-shared layers Standard DynamicCache(config) via self-registered layer types; no custom Cache class, no create_cache. * [Dory] last_loop inside the recurrent cache layers (read without writing), static recurrent layers * [Dory] later loops reuse the layer's own class; shorter CODEPATH note * [Dory] fix review findings: loop count check, rope init per rope_type, past recording in later loops, n_output_layers, reverse mapping * last cleanups * nits * try something simpler * cut some crap * up * simplify LayerPairCollapse * dory: cache only, no past_key_values deprecation * dory: model card date, tester overrides --------- Signed-off-by: Shantanu Acharya <shantanua@nvidia.com> Co-authored-by: OpenAI Codex <noreply@openai.com> Co-authored-by: Arthur <arthur.zucker@gmail.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>
Author
Parents
Loading