Add Dory model support (#50099)
* Add Dory model support
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Shantanu Acharya <shantanua@nvidia.com>
* Reuse shared attention and rotary implementations for Dory
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Shantanu Acharya <shantanua@nvidia.com>
* Simplify fixed Dory configuration and clarify normalized updates
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Shantanu Acharya <shantanua@nvidia.com>
* Align Dory defaults and simplify recurrent forward loops
Replace the nested forward helper with explicit input, recurrent, and output loops. Match released 4B defaults while retaining the 72-block checkpoint layout. Move chunked prefill/decode coverage into DoryModelTest and test configuration defaults.
Co-authored-by: OpenAI Codex <noreply@openai.com>
Signed-off-by: Shantanu Acharya <shantanua@nvidia.com>
* [Dory] collapse (attention, mlp) blocks into standard decoder layers
DoryL2Norm in Gemma2 sandwich slots, AfmoeAttention parent, LayerPairCollapse to load released weights;
config serializes back to the released 72-block layout.
* [Dory] keep the sub-norms before o_proj/down_proj (BitNet names) for exact parity
* make fix-repo: mark dory modeling as generated
* [Dory] recurrent cache layer types: per-loop histories, last_loop read like KV-shared layers
Standard DynamicCache(config) via self-registered layer types; no custom Cache class, no create_cache.
* [Dory] last_loop inside the recurrent cache layers (read without writing), static recurrent layers
* [Dory] later loops reuse the layer's own class; shorter CODEPATH note
* [Dory] fix review findings: loop count check, rope init per rope_type, past recording in later loops, n_output_layers, reverse mapping
* last cleanups
* nits
* try something simpler
* cut some crap
* up
* simplify LayerPairCollapse
* dory: cache only, no past_key_values deprecation
* dory: model card date, tester overrides
---------
Signed-off-by: Shantanu Acharya <shantanua@nvidia.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Arthur <arthur.zucker@gmail.com>
Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com>