[new model] Add Zyphra/ZAYA1-8B (#45862)
* zaya1 support
* add test
* update example
* new config
* remove empty line
* pass ci
* modify config, laguna-sytle rope
* use existing cache
* cca refine + use llama attn
* use dict for 2d/4d mask
* optimize, reuse existing code
* inherit from AfmoeForCausalLM,
but need to construct cache from _make_zaya_cache
* checkpoint conversion
* align with official implement, check 74b conversion
* easier test
* remove mapping since we convert the ckpt
* use default_swa_theta
* update date
* temp init
* modular
* better residual scaling
* better cache
* ops forget init again
* better naming
* llama decoderlayer
* improve
* update date
* Fix ZAYA residual stream precision regression
* code clean
* date
* update fsdp
* clean
* upstream
* improve comments
* date
* update link!
* update modeling
* Fix ZAYA config after main merge
* fix fa2 test, ep test
* fix CI
* simplify!
* update mask helper
* update doc date
* update CI
---------
Co-authored-by: Tarek Ziade <tarek@ziade.org>
Co-authored-by: Anton Vlasjuk <73884904+vasqu@users.noreply.github.com>
