openvino
6b80671c - [GPU][TRANSFORMATIONS] Support RMS Normalization Fusion without Learnable Affine Parameters (#33861)

Commit
62 days ago
[GPU][TRANSFORMATIONS] Support RMS Normalization Fusion without Learnable Affine Parameters (#33861) ### Details: - This PR enhances RMS normalization fusion to support pattern without learnable affine parameter (gamma), enabling optimization of transformer architecture like LTX-Video - The existing RMS fusion pass only supported pattern with constant gamma parameter. However, some transformer model (e.g., LTX-Video's attention layers) use RMS normalization followed by dynamic scaling operation where the scale factor is non-constant. These pattern was previously unfused, missing optimization opportunity - When `elementwise_affine=False` (equivalent to [Pytorch RMS's attribute](https://docs.pytorch.org/docs/stable/generated/torch.nn.modules.normalization.RMSNorm.html)), RMS normalization does not include learnable gamma parameters. The gamma is implicitly fixed to ones, reducing the decomposed graph pattern from: `x → Power(2) → ReduceMean → Add(eps) → Sqrt → Divide(1/√) → Multiply(x, 1/√) → Multiply(gamma) ` to: `x → Power(2) → ReduceMean → Add(eps) → Sqrt → Divide(1/√) → Multiply(x, 1/√) [NO gamma multiplication] ` <img width="648" height="924" alt="image-2026-01-26-22-53-05-973" src="https://github.com/user-attachments/assets/02b4580f-bbce-43ea-affd-438f0a5f4ea7" /> ### Tickets: - [CVS-179953](https://jira.devtools.intel.com/browse/CVS-179953) --------- Signed-off-by: Andrew Park <andrew.park@intel.com>
Author
Parents
Loading