DeepSpeed
397efe87 - Fix DeepCompile fallback handling for scalar outputs (#8149)

Commit
47 days ago
Fix DeepCompile fallback handling for scalar outputs (#8149) When DeepCompile lowered the graph-safe RNG wrapper around this operator, the fallback handler treated every non-`None` output as an Inductor buffer for never-reuse registration and called `get_name()` unconditionally. The scalar output therefore raised `AttributeError: 'int' object has no attribute 'get_name'` before the first compiled forward could run (See the following definition of `_scaled_dot_product_flash_attention`). ``` aten::_scaled_dot_product_flash_attention( Tensor query, Tensor key, Tensor value, float dropout_p=0.0, bool is_causal=False, bool return_debug_mask=False, *, float? scale=None ) -> ( Tensor output, Tensor logsumexp, Tensor cum_seq_q, Tensor cum_seq_k, SymInt max_q, SymInt max_k, Tensor rng_state, Tensor unused, Tensor debug_attn_mask ``` This PR limits "never-reuse" registration to Inductor IR outputs. This is applied to tensor buffers, while scalar and `None` leaves pass through without tensor-only method calls. Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Author
Parents
Loading