llvm-project

8fd8daeb - [AArch64] Expand FORM_TRANSPOSED_REG_TUPLE to copies before regalloc (#207205)

[AArch64] Expand FORM_TRANSPOSED_REG_TUPLE to copies before regalloc (#207205) Previously, we kept the FORM_TRANSPOSED_REG_TUPLE nodes around during register allocation. The problem with this approach is that it does not model the potential overlap in live ranges between the destination and source operands. As a result, the register allocator assumes it has complete freedom to allocate registers to the operands. For example, there is nothing stopping it from allocating: ``` {z0, z1, z2, z3} = FORM_TRANSPOSED_X4 z3, z2, z1, z0 ``` However, such cases are hard to expand later, either requiring spills or complex shuffles. The current expansions of FORM_TRANSPOSED_REG_TUPLE miscompile in cases like this because, when naively expanded into copies after register allocation, earlier copies can clobber values that are still needed by later ones. For the above case, the incorrect expansion would be: ``` z0 = COPY z3 // z0 clobbered z1 = COPY z2 // z1 clobbered z2 = COPY z1 // reads the wrong value of z1 z3 = COPY z0 // reads the wrong value of z0 ``` This patch fixes the issue by expanding FORM_TRANSPOSED_REG_TUPLEs into copy sequences immediately before register allocation (in `aarch64-post-coalescer`). For example: ``` %v4:zpr4mul4 = FORM_TRANSPOSED_X4 %v0:0, %v1:0, %v2:0, %v3:0 ``` Expands to: ``` undef %v4.zsub0:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v0:0 %v4.zsub1:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v1:0 %v4.zsub2:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v2:0 %v4.zsub3:zpr4mul4 = COPY_INTO_TRANSPOSED_TUPLE %v3:0 ``` This is similar to how REG_SEQUENCE is expanded and allows the register allocator to reason about how the copies may interfere with one another. To ensure our register allocation hints still apply, we encourage the scheduler to place FORM_TRANSPOSED_REG_TUPLE nodes immediately before their users. This keeps the live ranges of the hint nodes (COPY_INTO_TRANSPOSED_TUPLE) short while extending the live ranges of their operands. As a result, the register allocator is more likely to allocate registers to the copy sources first, which works best for our allocation hints.