llvm-project
fe1fc78c - [SmallVector] Out-of-line the trivially-copyable push_back grow path (#206213)

Commit
34 days ago
[SmallVector] Out-of-line the trivially-copyable push_back grow path (#206213) In the approximately trivially-copyable specialization, push_back's grow path does not early return. Both Clang and GCC likely keep `this` and `Elt` live across the out-of-line `grow_pod` call, saving and restoring them in the prologue/epilogue. Shrink wrapping can't sink it (the saved values are used in the store block the fast path also reaches). Move the grow-and-store into a noinline `growAndPushBack` helper and tail call it. The fast path needs no callee-saved registers. `push_back(int)` drops from 14 to 7 instructions on x86-64. ``` // void vec_pb_int(llvm::SmallVectorImpl<int>&v, int x){ v.push_back(x); } mov eax, dword ptr [rdi + 8] cmp eax, dword ptr [rdi + 12] jae _ZN4llvm23SmallVectorTemplateBaseIiLb1EE15growAndPushBackEi # TAILCALL mov rcx, qword ptr [rdi] mov dword ptr [rcx + 4*rax], esi inc dword ptr [rdi + 8] ret ``` `noinline` keeps the fast path frame-free and the cold grow path out of every call site (with a single COMDAT copy). It is load-bearing, as GCC and Clang otherwise inline it back for some code. However, the out-of-line call makes the element's address escape, which defeats construct-in-place for large element types. `T Tmp = Elt` copy preserves the internal-reference-during-grow case and is elided for by-value element types.
Author
Parents
Loading