llvm-project

42c4dc0d - [BPF] Lower unwinding to unwind_goto and unwind_exit (#233649)

[BPF] Lower unwinding to unwind_goto and unwind_exit (#233649) Replace the .bpf_cleanup table with two new instructions, so that neither a loader nor the kernel has to dispatch landing pads. unwind_goto pad BPF_JMP | BPF_JCOND, src_reg = 1 unwind_exit BPF_JMP | BPF_EXIT, src_reg = 1 'call f; unwind_goto pad' goes to the pad when f left with unwind_exit and falls through when it left with exit. No register says which one it was: how one instruction tells the other is up to the JIT. In everything else unwind_exit is exit. ret exit invoke f() unwind pad call f; unwind_goto pad resume r0 = 0; unwind_exit call _Unwind_RaiseException r0 = 0; unwind_exit invoke _Unwind_RaiseException goto pad A call that may unwind and has no landing pad gets one that only resumes. The first version of this change kept the bit in r3: 'r3 = 0; exit', 'r3 = 1; exit' and 'if r3 != 0 goto pad'. r3 is an ordinary register and an argument, and the kernel JITs had to recognize the pattern by its shape to keep the branch off the normal path. With the instructions there is nothing on the normal path to begin with: 'ret' is 'exit' again, the BTF decl_tag that told the kernel which functions set r3 is gone, and a nounwind function whose address is taken does not have to know that its caller looks. invoke, landingpad and resume only carry an unwind on. Something has to start it. With the Itanium ABI a language runtime calls _Unwind_RaiseException() for that, e.g. at the end of a panic handler. On other targets it is a function of the unwinder library. Here the backend lowers the call, unless the program defines the function. To start an unwind and to carry one on is the same thing: leave the frame with unwind_exit, after its own pad if it has one. By its prototype the function comes back when it fails. Here it cannot fail, so what follows the call is dropped. resume is left to DwarfEHPrepare, which turns it into a call of _Unwind_Resume() like on other targets. The backend takes that name from the libcall table. Neither function reaches the object file. Not being nounwind is the default in IR, so that alone does not opt a function in. It also needs uwtable, which rustc sets on every function with -C panic=unwind. clang needs -fexceptions -funwind-tables. A landing pad in a function without uwtable is an error. So is the start of an unwind in a function that cannot unwind: there is no unwind_goto after the calls of it. Every pad is entered whatever was thrown, so only catch-all pads are supported: cleanup, catch of a null type, and an empty filter, which is how rustc guards a function that must not unwind. A pad expects the exception object in r0. There is none, so a pad that looks at it starts with r0 = 0. unwind_exit returns what exit returns, hence 'r0 = 0' in front of it. A function that returns more than 8 bytes returns them in r0 and r2, and the kernel wants both to be set, so there it is 'r0 = 0; r2 = 0'. The kernel wants unwind_goto right after the call. A callee that does not return needs it as well: that is how its unwind_exit finds the pad. unwind_goto has a 16-bit offset. BPFMIPreEmitPeephole::adjustBranch() took the third operand of every conditional branch for its target. Now it takes the last one and retargets the branch in place. unwind_goto is inserted after register allocation. Until then the edge from the call to the pad is an EH edge, which keeps only r6-r9 and the stack alive over it. That is what a return of the callee preserves. clang defines __BPF_FEATURE_UNWIND. Add MachineFunction::clearLandingPads(), since the pads become ordinary blocks and there is no exception table to emit.