Backport a fix for GMP patches to `backports-release-1.10` (#51445)
This backports `Fixes and improvements for source builds (#51422)` PR to
the `backports-release-1.10` branch to make it buildable without binary
builder (we are monitoring that branch in our CI to make sure we are up
to date with the upcoming 1.10 release).
Co-authored-by: Tim Besard <tim.besard@gmail.com>