fix `mod` for mixes of `Signed` and `Unsigned` (#57853)
Previously this was just overfowing producing wrong answers (both for
the sign convention and just the wrong modulo class)
fixes https://github.com/JuliaLang/julia/issues/57851
(cherry picked from commit 056891701e8ec629834def549f565cf80c46ccf9)
