CI: Store Passes in results.json + various improvements. (#57686)
- Adds option to enable saving Test.Pass results to DefaultTestSet
(started as https://github.com/JuliaLang/julia/pull/57690 but more
complete here) and enable it when Base.runtests is called when CI=true,
to match when reports are saved
- Stops attributing testset duration to tests from that testset, it
doesn't make sense.
- Make the test name resemble (as far as possible) the original test
call. Making the name independent of test outcome should make it easier
to group tests and identify flaky ones.
- store repeat counts as tags, rather than in the name
- moves the save to inside the test workers, where all test info is
available, and the save can be done in parallel to save time , then
jsons are made at the end and uploaded as an artifact in a tarball
- speed improvements to make saving time acceptable
- fixes a couple of tests that seem to get a little flakier with this
Note on the WeakRef test fix: If a test like `@test x == y` passes, the
`Pass` object retains `x`. Previously to this the Pass object was still
made, but must have been GC-ed immediately. This PR remakes the Pass
object when appending it to DefaultTestSet without the captured data. I
think that just delayed the GC a little to make the test flaky, so I
moved the `@test` to outside the function to avoid capturing `x`.
Also buildkite had to fix an issue to make tests show up properly on our
jobs: https://forum.buildkite.community/t/test-failures-showing-up-in-test-digest-but-not-on-jobs/4306
