Invalidator: don’t call invalidate at all if there’s nothing to invalidate (#75706)
No behavior changes in this PR, just a small adjustment since while
debugging it was surprising that `invalidate` was called even though it
was a no-op with an empty array.
Previously, `invalidate` would unconditionally get called after
building, even if no work was queued.
Test Plan: CI