Fix race condition when setting client reference manifests (#71741)
Similar to how #71669 fixed a race condition when setting the manifests
singleton regarding the server action manifests, this PR fixes the same
race condition for the client reference manifests.