fix(chart): loosen the ALB health check on the prod search ingresses (#3422)
During the target-type rollout the search targets flapped in and out of both
ALBs, reaching 2 of 3 targets unhealthy at once with reason Target.Timeout.
The search service runs 2 uvicorn workers (capped to avoid OOM) and serves
/healthcheck from the same workers that run the search queries, so the health
check blocks whenever both workers are busy. The target group allowed 5s with
an unhealthy threshold of 2, so ~30s of load was enough to pull a pod out of
rotation. Its traffic then moved to the remaining pods, making them more
likely to time out in turn — the flapping is self-reinforcing, and with only
3 replicas all three could be evicted, which would return 503 on search.
Raise the timeout to 10s and the unhealthy threshold to 5. With the interval
left at 15s, a pod now has ~75s before the ALB pulls it, while a genuinely
dead pod is still removed within 75s. Kubernetes already tolerates 150s on
the same endpoint (failureThreshold 30 x periodSeconds 5), so the ALB no
longer evicts pods that kubelet still considers ready.
This does not remove the underlying contention on the 2 uvicorn workers; it
stops a transient slowdown from turning into lost capacity.