dataset-viewer
157d6a1d - fix(chart): loosen the ALB health check on the prod search ingresses (#3422)

Commit
40 days ago
fix(chart): loosen the ALB health check on the prod search ingresses (#3422) During the target-type rollout the search targets flapped in and out of both ALBs, reaching 2 of 3 targets unhealthy at once with reason Target.Timeout. The search service runs 2 uvicorn workers (capped to avoid OOM) and serves /healthcheck from the same workers that run the search queries, so the health check blocks whenever both workers are busy. The target group allowed 5s with an unhealthy threshold of 2, so ~30s of load was enough to pull a pod out of rotation. Its traffic then moved to the remaining pods, making them more likely to time out in turn — the flapping is self-reinforcing, and with only 3 replicas all three could be evicted, which would return 503 on search. Raise the timeout to 10s and the unhealthy threshold to 5. With the interval left at 15s, a pod now has ~75s before the ALB pulls it, while a genuinely dead pod is still removed within 75s. Kubernetes already tolerates 150s on the same endpoint (failureThreshold 30 x periodSeconds 5), so the ALB no longer evicts pods that kubelet still considers ready. This does not remove the underlying contention on the 2 uvicorn workers; it stops a transient slowdown from turning into lost capacity.
Author
Parents
Loading