Readiness and Liveness Probes: Health Checks That Hold in Production

Short answer: Readiness and liveness probes decide when a pod can receive traffic and when it must be restarted, and they are the simplest, highest‑leverage control you have to keep a vibecoded app alive under real load. A liveness probe should only answer “Is the process healthy enough to keep running?” while a readiness probe should answer “Can this instance serve requests right now?”. Most outages from rough rollouts come from conflating the two or from probes that are too slow, too strict, or check the wrong things. Use startupProbe for slow boots, flip readiness to false on SIGTERM for graceful shutdown, and keep probe handlers fast and isolated. Production‑ready apps ship with probes that reflect real serving ability without flapping.

Key takeaways

• Readiness and liveness probes have different purposes: readiness gates traffic; liveness triggers restarts.
• Liveness checks should be minimal and internal; readiness can verify critical dependencies but must be fast.
• Use startupProbe to shield slow initialization and migrations from premature restarts.
• Flip readiness to false on SIGTERM, drain, then exit within terminationGracePeriodSeconds for graceful shutdown.
• Probe thresholds, timeouts, and intervals must match your latency budget to prevent flapping and cascading failures.

What are readiness and liveness probes, and why do they matter?

Readiness and liveness probes are Kubernetes health checks that control traffic routing and restarts per pod. A readiness probe gates whether a pod is added to service endpoints, while a liveness probe determines whether the kubelet should restart the container.

Use a liveness probe to detect stuck processes that cannot recover and require a restart. Use a readiness probe to signal transient unavailability such as warming caches, backpressure, or controlled shutdowns. Production rollouts, autoscaling, and incident response depend on these signals to avoid sending users to broken pods.

How do readiness and liveness probes work in Kubernetes?

Kubernetes probes run on a schedule and evaluate an HTTP endpoint, a TCP socket, or a command in the container. Success and failure thresholds decide when the kubelet flips the pod’s readiness or triggers a restart.

• Liveness probe: when it fails for the configured threshold, the kubelet restarts the container.
• Readiness probe: when it fails, the pod is removed from service endpoints until the probe passes again.
• Startup probe: during startup, only startupProbe runs; when it passes, liveness and readiness commence. This prevents restarts during slow initialization.

Probe timing is controlled by periodSeconds (how often), timeoutSeconds (how long to wait), and failureThreshold/successThreshold (stability). These settings must match your app’s latency profile.

What should a production liveness probe actually check?

A production liveness probe should check only the container’s ability to make forward progress. Keep it minimal and internal so transient external failures do not cause needless restarts.

• Check the event loop or thread pool is responsive by serving a tiny in‑process handler.
• Ensure there is no deadlock or out‑of‑memory condition preventing basic handler execution.
• Do not call databases, queues, or external APIs in liveness; external outages should not force container restarts.
• Respond fast with a simple 200 OK; avoid logging-heavy or allocation-heavy code paths.

Restarts are expensive under load. Reserve liveness for unrecoverable states, not as a general error response.

What should a production readiness probe verify?

A production readiness probe should verify the instance can serve traffic right now. It can consider critical dependencies but must return quickly and avoid cascading failures.

• Confirm the HTTP server can accept connections and dispatch a trivial handler.
• Optionally verify the database connection pool has at least one healthy connection or can open one within a tight budget. Use a lightweight ping query, not a heavy transaction. For guidance on safe pool design, see Postgres connection pooling for MVPs.
• If request processing requires a specific cache, confirm the cache client is connected; do not flush or rebuild anything in the probe.
• If a critical message broker or stream is required to accept user writes, verify the client can publish to a health topic or is connected, but avoid persistent side effects.
• Expose backpressure: when queues exceed a safe depth or the node is out of file descriptors, report not ready to shed load.

Readiness should reflect serving ability under your SLO. If the app cannot meet the latency budget for a standard request due to dependency loss or resource pressure, it should fail readiness and stop taking traffic until recovered.

How to design probe endpoints that do not lie

Health endpoints must be explicit, fast, and stable under stress. A single /healthz that always returns 200 is worse than no health checks because it misroutes traffic during failures.

• Expose distinct handlers: one for liveness (pure in‑process) and one for readiness (in‑process + critical dependencies).
• Return 200 for healthy, 503 for unready, and 500 only for true liveness failures. Use explicit status codes rather than encoding state in JSON.
• Keep bodies tiny and optional; the probe does not need verbose payloads. Emit detailed diagnostics to logs and metrics instead.
• Set strict per‑handler timeouts shorter than your probe timeout to avoid head‑of‑line blocking in the server.
• Do not require authentication; run the handlers on an internal port or path with network policy restrictions.

Use consistent naming and document the contract for each handler. A clear contract prevents accidental changes that break rollout safety.

When should you use startupProbe?

Use startupProbe to shield slow or variable initialization from liveness restarts. Startup probes run before readiness and liveness and should verify the app has completed boot steps that cannot be interleaved with traffic.

• Long module loading, warm JITs, or runtime compilation steps.
• Data migrations or schema checks that must complete before serving.
• Large cache preloads that are required before the instance can respond within the SLO.

During startupProbe, liveness is disabled. Choose a conservative timeout and period to avoid false restarts, then transition into normal readiness/liveness once ready to serve.

How do probes coordinate graceful shutdown?

Graceful shutdown requires flipping readiness to false on SIGTERM, draining in‑flight work, and exiting within the terminationGracePeriodSeconds. This sequence prevents the load balancer from sending new requests while the pod finishes current ones.

1. On SIGTERM or preStop hook, set readiness to false immediately and respond to readiness checks with 503.
2. Stop accepting new connections and stop dequeuing new jobs from background workers.
3. Finish in‑flight HTTP requests and in‑flight jobs within the grace period; cancel or checkpoint long tasks.
4. Close network connections cleanly and exit before terminationGracePeriodSeconds.

Graceful shutdown is essential for zero‑downtime rollouts and for safe node drains. Liveness must stay green during drain, otherwise the kubelet may kill the container abruptly before it finishes work.

What thresholds and timings should you configure?

Probe timings must reflect your app’s latency, startup characteristics, and dependency budgets. Overly aggressive probes cause flapping; overly lax probes hide real failures.

• Period: choose a cadence that detects failure within your error budget without excess load, often a few seconds for readiness and slightly longer for liveness.
• Timeout: set shorter than your normal request timeout and matched to the handler’s work (hundreds of milliseconds to a couple seconds for most web apps).
• Failure threshold: allow brief jitter for readiness (a handful of consecutive failures) and fewer for liveness since it should be a simple in‑process handler.
• Success threshold: for readiness, require more than one success to rejoin rotation if your system is prone to flapping; liveness usually accepts a single success.
• Startup: give startupProbe a generous window if cold starts or cache preloads are variable, then switch to normal probes.

Think in terms of user experience: a readiness detection time longer than your traffic shift or rollout window can send users to broken pods. Align probe intervals with deployment strategies and HPA scaling intervals.

Which dependencies belong in readiness, and which do not?

Not every dependency belongs in readiness. Readiness should include only dependencies critical to serving the traffic the pod is routed.

• Include: the HTTP server, the primary database used in request paths, a mandatory cache that fronts queries, or a broker required to accept writes synchronously.
• Exclude: optional analytics sinks, asynchronous email queues, or batch data sources; degrade gracefully instead.
• Guard: if the database is occasionally slow, your readiness check should use a lightweight ping within a tight timeout rather than a full query.

If you rely on a relational database, verify the connection pool state rather than opening a new connection on every probe. Unbounded connection attempts from probes can exhaust pools. For tuning pools and safe timeouts, see our guide on Postgres connection pooling.

How should probes behave under backpressure?

Under sustained load, a healthy system may need to shed traffic. Readiness is the lever to reduce intake per pod while preserving liveness.

• Expose internal queue depth and CPU saturation to the readiness handler and mark unready when exceeding thresholds.
• Rate‑limit readiness computations to avoid thrashing; flip only when thresholds are crossed consistently.
• Coordinate with autoscalers that use CPU or custom metrics so pods rejoin rotation as pressure falls.

Backpressure‑aware readiness prevents tail latency from exploding and protects your error budget during traffic spikes.

Common anti‑patterns that break rollouts

Most production incidents tied to health checks come from a few predictable mistakes. Avoid these anti‑patterns when moving a vibecoded app to production.

• Liveness runs a database query or calls a third‑party API. External failures trigger container restarts and make recovery harder.
• Readiness performs heavy work like cache rebuilds or migrations. Probes become the load generator that sinks the pod.
• Single /healthz endpoint with complex logic. A single flag hides the semantics differences and encourages inappropriate checks.
• Probe timeouts exceed server timeouts. The probe hangs the thread pool longer than user requests, causing head‑of‑line blocking.
• No startupProbe on slow boots. The pod restarts in a loop before it can initialize.
• No readiness drop on SIGTERM. The pod receives traffic while shutting down, leading to spurious 5xx during rollouts.
• Health handlers log at WARN/ERROR every hit. Probes flood logs and mask real signals.

Make probes boring. Boring health checks are reliable and cheap.

How do you implement probes in common stacks?

Implementation is straightforward if you separate concerns and keep handlers tiny. The patterns are the same across languages and frameworks.

• HTTP servers: mount two routes, e.g., /live for liveness and /ready for readiness, each with strict timeouts enforced by the server or middleware.
• Background workers: expose a separate port or an HTTP endpoint from the worker, or use exec probes to check the worker’s ability to receive and process a no‑op job.
• Process signals: register a SIGTERM handler that flips readiness state and closes listeners; many frameworks expose graceful shutdown hooks you should wire to the readiness flag.
• Database checks: ping the pool or run a lightweight read, capped by a short timeout; never create new connections per probe.
• Cache/broker checks: verify clients report connected state; avoid publish/subscribe unless to a dedicated health channel without side effects.

Choose HTTP probes where possible; they are easier to time and instrument than exec probes and map cleanly to service health.

How do you test probes before production?

Test health checks under conditions they are meant to control: slow boots, dependency outages, rolling updates, and resource pressure. Tests that only assert 200 in development miss the point.

• Induce slow startup by delaying dependency availability; verify startupProbe gates restarts until initialization completes.
• Kill the database or inject DNS failures; verify readiness flips to 503 quickly while liveness stays 200.
• Send SIGTERM during a load test; confirm readiness drops immediately, in‑flight traffic completes, and no 5xx spike occurs.
• Simulate queue pileups; verify readiness fails under backpressure and recovers without flapping when pressure subsides.

Use realistic data volumes and request patterns in tests. If you need representative non‑sensitive data, see our guidance on test data management for MVPs.

How do probes interact with rollouts and traffic shifting?

Probes are the safety layer for zero‑downtime rollouts, canaries, and blue‑green strategies. If probes reflect real serving ability, the platform will naturally route around unhealthy pods during a rollout.

• New pods: startupProbe prevents premature restarts; readiness ensures pods only receive traffic after warmup.
• Terminating pods: readiness drop and graceful drain prevent 5xx spikes and partial writes.
• Canary: readiness keeps the canary out of rotation if it regresses, minimizing blast radius.

Combine probes with incremental traffic shifting and versioned configuration. If the app itself exposes precise health, the platform can make safe decisions without brittle deployment scripts.

Security and privacy considerations for health endpoints

Health endpoints should not expose secrets, user data, or verbose internals. Probes are frequent and automated; any leakage multiplies risk.

• Bind the health server to localhost or a dedicated port exposed only within the cluster.
• Return minimal bodies; never include environment variables, connection strings, or stack traces.
• Rate‑limit or isolate health endpoints from public ingress; they exist for the kubelet, not the internet.

Use standard HTTP security headers on externally exposed paths. For baseline safe defaults on web routes, see our article on HTTP security headers for vibecoded apps.

Operational metrics: what to observe from probes

Health checks become operational signals when you measure them. Probe outcomes and latencies should be first‑class metrics.

• Track readiness transitions over time per deployment, node, and zone.
• Measure probe handler latency distributions and error codes.
• Correlate readiness loss with resource metrics (CPU, memory, network, file descriptors) and dependency errors.

Alert on sustained readiness loss across a percentage of pods and on unexpected liveness restarts. Alerts that fire on a single pod flapping tend to be noise.

A minimal, production‑ready health check checklist

Use this short checklist to close the vibecoding‑to‑production gap for health checks.

• Separate /live and /ready endpoints with explicit status codes.
• Use startupProbe for variable initialization or heavy warmups.
• Keep liveness in‑process only; keep readiness fast with only critical dependencies.
• Flip readiness to false on SIGTERM; drain, then exit within the grace period.
• Choose timeouts, periods, and thresholds to match your latency budget and rollout plan.
• Instrument probe latencies and transitions; alert on sustained issues, not one‑offs.
• Load test with dependency failures and shutdowns to validate probe behavior.

How Moai Team approaches this

We treat health checks as an interface contract between your service and the platform. We start by writing explicit probes that reflect the real serving ability: a minimal in‑process liveness handler and a readiness handler that checks just the critical path within a tight budget. We wire SIGTERM to flip readiness to false, implement connection draining, and verify termination fits within the configured grace period.

We align probe timings with your deployment strategy and SLO so rollouts do not spike 5xx. We harden database checks to read from pools rather than open new connections and calibrate timeouts to prevent connection storms; for deeper database behavior, we pair this work with the patterns in our Postgres connection pooling guide. We instrument probe latencies and transitions as first‑class metrics and run failure drills that cut dependencies, apply backpressure, and send SIGTERM during load. This closes the vibecoding‑to‑production gap where most prototypes break: shutdowns, slow boots, and dependency wobble.

Frequently Asked Questions

Should my liveness probe check the database?

No. A liveness probe should only verify the process can handle a trivial request in‑process. Database failures are usually transient and should affect readiness, not trigger restarts.

When do I need startupProbe?

Use startupProbe when initialization is slow or variable, such as loading large models, warming JITs, or running migrations. It prevents premature liveness restarts while the app boots.

What HTTP status codes should readiness and liveness return?

Return 200 for healthy, 503 for unready, and 500 only when the process is unhealthy. Distinct status codes help the platform and operators understand state quickly.

How fast should my readiness probe be?

Keep readiness under a tight timeout, typically below your average request latency budget. If the readiness check is slow, it competes with real traffic and can cause flapping under load.

How do I avoid probe flapping?

Use appropriate failure and success thresholds, avoid checking optional dependencies, and base readiness on stable indicators like pool health and queue depth. Ensure handlers have shorter timeouts than the probe itself.

How do probes affect rolling updates?

Probes gate traffic during updates: new pods must pass readiness before receiving traffic, and terminating pods should drop readiness to drain cleanly. Proper probes prevent 5xx spikes during rollouts.

Shipping a prototype to real users? We embed forward‑deployed engineers to close the gap. Talk to Moai Team.