Unstable cluster? How to set up Kubernetes health checks

Why your applications crash out of nowhere in Kubernetes

Managing containerized systems often feels straightforward until an application silently hangs in production. Without automated supervision, the orchestrator keeps sending requests to an instance that has already lost its ability to respond, resulting in severe latency and user frustration. In practice, this means your infrastructure needs constant mechanisms to check service health, triggering corrective actions before problems impact the business. If you manage distributed systems, you know that preventing failures requires planning just as rigorous as setting up a proactive cluster monitoring and resource prediction architecture.

Kubernetes solves this requirement using health checks known as probes. These periodic tests allow the platform to discover precisely when a service is ready to accept traffic, when it needs a restart, or whether it even managed to initialize correctly. However, misconfiguring these parameters can create a severe side effect, killing perfectly healthy pods simply because they took a bit longer to process heavy workloads.

Understanding the three core types of probes

To build a solid defense, Kubernetes divides checks into three distinct categories, each with a well-defined role in the container lifecycle. The first is the startup probe, responsible for verifying whether the application has finished booting up. It is especially useful in legacy systems or robust enterprise applications that require time to load data into memory before accepting connections. While the startup probe runs, other checks remain disabled, preventing the orchestrator from killing the application prematurely.

The second category is the liveness probe, which indicates whether the container is still alive and functioning. If this check fails repeatedly, Kubernetes assumes the application entered an unrecoverable state and destroys it to spin up a fresh instance from scratch. Meanwhile, the readiness probe handles service readiness: it tells the load balancer whether the service is fit to receive network traffic at that exact moment. If an external database goes down, for example, the readiness probe fails, isolating the affected pod without needing to restart the entire process.

Common mistakes in timing and threshold configurations

One of the biggest misconceptions among engineering teams is treating all probes with the exact same timing values. Setting an overly short interval for initialDelaySeconds or an aggressive timeout often triggers the dreaded death loop, where the system attempts to restart a pod that was simply processing a heavy query. In practice, this exhausts cluster resources and worsens the instability the check was trying to prevent.

Another frequent issue is using overly complex endpoints for checks, such as routes querying multiple downstream microservices and databases in a cascade. If the primary database experiences a momentary hiccup, all dependent applications fail their liveness probes simultaneously, creating a catastrophic cascading reaction. The ideal approach is keeping the health check route lightweight, testing only internal process integrity without deep external dependencies.

Implementing liveness and readiness in practice

Configuring these guidelines requires attention to the details of the YAML manifest defining your deployment. Below is a practical example of how to structure these tests using basic HTTP requests to monitor service health safely.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-api
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: api
        image: sample-api:v1
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

In this example, the readiness probe begins testing the application after five seconds, repeating the check every ten seconds. Meanwhile, the liveness probe waits fifteen seconds to allow for full initialization and then periodically monitors process uptime. This spacing prevents false positives during normal CPU utilization spikes.

Best practices to keep your cluster stable

Maintaining a resilient production environment requires continuous stress testing and validation of configured limits. Adjust failureThreshold and successThreshold parameters to ensure temporary network hiccups do not result in unnecessary restarts. Additionally, ensure your application properly releases old connections and manages memory resources to prevent leaks that confuse liveness probes.

Investing time in properly modeling your container lifecycle separates fragile systems from truly elastic infrastructures. When your application communicates effectively with the orchestrator, mysterious crashes disappear, and operations gain the predictability needed to scale fearlessly. To complement this implementation in production environments, see our related technical analyses on How to run local generative AI with Ollama and ensure data privacy and Slow API? How to optimize N+1 queries in Hibernate.

Final considerations

The stability of a Kubernetes cluster depends not only on physical node capacity, but also on how each microservice communicates its actual health state. The balanced use of startup, liveness, and readiness probes creates an automated safety net protecting the system against unexpected failures. By avoiding generic configurations and matching response times to your software's reality, you achieve a much safer, more predictable production environment ready for continuous growth.