Containers restarting endlessly? How to investigate OOMKilled

The silent terror: what the OOMKilled code actually means

Anyone managing modern cloud applications will eventually run into a frustrating behavior: a container simply vanishes, restarts, and leaves vague traces in system logs. Most of the time, the culprit is the feared OOMKilled code, which stands for Out Of Memory Killed. In practice, this means the Linux kernel, the brain behind the operating system managing the hardware, realized the application was sucking up more RAM than allowed and terminated it summarily to keep the entire server from locking up.

To understand this mechanism, imagine Kubernetes works like a gated community where every resident—representing a container—has a strict quota of space and supplies. If a resident starts accumulating trash and blocking neighbors' hallways, building management intervenes immediately and evicts the offender. On servers, the judge and executioner is the OOM Killer, a Linux kernel subsystem monitoring RAM consumption second by second. When memory pressure reaches critical levels, it picks the process consuming the most proportional resources and shuts it down by sending a fatal signal.

Why resource limits save or destroy your cluster

One of the most common mistakes beginners make is leaving limits and requests fields blank in Kubernetes configuration files. Requests tells the scheduler how much memory a container needs to boot up, while limits defines the absolute ceiling it can never cross. Omitting these values gives the container carte blanche to consume all available memory on the node, frequently triggering a domino effect that impacts neighboring applications on the same physical server.

In corporate scenarios, lacking these constraints turns a sudden traffic spike into a widespread outage. Before planning fine-tuning, remember that infrastructure issues usually come with other operational pains, such as intermittent name resolution failures caused by CoreDNS or full disks locking the environment, as discussed when exploring how to keep the CI/CD pipeline running without running out of disk space in Docker. Defining clear limits isolates damage, ensuring that if a service suffers a denial-of-service attack or an internal leak, it dies alone without taking down the rest of the infrastructure.

How to track the exact moment of memory death

When a pod restarts, the first investigation tool is the standard Kubernetes inspection command. Running the pod listing will likely show statuses like OOMKilled or an exit code of 137. This number 137 is a mathematical signature resulting from adding 128 to code 9, which represents the SIGKILL signal sent by the operating system to forcefully terminate the runaway process. Knowing how to decode this number saves hours of trial and error at the engineering workbench.

To extract additional details from the cluster's recent event history, use the detailed describe command. It displays reboot history and the exact reason for the container's last termination. See how to inspect your pod's behavior directly in the terminal:

kubectl get pods -n production
kubectl describe pod my-web-service-7b99d-xyz9q -n production

If traditional textual logs disappear along with the process, the best alternative is to centralize infrastructure event and metric information before the container is exterminated. Maintaining a robust observability pipeline complements the diagnostic strategy, just as centralizing scattered cluster logs with Fluent Bit allows you to audit application behavior milliseconds before collapse.

Identifying memory leaks at the code root

Often, the OOMKilled error isn't the fault of a low limit, but rather a memory leak within the application. This happens when code allocates space in RAM to process data but forgets to release that reference when the task finishes. Managed languages like Java, Node.js, and Python feature automatic garbage collectors, but if global references accumulate objects in lists or infinite caches, the collector becomes useless and memory grows linearly until hitting the Kubernetes ceiling.

To investigate leaks in development or staging environments, use language-specific memory profiling tools like heapdump in Node.js or pprof in Go applications. Monitor container behavior under simulated load and check whether RAM consumption drops after traffic peaks or maintains a constant upward curve. If memory never returns to baseline after requests end, you've found a code leak that needs root-level fixing rather than simply raising resource limits in the YAML file.

Step-by-step guide to securely tuning requests and limits

If code analysis ruled out leaks and the application simply needs more breathing room to run comfortably, the next step is recalibrating the Kubernetes manifest. Never guess random values; base them on real metrics collected by tools like Prometheus and Grafana during peak operational hours. Follow the procedure below to update resource sizing for a production deployment:

1. Check the historical peak memory consumption of the container over the past two weeks using cluster metrics.
2. Edit your deployment's YAML manifest file, locating the resource section for the affected container.
3. Apply the updated changes to the cluster by running the rollout update command and validating stability.

See a practical example of how to structure resource blocks inside your Kubernetes deployment configuration file:

resources:
  requests:
    memory: "512Mi"
    cpu: "250m"
  limits:
    memory: "1Gi"
    cpu: "500m"

Final considerations

Investigating OOMKilled failures requires patience and a systemic vision spanning from code writing to infrastructure node configuration. Understanding that memory limits are not a punishment, but a stability contract, changes how we design resilient applications for cloud environments. Monitoring consumption trends, applying data-driven adjustments, and fixing internal leaks ensures your containers remain steady, delivering continuous value to users without unexpected late-night alarms.