Container restarting? How to debug Docker Exit Codes

Why is your Docker container restarting by itself?

When a Docker container suddenly shuts down and enters an infinite loop of restarts, the initial reaction is often panic. In practice, Docker does not shut down a system on a whim; it always leaves a valuable clue about what happened right before termination. This clue is the Exit Code, a numeric value returned by the process when it stops working. Understanding these numbers turns a mysterious error into a clear diagnostic, allowing you to fix the root cause rather than just restarting the service and praying it lasts longer.

To investigate thoroughly, we need to look beyond the basic 'docker ps' command. The Docker engine talks directly with the operating system kernel to manage process isolation. When something goes out of control—whether it is a lack of RAM memory, a low-level programming error, or a refused network connection—the kernel steps in and abruptly terminates the application. Before analyzing the details, it is worth noting that managing infrastructure failures requires patience and the right tools, much like investigating physical hardware faults, as discussed in our guide on how to test integrated circuit on board with oscilloscope.

Unlocking common exit codes: 137, 139, and 111

An exit code is an integer that the program hands over to the operating system upon finishing its execution. In the Docker universe, certain numbers appear so frequently that they deserve a permanent place in any engineer's memory. The infamous exit code 137, for instance, is the classic sign that the process was summarily executed due to a lack of memory. In practice, this means the application requested more RAM than allocated to the container, and the OOM Killer stepped in to save the host system. Another frequent code is 139, representing a segmentation fault, which occurs when software attempts to read or write to a forbidden or corrupted memory address.

Code 111 usually appears in connection refused scenarios, indicating that the application tried talking to a database or external API that wasn't ready or listening on that specific port. To inspect the exact exit code of a recently crashed container, you can run the following command in your terminal:

docker inspect --format='{{.State.ExitCode}}' container_name

This simple command reads directly from Docker's metadata structure and returns the exact number, saving investigative time and pointing you straight to the correct error category.

The critical role of PID 1 and signal management

Inside a container, the first process started receives the identifier PID 1. In Unix system theory, PID 1 carries immense responsibility: it acts as the 'boss' of all other spawned processes, charged with reaping orphan processes and passing vital signals, such as the SIGTERM termination command. When we write Dockerfile instructions incorrectly, we often place a simple script or an interpreter at PID 1 that cannot handle operating system signals properly. The practical result is that when you try to stop the container, it simply ignores the polite warning and must be forcibly killed after a ten-second wait.

To prevent this unwanted behavior, it is crucial to use the exec form in your Dockerfile or rely on lightweight process managers that correctly forward signals. When a container completely freezes and stops responding even to normal commands, operators must resort to brute-force command-line tools, a scenario we cover in detail in Docker container stuck? How to force stop via CLI. Ensuring PID 1 is a process aware of its role prevents data corruption and sudden drops of critical services.

Tracking kernel logs and identifying the OOM Killer

Often, your application's traditional logs (retrieved via 'docker logs') suddenly stop and show no explicit error prior to a crash. This happens because the failure occurred outside the programming language's reach, directly at the operating system level. When memory consumption exceeds the container's cgroup ceiling, the Linux kernel acts silently. To confirm if your container fell victim to this summary execution, you need to check the host system's general event log using the dmesg command in the server terminal:

dmesg -T | grep -i oom

This command filters kernel history showing out-of-memory events accompanied by readable timestamps. Inspecting the output reveals exactly which process was sacrificed and what the memory consumption was at that precise moment. Identifying this pattern allows you to adjust resource limits in your Docker Compose file, preventing further unwanted interruptions in production.

How to prevent restart loops and ensure resilience

Discovering why a crash happened is only half the battle; the other half is ensuring the system remains stable and predictable. Automatic restart policies like 'restart: always' are great for keeping services online after a sudden failure, but they can mask serious structural problems, creating a vicious cycle of CPU consumption and error recurrence. The ideal approach is to combine realistic memory limits, proper monitoring strategies, and load tests to validate application behavior before moving to production.

In short, mastering exit code reading and understanding the dynamic between the container and the Linux kernel eliminates guesswork when debugging infrastructure issues. By monitoring PID 1, correctly interpreting codes like 137 and 139, and checking dmesg when necessary, you gain total autonomy to diagnose and fix any failure in your containerized environments with speed and precision.