Deploy failing? How to inspect Kubelet logs on the node

What is the Kubelet and why does it fail during a deploy

When a deployment fails in Kubernetes, frustration is usually immediate. Generic messages like CrashLoopBackOff or ContainerCreating show up in the terminal, but they often fail to reveal the root cause. This is precisely where the Kubelet comes in—the invisible agent running on every machine in your cluster (the nodes) that turns theoretical specifications into actual containers running in practice.

In engineering routines, the Kubelet acts like the local manager of a major construction site: it receives blueprints sent by the central server and coordinates the workers, which in our case are tools like Containerd or Docker. If material is missing, if the foundation fails, or if there is a permission error, the Kubelet is the one recording the incident in the system's logbook. Understanding how to read this logbook directly on the machine is the key to unblocking any frozen infrastructure.

Often, startup failures are closely tied to exhausted resource problems on the host, as we discussed when analyzing why pods get OOM Killed in Kubernetes and how to adjust limits. When the operating system needs to forcibly kill processes to save itself, the Kubelet records the kernel's distress in its internal logs. Let's see how to access this valuable information using simple and direct commands.

How to access the node and isolate the Kubelet service

To inspect the Kubelet, the first step is to open a secure SSH connection directly to the virtual or physical machine experiencing the issue. Unlike traditional kubectl commands that talk to the central API, here we are standing on the factory floor, looking directly at the operating system supporting the containers.

The vast majority of modern Linux distributions manage services through Systemd, the standard initialization system. This means Kubelet logs are captured and stored centrally by a tool called Journald. To examine the recent behavior of our agent, we use the system's standard query utility.

Run the command below in the node terminal to view the most recent entries generated exclusively by the Kubelet service:

sudo journalctl -u kubelet -e --no-pager

This command filters out the chaos generated by the entire operating system and focuses only on the latest Kubelet lines, letting you quickly identify whether there were certificate rejections, API communication failures, or timeout errors while trying to pull an image.

Identifying image errors and disk failures on the node

One of the most common reasons for a pod to refuse startup is the inability to download the container image from the remote registry, whether due to invalid credentials, temporary network glitches, or a lack of disk space on the node. When this happens, the Kubelet repeatedly tries to download the package, generating a rich history of error messages.

When inspecting logs, look for terms like ImagePullBackOff, ErrImagePull, or failures related to the underlying filesystem. In some cases, severe I/O problems on the machine can corrupt local storage, much like what we investigate when a slow NVMe SSD on Linux requires terminal diagnostics before Kubernetes starts rejecting new workloads.

To monitor events in real-time while trying to recreate the problematic pod in another window, use the continuous tracking parameter of journalctl:

sudo journalctl -u kubelet -f -n 100

This continuous view allows you to trigger the deployment command and watch, second by second, how the Kubelet interacts with the container runtime to pull layers, mount volumes, and start internal processes.

Inspecting the runtime and resolving the freeze

Beyond Kubelet textual logs, we often need to look directly at the engine running the containers—usually Containerd. The Kubelet delegates the actual creation of processes to this engine, and a communication error between them can completely freeze the application lifecycle.

When the node exhibits widespread freezes and the Kubelet appears stuck, it is worth checking whether the kernel terminated critical processes due to excessive memory consumption, as detailed in our guide on how to audit OOM Killer with dmesg. Often, the Kubelet suffers abrupt crashes simply because the machine itself ran out of breath.

To safely restart the agent after applying configuration fixes or expired certificates, use the system service manager:

sudo systemctl restart kubelet

This action forces the Kubelet to reread its local configuration files, re-establish secure communication with the central control plane, and resume synchronizing pods assigned to that specific node.

Final thoughts on deep diagnosis in Kubernetes

Mastering Kubelet log inspection directly on the node transforms a software or DevOps engineer from a mere spectator into a complete problem solver. Instead of relying solely on high-level commands that often omit crucial infrastructure details, stepping down to the operating system level reveals exactly where the container ecosystem hit barriers.

Maintaining this habit of structured troubleshooting ensures that production outages stop being a frightening mystery and become incidents with root causes identified and resolved in minutes. Observability always begins where abstraction ends.