How to Inspect mTLS Traffic in Kubernetes Pods with cURL

Why mTLS Secures Microservices Inside Kubernetes

In modern infrastructure environments, ensuring that a container only communicates with authorized peers is an operational necessity. This is where mTLS, or mutual Transport Layer Security, comes into play—a protocol requiring both the client and the server to present valid digital certificates before any data exchange occurs. In practice, this means encryption alone is not enough; each microservice must prove its cryptographic identity across the cluster network, preventing unauthorized eavesdropping or message tampering between neighboring applications.

When adopting service meshes like Istio or Linkerd, this protection happens automatically through sidecar proxies injected alongside our application. However, when a handshake error occurs—meaning the initial security negotiation fails and the connection is rejected—finding the exact root cause can turn into a difficult puzzle. Traffic is often silently blocked, producing cryptic logs that point to no obvious culprit.

The Challenge of Debugging TLS Handshake Failures in Isolation

Network debugging in Kubernetes clusters is notoriously stressful because pods run in isolated namespaces, frequently lacking traditional diagnostic tools inside production container images. When traffic is masked by mutual encryption, capturing packets with network traffic inspection tools like tcpdump only reveals unreadable encrypted data blocks, failing to show why the server rejected the certificate.

To make matters worse, before troubleshooting the network, underlying infrastructure issues such as resource bottlenecks described in CPU and memory container audits can crash the proxy and simulate a phantom connection failure. Therefore, isolating the problem requires mimicking the exact call made by the microservice, manually injecting the same certificates distributed by the Service Mesh using a versatile HTTP client like cURL.

How to Extract Certificates and Keys from a Kubernetes Pod

Before executing cURL commands, we need to gather the cryptographic secrets that the cluster generated and injected into the source pod. mTLS relies on three primary elements: the public certificate authority (CA) certificate, the client certificate itself, and the matching private key proving ownership.

Typically, these files are stored in volumes mounted inside the container, often under /etc/certs or within Kubernetes secrets mounted via volumeMounts. To inspect these files without disrupting production, we can open an interactive shell in an ephemeral debug pod or copy the files directly using the Kubernetes remote copy utility:

kubectl cp my-namespace/my-pod-abc:/etc/certs/tls.crt ./tls.crt && kubectl cp my-namespace/my-pod-abc:/etc/certs/tls.key ./tls.key

With files saved locally, we can inspect expiration dates and Subject Alternative Names (SANs) using OpenSSL to ensure the certificate is valid and belongs to the service we intend to debug.

Executing cURL with mTLS to Simulate Pod-to-Pod Calls

With certificate files ready, the next step is building the cURL command pointing to the target pod's internal IP or the cluster's internal DNS service name. Because the server enforces mutual authentication, we must explicitly supply the client certificate (--cert), the private key (--key), and the CA validating the server certificate (--cacert).

To run this test effectively, follow this sequence from inside a utility pod equipped with cURL in the same cluster:

1. Copy the certificate and private key files into the working directory of your diagnostic pod.
2. Execute the cURL command passing the mTLS parameters pointing to the target microservice secure port.
3. Analyze the returned HTTP status code and the detailed handshake trace generated by the verbosity flag.

The practical command for this test follows this detailed structure:

curl -iv --cert /tmp/tls.crt --key /tmp/tls.key --cacert /tmp/ca.crt https://target-service.namespace.svc.cluster.local:8443/api/v1/status

The flags -i and -v are crucial here: they output response headers and the full TLS handshake establishment trace, making it easy to identify precisely where the negotiation failed.

Interpreting Common Handshake Errors and Troubleshooting

When cURL returns a connection error, the terminal output usually reveals the core issue. If the error reads SSL certificate problem: self signed certificate in certificate chain, it means the server presented a certificate signed by an unfamiliar CA—the fix is ensuring the file referenced in --cacert matches the exact root CA that issued the server certificate.

Another classic error occurs when the server rejects the client with a handshake alert code, indicating either an expired client certificate or a domain mismatch against the Service Mesh security policies. Validating these details eliminates invisible roadblocks and restores operational predictability across microservice ecosystems.

Final Thoughts on Network Diagnostics in Service Meshes

Inspecting mTLS traffic inside Kubernetes pods using cURL is an essential skill for engineers navigating complex distributed architectures. While service meshes automate most encryption layers, understanding underlying certificate mechanics and knowing how to forge manual requests drastically accelerates production incident resolution. Mastering this technique turns obscure network failures into fast, precise diagnostics.