How to inspect encrypted HTTPS traffic using mitmproxy

Why do we need to intercept secure connections?

When developing web or mobile applications, most communication with servers occurs over the HTTPS protocol, which ensures privacy and data integrity through end-to-end encryption. In practice, this means that even we, building the system on our own computers, cannot easily see what data is being sent across the wire, turning traffic into an opaque black box during debugging. To solve this dilemma without compromising production security, we use tools known as intercepting proxies, capable of temporarily decrypting traffic in a controlled development environment.

If you have ever needed to debug a container hanging or understand the exact behavior of a third-party API, you know that guessing the contents of a network packet is a frustrating waste of time. It is precisely in this scenario that mitmproxy stands out as an extremely powerful open-source alternative to Charles Proxy or Fiddler. It runs directly in the terminal, accepts complex automations via scripts, and consumes minimal computer resources, making it essential for engineers who need to inspect HTTP headers, request bodies, and status codes in real time.

What is mitmproxy and how does it work under the hood?

Mitmproxy is a suite of interactive tools focused on HTTP and HTTPS traffic analysis, consisting of an interactive command-line interface (mitmproxy), a user-friendly web version (mitmweb), and a command-line tool for pure automation (mitmdump). In practice, it acts as a legitimate 'man in the middle': your application sends the request to mitmproxy thinking it is the final server, mitmproxy decrypts the packet using its own certificate, reads or alters the content, and then forwards the call to the actual destination, repeating the process on the return journey.

For this magic to happen without triggering insurmountable security alerts in the operating system, we need to install the root certificate generated by mitmproxy itself onto the test machine. While setting a static IP on Linux ensures network route stability, trusting the proxy's certificate ensures that the browser or application accepts the intermediated connection without corrupting the SSL/TLS encryption handshake.

Step-by-step guide to installing and configuring mitmproxy

Installing mitmproxy can be done easily via the system package manager or through pip, the Python package manager. We will use the command-line environment to run the utility for the first time and generate the necessary configuration files in your user folder.

1. Install mitmproxy on your operating system by running the installation command in your development machine terminal.

   pip install mitmproxy

2. Start mitmproxy for the first time so it can create the SSL certificates in the hidden folder of your home directory.

   mitmproxy

3. Configure your browser or operating system to direct all HTTP and HTTPS traffic to the local address on the default port.

   export http_proxy='http://127.0.0.1:8080' && export https_proxy='http://127.0.0.1:8080'

After executing these initial steps, open your browser and visit the special address provided by the tool to download the certificate file and install it into your system's trusted certificate store. Without this step, any triggered HTTPS request will generate a fatal invalid certificate error in the monitored applications.

Inspecting and modifying requests in real time

With the proxy running and certificates properly trusted, any request made from the local machine will instantly appear in the mitmproxy interface. You can navigate through the lines using your keyboard arrow keys, select a specific request by pressing Enter to examine in detail the outgoing headers, session cookies, and the JSON payload sent in the body of a POST or PUT call.

One of the most useful features during staging tests is the ability to pause the request before it reaches the destination server, allowing you to manually alter crucial parameters to test how the system behaves under unexpected data inputs. For those managing local servers who need to clean Docker cache on Linux to free up disk space, monitoring container traffic through mitmproxy reveals exactly which old or corrupted requests are being repeatedly sent by the application.

Automating tasks with Python scripts

When manual visual inspection is no longer enough, mitmproxy shines by allowing the creation of Python scripts to intercept and modify traffic programmatically. We can write simple functions that listen to request lifecycle events and apply automatic rules, such as injecting an authentication header into all calls or simulating a microservice connection drop.

Below is a basic example of a script that intercepts all HTTP requests and adds a custom tracking header before forwarding them along:

from mitmproxy import http

def request(flow: http.HTTPFlow) -> None:
    # Adds a custom header to all intercepted requests
    flow.request.headers["X-Debug-Inspection"] = "Active"
    print(f"Intercepted request to: {flow.request.pretty_url}")

This level of programmability turns mitmproxy into a Swiss Army knife for integration testing, enabling the simulation of complex network failure scenarios that would be extremely difficult to reproduce in a real production environment.

Final thoughts on secure debugging

Mastering the use of an interception proxy like mitmproxy completely changes how we approach integration issues between systems and APIs. By decoding encrypted traffic in controlled environments, we eliminate guesswork and successfully isolate the root cause of complex bugs within minutes, saving hours of frustration at the development workbench.

Always remember that these powerful tools should be used strictly in development, testing, and staging contexts on secure machines. Never install interception certificates in production environments or on personal devices without rigorous security controls, thereby ensuring data integrity and the privacy of end-users.