How to debug latency and chaos in HTTP requests with Toxiproxy

Why does your application fail when the network slows down?

When we write software on fast computers connected to ultra-low-latency local networks, everything appears to work flawlessly. However, the real world is unforgiving: cloud servers suffer from internet congestion, submarine cables get severed, and mobile connections fluctuate constantly. In practice, this means your application must handle delays and interruptions gracefully without corrupting data or freezing the user interface. Testing these unfavorable scenarios in a controlled manner used to be complex, requiring manual routing or firewall changes.

To solve this headache without friction, engineers rely on network-layer chaos engineering tools. This is where Toxiproxy comes in, a fantastic utility created by Shopify that acts as an intelligent intermediary between your application and external databases or services. Instead of tweaking physical infrastructure, you spin up a local proxy that intercepts network traffic and applies artificial degradation rules, letting you observe exactly how your code behaves under heavy operational stress.

What is Toxiproxy and how does it work in practice?

Toxiproxy consists essentially of two parts: a lightweight application written in Go that manages TCP connections, and an HTTP API that lets you alter connection behaviors at runtime. When you launch Toxiproxy, it opens a port pretending to be your real service—for example, a PostgreSQL database. Your application connects to this Toxiproxy port instead of hitting the database directly. In practice, any data packet passing through can be delayed, corrupted, or simply dropped.

This modular approach completely transforms how we build automated integration tests. Instead of waiting for luck to catch a timeout error in production, you program your staging environment to intentionally inject two hundred milliseconds of latency during a payment routine. To ensure traffic remains transparent and auditable during these stress tests, engineering teams frequently combine these simulations with deep packet inspections, such as inspecting corrupted network traffic with eBPF on high-performance Linux environments.

Setting up Toxiproxy with Docker Compose

The cleanest and fastest way to run Toxiproxy on your development machine or CI server is using Docker. We can create a simple configuration file that spins up both the tool and a dummy service for testing. Below is a practical example of a setup file to deploy Toxiproxy alongside a backend application.

version: '3.8'
services:
  toxiproxy-server:
    image: shopify/toxiproxy:latest
    ports:
      - "8474:8474"
      - "5433:5433"
    command: ["/toxiproxy-server", "--host=0.0.0.0"]
  app:
    image: my-backend-service:latest
    environment:
      - DB_HOST=toxiproxy-server
      - DB_PORT=5433
    depends_on:
      - toxiproxy-server

With this file running, port 8474 is open to receive traffic modification commands, while port 5433 redirects packets to the final destination configured via API. This separation ensures tests occur completely isolated without interfering with other services in your cluster or homelab.

Creating your first chaos scenario with toxics

In Toxiproxy terminology, network degradation rules are called "toxics." You can add toxics using the official command-line interface or by making direct HTTP requests to the control API. To demonstrate how to inject latency and packet loss, we will use practical terminal commands executed from your test bench.

Before applying any chaos rule, we must register the route Toxiproxy will monitor, pointing to the original service address. Follow the steps below to configure the proxy and inject instability:

1. Register the proxy pointing to your actual database or external service:
   

   curl -X POST http://localhost:8474/proxies -d '{"name": "database", "listen": "0.0.0.0:5433", "upstream": "192.168.1.50:5432"}'

2. Add an artificial delay of five hundred milliseconds with a one-hundred-millisecond jitter:
   

   curl -X POST http://localhost:8474/proxies/database/toxics -d '{"type": "latency", "attributes": {"latency": 500, "jitter": 100}}'

3. Simulate a ten percent timeout drop rate to test retry resilience:
   

   curl -X POST http://localhost:8474/proxies/database/toxics -d '{"type": "timeout", "attributes": {"timeout": 1000}}'

Running these commands causes any request made to port 5433 to immediately experience the configured impact. This allows you to evaluate in real-time whether your services retry connections safely or collapse due to thread exhaustion.

Final thoughts on resilience in modern systems

Intentionally simulating network failures is no longer a luxury restricted to large tech companies; it has become a fundamental engineering requirement. Tools like Toxiproxy democratize chaos testing, enabling any developer to uncover hidden flaws in their architecture before they hit users in production. Knowing exactly how your software reacts to severe delays and corrupted packets ensures much more stable, secure applications ready for any infrastructure surprise.

Investing time in robust staging chaos scenarios protects businesses from financial losses and drastically reduces operations team stress during critical incidents. Combine these network resilience practices with rigorous hardware resource monitoring — such as measuring bare-metal server power consumption with IPMI — and you will gain a complete overview of your infrastructure health across both logical and physical layers.