How to Simulate Network Failures and Latency in Requests Using Toxiproxy

The Invisible Challenge of Network Instability

When developing modern applications, we usually test everything in controlled environments where the internet works flawlessly without delays or drops. In practice, the reality of everyday users is entirely different, marked by unstable connections, fluctuating mobile networks, and sluggish servers on the other side of the planet. Testing your software's resilience against these scenarios requires specialized tools capable of manipulating data traffic seamlessly. Without simulating these failures beforehand, problems are only discovered once the system is live, causing customer frustration and emergency middle-of-the-night pages.

To solve this dilemma, engineers often resort to manual workarounds like unplugging the network cable or throttling browser speeds, but these solutions are limited and hard to automate. This is where the concept of a TCP proxy comes in, serving as an intermediate server sitting between your application and the external service, intercepting and manipulating all passing traffic. In practice, a proxy acts like a strict gatekeeper that can choose to delay a message delivery, tear apart a data packet mid-flight, or simply pretend the recipient is not home. This surgical manipulation of network flow is the key to building fault-tolerant systems.

Understanding Toxiproxy and Its Architecture

Created by the Shopify team, Toxiproxy is an open-source tool specifically designed to test adverse network conditions in an automated and programmatic manner. The project is split into two main parts: a Go-written server that executes network manipulation rules and a client API available in multiple languages to control that server. In practice, you configure Toxiproxy to listen on a local port and redirect traffic to your real database or API, injecting called "toxics" along the way. This means your application keeps pointing to a local address without realizing the traffic is being subtly corrupted or delayed in transit.

The major advantage of this architecture is absolute isolation, as no application code needs modification for chaos tests to take place. You do not need to inject error-simulation logic into your production codebase, keeping the code clean and strictly focused on business logic. Furthermore, because Toxiproxy runs effortlessly as a Docker container, it integrates seamlessly into continuous integration environments where automated tests run with every new line of code written. This configuration ease turns resilience tests—which once demanded complex lab setups—into trivial tasks running on any developer's machine.

Setting Up the Environment and Creating the First Proxy

To get the tool up and running, the fastest approach is using Docker to spin up the Toxiproxy server alongside the CLI (command-line interface) to manage rules. The basic command to start the server consists of mapping the port where the proxy listens and the control API port, enabling you to send commands for creating virtual connections. In practice, you create a proxy specifying a friendly name, the local listening address, and the real service address you want to reach, such as a PostgreSQL database or payment microservice. Once created, any request made to the local port will be mediated by Toxiproxy before reaching its final destination.

We can exemplify this setup by creating a command-line proxy for a fictitious API service:

docker run --rm -d -p 8474:8474 -p 8080:8080 --name toxiproxy shopify/toxiproxy
toxiproxy-cli create -l 0.0.0.0:8080 -u api.example.com:443 my-api

In this example, Toxiproxy is now listening on port 8080 and forwarding all traffic to 'api.example.com'. The client application starts making requests to 'localhost:8080', allowing the developer to apply traffic modifications at runtime without restarting any services. This dynamic flexibility is essential for testing how the system reacts when the network degrades abruptly during usage.

Injecting Latency and Jitter to Test Timeouts

Latency is the temporal delay a data packet takes to travel from origin to destination, while jitter represents the unpredictable variation in delivery speed. In practice, real networks are never constant; a packet might take 50 milliseconds one moment and 400 milliseconds right after due to route congestion. To simulate this behavior in Toxiproxy, we add a 'latency' toxic to our previously configured proxy, specifying the base delay time and acceptable variation. This forces the application to handle prolonged waits and helps validate whether timeout limits are configured with realistic and safe margins.

Here is how to add latency using the tool's command-line interface:

toxiproxy-cli toxic add -t latency -a latency=1000 -a jitter=200 my-api

In practice, this command adds a one-second delay with a two-hundred-millisecond variance to all requests passing through the proxy. When your application tries talking to the API, it notices the response took much longer than usual, triggering internal protection mechanisms. If your system lacks well-defined timeouts, processing threads will hang waiting for responses, exhausting server resources and crashing the entire application in a cascade. Toxiproxy makes these architectural flaws visible before they reach production environments.

Simulating Packet Loss and Connection Drops

Beyond slowness, networks suffer from packet loss, where entire chunks of information vanish mid-way due to physical interference or routing glitches. When this happens, the transport protocol must resend lost data, generating retransmissions that drastically degrade overall distributed system performance. In Toxiproxy, we can simulate this disastrous scenario using the packet loss toxic, defining an exact percentage of data that will be randomly discarded by the proxy. This simulation immediately reveals whether your application code knows how to handle transient failures or if it breaks at the first sign of network instability.

Another extremely useful toxic for chaos testing is the 'timeout' toxic, which abruptly closes connections after a set period of inactivity or programmatically. This lets you verify if the system has robust retry policies combined with exponential backoff waiting strategies. In practice, if a connection drops mid-payment, the application must not simply duplicate the charge but safely verify the transaction state before trying again. The combined use of these toxics ensures software maintains data integrity even when operating over completely chaotic and hostile network infrastructure.

Automating Resilience Tests in CI/CD Pipelines

Testing networks manually is helpful during early development, but true operational maturity comes when these chaos tests run automatically within continuous integration pipelines. In tools like GitHub Actions or GitLab CI, you can spin up the Toxiproxy service in the background during integration and end-to-end test suite execution. The test suite can then programmatically turn different toxics on and off via Toxiproxy's REST API, validating specific failure scenarios for each test case run. Thus, any code change removing a network error handler or imprudently lowering a timeout is immediately blocked before reaching the main repository.

This automated approach turns network resilience into a measurable and testable requirement, exactly like traditional unit testing. Developers gain confidence to refactor legacy integration code knowing unstable networks won't surprise them on a Friday afternoon in production. Furthermore, documenting these test scenarios helps the team understand real operational limits, facilitating capacity planning and defining Service Level Agreements (SLAs). Modern engineering requires assuming failure is inevitable, and tools like Toxiproxy give us the power to rehearse that failure safely.

Final Thoughts on Resilience Engineering

Simulating failures and latency in network requests is no longer a luxury restricted to major tech companies; it has become a fundamental necessity for any modern distributed system. Throughout this article, we saw how Toxiproxy acts as a powerful, transparent intermediary, allowing the injection of delays, jitter, and packet loss without modifying a single line of application code. This practice uncovers hidden vulnerabilities, like misconfigured timeouts and lack of idempotency, which typically cause massive headaches in production environments. Adopting this chaos-testing mindset ensures your application remains steadfast and stable even when the surrounding world faces a digital storm.

Ultimately, system stability depends not only on isolated code robustness, but on how it interacts with real-world chaos. Integrating network simulations into automated tests elevates team technical maturity, shifting focus from fire-fighting to systematically and predictably preventing failures. With accessible and flexible tools, any engineer can start testing infrastructure limits today, building software truly prepared for the unexpected. Resilience ceases to be an abstract promise and becomes an engineering attribute validated at every development cycle.