How to track slow HTTP requests in Nginx with logs

How reverse proxy web traffic hides performance bottlenecks

When a user visits a modern website, the request rarely hits the application code directly. Instead, it passes through a reverse proxy server that acts as an intelligent gatekeeper, receiving connections, managing security certificates, and distributing traffic to backend application servers running behind the scenes. In practice, this architecture protects internal systems and optimizes resource usage, but it creates a veil of mystery when things slow down. If a page takes ten seconds to load, developers often struggle to know whether the delay happened in the network, at the edge server, or deep within the backend code.

Nginx is the most popular maestro of this edge architecture, renowned for its extreme efficiency in handling tens of thousands of simultaneous connections with minimal memory. However, the software's default logging configuration focuses only on essentials, such as the caller's IP address, the requested URL, and the returned HTTP status code, like the familiar 200 success or 500 internal error. Without tuning these metrics, diagnosing intermittent slowdowns becomes a guessing game. To go beyond the basics and discover exactly where time is being wasted, we need to open the black box of reverse routing and teach the server to detail every single millisecond of waiting.

Understanding the anatomy of a request and the upstream role

To track bottlenecks accurately, we must understand the concept of upstream, which represents the group of backend servers responsible for processing heavy application logic, such as writing database records or rendering dynamic pages. When Nginx forwards a request to this destination, a critical temporal window opens, encompassing the time the proxy takes to send data, the period the backend spends processing the request, and the moment the response finally returns to the edge. In practice, the total delay perceived by the client is the sum of all these steps combined with network latency.

Spot diagnostic tools help isolate momentary issues, such as how to inspect stuck HTTP requests using cURL, allowing you to test route behavior directly from the terminal. However, when issues affect thousands of users scattered globally at varying times, manual commands are no longer sufficient. We need continuous, structured historical logging that records the behavior of every call without burdening the server disk, turning raw data into actionable intelligence for the engineering team.

Customizing log_format to capture milliseconds

The secret to uncovering hidden latency lies in Nginx's customized logging format directive, known as log_format. By default, the system uses a combined format that records only generic information. However, Nginx provides advanced time and performance variables that can be injected directly into each logged line. Among the most valuable variables are total request time, connection time, and most importantly, specific origin server metrics.

To implement this improvement, edit the main configuration file, typically located in the service's installation directory, and add a custom line inside the http block. It is essential to choose a clear name for the new format, such as backend_audit, and list the desired variables in an organized way. In complex microservices environments, it is also useful to validate name resolution before traffic hits the proxy, similar to how to test DNS resolution and response time with dig to ensure slowdowns do not originate from misconfigured name lookups.

Implementing the directive in the configuration file

Let us get hands-on with a practical configuration example that adds vital upstream timing metrics. Open the main configuration file using a text editor with administrative privileges and locate the http block where global directives are declared. The following block defines a rich format that exposes the exact backend response time and the total delivery time to the client.

http {
    log_format backend_audit '$remote_addr - $remote_user [$time_local] '
                             '"$request" $status $body_bytes_sent '
                             '"$http_referer" "$http_user_agent" '
                             'rt=$request_time ut=$upstream_response_time '
                             'uct=$upstream_connect_time uht=$upstream_header_time';

    access_log /var/log/nginx/access_audit.log backend_audit;
}

In the code snippet above, the variable rt represents the total request time in seconds with millisecond precision, while ut exclusively reveals the time the backend took to respond. The variables uct and uht measure, respectively, the time taken to connect to the upstream and the time the server took to send the first response header. After saving the file, test the syntax by running a validation command in the terminal and reload the service to apply changes without dropping active connections.

Validating syntax and applying changes safely

Before pushing any configuration change to production, it is mandatory to test for typos or invalid directives to prevent catastrophic outages on the web service. Modern servers make this verification easy through a native command that analyzes all configuration files and points out the exact line of any mistake. In practice, this simple precaution ensures the ecosystem remains stable while you adjust traffic observability policies.

To run the validation test and reload Nginx without interrupting user requests, use the following commands in your Linux server terminal:

sudo nginx -t
sudo systemctl reload nginx

If the return message confirms that the syntax is correct and the test succeeded, the graceful reload will apply instantly. From this exact second onward, every incoming HTTP request will stamp the designated log file with detailed latency behavior, enabling you to cross-reference data and find bottlenecks that previously went completely unnoticed by the technical team.

Interpreting data and mining bottlenecks in logs

With new data flowing into the log file, the next challenge is extracting useful intelligence without spending hours staring at endless lines of text. Since logs now contain variables like ut (upstream time), you can quickly identify which routes or endpoints are choking the system. If total request time is high but upstream time is low, the bottleneck likely resides in client-side processing, data serialization, or the network between the edge and the browser. If upstream time is high, the investigation should immediately shift to application code, slow database queries, or a lack of caching.

To automate this scan across large volumes of textual data, engineers often combine native operating system utilities, such as how to use awk to extract log columns in terminal, to isolate specific metrics and sort the slowest requests of the day. This daily routine turns observability into a proactive ally, allowing you to fix performance flaws before customers notice any service degradation.

Final thoughts on observability and web performance

Tracking slow HTTP requests in Nginx goes far beyond simply editing a line of code in configuration files; it is about embracing a data-driven mindset in infrastructure management. By demystifying the reverse traffic flow and injecting granular upstream metrics into log_format, your team gains the surgical capability to diagnose complex problems in seconds, eliminating guesswork and directing efforts exactly where the application needs optimization most.

Maintaining visibility into actual system behavior ensures more resilient applications, satisfied customers, and a much smoother engineering operation. As traffic grows, combining structured proxy logs with real-time monitoring tools builds a fortress of reliability that supports the continuous growth of any digital business.