Disk full from logs? How to rotate with logrotate

The silent danger of disk full errors caused by server logs

When running applications on production servers, the continuous generation of textual records is essential for monitoring system health and auditing user access. However, without a proper management strategy, these files grow exponentially until they exhaust all available space on the hard drive. In practice, this means the operating system loses the ability to write temporary files, databases crash due to lack of transaction space, and the entire server simply stops responding. This is a classic scenario where the very mechanism created to help us diagnose issues ends up causing a total outage due to operational neglect.

To prevent this type of disaster, the standard tool on Linux-based operating systems is a utility called logrotate, a background program responsible for automating the cleanup, compression, and deletion of old log files. Before implementing any retention policy, it is very useful to understand how to diagnose disk space consumption in the environment. As we detail in our article on Docker out of space? How to clean cache and free disk, monitoring block and inode usage is the first step to prevent unexpected server shutdowns.

How the log file lifecycle works using logrotate

The logrotate tool operates on a schedule, usually via a cron job running every day during the early morning hours. The basic process consists of taking the current log file, renaming it with a numeric or date suffix, creating a brand new empty file so the application can keep writing without interruption, and optionally compressing the old file to save space. This cycle prevents a single text file from reaching dozens of gigabytes, which would make opening and searching for errors an extremely slow task for any engineer.

Another critical point in this process is ensuring that the running application realizes the file has been renamed and starts writing to the new file created by the system. If the application keeps the old file descriptor open in memory, it will continue dumping data into the renamed file, occupying disk space just the same until the process restarts. To solve this, logrotate uses post-processing directives, such as the copytruncate instruction or signals sent directly to the service so it reopens log files cleanly and safely.

Writing a robust retention policy for critical servers

Logrotate configuration is centralized in files located within a specific system directory, where each application can have its own customized rule. Let's look at a practical configuration example for a corporate web application that generates intense daily logs on a production server. This policy ensures daily rotation, keeps a thirty-day history, compresses old files, and prevents temporary file absence from causing utility errors.

/var/log/my-application/*.log {
    daily
    rotate 30
    compress
    delaycompress
    missingok
    notifempty
    create 640 www-data adm
    sharedscripts
    postrotate
        /usr/bin/systemctl reload my-application.service > /dev/null 2>&1
    endscript
}

Every line in this configuration block plays a decisive role in server stability. The daily directive forces checking and rotation every 24 hours, while rotate 30 specifies that only the thirty most recent files are kept, automatically deleting older ones. The compress option shrinks records using the gzip utility, and delaycompress ensures yesterday's rotated file is only compressed during tomorrow's rotation, preventing very recent write processes from conflicting with decompression. Additionally, the create directive defines exact security permissions for the newly generated file.

Validating and testing logrotate policies before applying

Deploying a new rotation rule directly to a production environment without prior testing is an unnecessary risk that can corrupt logs or interrupt application write flows. Logrotate offers a simulation execution flag, known as verbose and test mode, which allows you to verify exactly what actions the program will perform without altering any actual files on disk. Running this verification on the command line is a mandatory step in any infrastructure change process.

To perform a practical test of the configuration we created, we run the forced simulation command pointing to the specific configuration file. In practice, this means running the instruction below in the server terminal and carefully analyzing each line of output generated by the system.

sudo logrotate -d -f /etc/logrotate.d/my-application

Analyzing the report generated by this command helps identify syntax errors, incorrect log file paths, or failures in executing service restart scripts. When dealing with complex systems, maintaining visibility into resource behavior is just as important as solving the immediate problem. If you face issues related to service behavior or I/O bottlenecks on the machine, it is worth checking additional strategies in our guide on Docker container stuck? How to inspect logs and volumes to expand your diagnostic toolkit. To explore these architectural concepts further in production, see our related technical analysis on Pod network down? How to debug CNI in Kubernetes.

Final thoughts on automating the log lifecycle

Managing log storage on production servers is not just about freeing up disk space every now and then; it is about building a resilient and predictable operational foundation. The correct adoption of logrotate combined with rigorous testing ensures that audit and debugging history remain accessible without compromising the underlying infrastructure's stability. By carefully planning the frequency, retention, and compression of data generated by applications, the engineering team eliminates unpleasant surprises and ensures the system continues operating with high performance, even under heavy workloads.