Definition: What Is Log Rotation?

Log rotation is an automated process of managing log file size, preventing the logs from filling the storage space and slowing down the system. It involves renaming an existing log file and creating a new log with the same name to store the new information. This is usually done once a day or week.

Why Do You Need Log Rotation?

Logs are extremely valuable when it comes to troubleshooting and security investigations. However, despite its significance, the log size might become problematic, reaching gigabyte or even terabyte levels in the case of a busy e-commerce server, which can significantly impact storage space and performance. Here are some of the reasons why log rotation is essential:

• Storage space: If you don’t compress or delete log files, they will keep growing until the server’s hard drive is full. And, because the application can only write new events when space is available, critical services will fail, and log gaps will appear. Log rotation can be the ultimate solution for this.
• System performance: Logs help administrators monitor the system and keep track of events such as application errors, system failures, security issues, and more. For example, large files (typically measured in gigabytes) take longer to open, read, and write in case of insufficient memory. Hence, even with lots of disk space, your server may run poorly.
• Slow response time: Large log files make it harder to open and read the file, thus taking longer to detect issues, locate security breaches, or identify other system problems. Making it harder to maintain the system’s health and performance.

How Does Log Rotation Work?

Log rotation works by renaming existing log files, then creating a new log with the original file’s name to continue storing the incoming data. Older log files are often stored in one location before being migrated to servers or the cloud.

Log rotation can be automated by creating a configuration file with predetermined parameters that are utilized as a trigger when using a log management tool or application. Temporarily disabling log rotation is possible, but it’s not advised, especially for sensitive data or login entries that may be needed to trace security breaches.

For example, if we’d have a log file named mylogs.log, you will know the files were rotated because they will be renamed using one of the below formats:

• mylogs_yyyymmdd.log, i.e., date format
• mylogs_12_04_03_02_23.log, i.e., with timestamp %H%M%d%m%Y
• mylogs.log.1 or mylogs.log.2 – the higher the number, the older the log file is
• mylogs.log.2.gz where .gz represents the compressed rotated log files

If we use the date format ( %Y%m%d ) for daily log rotations, the updated log file will look like this:

mylogs.log
mylogs_20230220.log
mylogs_20230219.log
mylogs_20230218.log

What Commands Are Used for Log Rotation?

Though the log rotation term is mainly used in Linux systems, it can be used in any OS. Here are some of the popular OS and their commands:

• Linux: Rotation is typically done using the “logrotate” command.
• MacOS: The “newsyslog” command is used.
• Windows: “LogRotateWin” is the Windows version of the “logrotate” command.

What to Do With Old Log Files

Given the limited storage availability, it is essential to take care of old log files created on a regular basis. The following procedures may be implemented depending on their function and necessity:

• Deleting: Deleting older log files is one of the simplest options, but it might be problematic if your organization operates in a sector that requires such information for audits or security breach investigations.
• Archiving: Log data archiving helps boost network security. Log files can be retrieved and analyzed in case of cyber attacks or legal or technical needs. This includes off-site backups and cloud storage services.
• Compressing: Compressed log files can be stored with a fraction of the space required by uncompressed ones, thereby reducing storage costs. In addition, the older log files can be compressed and kept on the server. Compressed log files on Linux-based systems often have the.gz extension.
• Sending to a log management tool: Helps with efficient log analysis, followed by quick and easy execution of any of the above steps. While using the tool you can also choose to either manually execute it or automate the process.

How Often Should Logs Be Rotated?

Depending on how frequently the log file is updated and written, log rotation can occur anywhere from once per day, once per week, or a month. For instance, daily log rotation is recommended for a busy web server to ensure storage space is available and files can be opened easily.

Log Rotation Best Practices

Blindly rotating logs without proper measures is a bad idea. Here are some of the best practices of log rotation to get the max out of it:

• Date or timestamp style renaming of log files. This format is the most popular and easy to read.
• Keep the logs. Depending on the type of data collected and the amount of storage space you have, you might need to keep it anywhere between one week to a full year or more. Only duplicates are removed. Most log data are preserved for a year with monthly backups.
• Periodically move the log file to centralized servers. If the log files remain in the local server for longer duration not only do they occupy storage space but can also be tampered or easily accessed for malicious intent.
• Keep a centralized log file. By comparing the centralized logs to the host logs, it may be possible to find tampered log files and troubleshoot.

Analyze Old Logs with Sematext

Sematext Logs is a powerful log management solution designed to streamline the collection and analysis of logs from various sources across your infrastructure. It features preconfigured dashboards that allow you to easily centralize logs and gain valuable insights from them.

With Sematext Logs, you can analyze old logs and detect anomalies or hidden patterns that may impact system performance. The platform offers auto-discovery capabilities, enabling you to automatically start forwarding and monitoring logs directly from the Sematext user interface. This means you can seamlessly connect your log sources, such as log files or containers, without the need for additional configuration.

Once your logs are ingested, you can leverage the powerful visualization capabilities provided by Sematext to explore and analyze your logs in a user-friendly manner.

The platform includes much-needed alerting functionalities, allowing you to set up threshold-based alerts based on your log data. When the number of logs hits a predefined threshold, you will be promptly notified via the notification channel of your choice. Whether you prefer team chat applications or integration with third-party Incident Management solutions, Sematext has you covered.

Watch the video below to learn more about what Sematext Logs can do for you. Or, better yet, start the 14-day free trial and try it yourself!

Sematext Logs Product Overview | Centralized Logging for all of your Applications

Start Free Trial

See Also