RabbitMQ is one of the most popular open-source message brokers available. Its ability to be deployed in various configurations and on various platforms makes it a widely used tool; it also supports all major messaging protocols, making it very versatile.
Still, debugging issues with a tool like RabbitMQ can be challenging, especially when it’s deployed on a large cluster. RabbitMQ logs are one way to go, as they help you backtrack to an earlier point while debugging.
In this article, we’ll take you through some basic know-how when it comes to logging on RabbitMQ, including how to configure RabbitMQ logs to store them at a particular location, how to change the format of the logs, and how to rotate RabbitMQ log files.
What Are RabbitMQ Logs?
RabbitMQ logs are debug, error, and critical messages generated by RabbitMQ written to files or the console, or sent to syslog. These log messages provide a lot of information on events taking place in the RabbitMQ cluster. For example, a log message is generated whenever a new message is received in a queue, when that message is replicated in another queue, if a node goes down or comes up, or if any error is encountered.
These log messages act as one of the primary sources of information when you’re trying to gain insights into the health of your RabbitMQ cluster or debug issues. You can even use them to generate dashboards that present information in a way that’s much easier to understand.
Not having these messages can make debugging issues very difficult, as they help us understand what RabbitMQ is doing at any given point in time. So enabling RabbitMQ logging and storing these log messages for a certain period is crucial for troubleshooting and fixing problems fast.
RabbitMQ Logs Location
RabbitMQ is a platform-agnostic tool. Because of this, depending on the operating system on which you install RabbitMQ, the log location changes. But RabbitMQ uses a bunch of environment variables and configuration parameters that remain the same across platforms. You can use this to find the RabbitMQ log file location and change it if needed.
Before looking at the default RabbitMQ log locations, here are the environment variables involved:
RABBITMQ_BASE: A Windows-only environment variable that is the base directory inside which RabbitMQ’s database and log files are stored.RABBITMQ_LOG_BASE: This is the base directory inside which RabbitMQ server logs are stored.RABBITMQ_LOGS: This is the path where the RabbitMQ server’s Erlang log files are stored; this configuration cannot be overwritten on Windows.
There are various other environment variables available for RabbitMQ, but we’re only interested in these, as they’re related to logs and log files.
When you deploy an instance of RabbitMQ on a server or cluster, some of these variables will be set with default values during deployment, depending on the operating system.
*nix Systems
This includes all Unix- and Linux-based operating systems, such as Ubuntu, Debian, RedHat, Fedora, macOS, BSD, etc.
On *nix systems, RabbitMQ uses a variable called ${install_prefix}. When you deploy RabbitMQ on a macOS machine using Homebrew, it uses the Homebrew Cellar location as the default installation path. The ${install_prefix} is set to /usr/local by default. But for Debian and RPM packages, the ${install_prefix} path defaults to an empty value.
With this information, you can review the default values of the environment variables for these operating systems:
RABBITMQ_BASE: Not used, as this is only for Windows.RABBITMQ_LOG_BASE:${install_prefix}/var/log/rabbitmqRABBITMQ_LOGS:$RABBITMQ_LOG_BASE/$RABBITMQ_NODENAME.log
Windows
The Windows operating system has a hidden folder at the path C:\Users\<username>\AppData. This is the default path the variable %AppData% refers to in the configuration below. You can use this to store custom settings and other information needed for the proper functioning of various applications. RabbitMQ also uses this path as the base to store all configuration information, as well as logs.
The list of environment variables for Windows is the following:
RABBITMQ_BASE:%AppData%\RabbitMQRABBITMQ_LOG_BASE:%RABBITMQ_BASE%\logRABBITMQ_LOGS:%RABBITMQ_LOG_BASE%\%RABBITMQ_NODENAME%.log
Generic Binary Builds
This section is applicable when you simply uncompress a tar file without actually installing it from a package. These are again the default environment variable values that ship with the generic Unix tar file. Here, the $RABBITMQ_HOME variable refers to the directory or path where you uncompressed the tar file:
RABBITMQ_BASE: Not used, as this is only for Windows.RABBITMQ_LOG_BASE:$RABBITMQ_HOME/var/log/rabbitmqRABBITMQ_LOGS:$RABBITMQ_LOG_BASE/$RABBITMQ_NODENAME.log
This configuration is similar to the one you would get if you deployed RabbitMQ on a *nix-based operating system.
Docker
Another very popular way of deploying RabbitMQ is on Docker containers. Checking RabbitMQ Docker logs is by far the easiest. This is because by default, all of the RabbitMQ logs are available on standard output, along with the regular Docker logs.
So to check the logs, all you have to do is run the docker logs command, and you’ll see RabbitMQ logs there too:
docker logs rabbitmq_container_id
How to Change the Log Folder
RabbitMQ provides two methods for you to change the location of the log folder:
- Use the configuration file to set a custom path for saving a log file;
- Use the environment variable, which is set by default during installation.
Using the Configuration File
The first method is to use the rabbitmq.conf file. You can find this configuration file at /etc/rabbitmq/ on Debian- and RPM-based systems or at $RABBITMQ_HOME/etc/rabbitmq/ in the case that RabbitMQ is installed by uncompressing a tar file. On a Windows system, the file is located in the %APPDATA%\RabbitMQ\ path, and for macOS, you’ll find it at ${install_prefix}/etc/rabbitmq/. Within this file, you can configure numerous parameters to not only change the RabbitMQ log folder location but also customize logging to a great extent.
For example, you can uncomment the log.dir configuration to set a custom path for saving the log files. By default, this parameter is commented because the environment variable is set automatically while installing or deploying RabbitMQ. The default value of the log file location configuration in the rabbitmq.conf file is as follows:
log.dir = /var/log/rabbitmq
You will have to uncomment the log.dir parameter, set a new value, and remove the environment variable for this configuration to take effect.
Using the Environment Variable
Here, the environment variable takes precedence over the configuration file parameter for the log folder location. The default value for the log folder location is saved in an environment variable named RABBITMQ_LOGS. Changing this environment variable will change the RabbitMQ log folder.
RabbitMQ Log Levels
RabbitMQ offers various log levels, each having a different severity number. This number is important, as it decides, along with other factors, if a message has to be logged or not.
Supported Log Levels
| Log Level | Severity |
|---|---|
| debug | 128 |
| info | 64 |
| warning | 16 |
| error | 8 |
| critical | 4 |
| none | 0 |
The “debug” log level has the highest severity number of 128. And the “none” log level has the lowest severity level of 0. Apart from none, which stops all logging from RabbitMQ, the lower the severity number, the higher the criticality of the message. By default, RabbitMQ is configured to log all “info” level messages. This means that by default, you’ll get all critical messages, errors, and warnings.
You should enable debug logging only when you’re trying to debug an issue because RabbitMQ produces numerous debug messages that are usually not necessary and will lead to high storage usage. So avoid the debug log level for day-to-day usage.
Changing the Log Level
RabbitMQ provides two methods of changing the log level: via the configuration files or running a command using CLI tools.
Changing the log level in the configuration file requires a node restart for the change to take effect. But this change is permanent since it’s made directly in the configuration file. The configuration parameter for changing the log level is as follows:
log.default.level = info
You can change this value to any supported log level and restart the node, and the change will be reflected.
The second method, using CLI tools, is transient. This change will be effective only until a node restarts. After a restart, the log level will be read from the configuration file again. This method comes in handy when you want to temporarily enable the RabbitMQ debug log level to perform some quick debugging:
rabbitmqctl -n rabbit@target-host set_log_level debug
The last option passed to this command is the desired log level. You can pass any of the supported log levels. Changing the log level this way does not require a node restart, and the changes will take effect immediately.
Log Message Categories
Along with log levels, RabbitMQ offers log categories to better fine-tune logging and debugging. Log categories are various categories of messages, which can be visualized as logs coming from various modules in RabbitMQ.
This is the list of log categories available in RabbitMQ:
- Connection: Connection lifecycle event logs for certain messaging protocols
- Channel: Errors and warning logs for certain messaging protocol channels
- Queue: Mostly debug logs from message queues
- Mirroring: Logs related to queue mirroring
- Federation: Federation plugin logs
- Upgrade: Verbose upgrade logs
- Default: Generic log category; no ability to override file location for this category
You can use these to override the default log levels configured at the global level. For example, suppose you’re trying to debug an issue and have configured the global log level as debug. But you want to avoid debug log messages for the upgrade category, or all log messages for the upgrade category.
For this, configure the upgrade log category to not send any log messages:
log.<category>.level = none
Similarly, you can configure all messages from a certain category to log to a separate file, using the following configuration:
log.<category>.file
This way, RabbitMQ makes it easy to segregate log messages based on both the log category and the log level.
How to Rotate RabbitMQ Logs
Log files can get very large very quickly. One easy way to avoid this is to rotate log files periodically. By default, RabbitMQ will not rotate log files and keeps appending logs to the same file. This is done to preserve the log history. But RabbitMQ does offer two ways of rotating log files out of the box.
Based on Date
You can use a configuration parameter to automatically rotate log files based on the date or a given hour of the day:
log.file.rotation.date = $D0
The $D0 option will rotate a log file every 24 hours at midnight. But if you want to instead rotate the log file every day at 11 p.m., use the following configuration:
log.file.rotation.date = $D23
Based on File Size
RabbitMQ also allows you to rotate log files based on their size. So, whenever the log file hits the configured file size, a new file is created:
log.file.rotation.size = 10485760
The configuration above rotates log files every time the file size hits 10 MB.
Number of Archived Log Files to Retain
When log file rotation is enabled, it’s also important to make sure you configure how many log files to retain. Otherwise, all older log files will be retained on disk, consuming unnecessary storage. RabbitMQ has a built-in configuration for this as well, and you can control the number of files to retain using the following configuration:
log.file.rotation.count = 5
In this example, five log files will be retained on the storage disk.
Log Rotation Using Logrotate
Along with the built-in log rotation mechanism, RabbitMQ also supports the logrotate package on *nix systems. Various tools use logrotate to effectively rotate log files. Both Debian and RPM systems with RabbitMQ running can set up logrotate to run weekly on the default log location to clean up the directory.
How to Format RabbitMQ Logs
In most production environments, you would want to use a third-party tool to analyze all of your logs. This becomes easy when the logs are formatted in a way that’s easy to understand by an outside tool. In most cases, this is the JSON format.
How to Log RabbitMQ in JSON
RabbitMQ provides support for formatting logs as JSON messages out of the box. You can use a parameter in the configuration file to change the format of the RabbitMQ log messages to JSON. This takes effect irrespective of the log level you have configured or the category of logs you’re logging.
To change the log format to JSON, you just have to add or uncomment the following configuration in the configuration file:
log.file.formatter = json
This will convert all log messages written to a file to JSON format. Or, change the format of the logs written to the console or syslog to JSON using the following configurations, respectively:
log.console.formatter = json log.syslog.formatter = json
If you want to revert these changes, that is, if you don’t want the log messages to be in JSON format, then simply comment out these configuration parameters.
RabbitMQ Logging Configuration
RabbitMQ offers three target options for logging messages: to a file, the console, or syslog. By default, RabbiMQ directs all log messages to files. You can use the RabbitMQ configuration file to control this.
Logging to File
The following configuration will enable or disable RabbitMQ logging to a file:
log.file = true
There are numerous other configuration parameters for tuning RabbitMQ logs to files. For example, you can control the log level to files, create separate files for each log category, use the built-in log rotation feature, or even integrate with a third-party log rotation package, as discussed above. When you want to stop logging messages this way, all you have to do is change the configuration value to false.
Logging to Console
Logging to the console is similar to logging to a file. You have a configuration in the RabbitMQ configuration file to either enable or disable logging to the console. The configuration is as follows:
log.console = true
As with logging to a file, you can change the value to false to stop the logging. Also, there are numerous options for formatting log messages sent to the console.
Logging to Syslog
Logging to syslog takes a bit more configuration than logging to a file or the console. Once you enable logging to syslog, you also have to provide other related configurations, like the transport mechanism (TCP or UDP), transport protocol (RFC 3164 or RFC 5424), syslog service IP address or hostname, port, TLS configuration if any, and more.
To enable logging to syslog, use the following configuration:
log.syslog = true
By default, RabbitMQ uses the UDP transport mechanism on port 514 and the RFC 3164 protocol. To change the transport and the protocol, use the following two configurations:
log.syslog.transport = tcp log.syslog.protocol = rfc5424
If you want to use TLS, do so using the following configuration:
log.syslog.transport = tls log.syslog.protocol = rfc5424 log.syslog.ssl_options.cacertfile = /path/to/ca_certificate.pem log.syslog.ssl_options.certfile = /path/to/client_certificate.pem log.syslog.ssl_options.keyfile = /path/to/client_key.pem
Next, you need to configure the syslog IP address or hostname and the port using the following configuration:
log.syslog.ip = 10.10.10.10
or
log.syslog.host = my.syslog-server.local
and
log.syslog.port = 1514
Along with these, you can also change the format of the log messages to JSON as we discussed earlier.
RabbitMQ Log Aggregation and Analysis with Sematext
Sematext Logs is a log management solution that can help you leverage your RabbitMQ log messages to gain insights into RabbitMQ’s performance and debug issues. Sematext agents can parse RabbitMQ logs out of the box without any added effort from users. Once the logs are parsed, they can be explored using the predefined dashboards available on Sematext Cloud.
Sematext Logs is part of the Sematext Cloud observability platform with support for a number of technologies and services, including RabbitMQ monitoring. So along with aggregating and analyzing RabbitMQ logs, you can also monitor all key RabbitMQ metrics and set up anomaly detection for threshold-based alerts. That way, you can easily correlate logs with metrics to speed up RabbitMQ troubleshooting.
Want to learn more about Sematext Logs? Then check out the short video just below:
Conclusion
RabbitMQ provides numerous features and options related to logs. These allow you to customize logs and send them to the right system in the way you want, letting you analyze performance and debug issues easily and quickly.
There are two ways to use log data. One is to retroactively look at what happened to debug an issue, while the other is to proactively monitor what the logs are saying about the health of your system. Sematext Logs’ ability to parse RabbitMQ logs out of the box makes it easy to do both, allowing you to troubleshoot issues while gaining insights into the overall performance of RabbitMQ.
Start your free trial today and let us know what you think!
Author Bio
Sunny Srinidhi
Sunny has been working in the data space for over seven years. He writes microservices to work with data at scale and has experience using a variety of databases, including MySQL, MongoDB, DynamoDB, Couchbase, Amazon Athena, and Apache HBase. He is currently working on a Customer 360 product that is intelligently unifying customer data from multiple sources at scale.