Skip to content
share

Journald Logs Integration

You can send Linux journal logs to Sematext in a few ways:

  • Logagent calls journalctl and tails the journal. Journal entries will be forwarded via HTTPS to Sematext
  • journal-upload sends data directly to Sematext. Unfortunately, in most systemd versions you can't use HTTPS as a transport, only HTTP
  • journal-upload sends journal entries to a local Logagent that forwards them to Sematext via HTTPS

Whichever method you choose, once logs are in, you can explore them and drill into specifics, such as SSH or Kernel logs.

Linux Logs Overview

With Node.js installed, you'd first need to install Logagent:

sudo npm i -g @sematext/logagent

Then you can use logagent-setup to configure it to push logs to your Logs App:

sudo logagent-setup -i $YOUR_LOGS_TOKEN -u https://logs-token-receiver.sematext.com -l

Note: if you're in the EU region, use -u https://logs-token-receiver.eu.sematext.com instead.

Option 2: journal-upload to Sematext

First install systemd-journal-remote on your machine.

Red Hat CentOS Debian Ubuntu Amazon Fedora 
sudo yum install systemd-journal-remote
sudo yum install systemd-journal-remote
sudo apt-get install systemd-journal-remote
sudo apt-get install systemd-journal-remote
sudo yum install systemd-journal-remote
sudo dnf install systemd-journal-remote

Edit the /etc/systemd/journal-upload.conf file and change the URL property.

[Upload]
URL=http://logsene-journald-receiver.sematext.com:80/YOUR_LOGS_TOKEN

# For EU Region
# URL=http://logsene-journald-receiver.eu.sematext.com:80/YOUR_LOGS_TOKEN

If you don't feel like editing files manually, you can run a sed command instead that will edit the file.

Sematext US Region Sematext EU Region 
sudo sed -i -E 's/(\#\s)?URL=.*/URL=http:\/\/logs-journald-receiver.sematext.com:80\/YOUR_LOGS_TOKEN/g' /etc/systemd/journal-upload.conf
sudo sed -i -E 's/(\#\s)?URL=.*/URL=http:\/\/logs-journald-receiver.eu.sematext.com:80\/YOUR_LOGS_TOKEN/g' /etc/systemd/journal-upload.conf

Once the configuration is done, you enable and run the systemd-journal-remote.

systemctl enable systemd-journal-upload
systemctl start systemd-journal-upload

It behaves like any other Systemd daemon. You can check the status with:

systemctl status systemd-journal-upload

Note: A bug in systemd-journal-upload service does not allow using HTTPS URL. You could use a local HTTPS reverse proxy to https://logsene-journald-receiver.sematext.com

Option 3: journald-upload to Sematext via Logagent

With Node.js installed, you'd first need to install Logagent:

sudo npm i -g @sematext/logagent

Then use logagent-setup to configure journal-upload to send data to it, and then to Sematext:

sudo logagent-setup -i $YOUR_LOGS_TOKEN -u https://logs-token-receiver.sematext.com -j

Note: if you're in the EU region, use -u https://logs-token-receiver.eu.sematext.com instead.

Exploring logs

Once data is in, you can explore it using the built-in reports or create your own. For example, you can use the Auth report to explore sudo commands:

Linux Logs Auth Report

Other built-in reports include:

  • Kernel: Logs filtered by the facility 0 (kernel). Here you will find all your startup logs, information about crashes, all that you typically see via dmesg
  • SSH: Logs generated by the SSH daemon.
  • Services: Logs from systemd saying starting/started/stopping/stopped. Look here for unexpected service restarts, for example.
  • Startup&Shutdown: Logs from the system-shutdown service, as well as the kernel message telling you the Linux version on startup. Look here for reboots.
  • Audit: Logs from the auditd service, with a syslog tag of audit and kernel messages including selinux or audit
  • Cron: Logs sent to the cron facility (9). For example, you shell see here if logrotate ran properly.
  • YUM/Snap: Logs labeled with either yum or snapd syslog tag. Look here for more info on package management.
  • Mail: Logs sent to the mail facility (2). You can check on your postfix here.
  • DNS: Messages from the systemd-resolved service. Look here if you suspect DNS resolution issues.

Troubleshooting

If you have trouble with journalctl on the first run, it might be that the pipe buffer gets overrun. logagent-setup configures it for 100MB (look for maxBuffer in /etc/sematext/logagent.conf) and tries to pull logs since one week before (look for initialQueryTime). Logagent remembers where it left off in /var/run/logagentLastQueryTimeFile and calls journalctl with a 1 second delay between each run. This means subsequent runs should pull very little data each time.

If the default 100MB buffer is too small for the initial pull, you can do one of the following:

  • make Logagent pull more recent data by changing initialQueryTime
  • increase maxBuffer (value is in bytes). But if you need more than e.g. 200MB you might need to allocate more memory to Logagent itself: the default is 300MB (look for --max-old-space-size=300 in /usr/bin/logagent)
  • truncate the journal to <100MB. For example journalctl --vacuum-size=90M

If journal-upload is involved (options 2 and 3), you might bump into the 10MB HTTP request size limit. This happens if the last stored cursor is larger than 10 MB. You can write a new cursor to /var/lib/systemd/journal-upload/state, or take the journal cursor from the output of journalctl -f | head -c 1 | grep __CURSOR.

Or, if your old logs are less important, you can truncate the journald to less than 10 MB:

journalctl --vacuum-size=9M

With options 2 and 3, you'll need to restart journal-upload if you make changes to the cursor or journal itself. You'll also need to restart it after restarting Logagent (it doesn't restore the connection on its own):

systemctl restart systemd-journal-upload