In part 1 of the “encrypted logs” series we discussed sending logs to Elasticsearch over HTTPS. This second part is about TLS syslog.
If you wonder what this has to do with Elasticsearch, the point is that TLS syslog is a standard (RFC-5425): any decent version of rsyslog, syslog-ng or nxlog works with it. So you can forward logs over TLS to a recent, “intermediary” rsyslog. Then, you can either use omelasticsearch with HTTPS to ship your logs to Elasticsearch, or you can install rsyslog on an Elasticsearch node (and index logs over HTTP to localhost).
Such a setup will give you the following benefits:
- it will work with most syslog daemons, because TLS syslog is so widely supported
- the “intermediate” rsyslog can act as a buffer, taking that pressure off your application servers
- the “intermediate” rsyslog can be used for processing, like parsing CEE-formatted JSON over syslog. Again, taking load off your applicaton servers
Our log analytics SaaS, Logsene, gives you all the benefits listed above through the syslog endpoint:
Client Setup
Before you start, you’ll need a public key from the Certificate Authority, which will be used to validate the encryption certificate from the syslog destination (more about the server side later).
If you’re using Logsene, you can download the CA certificates directly. If you’re on a local setup, or you just want to consolidate your logs before shipping them to Logsene, you can use your own certificates or generate self-signed ones. Here’s a guide to generating certificates that will work with TLS syslog.
With the CA certificate(s) in hand, you can start configuring your syslog daemon. For example, the rsyslog configuration can look like this:
module(load="imuxsock") # listens for local logs on /dev/log global ( # global settings defaultNetstreamDriver="gtls" # use TLS driver when it comes to transporting over TCP defaultNetstreamDriverCAFile="/opt/rsyslog/ca_bundle.pem" # CA certificate. Concatenate if you have more ) action( # how to send logs type="omfwd" # Forward them target="logsene-receiver-syslog.sematext.com" # to Logsene's syslog endpoint port="10514" # on port X protocol="tcp" # over TCP template="RSYSLOG_SyslogProtocol23Format" # using the RFC-5424 syslog format StreamDriverMode="1" # via the TLS mode of the driver defined above. StreamDriverAuthMode="x509/name" # Request the machine certificate of the server StreamDriverPermittedPeers="*.sematext.com" # and based on it, just allow Sematext hosts )
This is the new-style configuration format for rsyslog, that works with version 6 or above. For the pre-v6 format (BSD-style), check out the Logsene documentation. You can also find the syslog-ng equivalent there.
Server Setup
If you’re using Logsene, you might as well stop here, because it handles everything from buffering and indexing to parsing JSON-formatted syslog.
If you’re consolidating logs before sending them to Logsene, or you’re running your local setup, here’s an excellent end-to-end guide to setting up TLS with rsyslog. The basic steps for the server are:
- use the same CA certificates as the client, so they have the same basis
- generate the machine public-private key pair. You’ll have to provide both in the rsyslog configuration
- set up the TLS rsyslog configuration
Explore
Once you start logging, the end result should be just like in part 1. You can use Logsene’s hosted Kibana, your own Kibana or the Logsene UI to explore your logs:
As always, feel free to contact us if you need any help:
- Logsene questions and feedback are always welcome
- if you need help with your local setup, we’d be glad to offer you logging consulting and production support
- if, like us, you’re passionate about these things, we’re hiring worldwide