Sending your Windows Event Logs to Sematext using NxLog and Logstash

There are a lot of sources of logs these days. Some may come from mobile devices, some from your Linux servers used to host data, while other can be related to your Docker containers. They are all supported by Sematext. What’s more, you can also ship logs from your Microsoft Windows based hosts and visualize them. In this blog post we’ll show how to send your Windows Event Logs in a way that will let you build great visualizations and really see what is happening on your Windows-based systems. You may also want to learn about using Fibratus for Windows Event Tracing.

Log Shipping Tools

We will start with the list of tools we will use:

• NxLog – open source log management tool available at no cost as the community edition. Available at nxlog.org. In this blog post we will use NxLog only as a log shipper, because it is light and fast. Each of the Microsoft Windows machines will have NxLog installed and will be sending data to Logstash.
• Logstash – open source log management tool, very flexible with a lot of plugins available. We will use Logstash for parsing the data received from NxLog, giving it a proper log structure and sending to Sematext. Logstash will be running on a separate server, providing a centralized point in your organization where data from log sources will be sent and processed to be ready for sending to Sematext. See 5-minutes Logstash tutorial.
• Sematext Logs – indexes your logs and makes them searchable. Exposes Elasticsearch API, Syslog, etc.

Log Ingestion Pipeline

The overall architecture of your log shipping infrastructure will look as follows:

Setting up Logstash for Windows

Let’s start with configuring and running Logstash. We will use a pretty simple Logstash configuration stored in a file called logstash.conf, which looks as follows:

input {
  tcp {
    port => 3515
    codec => "line"
    type => "WindowsEventLog"
  }
}

filter {
  if [type] == "WindowsEventLog" {
    json {
      source => "message"
    }
    if [SourceModuleName] == "eventlog" {
      mutate {
        replace => [ "message", "%{Message}" ]
      }
      mutate {
        remove_field => [ "Message" ]
      }
    }
  }
}

output {
  elasticsearch {
    hosts => "logsene-receiver.sematext.com:443"
    ssl => "true"
    index => "YOUR_TOKEN"
    manage_template => false
  }
}

Let’s discuss the configuration a bit. We configured the input module to listen on TCP socket on port 3515, we’ve given it a type of WindowsEventLog and we told Logstash that data is line-oriented – we’ve used the line codec for that.

We also need some data manipulation and for that we’ve configured the filter section to use the json filter for the events that are of type WindowsEventLog, so the data we expect from our input. We told the json filter that our data is in the message field. We also use the mutate filter to replace the message field with the contents of the Message field and we remove the Message field.

Finally, we’ve configured the output module, which uses the Elasticsearch HTTP API to send the data to Sematext. One thing to note here is replacing YOUR_TOKEN string with the actual token of your Sematext Log App. You can find your token in the integration instructions for your application here.

Next, we can just start Logstash by running the following command from Logstash home directory:

binlogstash -f logsene.conf

Setting up NxLog

Let’s now send data from our Microsoft Windows hosts to our centralized Logstash. Each of our NxLog instances will be configured as follows (this configuration should work for most modern Windows systems):

define ROOT C:\Program Files (x86)\nxlog
define ROOT_STRING C:\Program Files (x86)\nxlog
define CERTDIR %ROOT%\cert

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
  Module xm_syslog
</Extension>

<Extension json>
  Module xm_json
</Extension>

<Input in>
  Module im_msvistalog
  SavePos TRUE
  Query <QueryList><Query Id="0"><Select Path="Application">*</Select><Select Path="System">*</Select><Select Path="Security">*</Select><Select Path="Setup">*</Select></Query></QueryList>
  Exec $EventReceivedTime = integer($EventReceivedTime) / 1000000; to_json();
</Input>

<Output out>
  Module om_tcp
  Host   192.168.1.15
  Port   3515
</Output>

<Route 1>
  Path in => out
</Route>

The configuration is not very complicated. We start by defining the paths and two extensions. Then we start with the interesting stuff – we define the input section. We tell it that we will use the im_msvistalog module to provide us with Windows Event Log data (read more at the module here). We also provide the query to tell NxLog which events we are interested in. Finally, we modify the event received time and we make it a JSON by running the to_json() function.

As you remember, our Logstash expects data from NxLog, so we need to define the output module accordingly. We use the om_tcp module, and provide the host and port.

Finally, we define a route, which tells NxLog what to do with input and output. We just specify that data from input should be sent to output. After that we can start NxLog and Logstash will start receiving our data.

Windows Event Logs Dashboard

We can now play with data we have in Sematext. We can quickly display it and watch them live using our live tail functionality, for example:

You can create alerts and be notified about certain events (learn more) or you can use Kibana 4 instance and build great dashboards that will help you understand your Windows Event Log data:

Windows Log Shipping Alternatives

As usual, the way of approaching the problem of sending the structured Windows Event Logs from your Microsoft Windows hosts to Sematext can be approached in different ways. We could install Logstash on all Windows hosts and push data directly to Sematext or make NxLog parse the data, structure it and send it without anything in the middle. However, if your Windows hosts don’t have internet access or you only want them to access certain number of hosts, the presented approach should work quite well. See log shipping integrations for more log shipping tools.

Next Steps

If you’re not using Sematext yet you can have a 30-day trial up and running in minutes — just sign up for a free account!  There’s no commitment and no credit card required.  And drop us an email or hit us on Twitter with suggestions, questions or comments about this post.