5-minute Logstash: Parsing and Sending a Log File

SIDE NOTE: We run Elasticsearch and ELK trainings, which may be of interest to you and your teammates.

NOTE: this post has been updated for Logstash 2.x.

We like Logstash a lot at Sematext, because it’s a good (if not the) swiss-army knife for logs. Plus, it’s one of the easiest logging tools to get started with, which is exactly what this post is about. In less than 5 minutes, you’ll learn how to send logs from a file, parse them to extract metrics from those logs and send them to Logsene, our hosted ELK logging service.

NOTE: Because Logsene exposes the Elasticsearch API, the same steps will work if you have a local Elasticsearch cluster.

NOTE: If this sort of stuff excites you, we are hiring world-wide for positions from devops and core product engineering to marketing and sales.


As an example, we’ll take an Apache log, written in its combined logging format. Your Logstash configuration would be made up of three parts:

  • a file input, that will follow the log
  • a grok filter, that would parse its contents to make a structured event
  • an elasticsearch output, that will send your logs to Logsene via HTTP, so you can use Kibana or its native UI to explore those logs. For example, with Kibana you can make a pie-chart of response codes:


The Input

The first part of your configuration file would be about your inputs. Inputs are modules of Logstash responsible for ingesting data. You can use the file input to tail your files. There are a lot of options around this input, and the full documentation can be found here. For now, let’s assume you want to send the existing contents of that file, in addition to the new content. To do that, you’d set the start_position to beginning. Here’s how the whole input configuration will look like:

input {
  file {
    path => "/var/log/apache.log"
    type => "apache-access"  # a type to identify those logs (will need this later)
    start_position => "beginning"

The Filter

Filters are modules that can take your raw data and try to make sense of it. Logstash has lots of such plugins, and one of the most useful is grok. Grok makes it easy for you to parse logs with regular expressions, by assigning labels to commonly used patterns. One such label is called COMBINEDAPACHELOG, which is exactly what we need:

filter {
  if [type] == "apache-access" {   # this is where we use the type from the input section
    grok {
      match => [ "message", "%{COMBINEDAPACHELOG}" ]

If you need to use more complicated grok patterns, we suggest trying the grok debugger.

The Output

To send logs to Logsene (or your own Elasticsearch cluster) via HTTP, you can use the elasticsearch output. You’ll need to specify that you want the HTTP protocol, the host and port of an Elasticsearch server.

For Logsene, those would be logsene-receiver.sematext.com and port 80. Another Logsene-specific requirement is to specify the access token for your Logsene app as the Elasticsearch index. You can find that token in your Sematext account, under Services -> Logsene.

The complete output configuration would be:

output {
  elasticsearch {
    hosts => "logsene-receiver.sematext.com:443" # it used to be "host" and "port" pre-2.0
    ssl => "true"
    index => "your Logsene app token goes here"
    manage_template => false
    #protocol => "http" # removed in 2.0
    #port => "443" # removed in 2.0

Wrapping Up

To start sending your logs, you’d have to download Logstash and put the three configuration snippets above in a file (let’s say, /etc/logstash/conf.d/logstash.conf). Then start Logstash. Once your logs are in, you can start exploring your data by using Kibana or the native Logsene UI.  Remember, Logsene is free to play with and it frees you up from having to manage your own Elasticsearch cluster.

73 thoughts on “5-minute Logstash: Parsing and Sending a Log File

  1. Hello Radu,

    Need one help on logstash… I am able to generate the logs however my NAS mount of 20 GB filling up within no time. So I want to come up with some change in script so that it can take the backup of existing log file after every 2GB in zip format and clear the log. So that I can write a cronjob to mvoe those backup files into S3 bucket for reference…. Any suggestion on this please…


      1. Hello Radu,

        Thanks for your reply… as per your suggestion I made changes as mentioned below… somehow the logstash is not running it is failing every one minute… can you please suggest me where I went wrong

        more logstash.conf
        input {
        beats {
        port => ‘5044’

        source => “message”
        if ([message] =~ “HealthChecker”) {
        drop {}

        output {
        elasticsearch {
        hosts => “search-elkamino-lp-jyv4ux52nns7qo6jwv3hnnup4q.us-east-1.es.amazonaws.com:80”
        codec => “json”
        access_key_id => “xxxxxxxx”
        secret_access_key => “xxxxxxxxxxx”
        region => “us-east-1c”
        bucket => “elk-fb-prod/”
        size_file => 2048
        time_file => 5
        format => “plain”
        canned_acl => “public_read_write”

        1. when I checked logstash logs I see below errors

          [2017-04-13T16:22:19,875][ERROR][logstash.outputs.s3 ] Unknown setting ‘format’ for s3
          [2017-04-13T16:22:19,884][ERROR][logstash.agent ] fetched an invalid config {:config=>”input {\n beats {\n port => ‘5044’\n
          }\n}\n\nfilter{\n json{\n source => \”message\”\n }\n if ([message] =~ \”HealthChecker\”) {\n drop {}\n }\n}\n\
          noutput {\n elasticsearch {\n hosts => \”search-elkamino-lp-jyv4ux52nns7qo6jwv3hnnup4q.us-east-1.es.amazonaws.com:80\”\n codec =>
          \”json\”\n }\ns3{\n access_key_id => \”AKIAJDOLZMAHM3UGIBSA\”\n secret_access_key => \”s45/NZWGlUcV7jU4Bldb20WCzD8hKyWCFRW8gC/a\
          “\n region => \”us-east-1c\”\n bucket => \”elk-fb-prod/\”\n size_file => 2048\n time_file => 5\n format =>
          \”plain\”\n canned_acl => \”public_read_write\”\n }\n}\n\ninput {\n beats {\n port => ‘5044’\n }\n}\n\nfilter{\n json{\n
          source => \”message\”\n }\n if ([message] =~ \”HealthChecker\”) {\n drop {}\n }\n}\n\noutput {\n elasticsearch {\n
          hosts => \”search-elkamino-lp-jyv4ux52nns7qo6jwv3hnnup4q.us-east-1.es.amazonaws.com:80\”\n codec => \”json\”\n }\n stdout { codec
          => \”json\” }\n}\n\n”, :reason=>”Something is wrong with your configuration.”}

          1. Hello,

            It complains about format=>plain in your S3 output. I think it should be “codec” there. I guess you picked that up from the documentation, which has “format” in the example.

          2. Hello Radu,

            Thank you for your reply…

            You are right… After removing the Format context it is start working… however the logs are not copying into S3 bucket so upon research I came to know that we need to install s3 plugin.



  2. Hi Guys,
    How will i parse the following messages in logstash or filebeat?

    “message” => “\n \n\n\n\n\n \n \n \n \n \n \n \n \n \n \n \n\n\n \n \n\n\n \n \n \n \n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n\n\n\n”,

    Thanks in advance

    1. Hello,

      Maybe it was a copy-paste error, but your message has only newlines. You can parse multiline messages with Logstash if you add the multiline flag. For example, (?(.|\r|\n)*) will put the whole (multiline) message in a “msg” field.

  3. Hello Radu, we are using ec2 containers service to run our app. so we needed a centralized location for all the logs, and we went on to install elk stack on a different ec2 instances. and we can see our logs in kabana. But if we want see the raw logs, where are they stored. like ES or logstash or kibana?

    1. Hi Vamsiram,

      Elasticsearch is where your data is stored. Logstash is the process that pushes data to Elasticsearch and Kibana is the tool that helps visualizing data from Elasticsearch. In this example, you can use our logging SaaS, Logsene, where we’d manage Elasticsearch for you and you have integrated Kibana as well: https://sematext.com/logsene/

  4. Hello Radu,
    I am newbie on Logstash.So, here are some questions: I need logstash to process two kind data : logs and metrics ,which come from Kafka. Using elasticsearch to process logs, and metrics for opentsdb.so the configuration should be what like,the log like:
    2016-10-20 10:45:55.037 [metrics-logger-reporter-1-thread-1] INFO com.example.metrics – type=GAUGE, name=com.apache.QueueManager.name.size, value=10,
    the problem boring me server days. You are so helpful.

    1. Hi Reed,

      Thanks for your kind words!

      I’m not sure I get your question – where did you get stuck with your configuration?

  5. Hi
    I have been struggling with logstash and Kibana and looking to find a best place where I can get complete answers to my queries but haven’t succeeded as yet. So my questions are as follows:

    1- If I have offline log files and I want to parse them into logstash do i still need filebeat.
    2- Why do i get an error on kibana saying “unable to fetch mapping. Do you have indices matching the pattern.” while i am trying to create indices for logstash
    3- How do i check if logstash is parsing my logs successfully and where can i troubleshoot errors.

    Last but not least is there a complete tutorial for logstash.


    1. Hi Tahir,

      To get complete answers for your particular use-case, we offer ELK consulting: https://sematext.com/consulting/logging/

      To answer your particular questions:
      1) you can tail files with Filebeat or Logstash, so Filebeat isn’t necessarily needed
      2) you’ll probably need to create an index pattern to tell Kibana which indices it should search. You can get some more details here: https://sematext.com/blog/2015/04/21/how-to-use-kibana-4-log-management/
      3) for that I normally use stdout (https://www.elastic.co/guide/en/logstash/current/plugins-outputs-stdout.html) but for grok patterns in particular I use the online Grok Debugger: http://grokdebug.herokuapp.com/

      I hope this helps, but please feel free to get in touch if you need more elaborate help.

  6. Hey Radu , i am new to logstash actualy elastic search take some time to load data and can not able toload files durng runtime i want to reduce that time please help me..?

      1. Thanx for Reply i want to filter my data but i dont know from where i can collect all imformation about logstash filter…? please give me guideness.

  7. Hi !!
    I’m new in logstash, I have some problem to filter logs from access.log to mongodb data base. Please is somebody who know how to parse log data for mondb data base?
    Thank you !!!

    1. Hi Chris,

      Yes, it sounds like you need Mongodb grok patterns, instead of the Apache one listed here. Depending on your version of Mongo, the ones that come with Logstash might work: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/mongodb

      Alternatively, and especially if you’re short on time, have a look at our Logagent (http://sematext.com/logagent/). It comes with MongoDB patterns already configured and, if Mongo logs under /var/log, Logagent will pick them up and parse them automatically. If not, you only need to point Logagent to your Mongo logs.

  8. Hi Radu,
    Not able to push data from logstash 2.4 to solr 6. Logstash config has
    input {
    file {
    path => “/log_ms1.log”
    start_position => “beginning”
    ignore_older => 0
    sincedb_path => “/dev/null”



    solr_http {
    solr_url => “http://localhost:8983/solr/ore2e”

    ore2e is solr collection.
    Logstash is not even throwing any error.
    Any suggestion?

    1. Hi Vishwas,

      Interesting, I’ve just done a quick test with 2.4 and Solr 6.1 and it worked. If Logstash doesn’t show any error, maybe there’s something on the Solr side that explains it?

      I’d suggest to look at the logs and also to make sure that either you have all the fields defined in the schema (use the stdout output to have Logstash print what it would send to Solr) or use a “schemaless” config (e.g. the schemaless example worked for me in this test, and you can take it from there).

  9. Hi,
    I’m newbie on Logstash. I have setup ELK on one Wndows server, called server A. I have many testing servers need to be monitored log files. I have installed filebeat agent to testing servers and do config to put log files to ELK server A (use Logstash as output in filebeat config) . The problem is I only want check error/warning/exception logs , but filebeat not support much on log filter. Searching on internet indicates that Elasticsearch and logtash both support filter quite good. Which one should I choose and easy to configure? Please advice me.
    My log format:
    08/11/2016 00:00:51 [ERROR] MeasConditions processAlarms: Error: com.lucent.vital.core.vException: DBStmt:executeUpdate failed: Field ‘alarmId’ doesn’t have a default value
    Query is : INSERT INTO VFRTAlarms(SrcId, DatasetTypeId, StatIP, ExceptionId, domainId, state, seq, alarmLevel, alarmIndex, severity)
    select -2002, M.DataSetTypeId, M.StatIP, M.ExceptionId, M.domainId, 0, -1, 0, 100, 0
    from ( select DataSetTypeId, StatIP, ExceptionId, domainId,
    sum( deltaTime ) as secs
    from VFRTMeasEvent
    where alarmLevel = 4
    group by DataSetTypeId, StatIP, ExceptionId, domainId) M
    INNER JOIN RTMeasCondLevel C on (M.ExceptionId = C.ExceptionId and C.alarmLevel = 4 and
    secs >= seconds)
    left join VFRTAlarms A
    on M.DataSetTypeId = A.DataSetTypeId and M.StatIP = A.StatIP and
    M.ExceptionId = A.ExceptionId and M.domainId = A.domainId and A.state 3
    where A.DataSetTypeId is null

    1. Hello,

      I think Logstash is the place to drop data. Ideally, you’d parse those logs with Grok (like in this post) and Severity will land in its own field. Then you could do something like:
      output {
      if [severity] == “ERROR” or [severity] == “WARN” {
      elasticsearch { … }

      If not, you can still do a regular expression on the whole message, but that will be slower and less precise:
      if [message] =~ “ERROR” { …

      Looking forward, in 5.x you’d be able to do filtering directly in Filebeat, which will certainly be faster: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-processors.html

  10. hi

    I installed log-stash on centos 6 but how to get maillog form postfix forwarder machine kindly send me the steps or file to configure or get maillog on logstash server. logstash succesffuly get syslog and messages logs but var/log/maillog not fetch, i installed postfix 2.10 on forwarder machine.

    appreciate if some trying to help me

    1. Hi Junaid,

      How is your Logstash getting data from /var/log/messages? Via the file input as described in this post? If yes, you can simply add /var/log/mail.log as another file under the “input” block. Or you can modify it to read from /var/log/* if you want to tail all files from there.

  11. Hi Radu,
    I’m new to elasticsearch and how can i send logs from my local machine to kibana locally and recieve logs from other machines.

  12. Hi Radu,
    Good Day. I have one doubt from below syntax.
    cat /etc/rsyslog.d/oslog.conf
    $WorkDirectory /var/lib/rsyslog # where to place spool files
    $ActionQueueFileName fwdRule1 # unique name prefix for spool files
    $ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
    $ActionQueueSaveOnShutdown on # save messages to disk on shutdown
    $ActionQueueType LinkedList # run asynchronously
    $ActionResumeRetryCount -1 # infinite retries if host is down
    template(name=”Dynfile” type=”string” string=”/var/log/syslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/%PROGRAMNAME%”)
    # Send messages to central server over TCP using the template.
    action(type=”omfwd” protocol=”tcp” target=”central server” port=”514″ template=”Dynfile”)

    Above syntax, there is no syntax error from rsyslogd -N1 but there is no output in this syntax. I am not receiving logs to central server through template. Is there any missing in this syntax.


    1. The only thing that looks strange here is the mix between new and old syntax. Since you’re using the action() directive, I would put all the queue settings there instead of using $Action.. directives.

      If you have future questions, please post any rsyslog-related ones in a rsyslog-related post (this is about Logstash) or on the rsyslog mailing list.

  13. Hi Radu,
    Good Day. I have already sent message about issue. But i have to provide information extra about this.
    1) First one didn’t well log flow from client server to central server.
    2) I am getting below some error from central server.
    1 2016-06-03T04:02:23.588635-04:00 centralserver rsyslogd – – cannot create ‘/dev/log’: Address already in use
    1 2016-06-03T04:02:23.589201-04:00 centralserver rsyslogd – – imuxsock does not run because we could not aquire any socket
    3) Os version red hat 6.7 and rsyslog version -5.8.10
    Could you provide solution how to fix this issue and Is this error affected log flow?

    1. Hello,

      It sounds like you have multiple rsyslogd (or maybe other syslog daemons?) running. The first one creating /dev/log will get the syslog messages, the others will generate the quoted error.

  14. Hi Radu,
    Good Day.
    I am facing one issue that we have central syslog server and configured many servers on this. syslog port connectivity is good between central log and client server. but log is not flowing to central server. Once service rsyslog restarted, some logs sent to central server. How do i fix this issue and do you have any idea about this.

  15. Hi Radu,
    Good Day
    We have done one script for user command history log. It is working properly and this log is going to syslog server. But Log is again writing local server. If any issue on server , we are not able to trouble shoot because history log seems a lot of /var/log/messages.

    Apr 23 01:17:00 clientmachine history_log#ec*f8*57*51*82*19*b2*ab*60*7d*2e*a1*95*fa*0b*c6##01*09_CDT-0500#2016-04-23#pts1#user_name: 1011 2016-04-23 01:17:00 ls -l
    I am using below syntax but it is not working
    If $syslogtag == “history_log” then {
    action(type=”omfwd” target=”syslog” port=”514″ protocol=”tcp”)
    How do i filter this log from /var/log/messages. Please provide your input

    1. Hello,

      I think logs show in /var/log/messages because there’s a line like

      *.* /var/log/messages

      before your “if” statement. If you move your “if” statement abpve it, your history logs shouldn’t get processed by the /var/log/messages action.

  16. Hi Radu,
    Below syntax is working.
    input(type=”imfile” file=”/var/log/apache/error_log” tag=”error_log:” statefile=”stat-apache-error”)
    input(type=”imfile” file=”/var/log/apache/access_log” tag=”access_log:” statefile=”stat-apache-access”)

    if $syslogtag == “error_log:” then {
    action(name=”error_log” type=”omfwd” target=”applogserver” port=”514″ protocol=”tcp”)
    action (name=”error_log” type=”omfwd” target=”syslogserver” port=”514″ protocol=”tcp”)
    if $syslogtag == “access_log:” then {
    action(name=”access_log” type=”omfwd” target=”applogserver” port=”514″ protocol=”tcp”)
    action (name=”access_log” type=”omfwd” target=”syslogserver” port=”514″ protocol=”tcp”)
    If you have any different , you can share me.
    Thanks so much Raddu

    1. Hello,

      This looks good to me. Though I’m not sure what you mean in your other question – the one with /var/log/messages. You don’t know how to forward those to one of the servers? Or maybe you don’t want to forward local logs in which case the config above should work just fine.

      1. Hi Radu,
        I need one thing that i am looking forward os logs to rsyslog server. My currently syntax is *.* @@rsyslogserver:514. I feel i didn’t get proper log for os logs. I need all os logs from server(For example If any issue , i can take information rsyslog server). I am struck with this syntax.
        Could you provide proper syntax for Os. (That logs is generating local and forward to rsyslog server. Don’t wan repeat logs for this.
        Rsyslog is consider log is log. I am not spilt proper for os logs
        If i get this syntax, really helpful for me. Please help me for this requirement

        1. Hi,

          *.* @@rsyslogserver:514 should forward everything, unless you have a previous action followed by “stop” or the ~ sign. They both mean the same thing, in new and old syntax respectively: stop processing messages that get here.

          Normally, “stop” statements go under a conditional, like:

          if $syslogfacility == ‘mail’ then {
          action(type=”file” file=”/var/log/mail”

          or, in old syntax:

          *.mail /var/log/mail

          If you have something like this, and then you have:

          *.* @@rsyslogserver:514

          then messages with facility set to “mail” won’t be forwarded. To work around this, you can bump this line closer to the beginning of the configuration file, before any “stop” statement. Or, you can rework your config to avoid using “stop” altogether (more work, but should be cleaner).

  17. Hi Radu,
    Really helpful.
    I am using rsyslog version 7.4.7 and below one needful for my requirement
    Finally i need one syntax but i tried , not getting exactly output.
    (1) i need os logs and apache_error logs only from application side will send to syslogserver through input module(new sysntax). don’t need other logs likes apache_access_log.
    (2) i need Os logs and all applications logs(error_log,access_logs,etc). will send to another syslogserver through input module(new syntax)
    Could you please tell me how do write syntax for this.

    1. Hi,

      I would do something like:

      # for OS logs

      # for tailing log files
      # then for every file you need. Wildcards should also work:

      if $syslogtag == ”apache_access_log:” then {
      action(type=”omfwd”… <— send to the server which should get all the logs (the one at point 2)
      stop # don't process these particular logs from this point on
      action(type=”omfwd”… <— send all other logs to the server which should get all the logs (from point 2)
      action(type=”omfwd”… <— also send them to the server at point (1)

      1. Hi Radu,
        Wildcards should work in rsyslogv8. Is it possible also two type syntax (1)input(type=”imfile” file=”/var/log/tomcat6/*.log” tag=”catalina.log:” statefile=”/var/log/stat-catalina-log”)
        (2)input(type=”imfile” file=”*.*” tag=”catalina.log:” statefile=”/var/log/stat-catalina-log”) in rsyslog version 7
        How to discard remote logs in client side( i mean i am getting same logs /var/log/messages. so i want discard forward logs from /var/log/messages.)
        I am helpful your comments. i am facing some issue that why i asked question here.
        I hope you can help me for above comments
        Thanks Radu

        1. Hello,

          A few things that should hopefully help:
          – indeed wildcards are only supported on version 8. To upgrade, you can get packages for various distros here: http://www.rsyslog.com/downloads/download-other/
          – I haven’t tried tailing the same file from two different imfile inputs. I’m not sure it works, though it may, if you specify different state files. Normally, you’d have one input per file, or per category of files (e.g. access logs in one input, error logs in another. Filtering can be done later with conditionals
          – normally, /var/log/messages isn’t a file that rsyslog tails. It’s a file that rsyslog writes to. The input is typically the imuxsock module (that listens to the local syslog socket – typically /dev/log) and the imklog module (that listens for kernel logs). I assume logs end up in /var/log/messages because you have something like this in the config:

          *.* /var/log/messages

          Which translates to “write everything to /var/log/messages) via the file output module: http://www.rsyslog.com/doc/master/configuration/modules/omfile.html

          If you want to only forward logs that you tail via imfile (and not logs picked up via imuxsock or imklog), I see two options:
          1) add conditionals. For example, you can filter by tag, like:
          if $syslogtag == “catalinalog:” then {
          2) use rulesets. This means you’d process tailed logs in a separate queue. To do that, you’d bind each imfile input to the ruleset you’ll use for forwarding (let’s call it “catalina”):
          input(type=”imfile” file=”/var/log/tomcat6/*.log” tag=”catalina.log:” statefile=”/var/log/stat-catalina-log” ruleset=”catalina”)

          Then you’d define the ruleset with all the forward actions within the ruleset:


          Notice how you don’t need any stop action here. Logs tailed from files will be processed separately, they won’t end up in /var/log/messages. You’ll find more info about rulesets in the last part of this post: https://sematext.com/blog/2015/04/13/monitoring-rsyslogs-performance-with-impstats-and-elasticsearch/

          1. Hi Radu,
            Wonderful and got something from your comments. I have tried this syntax what your comments.
            My goal is that i need two files will send to two rsyslog server and only write to syslog server (mentioned path) and not write local(client side) /var/log/messages and also not write server local(/var/log/messages).

            server side
            $template SyslogAuth, “/syslog/%$YEAR%/%$MONTH%/%$DAY%/%HOSTNAME%/%PROGRAMNAME%”
            *.* ?SecurityAuth
            Client side
            input(type=”imfile” file=”/var/log/httpd/access_log” tag=”apache_access_log:” statefile=”/var/log/stat-apache-access” ruleset=”access”)

            action (type=”omfwd” target=”rsyslogserver1″ port=”514″ protocol=”tcp”)
            action (type=”omfwd” target=”rsyslogserver2″ port=”514″ protocol=”tcp”)

            input(type=”imfile” file=”/var/log/httpd/error_log” tag=”apache_access_log:” statefile=”/var/log/stat-apache-access” ruleset=”error”)

            action (type=”omfwd” target=”rsyslogserver1″ port=”514″ protocol=”tcp”)
            action (type=”omfwd” target=”rsyslogserver2″ port=”514″ protocol=”tcp”)
            Is it possible all forwarding logs write to rsyslog server template path?. i need ignore client local(/var/log/messages) and server local(/var/log/messages) directly will write server template path(/syslog/year/month/day/hostname/ path).
            Could you please help for this one. I am searching but struck with rsyslog syntax.
            I hope you can help me for this one

  18. Hi Radu,
    Could you please help me for rsyslog setup. I need log(/var/log/httpd/error_log) forward to two servers
    through input syntax.
    like this ” input(type=”imfile” file=”/var/log/httpd/access_log” tag=”apache_access_log:” statefile=”/var/log/stat-apache-access”)”. it is going one server but i need forward this log to two server through above syntax

    1. Hi,

      This is a Logstash post, so maybe the configs from here will be more helpful:

      Like with Logstash, in rsyslog’s case inputs take care of data ingestion. It’s the actions of output modules (simply outputs, in Logstash land) which take care of sending data to the next set of servers. You can find more info about the flow of data in rsyslog here:

      If you want to forward to two servers in rsyslog, you’d have to define two actions. For example, if you want to send the same messages to two target rsyslog servers via TCP you can do this:
      action(name=”SendToServer1″ type=”omfwd” Target=”server1″ Port=”514″ Protocol=”tcp” )
      action(name=”SendToServer2″ type=”omfwd” Target=”server2″ Port=”514″ Protocol=”tcp” )

      1. Hi Radu,
        I am used this syntax but i am getting syntax. i want error log only will send to target server.
        filename = /var/log/httpd/error_log
        Could you please correct my syntax ?

        1. I think the syntax error is here:

          filename = /var/log/httpd/error_log

          The point is, omfwd is an action (it stands for Output Module for ForWarDing messages over the network). That’s why you have protocol=”tcp” there, to tell the action that you want to use TCP for forwarding.

          For tailing the file, you’d use a separate statement: an input (not an action). Something like:
          input(type=”imfile” file=”/var/log/httpd/error_log” tag=”apache_error_log:”)

          rsyslog will tail that file, and then output to whichever destination you prefer via actions. Like the one you just quoted, except that instead of a “filename”, you can provide an (optional) name:
          name = “error_log”

          The name is only really useful for debugging (i.e. if you start rsyslog with the -dn command line) and for monitoring rsyslog’s performance indicators, like we showed in this post: https://sematext.com/blog/2015/04/13/monitoring-rsyslogs-performance-with-impstats-and-elasticsearch/

    2. Hi Radu,
      Thanks for response.
      I need one more help for syntax. I want only apache_error log forward to central server2 through rsyslog. Don’t need other files in central server. How i do create syntax through input module.

      1. Hi,

        You can use a conditional for that. For example, if your input looks like this:

        input(type=”imfile” file=”/var/log/httpd/error_log” tag=”apache_error_log:”)

        Then you can call the action only for logs with the ”apache_error_log:” tag:

        if $syslogtag == ”apache_error_log:” then {
        name = “error_log”

        1. Hi Radu,
          I have checked your syntax what you posted. It is working fine. I have two configuration
          First one
          # Apache access file:
          $InputFileName /var/log/httpd/access_log
          $InputFileTag apache_access_log
          $InputFileStateFile /var/log/stat-apache-access
          $InputFileSeverity info
          $InputFilePersistStateInterval 20000
          if $programname == ‘apache_access_log’ then @@servertest:514
          & ~
          second one
          input(type=”imfile” file=”/var/log/httpd/access_log” tag=”apache_access_log:”
          (1)Which one configuration fast send to syslog server.
          I need full data from client to syslog server and also There is no data loss through rsyslog setup.
          (2)Could you please how to tune rsyslog setup client and server side.
          If you have any suggestion from rsyslog syntax , you can share here.

          1. Hi,
            (1) the two are somewhat equivalent performance-wise, except that in the new syntax you don’t have that conditional. I would recommend to use the new syntax as it’s easier to maintain (many new features are only supported via the new syntax)
            (2) with a default setup deployed on each apache host, I doubt that you’ll need to tune rsyslog on the client side. It should be light and fast enough as it is. On the server side, it depends on what you need rsyslog to do. I posted a couple of links before, I’m not sure if you had the time to go over them. This one explains how rsyslog works and what I did for tuning:

            This one explains how to parse apache logs and also gives more info on the performance front: https://sematext.com/blog/2015/10/05/recipe-apache-logs-rsyslog-parsing-elasticsearch/

  19. Question is that in /etc/logstash/conf.d there is no files after install to configure as expected. Is this where we set the values described in the above three links? do we make that file or is there a place to edit an existing one to make config changes? confused.

    1. Yes, you’ll have to create a new file, for example /etc/logstash/conf.d/logstash.conf, and then fill it with the snippets shown in the post. Though you might need to change the file name (for the file you want to tail), the Logsene token or the Elasticsearch host&port if you’re using a local Elasticsearch cluster.

    1. Hi Ravi,

      The file input (as described above) supports wildcards, so you can say something like path => “/var/log/*” and when a new file comes up, Logstash will tail it.

      Then, if you want to upload to Logsene (which includes Kibana out of the box), the config is written above. If you want to upload to your own Elasticsearch cluster, just specify the host (and I’d also recommend setting ‘protocol’ to ‘http’ because it’s easier to upgrade). Once your logs are in, install Kibana and it should work out of the box, provided that you point it to the same Elasticsearch cluster.

      1. Hi Radu,

        Thank you very much for your response. I am very new to elk stack and here although I made changes in logstash config file, I don’t see any changes at my kibana dashboard. It might be simple but I am asking you, how to trigger them.

      2. when i run logstash using this command:
        logstash# /usr/java64/latest/bin/jaa -jar logstash-1.3.2-flatjar.jar agent -f logstash-1.3.2.conf &

        I see, port settings: my ip:9300

        my config file:
        file {
        type => “apps1”
        add_field => [ “componentName”, “ScheduleUI” ]
        path => “/tmp/test1.log”
        sincedb_path => “/root”
        filter {
        output {
        elasticsearch {
        host => “myip”
        port => 9300
        cluster => “CLPcluster”

        I dont see any change in my kibana dashboard

        1. Hello Radu,
          I got it resolved. I added start_position in my config file. Its taking all the contents i added to my folder.

          Could you suggest me some material how to generate reports using kibana. I am very new to this.

          Thank you.

  20. Great writeup. The three logstash links in the overview section are incorrect because they are relative (lack the http protocol).

Leave a Reply