Elasticsearch for Logging

Upcoming Elasticsearch Classes 2018

DatePriceRegistration
Sept 24-25$800 / person
Only $720 / person before July 20
Register Now
Dec 10-11$800 / person
Only $720 / person before Sept 30
Register Now

Overview

Radu Gheorghe

Comprehensive 2-day sessions (two 4-hour sessions), this Elasticsearch online class is taught by Radu Gheorghe, a seasoned Elasticsearch instructor, and consultant from Sematext, author of “Elasticsearch in Action”, and frequent conference speaker.

After taking this course you will know how to:

  • set up and use Kibana and Timelion
  • build different types of visualizations
  • create dashboards, dig in with sub-aggregations, and use Kibana to search through data.

We’ll cover log shipping with Logstash, various Beats and Logagent. This will cover various inputs, outputs, using Elasticsearch Ingest node, using grok, and so on. Each section is followed by a lab with multiple hands-on exercises. See course outline below for more.

Who Should Attend

The course is designed for technical attendees with basic Elasticsearch experience, as we’ll focus on the tooling around Elasticsearch. A person should be able to index data to Elasticsearch, run queries and aggregations, work with mappings and analysis.

Experience with Linux systems is not a must, but a basic familiarity with running shell commands (e.g., using curl command) will make the course more enjoyable. If you do not have prior Elasticsearch experience, we strongly suggest you consider attending our Core Elasticsearch training first.

For running a logging setup in production, with a non-trivial volume of logs, one needs a good understanding of performance, scaling, monitoring and administering the components involved. While we cover these aspects for ETL tools (Logstash, Logagent, etc) here, the equivalent Elasticsearch part is covered in our Elasticsearch Operations course.

Prerequisites

Intro to Elasticsearch or pre-existing knowledge of Elasticsearch concepts covered in Intro to Elasticsearch

Why Attend

The virtual Elasticsearch training gives you and your team the skills needed to successfully use Elasticsearch capabilities by improving your workflow and increasing efficiency.
Further benefits:

  • a customized learning experience
  • same high-quality instruction as our public or private Elasticsearch classes
  • more affordable than public training
  • more flexible – no need to travel

Things to Remember

For the online training, all participants must use their own computer with OSX, Linux, or Windows, with the latest version of Java installed.  Participants should be comfortable using a terminal / command line.
Sematext provides:

  • a digital copy of the training material
  • a VM with all configs, scripts, exercises, etc.

Course Outline

Modules

  1. Data visualisation through Kibana
    • installation and configuration
    • index patterns; refreshing the fields list
    • discovering and searching raw data
    • Lucene query syntax vs Kuery syntax
    • visualizing data; types of visualizations and what they’re used for
    • Timelion charts; using the Timelion query language
    • building dashboards
    • Lab
      • building complex queries through the Lucene query syntax
      • digging deeper into data through sub-aggregations
      • building dashboards on top of saved searches and visualizations
      • comparing different data series in Timelion (raw average vs moving average)
  2. Data ingestion through Logstash
    • installation
    • inputs: popular input plugins and their configuration options
    • codecs: parsing JSON and multiline logs
    • filters: using grok and geoip to parse and enrich data
    • outputs: popular output plugins and their options
    • pipeline pattern: using Logstash on every logging box
    • using Logstash with Kafka and Redis as a buffer
    • adjusting pipeline workers and batch sizes
    • adjusting Logstash heap size
    • specific plugin tunables
    • Lab
      • configuring Logstash to parse and enrich Apache logs
      • tuning Logstash for throughput
      • using Logstash with Kafka
  3. Data collection using Beats
    • installation: Packetbeat, Topbeat, Filebeat
    • Filebeat tunables
    • parsing JSON logs
    • sending logs directly to Elasticsearch
    • using Ingest nodes
    • sending logs directly to Logstash
    • sending logs to Logstash via Kafka
    • Lab
      • shipping parsing Apache logs via Filebeat and Ingest node
      • shipping and parsing Apache logs via Filebeat and Logstash
  4. Data collection using Logagent
    • installation
    • running on-demand or as a service
    • parsing rules
    • GeoIP matching and database updates
    • using Kafka vs its own buffer
    • UDP syslog and other listeners
    • Lab
      • parsing and sending local Apache and syslog to Elasticsearch
      • build a pipeline with Logagent and Kafka