Elasticsearch for Logging
Upcoming Elasticsearch Classes 2019
|Mar 5, 2019||Kibana and Logstash Fundamentals||Register Now|
Comprehensive 2-day sessions (two 4-hour sessions), this Elasticsearch online class is taught by Radu Gheorghe, a seasoned Elasticsearch instructor, and consultant from Sematext, author of “Elasticsearch in Action”, and frequent conference speaker.
After taking this course you will know how to:
- set up and use Kibana and Timelion
- build different types of visualizations
- create dashboards, dig in with sub-aggregations, and use Kibana to search through data.
We’ll cover log shipping with Logstash, various Beats and Logagent. This will cover various inputs, outputs, using Elasticsearch Ingest node, using grok, and so on. Each section is followed by a lab with multiple hands-on exercises. See course outline below for more.
Who Should Attend
The course is designed for technical attendees with basic Elasticsearch experience, as we’ll focus on the tooling around Elasticsearch. A person should be able to index data to Elasticsearch, run queries and aggregations, work with mappings and analysis.
Experience with Linux systems is not a must, but a basic familiarity with running shell commands (e.g., using curl command) will make the course more enjoyable. If you do not have prior Elasticsearch experience, we strongly suggest you consider attending our Core Elasticsearch training first.
For running a logging setup in production, with a non-trivial volume of logs, one needs a good understanding of performance, scaling, monitoring and administering the components involved. While we cover these aspects for ETL tools (Logstash, Logagent, etc) here, the equivalent Elasticsearch part is covered in our Elasticsearch Operations course.
Intro to Elasticsearch or pre-existing knowledge of Elasticsearch concepts covered in Intro to Elasticsearch
The virtual Elasticsearch training gives you and your team
the skills needed to successfully use Elasticsearch capabilities by improving your workflow and increasing efficiency.
- a customized learning experience
- same high-quality instruction as our public or private Elasticsearch classes
- more affordable than public training
- more flexible – no need to travel
Things to Remember
For the online training, all participants must use their own computer with OSX, Linux, or Windows, with the latest version
of Java installed. Participants should be comfortable using a terminal / command line.
- a digital copy of the training material
- a VM with all configs, scripts, exercises, etc.
Data visualisation through Kibana
- installation and configuration
- index patterns; refreshing the fields list
- discovering and searching raw data
- Lucene query syntax vs Kuery syntax
- visualizing data; types of visualizations and what they’re used for
- Timelion charts; using the Timelion query language
- building dashboards
- building complex queries through the Lucene query syntax
- digging deeper into data through sub-aggregations
- building dashboards on top of saved searches and visualizations
- comparing different data series in Timelion (raw average vs moving average)
Data ingestion through Logstash
- inputs: popular input plugins and their configuration options
- codecs: parsing JSON and multiline logs
- filters: using grok and geoip to parse and enrich data
- outputs: popular output plugins and their options
- pipeline pattern: using Logstash on every logging box
- using Logstash with Kafka and Redis as a buffer
- adjusting pipeline workers and batch sizes
- adjusting Logstash heap size
- specific plugin tunables
- configuring Logstash to parse and enrich Apache logs
- tuning Logstash for throughput
- using Logstash with Kafka
Data collection using Beats
- installation: Packetbeat, Topbeat, Filebeat
- Filebeat tunables
- parsing JSON logs
- sending logs directly to Elasticsearch
- using Ingest nodes
- sending logs directly to Logstash
- sending logs to Logstash via Kafka
- shipping parsing Apache logs via Filebeat and Ingest node
- shipping and parsing Apache logs via Filebeat and Logstash
Data collection using Logagent
- running on-demand or as a service
- parsing rules
- GeoIP matching and database updates
- using Kafka vs its own buffer
- UDP syslog and other listeners
- parsing and sending local Apache and syslog to Elasticsearch
- build a pipeline with Logagent and Kafka