2017 is almost here and, like last year, we thought we’d share how 2016 went for us. We remain committed to be your “one-stop shop” for all things Elasticsearch and Solr: from Consulting, Production Support, and Training, to complementing that with our Logsene for all your logs, and SPM for all your monitoring needs.
Docker
It’s safe to say 2016 was the year of Docker and by extension Kubernetes, Mesos, Docker Swarm, among others, too. They stopped being just early adopters’ toys and have become production-ready technologies used by many. This year we’ve added excellent support for Docker monitoring with SPM and logging with Logsene via the open-source Sematext Docker Agent.
But, let’s walk through 2016 highlights in the chronological order, month by month.
January
We started the year fresh and published a lot of great content on our blog: from a bunch of useful articles about handling log management (such as: How to forward CloudTrail (or other logs from AWS S3) to Logsene, announcing PagerDuty and Logsene integration or using Filebeat to send Elasticsearch logs to Logsene) to hot topics about monitoring key metrics (such as: introducing NetMaps, a new feature in SPM or presenting the Sematext Docker Agent as a turnkey-solution for Docker Logs, Metrics and Events). If you, like many of our readers, run Elasticsearch and are considering running it inside Docker, have a look at how to run high performance and fault tolerant Elasticsearch Clusters on Docker.
February
We continued our journey through the logging field by describing how to send your Windows Event Logs to Logsene using NxLog and Logstash. In February we introduced Logagent – a log parser and shipper with log patterns for a number of popular log formats – from various Docker images including Nginx, Apache, Linux and Mac system logs, to Elasticsearch, Redis, Solr, MongoDB and more. Logagent detects the log format automatically using the built-in pattern definitions (also lets you provide your own, custom patterns). You may read more here. Also, we thought about our team needs too, as Sematext team is highly distributed and we prepared a insightful blog post showing how to index your Slack messages in Logsene for the purpose of archiving, searching, and analytics.
March
Spring was good to us. We announced the launch of on-demand Java Profiling that lets you profile any app running on top of JVM, even in production. If you are using AWS like us and want to collect all your application logs, you may want to start sending AWS CloudWatch Logs (e.g. VPC logs) to Logsene. As Elasticsearch is our expertise, March ended with two related and popular topics: reindexing data with Elasticsearch and documents update by query with Elasticsearch.
April
Docker, Solr and Elasticsearch were our focus in April. We went to London to give several public Elasticsearch and Solr classes: Core Solr Training, Elasticsearch for Developers; Elasticsearch for Logging and Elasticsearch Operations. While not in London we found time to write about Docker Cloud: Monitoring & Logging, Automatic Geo-IP Enrichment for Docker Logs and Monitoring Kafka on Docker Cloud (just to name a few).
May
We were recognized by Docker as the Ecosystem Technology Partner (ETP) for monitoring and logging. This designation indicates that our products: SPM Performance Monitoring and Logsene have demonstrated working integration with the Docker platform via the Docker API and are available to users and organizations that seek solutions to monitor their Dockerized distributed applications. Just take a look at Sematext Docker Agent, which is extremely easy to deploy with Kubernetes, Docker Swarm, Docker Cloud and Docker Datacenter.
We covered a few more interesting Solr and Elasticsearch topics, such as: Solr streaming expressions for collection auto-updating, DocValues reindexing with Solr streaming expressions, or scalable and flexible Elasticsearch reindexing via rsyslog.
June
The middle of the year found us in New York, holding public Elasticsearch / Elastic Stack Training and Apache Solr Training. While some of our colleagues were in New York, others were giving talks about running Elasticsearch in Docker at Berlin Buzzwords. Indeed, monitoring of Docker environments is challenging enough, so we shared our Docker DevOps experience in Top Docker Metrics to Watch.
July
We created a handy mindmap showing a number of open source Docker Monitoring & Logging tools and we published a reference guide for Monitoring Docker Datacenter Logs & Metrics.
As the ecosystem of DevOps tools and resources – for monitoring, for logging, for alerting, for continuous integration and deployment, configuration management, etc. – is growing, finding answers as quickly and efficiently as possible becomes extremely important. We are DevOps like many of our followers and often need to research specific DevOps tools. To help with that we launched search-devops.com, which aggregates, indexes and makes searchable all content repositories (mailing lists, source code, wikis, issue trackers, etc.) for a number of open source DevOps projects.
August
August was about logging all the way. If you are interested in Elastic Stack Import-Export with Logstash & Logsene or RancherOS Monitoring and Logging Support these are the how-to articles to read.
September
When it comes to centralizing logs to Elasticsearch the first log shipper that comes to mind is Logstash. However, Logstash is far from being the only game in town. In 5 Logstash Alternatives we covered other log shippers you may want to consider. We couldn’t let September pass without covering Docker, of course. Here’s how to set up full Docker Swarm Cluster Monitoring & Logging with 1 Command.
October
We participated in an exciting community-organized Swarm3K project by providing Docker Swarm monitoring tooling.You can read more about that in taming SwarmZilla: 150k Containers in 3K+ Docker Swarm Nodes. We flew to San Francisco where we held public Elasticsearch trainings and Solr trainings.
November
This was a conference month for us. We talked about tuning Solr & pipeline for Logs and about running Solr in Docker at Lucene/Solr Revolution conference in Boston. Then, we presented a deep dive into using Elasticsearch for logs and metrics at O’REILLY Velocity 2016. November brought us two great guest posts: one about Docker Swarm lessons from Swarm3K by Prof. Chanwit Kaewkasi (Docker Captain who organized Swarm3K – the largest Docker Swarm cluster to date) and the other about exploring Windows Kernel with Fibratus and Logsene by Nedim Šabić, developer of Fibratus, a tool for exploration and tracing of the Windows kernel.
December
Still thinking about Logging Libraries vs Log Shippers? You may see some pros and cons here. We were a bit disappointed when we couldn’t find any good, modern DataTable alternative that works well with React and Redux, so we built Sematable – ReactJS & Redux Table, a clean and easy to use React and Redux friendly data table.
Again some Docker knowledge that we shared, such as: how and why running Solr in Docker, logging and monitoring support for Kubernetes containers or making Elasticsearch in Docker Swarm Elastic.
And that was it with 2016 highlights! 🙂
However, we cannot end this post without pointing out a few topics that you liked and continued reading this year, such as: 5-minute Logstash: Parsing and Sending a Log File or Elasticsearch Refresh Interval vs Indexing Performance.
Thanks for reading our blog and see you here, on the blog, in 2017!