Point out key Kubernetes metrics and log elements to help you troubleshoot and tune Docker and Kubernetes
Managing microservices in containers is typically done with Cluster Managers and Orchestration tools such as Google Kubernetes, Apache Mesos, Docker Swarm, Docker Cloud, Amazon ECS, Hashicorp Nomad just to mention a few. However, each platform has slightly different of options to deploy containers or schedule tasks to each cluster node. This is why we started a Series of blog post with Docker Swarm Monitoring, and continue today with a quick tutorial for Container Monitoring and Log Collection on Kubernetes.
This is a guest post by Prof. Chanwit Kaewkasi, Docker Captain who organized Swarm3K – the largest Docker Swarm cluster to date.
Swarm3K was the second collaborative project trying to form a very large Docker cluster with the Swarm mode. It happened on 28th October 2016 with more than 50 individuals and companies joining this project.
Sematext was one of the very first companies that offered to help us by offering their Docker monitoring and logging solution. They became the official monitoring system for Swarm3K. Stefan, Otis and their team provided wonderful support for us from the very beginning.
To my knowledge, Sematext is one and the only Docker monitoring company which allow to deploy the monitoring agents as the global Docker service at the moment. This deployment model provides for a greatly simplified the monitoring process.
Swarm3K Setup and Workload
There were two planned workloads:
MySQL with WordPress cluster
The 25 nodes formed a MySQL cluster. We experiences some mixing of IP addresses from both mynet and ingress networks. This was the same issue found when forming a cluster of Apache Spark in the past (see https://github.com/docker/docker/issues/24637). We prevented this by binding the cluster only to a single overlay network.
A WordPress node was scheduled somewhere on our huge cluster, and we intentionally didn’t control where it should be. When we were trying to connect a WordPress node to the backend MySQL cluster, the connection kept timing out. We concluded that a WordPress / MySQL combo would be set to run correctly if we put them together in the same DC.
We aimed for 3000 nodes, but in the end we successfully formed a working, geographically distributed 4,700-node Docker Swarm cluster.
What we also learned from this issue was that the performance of the overlay network greatly depends on the correct tuning of network configuration on each host.
When the MySQL / WordPress test failed, we changed the plan to try NGINX on Routing Mesh.
The Ingress network is a /16 network which supports up to 64K IP addresses. Suggested by Alex Ellis, we then started 4,000 NGINX containers on the formed cluster. During this test, nodes were still coming in and out. The NGINX service started and the Routing Mesh was formed. It could correctly serve even as some nodes kept failing.
We concluded that the Routing Mesh in 1.12 is rock solid and production ready.
We then stopped the NGINX service and started to test the scheduling of as many containers as possible.
This time we simply used “alpine top” as we did for Swarm2K. However, the scheduling rate was quite slow. We went to 47,000 containers in approximately 30 minutes. Therefore it was going to be ~10.6 hours to fill the cluster with 1M containers. Unfortunately, because that would take too long, we decided to shut down the manager as it made no point to go further.
Scheduling with a huge batch of containers stressed out the cluster. We scheduled the launch of a large number of containers using “docker scale alpine=70000”. This created a large scheduling queue that would not commit until all 70,000 containers were finished scheduling. This is why when we shut down the managers all scheduling tasks disappeared and the cluster became unstable, for the Raft log got corrupted.
One of the most interesting things was that we were able to collect enough CPU profile information to show us what was keeping the cluster busy.
Here we can see that only 0.42% of the CPU was spent on the scheduling algorithm. I think we can say with certainty:
The Docker Swarm scheduling algorithm in version 1.12 is quite fast.
I would like to thanks Sematext again for the best-of-class Docker monitoring system, DigitalOcean for providing all resources for huge Docker Swarm managers, and the Docker Engineering team for making this great software and supporting us during the run.
While this time around we didn’t manage to launch all 150,000 containers we wanted to have, we did manage to create a nearly 5,000-node Docker Swarm cluster distributed over several continents. Lessons we’ve learned from this experiment will help us launch another huge Docker Swarm cluster next year. Thank you all and I’m looking forward to the new run!
Until recently, automating the deployment of Performance Monitoring agents in Docker Swarm clusters was challenging because monitoring agents had to be deployed to each cluster node and the previous Docker releases (<Docker engine v1.12 / Docker Swarm 1.2.4) had no global service scheduler (Github issue #601). Scheduling services with via docker-compose and scheduling constraints required manual updates when the number of nodes changed in the swarm cluster – definitely not convenient for dynamic scaling of clusters! In Docker Swarm Monitoring and Logging we shared some Linux shell acrobatics as workaround for this issue.
The good news: All this has changed with Docker Engine v1.12 and new Swarm Mode. The latest release of Docker v1.12 provides many new features for orchestration and the new Swarm mode made it much easier to deploy Swarm clusters.
If you’ve missed our Core Solr training in October 2015 in New York, here is another chance – we’re running the 2-day Core Solr class again next month – June 13 & 14, 2016.
This course covers Solr 5.x as well as Solr 6.x! You can see the complete course outline under Solr & Elasticsearch Training Overview . The course is verycomprehensive — it includes over 20 chapters and lots of hands-on exercises. Be prepared to learn a lot!
$1,200 early bird rate (valid through June 1) and $1,500 afterwards.
There’s also a 50% discount for the purchase of a 2nd seat!
We added Account Sharing to all our products, thus making it easier to share SPM, Logsene, and Site Search Analytics apps by teams. We made a big contribution to Kafka 0.8.2 by reworking pretty much all Kafka metrics and making them much more useful and consumable by Kafka monitoring agents. We also added support for HAProxy monitoring to SPM.
We announced Node.js / io.js monitoring. This was a release of our first Node.js-based monitoring monitoring agent – spm-agent-nodejs, and our first open-source agent. The development of this agent resulted in creation of spm-agent – an extensible framework for Node.js-based monitoring agents. HBase is one of those systems with tons of metrics and with metrics that change a lot from release to release, so we updated our HBase monitoring support for HBase 0.98.
The SPM REST API was announced in April, and a couple of weeks later thespm-metrics-js npm module for sending custom metrics from Node.js apps to SPM was released on Github.
A number of us from several different countries gathered in Krakow in May. The excuse was to give a talk about Tuning Elasticsearch Indexing Pipeline for Logs at GeeCon and give away our eBook – Log Management & Analytics – A Quick Guide to Logging Basics while sponsoring GeeCon, but in reality it was really more about Żubrówka and Vișinată, it turned out. Sematext grew a little in May with 3 engineers from 3 countries joining us in a span of a couple of weeks. We were only dozen people before that, so this was decent growth for us.
Logsene users got Live Tail in November, while SPM users welcomed the new Top Database Operations report. Live Tail comes in very handy when you want to semi-passively watch out for errors (or other types of logs) without having to constantly search for them. While most SPM users have SPM monitoring agents on their various backend components, Top Database Operations gives them the ability to gain more insight in performance between the front-end/web applications and backend servers like Solr, Elasticsearch, or other databases by putting the monitoring agents on applications that act as clients for those backend services. We worked with O’Reilly and produced a 3-hour Working with Elasticsearch Training Video.
Using Cloud (aka SaaS) applications is natural for most of us — simply sign up with your email, login and then use the service within minutes. The Cloud works particularly well with consumer-oriented services. Businesses, however, have slightly different needs.
Up until now, Sematext Apps offered only App sharing. For instance, if you had an SPM App for monitoring your Elasticsearch cluster, another SPM App for monitoring Spark, and a Logsene App where you shipped all your logs, you could invite one or more of your teammates to each one of those apps separately, and give them either Admin or User role. This works fine when you have just a few teammates or just a few Apps.
Larger organizations (aka Enterprises), however, have more complex needs due to their size and structure: they often have multiple departments and teams, many of whom have different roles and permissions.
With more and more large enterprises using Sematext Apps we recently introduced a new feature called “Account Sharing” to our SPM, Logsene and Site Search Analytics solutions.
Example of Account Sharing with different Roles assigned to team members
As the diagram above shows, Account Sharing lets the Account Owner add more people to his Account and give them different roles – Admin, User, or Billing Admin. The benefit of Account Sharing over App Sharing is that by sharing your Account you can easily share all your Apps, Alerts, etc. with your teammates, without having to resort to all people using the same username/password for the shared account. Admins can invite additional people, Billing Admins have access to payment details, and Users can’t modify other people’s Alerts and other settings, though they can create, modify, and delete their own.
Once you have access to multiple Accounts you can easily switch between them, as shown here:
One can have access to an unlimited number of Accounts. Switching between them is seamless and does not require one to log out and log in. While App sharing is still possible (and we have no plans to remove it), Account Sharing makes teamwork much simpler!
About 10 days ago we ran a a poll about which languages/APIs people use when writing their Apache Kafka Producers and Consumers. See Kafka Poll: Producer & Consumer Client. We collected 130 votes so far. The results were actually somewhat surprising! Let’s share the numbers first!
What do you think? Is that the breakdown you expected? Here is what surprised us:
Python is clearly popular and gaining in popularity, but at 13% it looks like it’s extra popular in Kafka context.
Go at 10-11% seems quite popular for a relatively young language. One might expect Ruby to have more adoption here than Go because Ruby has been around much longer.
We put C/C++ in the poll because these languages are still in use, though we didn’t expect it to get 6% of votes. However, considering C/C++ are still quite heavily used generally speaking, that’s actually a pretty low percentage.
The “Other” category is relatively big, at a bit over 12%. Did we forget some major languages people often use with Kafka? Scala? See info about the Kafka Scala API here.
Everyone and their cousin is using Kafka nowadays, or at least that’s what it looks like from where we at Sematext sit. However, because of the relatively high percentage of people using Python and Go, we’d venture to say Kafka adoption is much stronger among younger, smaller companies, where Python and Go are used more than “enterprise languages”, like Java, C#, and C/C++.
Kafka has become the de-facto standard for handling real-time streams in high-volume, data-intensive applications, and there are certainly a lot of those out there. We thought it would be valuable to conduct a quick poll to find out which which implementation of Kafka Producers and Consumers people use – specifically, which programming languages do you use to produce and consume Kafka messages?
Please tweet this poll and help us spread the word, so we can get a good, statistically significant results. We’ll publish the results here and via @sematext (follow us!) in a week.
NOTE #: If you choose “Other”, please leave a comment with additional info, so we can share this when we publish the results, too!
The Solr Redis Plugin is an extension for Solr that provides a query parser that uses data stored in Redis. It is open-sourced on Github by Sematext. This tool is basically a QParserPlugin that establishes a connection to Redis and takes data stored in SET, ZRANGE and other Redis data structures in order to build a query. Data fetched from Redis is used in RedisQParser and is responsible for building a query. Moreover, this plugin provides a highlighter extension which can be used to highlight parts of aliased Solr Redis queries (this will be described in a future).
Use Case: Social Network
Imagine you have a social network and you want to implement a search solution that can search things like: events, interests, photos, and all your friends’ events, interests, and photos. A naive, Solr-only-based implementation would search over all documents and filter by a “friends” field. This requires denormalization and indexing the full list of friends into each document that belongs to a user. Building a query like this is just searching over documents and adding something like a “friends:1234” clause to the query. It seems simple to implement, but the reality is that this is a terrible solution when you need to update a list of friends because it requires a modification of each document. So when the number of documents (e.g., photos, events, interests, friends and their items) connected with a user grows, the number of potential updates rises dramatically and each modification of connections between users becomes a nightmare. Imagine a person with 10 photos and 100 friends (all of which have their photos, events, interests, etc.). When this person gets the 101th friend, the naive system with flattened data would have to update a lot of documents/rows. As we all know, in a social network connections between people are constantly being created and removed, so such a naive Solr-only system could not really scale.
Social networks also have one very important attribute: the number of connections of a single user is typically not expressed in millions. That number is typically relatively small — tens, hundreds, sometimes thousands. This begs the question: why not carry information about user connections in each query sent to a search engine? That way, instead of sending queries with clause “friends:1234,” we can simply send queries with multiple user IDs connected by an “OR” operator. When a query has all the information needed to search entities that belong to a user’s friends, there is no need to store a list of friends in each user’s document. Storing user connections in each query leads to sending of rather large queries to a search engine; each of them containing multiple terms containing user ID (e.g., id:5 OR id:10 OR id:100 OR …) connected by a disjunction operator. When the number of terms grows the query requests become very big. And that’s a problem, because preparing it and sending it to a search engine over the network becomes awkward and slow.
How Does It Work?
The image below presents how Solr Redis Plugin works.