Skip to content
share library_books

SolrCloud Monitoring Integration

Sematext offers simple and versatile SolrCloud monitoring agent written in Java and Golang with minimal CPU and memory overhead. It's easy to install and require no changes in the SolrCloud source code or your application's source code.

Sematext SolrCloud Monitoring Agent

This lightweight, open-source Monitoring Agent collects SolrCloud performance metrics and sends them to Sematext. It comes packaged with a Golang-based agent responsible for Operating System level metrics like network, disk I/O, and more. The SolrCloud Monitoring Agent can be installed with RPM/DEB package manager on any host running Linux or in a containerized environment using sematext/spm-client.

The Sematext SolrCloud Monitoring Agent can be run in two different modes - in-process and standalone. The in-process one is run as a Java agent, it is simpler to initially set up, but will require restarting your Solr node when you will want to upgrade your monitoring Agent, i.e. to get new features. The benefit of the standalone agent mode is that it runs as a separate process and doesn't require a Solr restart when it is installed or upgraded.

After creating a SolrCloud App in Sematext you need to install the Monitoring Agent on each host running your SolrCloud nodes to have the full visibility over the metrics from each host. The full installation instructions can be found in the setup instructions displayed in the UI.

For example, on CentOS, you need to add Sematext Linux packages and install them with the following command:

sudo wget https://pub-repo.sematext.com/centos/sematext.repo -O /etc/yum.repos.d/sematext.repo
sudo yum clean all
sudo yum install spm-client

After that, setup the SolrCloud Monitoring Agent by running a command like this:

sudo bash /opt/spm/bin/setup-sematext  \
    --monitoring-token <your-monitoring-token-goes-here>   \
    --app-type solrcloud  \
    --agent-type javaagent  \
    --infra-token <your-infra-token-goes-here>

The command above will set up your SolrCloud Monitoring Agent in the in-process mode. To have it running in the standalone mode, run the command below instead of the one above:

sudo bash /opt/spm/bin/setup-sematext  \
    --monitoring-token <your-monitoring-token-goes-here>   \
    --app-type solr  \
    --agent-type standalone  \
    --infra-token <your-infra-token-goes-here>  \
    --jmx-params '-Dspm.remote.jmx.url=localhost:3000'

Keep in mind that your need to provide the Monitoring token and Infra token. They are both provided in the installation instructions for your SolrCloud App.

Finally, the last thing that needs to be done is adjusting the solr.in.sh file and add the following section:

SOLR_OPTS="$SOLR_OPTS -Dcom.sun.management.jmxremote -javaagent:/opt/spm/spm-monitor/lib/spm-monitor-generic.jar=<your-monitoring-token-goes-here>::default"

Or if you would like to run the Solr Monitoring Agent in the standalone mode add the following section to the solr.in.sh file:

SOLR_OPTS="$SOLR_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=3000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

Make sure that tag <jmx /> is enabled in your solrconfig.xml file.

You need to restart your SolrCloud node after the changes above.

Collected Metrics

The Sematext SolrCloud monitoring agent collects the following metrics.

Operating System

  • CPU usage
  • CPU load
  • Memory usage
  • Swap usage
  • Disk space used
  • I/O Reads and Writes
  • Network traffic

Java Virtual Machine

  • Garbage collectors time and count
  • JVM pool size and utilization
  • Threads and daemon threads
  • Files opened by the JVM

Solr

  • Requests rate and latency
  • Solr index stats and file system stats
  • Added and pending documents
  • Deletes by id and queries
  • Filter cache statistics
  • Document cache statistics
  • Query result cache statistics
  • Per segment filter cache statistics
  • Commit events
  • Warmup times

Troubleshooting

If you are having issues with Sematext Monitoring, i.e. not seeing SolrCloud metrics, see How do I create the diagnostics package.

For more troubleshooting information please look at Troubleshooting section.

Integration

Metrics

Metric Name
Key (Type) (Unit)
Description
cache lookups
solr.cache.lookups
(long counter)
lookups count
cache hits
solr.cache.hits
(long counter)
hits count
cache size
solr.cache.size
(long gauge)
cache size (count of elements)
cache evictions
solr.cache.evicted
(long counter)
count of evictions
warmup time
solr.warmup.time
(long counter) (ms)
warmup time
cache memory used
solr.cache.size.bytes
(long gauge) (bytes)
cache size in bytes
cache max size
solr.cache.size.max
(long gauge)
cache max size
autowarm count or %
solr.cache.autowarm.count
(long gauge)
cache autowarm count or %
request time
solr.requests.time
(long counter) (ms)
request time
req.count
solr.requests
(long counter) (req)
request count
avg. request latency
solr.requests.latency.avg
(double gauge)
avg. request latency
request error count
solr.requests.error.count
(long counter)
request error count
request timeout count
solr.requests.timeout.count
(long counter)
request timeout count
index max doc
solr.index.docs.max
(long gauge) (docs)
max doc in the index
index num docs
solr.index.docs
(long gauge) (docs)
number of docs in the index
index segments
solr.index.segments
(long gauge)
index segments count
index num of files
solr.index.files
(long gauge)
number of files in solr index
index size on the disk
solr.index.files.size
(long gauge) (bytes)
size of solr index on the disk
commits
solr.indexing.commits
(long counter)
total count of commits
optimizes
solr.indexing.optimizes
(long counter)
count of optimizes
rollbacks
solr.indexing.rollbacks
(long counter)
count of rollbacks
expunge deletes
solr.indexing.deletes.expunge
(long counter)
count of expunge deletes
index docs added
solr.indexing.docs.added
(long counter) (docs)
added docs
deletes by id
solr.indexing.deletes.id
(long counter)
deletes by id
deletes by query
solr.indexing.deletes.query
(long counter)
deletes by query
update errors
solr.indexing.errors
(long counter)
count of update errors
autocommits
solr.indexing.commits.auto
(long counter)
count of auto commits
soft autocommits
solr.indexing.commits.soft
(long counter)
count of soft auto commits
index docs pending
solr.indexing.docs.pending
(long gauge) (docs)
count of pending docs
autocommit max time
solr.indexing.commits.auto.time.max
(long gauge) (ms)
autocommit max time