Skip to content
share

Solr Monitoring Integration

Sematext offers simple and versatile Solr monitoring agent written in Java and Golang with minimal CPU and memory overhead. It's easy to install and require no changes in Solr source code or your application source code.

Sematext Solr Monitoring Agent

This lightweight, open-source Monitoring Agent collects Solr performance metrics and sends them to Sematext. It comes packaged with a Golang-based agent responsible for Operating System level metrics like network, disk I/O, and more. The Solr Monitoring Agent can be installed with RPM/DEB package manager on any host running Linux or in a containerized environment using sematext/sematext-agent.

The Sematext Solr Monitoring Agent can be run in two different modes - in-process and standalone. The in-process one is run as a Java agent, it is simpler to initially set up, but will require restarting your Solr node when you will want to upgrade your monitoring Agent, i.e. to get new features. The benefit of the standalone agent mode is that it runs as a separate process and doesn't require a Solr restart when it is installed or upgraded.

After creating a Solr App in Sematext you need to install the Monitoring Agent on each host running Solr that you want to monitor. The full installation instructions can be found in the setup instructions displayed in the UI.

For example, on Ubuntu, you need to add Sematext Linux packages and install them with the following command:

echo "deb http://pub-repo.sematext.com/ubuntu sematext main" | sudo tee
/etc/apt/sources.list.d/sematext.list > /dev/null
wget -O - https://pub-repo.sematext.com/ubuntu/sematext.gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install sematext-agent

After that, setup the Solr Monitoring Agent by running a command like this:

sudo bash /opt/spm/bin/setup-sematext  \
    --monitoring-token <your-monitoring-token-goes-here>   \
    --app-type solr  \
    --agent-type javaagent  \
    --infra-token <your-infra-token-goes-here>

The command above will set up your Solr Monitoring Agent in the in-process mode. To have it running in the standalone mode, run the command below instead of the one above:

sudo bash /opt/spm/bin/setup-sematext  \
    --monitoring-token <your-monitoring-token-goes-here>   \
    --app-type solr  \
    --agent-type standalone  \
    --infra-token <your-infra-token-goes-here>  \
    --jmx-params '-Dspm.remote.jmx.url=localhost:3000'

Keep in mind that your need to provide the Monitoring token and Infra token. They are both provided in the installation instructions for your Solr App.

Finally, the last thing that needs to be done is adjusting the solr.in.sh file and add the following section:

SOLR_OPTS="$SOLR_OPTS -Dcom.sun.management.jmxremote
-javaagent:/opt/spm/spm-monitor/lib/spm-monitor-generic.jar=<your-monitoring-token-goes-here>::default"

Or if you would like to run the Solr Monitoring Agent in the standalone mode add the following section to the solr.in.sh file:

SOLR_OPTS="$SOLR_OPTS -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=3000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"

Make sure that tag <jmx /> is enabled in your solrconfig.xml file.

You need to restart your Solr node after the changes above.

Collected Metrics

The Sematext Solr monitoring agent collects the following metrics.

Operating System

  • CPU usage
  • CPU load
  • Memory usage
  • Swap usage
  • Disk space used
  • I/O Reads and Writes
  • Network traffic

Java Virtual Machine

  • Garbage collectors time and count
  • JVM pool size and utilization
  • Threads and daemon threads
  • Files opened by the JVM

Solr

  • Requests rate and latency
  • Solr index stats and file system stats
  • Added and pending documents
  • Deletes by id and queries
  • Filter cache statistics
  • Document cache statistics
  • Query result cache statistics
  • Per segment filter cache statistics
  • Commit events
  • Warmup times

Correlation with Logs

Be sure to check out the Solr Logs integration as well, to get a complete view on Solr: what kind of errors or warnings are happening, how fast is each request, when Solr was restarted and so on.

You can also correlate logs with metrics using Split Screen. For example, if you have a slow query in the logs, you can bring your Solr Monitoring App on the same screen and dig deeper in four steps:

  1. Click the Split Screen button to bring a new App to your screen.
  2. Select your Solr Monitoring App.
  3. Select the report/dashboard you're interested in, such as the CPU and Memory report.
  4. Hover the mouse over the latency spike and you'll see the vertical bar showing the same time on all charts from both sides of your screen.

A full example of Solr metrics and logs correlation can be found in this article.

Troubleshooting

If you are having issues with Sematext Monitoring, i.e. not seeing Solr metrics, see How do I create the diagnostics package.

For more troubleshooting information please look at Troubleshooting section.

Integration

Metrics

Metric Name
Key (Type) (Unit)
Description
cache lookups
solr.cache.lookups
(long counter)
lookups count
cache hits
solr.cache.hits
(long counter)
hits count
cache size
solr.cache.size
(long gauge)
cache size (count of elements)
cache evictions
solr.cache.evicted
(long counter)
count of evictions
warmup time
solr.warmup.time
(long counter) (ms)
warmup time
cache memory used
solr.cache.size.bytes
(long gauge) (bytes)
cache size in bytes
cache max size
solr.cache.size.max
(long gauge)
cache max size
autowarm count or %
solr.cache.autowarm.count
(long gauge)
cache autowarm count or %
request time
solr.requests.time
(long counter) (ms)
request time
req.count
solr.requests
(long counter) (req)
request count
avg. request latency
solr.requests.latency.avg
(double gauge)
avg. request latency
request error count
solr.requests.error.count
(long counter)
request error count
request timeout count
solr.requests.timeout.count
(long counter)
request timeout count
index max doc
solr.index.docs.max
(long gauge) (docs)
max doc in the index
index num docs
solr.index.docs
(long gauge) (docs)
number of docs in the index
index segments
solr.index.segments
(long gauge)
index segments count
index num of files
solr.index.files
(long gauge)
number of files in solr index
index size on the disk
solr.index.files.size
(long gauge) (bytes)
size of solr index on the disk
commits
solr.indexing.commits
(long counter)
total count of commits
optimizes
solr.indexing.optimizes
(long counter)
count of optimizes
rollbacks
solr.indexing.rollbacks
(long counter)
count of rollbacks
expunge deletes
solr.indexing.deletes.expunge
(long counter)
count of expunge deletes
index docs added
solr.indexing.docs.added
(long counter) (docs)
added docs
deletes by id
solr.indexing.deletes.id
(long counter)
deletes by id
deletes by query
solr.indexing.deletes.query
(long counter)
deletes by query
update errors
solr.indexing.errors
(long counter)
count of update errors
autocommits
solr.indexing.commits.auto
(long counter)
count of auto commits
soft autocommits
solr.indexing.commits.soft
(long counter)
count of soft auto commits
index docs pending
solr.indexing.docs.pending
(long gauge) (docs)
count of pending docs
autocommit max time
solr.indexing.commits.auto.time.max
(long gauge) (ms)
autocommit max time

FAQ

How do I enable JMX in Solr?

Add or uncomment the <jmx/> directive in solrconfig.xml and restart Solr. Seehttps://wiki.apache.org/solr/SolrJmxfor more info.

I don't see any data on Solr and JVM reports, what is the problem?

You should probably enable JMX in your Solr.Add or uncomment the<jmx />directive insolrconfig.xmland restart Solr. Seehttps://wiki.apache.org/solr/SolrJmxfor more info.

I don't see any data only in Solr Components or Errors reports, what should I do?

Most likely you are using the standalone variant of App Agent. In that case, App Agent can't collect metrics which are available only when running in-process. If so, switch to in-process (javaagent) version of App Agent.

Why don't I see Solr index file size metric?

App Agent runs as 'spmmon' user. Make sure this user has appropriate access permissions on Solr's index directories. If you cannot give 'spmmon' user the read permissions you can switch to the in-process (javaagent) version of the App Agent.