share share share 

Sematext Monitoring FAQ

General¶

What should I do if I can't find the answer to my question in this FAQ?¶

Check the general FAQ for questions that are not strictly about Sematext Monitoring. If you can't find the answer to your question please email support@sematext.com or use our live chat.

Is there a limit to how many servers/nodes/containers I can monitor with Sematext?¶

There are no limits on the number of servers, nodes, or containers. Each Sematext Monitoring plan covers monitoring of a certain number of containers at no extra charge.

What types of applications and infrastructure can Sematext monitor?¶

See integrations.

What are the various configuration files present in sematext-agent package?¶

The configuration files of sematext-agent are:

/opt/spm/properties/st-agent.yml - Contains Sematext Agent related configurations.
/opt/spm/spm-monitor/conf/spm-monitor-config-<YOUR_TOKEN>-<JVM_NAME>.properties - One file for each of the App Agents setup on this host.
/opt/spm/properties/agent.properties - Configurations that are common to Sematext Agent and App Agents.

Can I reduce the amount of logs generated by Sematext Agent?¶

Yes. You can make Sematext Agent reduce the number of messages it logs every minute (otherwise it logs detailed info about its monitoring lifecycle).

For Sematext Agent, you can configure the following properties in /opt/spm/properties/st-agent.yml

logging:
    level: error
    write-events: false

For App Agents, you can find the property files for your monitoring apps with:

ls /opt/spm/spm-monitor/conf/spm-monitor-config-*.properties

You can adjust one or more of them, depending on which application's monitor log output you want to reduce. At the bottom of those files add the following line:

SPM_MONITOR_LOGGING_LEVEL=reduced

After you have made the changes, restart your Sematext Agent.

sudo service sematext-agent restart

If you are running Embedded App Agents, restart your application/java process.

How frequently does Sematext Agent collect metrics and can I adjust that interval?¶

Sematext Agent by default collects metrics every 10 seconds. As example, to reduce this frequency to 30 seconds, for Sematext Agent, set interval property /opt/spm/properties/st-agent.yml. To set it to 30 seconds, set the value to 30s.

For App Agents simply add the following line to your Sematext Agent properties files located in /opt/spm/spm-monitor/conf:

SPM_MONITOR_COLLECT_INTERVAL=30000

The value is expressed in milliseconds and it can be adjusted from 10000 to 60000. If you are adjusting it, we recommend setting it to 30000. With bigger values it is possible some 1-minute intervals would be displayed without the data in the UI.

After you have made the changes, restart your Sematext Agent.

sudo service sematext-agent restart

If you are running Embedded App Agents, restart your application/java process.

How much memory is standalone App Agent using and can it be adjusted?¶

By default, each standalone App Agent process is started with "-Xmx192m" setting. This means that its JVM heap will use 192 MB at most. In many cases App Agent doesn't actually need or use that much memory. If you want to be absolutely sure about it, simply lower this number in /opt/spm/spm-monitor/bin/spm-monitor-starter.sh, with the following variable (around line 63):

JAVA_OPTIONS="$JAVA_OPTIONS -Xmx384m -Xms128m -Xss256k"

You can change "-Xmx" part to "-Xmx192m" for example and see how it goes (check if App Agent is still working over the next few hours/days and is still not using much CPU).

We don't recommend changing this setting if you are monitoring Elasticsearch or Solr clusters with a high number of indices or shards. In those cases App Agent has to parse and gather large amounts of metrics data returned by Elasticsearch or Solr. Contact us in chat or via support@sematext.com if you'd like to get more info and help making this adjustment.

Can I use Sematext to monitor multiple applications running on the same server / VM?¶

Yes. There are really 2 different scenarios here:

If each of those applications should be monitored under a different Monitoring App (e.g., you could have Solr running on your server along with some Java app and you want to monitor both - Solr would be monitored with Monitoring App of Solr or SolrCloud type, while the Java app would be monitored with Monitoring App of JVM type), just complete all installation steps (which are accessible from https://apps.sematext.com/ui/monitoring, click Actions > Install Monitor for app you are installing) for each of them separately.
If you want them monitored under the same Monitoring App (e.g., you have 3 Solr instances running on a server), you must use different JVM name for each of them. To do this, "Agent Installation" step should be run only once on this machine, while "Agent Setup" step should be run once for each of the 3 Solr instances (installation instructions are accessible from https://apps.sematext.com/ui/monitoring, click Actions > Install Monitor for app you are installing). When running script /opt/spm/bin/setup-sematext in step "2. Client configuration setup", you should add jvm-name parameter (and value) at the end of parameter list, like this:

sudo bash /opt/spm/bin/setup-sematext --monitoring-token 11111111-1111-1111-1111-111111111111 --app-type solr --agent-type javaagent --jvm-name solr1

In this example, we are setting up things for 3 separate Solr processes, monitored under JVM names: solr1, solr2 and solr3 (you choose any names), so this is adjusted command for solr1 instance.

In the remaining sub-steps of "Agent Setup", replace word "default" with the jvm-name value you just used.

"Agent Setup" step will have to be repeated several times, once for each monitored application (in our example 3 times with 3 different jvm-name values).

Note: By using this kind of setup, you will be able to see JVM stats of all 3 processes separately (JVM name filter is used to do the filtering). When it comes to other, non-JVM stats, they will be aggregated into single value (for instance, request rate chart will show sum of request rate value of these 3 Solr instances). If you want to see even non-JVM stats separately, you will have to create 3 separate SPM Solr applications (one for each Solr instance running on this machine).

Can I use Sematext to monitor my service which runs on Windows or Mac?¶

Sematext Agent installers currently exist only for Linux, however there is still a way to monitor your service. If you are OK with installing Sematext Agent on separate linux box, and pointing it (more about that further below) to your service (which should be monitored) running on Windows or Mac, you can use Sematext to monitor all non-OS related metrics. You can stop and disable OS metric collector using:

sudo service spm-monitor-sta stop
sudo systemd disable spm-monitor-sta # systemd supported OS
echo manual | sudo tee /etc/init/spm-monitor-sta.override  # init.d supported OS

Metrics specific to your process will be displayed (for example, in case of Solr, you will see all search, index, cache metrics along with all JVM metrics like pool memory, GC stats, JVM threads etc; as another example, in case of MySQL you will see all metrics related to connections, users, queries, handlers, commands, MyISAM, InnoDB, MySQL traffic, etc) .

When monitoring Solr, Elasticsearch, HBase, Hadoop and other JVM-based services, you will have an option to choose between using Embedded App Agent or Standalone App Agent. The workaround described here requires the use of standalone App Agent variant. Here's what you'd need to do to see your metrics in Sematext Cloud:

Install sematext-agent on any Linux box (you can use this box for anything else, it is needed here just to run a process which collects metrics and sends them to Sematext). Installation instructions are accessible from https://apps.sematext.com/ui/monitoring, just click Actions > Install Monitor for app are installing. In step 1 choose the "Other" tab to create a minimal installation. This will not install modules needed for monitoring OS metrics - if those modules were installed, they would collect OS metrics of your Linux box, which is probably something you don't need. If you decide to follow instructions from some other tab, keep in mind OS metrics displayed in Sematext UI will not be OS metrics of your Windows/Mac machine.
In step 2, if you are given a choice between Embedded and Standalone App Agent, choose Standalone monitor. It will use remote JMX to collect metrics from your Windows/Mac machine. Just follow instructions given on Standalone tab. The only difference you will want to make is in -Dspm.remote.jmx.url parameter used in monitor properties. You will know where to adjust it if you follow standard instructions for Standalone monitor. While this parameter will usually have a value like -Dspm.remote.jmx.url=localhost:3000, in this case you will have to replace localhost with the address of your Windows/Mac machine(s) by which it can be reached from your helper Linux box.
If you are monitoring something that doesn't offer a choice between Embedded and Standalone monitor, installation instructions will explain where you can define your own address. Instructions typically use localhost, but you can instead use something like 10.1.2.3 or my-win-solr-server.mycompany.com to point to the machine that hosts the service you intend to monitor.

In case you want to monitor multiple machines belonging to the same cluster in this way, you can still use Sematext Agent installed on a single Linux helper box. Just do this:

Create a separate Monitoring App (https://apps.sematext.com/ui/integrations) for each of the machines which should be monitored
For each of those Monitoring Apps (and your monitored machines), go through installation process presented after the app was created (installation instructions are accessible from https://apps.sematext.com/ui/monitoring, just click Actions > Install Monitor for app your are installing). Note: Since each Monitoring App uses its own token, they all have slightly different installation commands. Besides different token, you will also use different addresses of Windows/Mac machines that host the monitored service.

Can Sematext monitor metrics via http-remoting-jmx protocol (e.g. WildFly, JBoss, etc.)?¶

Yes. The following steps are needed:

In /opt/spm/spm-monitor/conf/spm-monitor-config-YOUR_TOKEN-default.properties add the following property.

SPM_MONITOR_JAR="/opt/spm/spm-monitor/lib/spm-monitor-generic.jar:/path/to/your/wildfly-client-all.jar"

Change the value of SPM_MONITOR_JMX_PARAMS in /opt/spm/spm-monitor/conf/spm-monitor-config-YOUR_TOKEN-default.properties. Of course, you can append to that additional JMX parameters, for example with password file location etc:

SPM_MONITOR_JMX_PARAMS="-Dspm.remote.jmx.url=service:jmx:remote+http://localhost:9990"

Restart spm-monitor with: sudo service sematext-agent restart

At this point the metrics will start appearing in charts. If they don't, run the diagnostics script to get fresh output of errors: sudo bash /opt/spm/bin/spm-client-diagnostics.sh

If you see errors similar to:

Caused by: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:

java.io.FileNotFoundException: /opt/wildfly/standalone/tmp/auth/local8363680782928117445.challenge (Permission denied)

javax.security.sasl.SaslException: DIGEST-MD5: Cannot perform callback to acquire realm, authentication ID or password [Caused by javax.security.auth.callback.UnsupportedCallbackException

it means there is permissions issue on WildFly dirs. There are two ways to get around this:

Run Sematext Agent with the same user that is running WildFly. To do that, add an entry like this to the end of /opt/spm/spm-monitor/conf/spm-monitor-config-YOUR_TOKEN-default.properties:

SPM_MONITOR_USER="wildfly"

After that restart Sematext Agent with: sudo service sematext-agent restart

Change permissions for the problematic directory, adjusting the path to match your environment:

chmod 777 /opt/wildfly/standalone/tmp/auth

This approach is not encouraged because of the obvious security problem, so use such approach only while testing, or if other options are not possible. As usual, restart spm-monitor after this change.

When should I run Standalone and when Embedded App Agent?¶

Standalone App Agent runs as a separate process, while the Embedded monitor runs embedded in the Java/JVM process. Thus, if you are monitoring a non-Java application, Standalone monitor is the only option. Standalone monitor is a bit more complex to set up when one uses it to monitor Java applications because it typically requires one to enable out-of-process JMX access, as described on Standalone App Agent page. With Embedded monitor this is not needed, but one needs to add the Sematext Agent to the Java command-line and restart the process of the monitored application. When running Standalone monitor one can update the App Agent without restarting the Java process being monitored, while a restart is needed when the Embedded App Agent is being used.

Can I move Sematext Agent to a different directory?¶

Yes. Soft move script moves all SPM files/directories to a new location, but symlinks /opt/spm to the new location. Use this script if you are OK with having /opt/spm symlinked. This script is recommended for most situations since it keeps your Sematext Agent installation completely in line with standard setup (all standard Sematext Agent commands and arguments are still valid).

It accepts 1 parameter: new directory where Sematext Agent should be moved to (if such directory doesn't exist, it will be created)

sudo bash /opt/spm/bin/move-spm-home-dir-soft.sh /mnt/some_dir

And that is it.

Is there an HTTP API?¶

Yes, see API Reference.

I have multiple Monitoring Apps installed on my machine, can I uninstall just one of them?¶

Yes, you can use the following command for that (it accepts only one parameter, the token of the Monitoring App you want to uninstall):

sudo bash /opt/spm/bin/remove-sematext 11111111-1111-1111-1111-111111111111

Can I disable Sematext Agent without uninstalling it?¶

Yes, just find its .properties file in /opt/spm/spm-monitor/conf and add to it:

SPM_MONITOR_ENABLED=false

After that restart the monitor to apply the change. In case of standalone agent, run:

sudo service sematext-agent restart

And in case of the Embedded App Agent, just restart the service that has this App Agent's javaagent parameter.

JMX attaching is not working out of the box¶

If your JVM applications are guarded by the Security Manager you are likely to see the following exception when Sematext Agent tries to bootstrap the JMX connector:

javax.naming.NoInitialContextException: Need to specify class name in environment or system property, or in an application resource file: java.naming.factory.initial

See enabling Security Manager permissions to allow automatic JMX connector exposure.

Can I reconfigure the agent to send metrics to a different Monitoring App?¶

Yes, this is how:

If you are using automonitoring via autodiscovery just change the App in the UI

Otherwise:

In /opt/spm/spm-monitor/conf directory look for a .properties file whose name contains the token of the Monitoring App where the metrics are being sent
Rename this file, replacing the App token part with the token of the Monitoring App where you want the agent to send metrics
If in-process agent is used also change the token in the javaagent definition in the startup script of the monitored service
Run sudo service sematext-agent restart

Can I switch a host to another Infra App? Can I reconfigure the Sematext Agent to send metrics to a different Infra App?¶

Yes, switching a host to another Infra App is possible by reconfiguring the Sematext Agent running on that host. This is needed to update the Infra App token that is configured in the Agent. Here is how:

Go to Fleet & Discovery > Agents and click "+Install Agent" in the upper right corner of the page
Select your environment and the new Infra App that you wish to switch to
Follow the installation instructions. If you are using a baremetal/VM environment, you can skip steps 1-3 from the Agent Installation section and follow only the instructions in the Agent Setup section.

See sharing FAQ.

What is the difference between OWNER, ADMIN, BILLING_ADMIN, and USER roles?¶

See info about user roles in sharing FAQ.

Agent Automation¶

Is there an Ansible Playbook for the Sematext Agent?¶

Yes, see the Ansible instructions for installing Sematext Agent in multiple hosts.

Agent Updating¶

How do I upgrade the Sematext Agent?¶

For detailed instructions on how to update or upgrade the Sematext Agent in one of the supported environments, see the Agent Installation documentation.

Agent Uninstalling¶

How do I uninstall the Sematext Agent?¶

For detailed instructions on how to uninstall the Sematext Agent in one of the supported environments, see the Agent Installation documentation.

Agent Overhead¶

What's the memory usage overhead for Sematext Agent running in a container?¶

The general memory overhead for Sematext Agent is influenced by the number of monitored containers. It also depends on which features are enabled for collecting data. For a vast majority of workloads, the memory consumption is between 50MB and 90MB.

What's `--restart always`?¶

The --restart always policy instructs the Docker daemon to restart the container automatically in case of failures.

If you're still concerned about memory usage, you can always enforce the limits with cgroup controllers. Example:

docker run -d  --restart always --memory 168m --memory-swap=236m --privileged -P --name st-agent \
  -v /:/hostfs:ro \
  -v /sys/:/hostfs/sys:ro \
  -v /var/run/:/var/run/ \
  -v /sys/kernel/debug:/sys/kernel/debug \
  -v /etc/passwd:/etc/passwd:ro \
  -v /etc/group:/etc/group:ro \
  -e INFRA_TOKEN=<your-infra-token> \
  -e REGION=US \
  sematext/agent:latest

What's the CPU usage overhead for Sematext Agent running in a container?¶

Generally under 1%.

Agent installation¶

How do I install the Sematext Agent?¶

For detailed instructions on how to install the Sematext Agent in one of the supported environments, see the Agent Installation documentation.

Alerts¶

Troubleshooting¶

Can I enable debugging in the Sematext Agent?¶

Yes. For Sematext Agent, edit logging.level property in /opt/spm/properties/st-agent.yml file. Available levels are: panic, fatal, error, warn, info, debug

For App Agents, simply add or edit the SPM_MONITOR_LOGGING_LEVEL property in any of the /opt/spm/spm-monitor/conf/spm-monitor/*.properties files and restart the agent (or the process the agent is attached to). Available levels are: FATAL, ERROR, WARN, INFO, DEBUG, TRACE

Can I install Sematext Agent on servers that are behind a proxy?¶

Yes. If you are installing the RPM, add this to /etc/yum.conf:

proxy=http or https://proxy-host-here:port
proxy_username=optional_proxy_username
proxy_password=optional_proxy_password

If you are using apt-get, set the http_proxy environmental variable:

export http_proxy=http://username:password@yourproxyaddress:proxyport

Can Sematext Agent send data out from servers that are behind a proxy?¶

Yes. You can update the proxy settings using the following command:

sudo bash /opt/spm/bin/setup-env --proxy-host "HOST" --proxy-port "PORT" --proxy-user "USER" --proxy-password "PASSWORD"

After this, run the following command to apply the settings:

sudo service sematext-agent restart

Can I change the region settings for the Sematext Agent installation?¶

Yes. By default the region is set to US. You can change it to EU using:

sudo bash /opt/spm/bin/setup-env --region eu

When installing Sematext Agent, I see "The certificate of pub-repo.sematext.com is not trusted" or a similar error. How can I avoid it?¶

There can be multiple reasons for this, most likely:

system time on your machine is not correct and adjusting it should fix the problem
you are installing Sematext Agent as root user - in some cases that can cause this error and installing Sematext Agent as a different user may help
you are missing the ca-certificates package. Use one of the following commands to install it:

sudo apt-get install ca-certificates
sudo yum install ca-certificates

If none of the above eliminates the problem for you try adding a flag to avoid certificate checking. In case the command with wget is failing in your case, add --no-check-certificate as wget argument. If command with curl is failing, add -k flag.

When installing Sematext Agent, I see "gnupg do not seem to be installed" or a similar error. How can I avoid it?¶

Debian based sematext-agent installation requires adding gpg key to your system and to be able to add a gpg key, you need to have gnupg package installed.

sudo apt install gnupg

After that, you should be able to add the key to your system.

wget -O - https://pub-repo.sematext.com/debian/sematext.gpg.key | sudo apt-key add -

How to prevent "Pending" status when installing Sematext Agent on a Kubernetes Cluster?¶

If you notice that the pods for Sematext Agent are stuck in a "Pending" status with an error similar to "Port is already in use" during the installation on your Kubernetes Cluster, it might be related to hostNetwork setting.

Sematext Agent requires hostNetwork access in order to access and monitor the Kubernetes control plane components. However, if you enable hostNetwork and the default port (8675) is already being used on your host, it can result in the Sematext Agent pods remaining in a "Pending" state. To resolve this problem, you need to modify the port being used.

If you have concerns about hostNetwork usage or you don't want to use it in your system, please refer to this page.

If you want to proceed with hostNetwork (which is required for fully featured Kubernetes monitoring), please follow the steps below, depending on the installation method you are using:

kubectl Installation

Update the API_SERVER_PORT environment variable with the new <Free-Port> value in your YAML file. The new <Free-Port> value should be a port that is not in use. Additionally, make sure to update the port values for livenessProbe, startupProbe, and ports accordingly, using the same <Free-Port> value.

Helm Installation

Include the following parameter in your regular helm installation command: --set agent.service.port=<Free-Port>. Replace <Free-Port> with a port that is not in use.

By following these instructions, you should be able to avoid the "Pending" status and successfully install Sematext Agent on your Kubernetes Cluster.

How do I create the diagnostics package?¶

Preparing diagnostics data differs depending on the setup you have.

Classic installation (rpm/deb packages)

If you are having issues with Sematext Monitoring, you can create diagnostics package on affected machines where Sematext Agent was installed by running:

sudo bash /opt/spm/bin/spm-client-diagnostics.sh

The resulting package will contain all relevant info needed for our investigation. You can send it, along with short description of your problem, to support@sematext.com or contact us in chat.

Container setups

Two types of agents are running in container setups and spawn from the following images:

sematext/agent (also known as STA) - collects infrastructure data (OS and container info, metrics and events)
sematext/app-agent (known as application agent or AA) - collects application metrics (e.g. metrics of your Elasticsearch or Kafka). For each monitored service instance one such application agent container will exist. These containers are automatically created and managed by STA.

sematext/agent

Docker:

sudo docker logs st-agent > /tmp/st-agent.log 2>&1 && tar -cvzf /tmp/st-agent.tgz /tmp/st-agent.log

Swarm:

sudo docker service logs st-agent > /tmp/st-agent.log 2>&1 && tar -cvzf /tmp/st-agent.tgz /tmp/st-agent.log

Kubernetes:

This will produce multiple tgz files in /tmp dir, one for each running sematext/agent instance:

for pod in $(kubectl get --no-headers -o=custom-columns=NAME:.metadata.name pods --selector=app=sematext-agent);do kubectl logs $pod -c agent | gzip -9 > /tmp/st-agent-$pod.gz;done

sematext/app-agent

App Agent writes multiple logs to container file system. You can list all App Agent pods with:

Docker/Swarm: sudo docker ps -f ancestor=sematext/app-agent
Kubernetes: kubectl get pods --selector=service=st-aa

You will notice that each App Agent container/pod name matches the name of the monitored service (e.g. service my_service-7db849967c-kdmgz will have associated App Agent container/pod my_service-7db849967c-kdmgz-aa-mqdt9). When you find the problematic App Agent, you can copy (using e.g. docker cp or kubectl cp command):

its config file from /opt/spm/spm-monitor/conf dir
its spm-monitor and spm-monitor-stats logs from /opt/spm/spm-monitor/logs/applications/YOUR_TOKEN/MONITORED_SERVICE_POD_NAME dir

To run the diagnostics script you'll have to execute the following command depending on your deployment type.

Docker/Swarm:

docker exec -it <target-aa-container> /opt/spm/bin/spm-container-diag

Kubernetes:

kubectl exec -it <target-aa-pod> -- /opt/spm/bin/spm-container-diag

Once the script execution has completed, you'll get the docker/kubectl command that copies the diagnostics package from the container file system to your machine.

You can send the diagnostics package, along with a short description of your problem, to support@sematext.com or contact us in chat.

I see only my system metrics (e.g. CPU, Memory, Network, Disk...), but where is the rest of my data?¶

Make sure you have followed all steps listed on installation instructions page. Agent Installation steps should be done first, followed by Agent Setup. If you have done that and you still don't see application metrics, see How do I create the diagnostics package.

I do not see any system metrics (e.g. CPU, Memory, Network, Disk), what could be the problem?¶

Make sure you have followed all steps listed on installation instructions page. It is possible you missed Agent Setup step. If you have done that and you still don't see application metrics, see How do I create the diagnostics package.

I am using trying to monitor Solr / Elasticsearch. My Request Rate and Latency charts are empty?¶

If other Solr/ES charts are showing the data, it is most likely that there were no requests sent to your Solr/ES in the time range you are looking at. Try sending some queries and see if request rate/latency charts will show them. If they don't, please send us an email to support@sematext.com or contact us in chat.

I am trying to monitor Elasticsearch. My HTTP Metrics charts are empty?¶

If other ES charts are showing data, it is most likely that HTTP metrics aren't enabled in elasticsearch.yaml. Setting http.enabled: true will fix this issue. If this doesn't fix it, please send us an email to support@sematext.com or contact us in chat.

I am trying to monitor Elasticsearch. Node stats are not tracked over time?¶

Make sure to set the node.name value in in elasticsearch.yaml. Elasticsearch will otherwise generate a random node name each time an instance starts, making tracking node stats over time impossible. If this doesn't fix your issue, please send us an email to support@sematext.com or contact us in chat.

I am trying to monitor MySQL. I see an error in Agent logs - java.sql.SQLException: The server timezone value 'XXX' is unrecognized or represents more than one timezone.¶

Try adding serverTimezone=UTC to ST_MONITOR_MYSQL_DB_ADDITIONAL_PARAMS attribute in your Sematext MySQL Agent config. When using setup-sematext script to create agent config, you can adjust that parameter into:

--ST_MONITOR_MYSQL_DB_ADDITIONAL_PARAMS 'autoReconnect=true&serverTimezone=UTC'

I am not seeing any data in Monitoring charts. How do I check if network connectivity is OK?¶

Sematext Agent sends data to Sematext via HTTP(S) so it is important that servers where you install Sematext Agent can access the internet. Things to check to ensure network connectivity is ok:

1. Try connecting to spm-receiver.sematext.com / spm-receiver.eu.sematext.com (if using Sematext Cloud Europe) with the following command:

nc -zv -w 20 spm-receiver.sematext.com 443

The output should show something like: Connection to spm-receiver.sematext.com 443 port [tcp/https] succeeded!

In case you see some other result:

if your server requires proxy to access the internet, you can define its settings using /opt/spm/bin/setup-env command. After that restart Sematext Agent.
if firewall is used to protect your server, it may be blocking outbound traffic from it. Sematext Agent sends the data over port 443, so please ensure with your network admins that port 443 is open for outbound traffic
check your DNS (see below)

2. Check if your DNS has correct entries for Sematext Monitoring Receiver:

nslookup spm-receiver.sematext.com
nslookup spm-receiver.eu.sematext.com

The output of this command should look like this, although the IP addresses and names may be somewhat different, as they change periodically:

Server:        127.0.1.1
Address:    127.0.1.1#53

Non-authoritative answer:
spm-receiver.sematext.com    canonical name = SPM-Prod-Receiver-LB-402293491.us-east-1.elb.amazonaws.com.
Name:    SPM-Prod-Receiver-LB-402293491.us-east-1.elb.amazonaws.com
Address: 50.16.206.179
Name:    SPM-Prod-Receiver-LB-402293491.us-east-1.elb.amazonaws.com
Address: 107.20.222.136

If you see different output, you may want to check with your network admins if everything with your DNS is ok or may need adjustment to be able to reach spm-receiver.sematext.com / spm-receiver.eu.sematext.com.

I don't see any of my metrics / I am getting errors when starting Sematext Agent or when starting my service which uses Embedded version of App Agent. What should I check?¶

Here are a few things to check and do:

Log into your monitored servers and make sure there are running Sematext Agent processes (there should be more than one of them)
Check if system time is correct. If not, you should adjust the time, restart the Sematext Agent with:
```
sudo service sematext-agent restart
```
and restart any other Embedded App Agents by restarting your server which is being monitored.
Check network connectivity as described elsewhere in the FAQ
Make sure disks are not full

Make sure user spmmon can have more than 1024 files open:

sudo vim /etc/security/limits.conf
spmmon     -    nofile    32000

and

sudo vim /etc/pam.d/su
session    required   pam_limits.so

Restart Sematext Agent after the above changes.

Check if hostname of your server is defined in /etc/hosts
If you are starting your Jetty (or some other server) with command like java ... -jar start.jar ... and using the Embedded version of App Agent, make sure -D and -javaagent definitions occur before "-jar start.jar" part in your command
If none of the suggestions helped, see How do I create the diagnostics package.

My server stopped sending metrics, so why do I still see it under Hosts Filter?¶

Filters have 2 hour granularity, which means that a server will be listed under Hosts filter until 2 hours since it last sent data have passed. For example, if a server stopped sending data at 1 PM and if at 2:30 PM you are looking at the last 1 hour of data (for a period from 1:30 PM until 2:30 PM) you will not see data from this server on the graph, but you will still see this server listed under the Hosts filter until 3 PM. After 3 PM this server should disappear from the Hosts filter.

I rebooted my server and now I don't see any data in my graphs. What should I check?¶

Here are a few things to check and do:

Make sure Sematext Agent is running: sudo service sematext-agent restart
Make sure disk is not full: df -h
Make sure maximal open files limit was not reached: see "I registered for Sematext Monitoring more than 5 minutes ago and I don't see any of my data, what should I check?"

How come Disk Space Usage report shows more free disk space than df command?¶

Sematext Monitoring reports both free and reserved disk space as free, while df does not include reserved disk space by default.

I changed my server's hostname and now I don't see new data in my graphs. What should I do?¶

Simply restart Sematext Agent:

sudo service sematext-agent restart

and restart any process which is using the Embedded version of App Agent.

Can I specify which Java runtime the App Agent should use?¶

Yes, you can edit the /opt/spm/properties/java.properties file where you can specify the location of Java you want the App Agent to use.

Can Sematext Agent use HTTP instead of HTTPS to send metrics from my servers?¶

Yes, although we recommend using HTTPS.

Sematext Agent by default uses HTTPS to send metrics data to Sematext. If you prefer to use HTTP instead (for example, if you are running Sematext Enterprise on premises or if you don't need metric data to be encrypted when being sent to Sematext over the Internet), you can adjust that in /opt/spm/properties/agent.properties by changing protocol to http in property:

server_base_url=https://spm-receiver.sematext.com

Why are some system information fields empty?¶

It may happen that some of the system information fields are unavailable. Depending on the architecture, since the Kernel version 2.6.26, the CONFIG_STRICT_DEVMEM kernel configuration option limits access to the /dev/mem character device by default. This character device provides userspace access to all physical memory. It is primarily used to access the system's IO memory addresses in relation to peripheral hardware. When CONFIG_STRICT_DEVMEM is enabled in the Kernel, access to the /dev/mem character device is disabled from userspace even for the root user. The Sematext Agent uses this character device to gather machine information. With CONFIG_STRICT_DEVMEM enabled in the Kernel, the Sematext Agent may log errors relating to /dev/mem and will not be able to gather complete machine information, resulting in some missing system information fields. To resolve this, it is necessary to recompile the kernel with CONFIG_STRICT_DEVMEM=n set in the kernel configuration options. It is usually possible to check if CONFIG_STRICT_DEVMEM is currently enabled in the kernel, by running the following command:

cat /boot/config-$(uname -r) | grep CONFIG_STRICT_DEVMEM

Security¶

What information are App Agents sending?¶

App Agents agents are sending metrics and data used for filtering those metrics, such as hostnames (which can be obfuscated). To see what exactly is being sent you can use tcpdump or a similar tool to sniff the traffic. App Agents ship data via HTTPS, but can also ship it via HTTP. JVM-based agents let you change the protocol in appropriate files under /opt/spm/properties/ directory, while Node.js-based agents let you change it via the SPM_RECEIVER_URL environmental variable.

Can hostnames in Sematext Monitoring be obfuscated or customized?¶

Yes, you can obfuscate or alias hostnames. This lets you:

never send your real hostnames over the internet
use custom hostname in Sematext UI in case original hostname is too cryptic (e.g. have Sematext show "my-solr-host1" instead of "ip-12-123-321-123")

How this is done depends on the environment in which you've deployed Sematext Agent. In container environments (Kubernetes, Docker Swarm, etc.) set the hostname alias using the HOSTNAME_ALIAS environment variable. This needs to be set before starting the Sematext Agent container.

In non-container environments, after Sematext Agent is installed, open /opt/spm/properties/agent.properties file and just add desired hostname as the value of the hostname_alias property, e.g.

hostname_alias=web1.production.

After that, restart Sematext Agent with:

sudo service sematext-agent restart

and restart any Java process which was using the Embedded App Agent.

Note:

old data will still be seen in Sematext under the old hostname, while new data (after hostname change) will be displayed under the new hostname
if you are installing Sematext Agent for the first time and you want to be 100% sure its original hostname never leaves your network, define your hostname alias in agent.properties file immediately after you complete the "Agent Installation" step and before you begin with the "Agent Setup" step (installation instructions can be accessed from https://apps.sematext.com/ui/monitoring, click Actions > Install Monitor on app you are installing)

Plans & Pricing¶

How much does your monitoring solution cost? Do you offer different plans? Check the Infra Monitoring and Service Monitoring Pricing.

Billing¶

How do you bill for infrastructure and server monitoring?¶

Usage is metered hourly on a per-agent basis. There are two types of Monitoring Apps, and they are priced differently:

Infra Monitoring Apps, starting at $0.004/hour per host for the Basic plan and 1 day of data retention.
Monitoring Apps for our supported Integrations, starting at $0.012/hour per agent for the Basic plan and 1 day of data retention.

For example:

If you send metrics from a server A to Infra Monitoring App Foo between 01:00 and 02:00 that's $0.004 for the Standard plan and 7 days of data retention.

If another agent is monitoring something else, even if that is running on the same server A, and sending metrics to a different Infra Monitoring App Bar, that's another $0.004.

If you are not sending metrics from a server A for a Infra Monitoring App Foo between 02:00 and 03:00 then you pay $0 for that hour.

If you send metrics from a server A to Integration Monitoring App Baz between 01:00 and 02:00 that's $0.012 for the Basic plan and 1 day of data retention.

A single agent monitoring 24/7 will end up being $2.8/month. If you run another agent on another server it will be 2 x $2.8/month. And for each integration, it will be an extra $8.64/month.

Note:

For each Integration Monitoring App, an Infra Monitoring App is required for monitoring your OS, container, and process metrics, along with info about your hosts, package and container image inventory, system events, and more.

How do you bill for container monitoring?¶

Container monitoring is based on the base price and per-container price. The base price includes monitoring of a container host and free monitoring of up to N containers. Per-container price is applied only if you run more than N containers per host. The number of containers per host is averaged for the whole App. The base price and the number of containers included in it depends on the plan. Note that monitoring of container host and containers is independent of monitoring of applications you run in those containers. Containerized applications monitored by Sematext are metered as separate hosts. In other words, whether the monitored application is running in a container or in a VM or directly on a server or in a public cloud instance is the same as far as metering and billing is concerned. For plans and price details see https://sematext.com/pricing/.

Containers are monitored and charged through Infra Apps which can also monitor your bare-metal servers and virtual machines.

Which credit cards are accepted?¶

See billing FAQ.

Can I be invoiced instead of paying with a credit card?¶

See billing FAQ.

How often will I get billed?¶

See billing FAQ.

Can the billing email be sent to our Accounts Payable/Accounting instead of me?¶

See billing FAQ.

Do I have to commit or can I stop using Sematext at any time?¶

See billing FAQ.

Can I get invoices?¶

See billing FAQ.

Sematext Monitoring FAQ

General¶

What should I do if I can't find the answer to my question in this FAQ?¶

Is there a limit to how many servers/nodes/containers I can monitor with Sematext?¶

What types of applications and infrastructure can Sematext monitor?¶

What are the various configuration files present in sematext-agent package?¶

Can I reduce the amount of logs generated by Sematext Agent?¶

How frequently does Sematext Agent collect metrics and can I adjust that interval?¶

How much memory is standalone App Agent using and can it be adjusted?¶

Can I use Sematext to monitor multiple applications running on the same server / VM?¶

Can I use Sematext to monitor my service which runs on Windows or Mac?¶

Can Sematext monitor metrics via http-remoting-jmx protocol (e.g. WildFly, JBoss, etc.)?¶

When should I run Standalone and when Embedded App Agent?¶

Can I move Sematext Agent to a different directory?¶

Is there an HTTP API?¶

I have multiple Monitoring Apps installed on my machine, can I uninstall just one of them?¶

Can I disable Sematext Agent without uninstalling it?¶

JMX attaching is not working out of the box¶

Can I reconfigure the agent to send metrics to a different Monitoring App?¶

Can I switch a host to another Infra App? Can I reconfigure the Sematext Agent to send metrics to a different Infra App?¶

Sharing¶

How can I share my Sematext Apps with other users?¶

What is the difference between OWNER, ADMIN, BILLING_ADMIN, and USER roles?¶

Agent Automation¶

Is there an Ansible Playbook for the Sematext Agent?¶

Agent Updating¶

How do I upgrade the Sematext Agent?¶

Agent Uninstalling¶

How do I uninstall the Sematext Agent?¶

Agent Overhead¶

What's the memory usage overhead for Sematext Agent running in a container?¶

What's --restart always?¶

What's the CPU usage overhead for Sematext Agent running in a container?¶

Agent installation¶

How do I install the Sematext Agent?¶

Alerts¶

Can I send alerts to PagerDuty, Slack, Nagios, or other WebHooks?¶

What are Threshold-based Alerts?¶

What is Anomaly Detection?¶

What are Heartbeat Alerts?¶

Troubleshooting¶

Can I enable debugging in the Sematext Agent?¶

Can I install Sematext Agent on servers that are behind a proxy?¶

Can Sematext Agent send data out from servers that are behind a proxy?¶

Can I change the region settings for the Sematext Agent installation?¶

When installing Sematext Agent, I see "The certificate of pub-repo.sematext.com is not trusted" or a similar error. How can I avoid it?¶

When installing Sematext Agent, I see "gnupg do not seem to be installed" or a similar error. How can I avoid it?¶

How to prevent "Pending" status when installing Sematext Agent on a Kubernetes Cluster?¶

How do I create the diagnostics package?¶

I see only my system metrics (e.g. CPU, Memory, Network, Disk...), but where is the rest of my data?¶

I do not see any system metrics (e.g. CPU, Memory, Network, Disk), what could be the problem?¶

I am using trying to monitor Solr / Elasticsearch. My Request Rate and Latency charts are empty?¶

I am trying to monitor Elasticsearch. My HTTP Metrics charts are empty?¶

I am trying to monitor Elasticsearch. Node stats are not tracked over time?¶

I am trying to monitor MySQL. I see an error in Agent logs - java.sql.SQLException: The server timezone value 'XXX' is unrecognized or represents more than one timezone.¶

I am not seeing any data in Monitoring charts. How do I check if network connectivity is OK?¶

I don't see any of my metrics / I am getting errors when starting Sematext Agent or when starting my service which uses Embedded version of App Agent. What should I check?¶

My server stopped sending metrics, so why do I still see it under Hosts Filter?¶

I rebooted my server and now I don't see any data in my graphs. What should I check?¶

How come Disk Space Usage report shows more free disk space than df command?¶

I changed my server's hostname and now I don't see new data in my graphs. What should I do?¶

Can I specify which Java runtime the App Agent should use?¶

Can Sematext Agent use HTTP instead of HTTPS to send metrics from my servers?¶

Why are some system information fields empty?¶

Security¶

What information are App Agents sending?¶

Can hostnames in Sematext Monitoring be obfuscated or customized?¶

Plans & Pricing¶

Billing¶

How do you bill for infrastructure and server monitoring?¶

How do you bill for container monitoring?¶

Which credit cards are accepted?¶

Can I be invoiced instead of paying with a credit card?¶

How often will I get billed?¶

Can the billing email be sent to our Accounts Payable/Accounting instead of me?¶

Do I have to commit or can I stop using Sematext at any time?¶

Can I get invoices?¶

What's `--restart always`?¶