Skip to content
share library_books

Monitoring Events

Events: What, Why, How?

Sematext Cloud can graph not only performance and custom metrics or logs, but also events. Such events may represent what is happening with a server or cluster, with an application, etc. Think application or server restarts, builds, deployments, alerts, etc. Events are graphed in timeseries charts and these charts can be shown next to metrics or logs charts. This makes it possible to easily correlate events with metrics and/or logs. In addition to showing events as timeseries charts, a detailed listing of events can be seen and, of course, events can have tags, and can be searched and filtered.

Beyond events that you want to see as part of your operations intelligence think about events that matter to your team or your organization in general. Such "business events" can be shipped to Sematext, too.

Besides being shown in the UI events are also exposed via a REST API that lets you post, retrieve, and search your events. This REST API matches the Elasticsearch API, so you can use any Elasticsearch tool or client to post, get, and search events.

Event Fields

An event has the following set of fields, most of which are optional:

Field Name Field Type Required Notes
timestamp date no Represents time when event happened (if not specified, current time will be assumed). The format is dateOptionalTime e.g.: 2014-02-17T21:37:04+0100 or 2014-02-17T14:15:01.534471+02:00 or ...
message string yes Short description of event, e.g. "Elasticsearch node03 on host somehost06 restarted". This is a default search field in Sematext UI, so it is good to keep it concise, but search-friendly.
name string no Event name, can be used as a short label for event, e.g. "Elasticsearch restart".
tags string array no Multivalued field. Each tag should be specified as a separate array element (e.g., "tags":[ "elasticsearch", "restart", "emergency fix"])
priority string no You can use any values that make sense to you, like "high", "very high" or 7. Note that sorting on this field will sort in lexicographical order.
creator string no Person, application, or component that created an event. E.g., "John Smith", "Elasticsearch", "Some Batch Job"
data string no Additional event data. It can be anything you may find useful to have along inside of event object. E.g., it could be stacktrace in case of "app_error" event, base64 encoded content of file generated during "user_registered" event, etc.

Adding Events

Events can be added interactively via the UI, but you can also add them via the API:

https://event-receiver.sematext.com/APPLICATION_TOKEN/event

Because an event is always associated with a Sematext App, the App token must be specified in the URL. Thus, to send multiple events associated with multiple Apps, separate call to the API will need to be made for each App.

Event Types

Each event has a type. This helps you distinguish between different kinds of events. You can specify the event type as a field in the event JSON structure as shown further below. You may want to use types such as alert, app_restart, server_restart, reboot, deployment... To get the most value out of typed events we strongly suggest using a smaller number of distinct event types (1-10) to keep things manageable.

Note: when using curl to call the Events API, you may experience "SSL certificate problem" errors. The reason is that curl doesn't bundle any CA certs any more. For more info see this. Regardless of curl errors, HTTPS communication should be successful.

Example 1

Consider an App whose token (your App tokens are at: https://apps.sematext.com/ui/integrations/apps) is 1111111-2222-3333-4444-555555555555. To send a server_restartevent call the Events API with the App token in the URL:

https://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/event

and with POST content, including event type, in JSON format like this:

{
  "timestamp" : "2014-02-17T15:29:04+0100",
  "message": "Application MyApp on MyHost04 restarted",
  "type" : "server_restart"
}

To add the above event with curl use:

curl -XPOST "https://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/event" -d '
{
  "timestamp" : "2014-02-17T15:29:04+0100",
  "message" : "Application MyApp on MyHost04 restarted",
  "type" : "server_restart"
}
'

Example 2

Same App, but we want to post a deployment event with more event properties populated. In this case the HTTP endpoint would be:

https://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/event

with HTTP POST content:

{
  "timestamp" : "2018-02-17T15:58:04+0100",
  "message": "Solr 7.0.0 version deployed on prodhost06",
  "name" : "Solr 7.0.0 deployment",
  "tags" : ["solr", "7.0.0", "deployment", "upgrade"],
  "priority" : "High",
  "creator" : "John Smith",
  "type" : "deployment"
}

or, again with curl:

curl -XPOST "https://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/event" -d '
{
  "timestamp" : "2018-02-17T15:58:04+0100",
  "message" : "Solr 7.0.0 version deployed on prodhost06",
  "name" : "Solr 7.0.0 deployment",
  "tags" : ["solr", "7.0.0", "deployment", "upgrade"],
  "priority" : "High",
  "creator" : "John Smith",
  "type" : "deployment"
}
'

Searching Events

Sematext lets you find events, metrics, and logs from a specific time period. Additionally, the event chart has a search box where you can further narrow down events to only those that match the input query. You can search on any event field you included in the event when posting it. The query syntax is the same as the logs search syntax.

Event Search API

Sematext exposes the Events Search HTTP API - as Elasticsearch search API-so events can be searched and retrieved programmatically via HTTP(S), using curl or any other Elasticsearch client. The API endpoint is:

https://event-receiver.sematext.com/APPLICATION_TOKEN

Alternatively, you can also use the same endpoint which was used when adding events, where event type is specified, in which case the matching events will be limited to the type specified in the URI:

https://event-receiver.sematext.com/APPLICATION_TOKEN/event

The simplest way to run a query is using URI search, like this:

$ curl -XGET "https://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/_search?q=creator:john"

More query options are available when using request body search, e.g.:

curl -XGET "https://event-receiver.sematext.com/1111111-2222-3333-4444-555555555555/_search" -d '
  "query" : {
    "query_string" : {
      "query" : "MyHost04",
      "default_field" : "message"
    }
  } 
'

This example shows how to use one of the simpler query types - query_string. To see which other query types are available, please check Elasticsearch docs.