Got Alarm Fatigue?
If so, you are not alone! We talk to a lot of people who want to reduce the frequent “noise” from monitoring alarms. To solve this common problem, Sematext added anomaly detection for alerts and PagerDuty integration to its SPM Performance Monitoring solution to dramatically reduce this noise compared with simple threshold-based alerting mechanisms. The integration with PagerDuty helps DevOps with incident management, i.e., managing escalation and routing alerts to the right person by defined schedules and communication channels.
PagerDuty is an alarm aggregation and dispatching service for system administrators and support teams. It collects alerts from your monitoring tools, gives you an overall view of all of your monitoring alarms, and alerts an on-duty engineer if there’s a problem. PagerDuty allows you to build sophisticated alerting rules to determine who to contact when problems occur. You can build on-call schedules to equitably share on-call responsibilities. You can also set up multiple levels of coverage, so if the “primary” on-call person doesn’t respond to an alert in a timely fashion, it’s automatically escalated to a “secondary” person, and so on.” – Source: PagerDuty FAQ.
SPM Performance Monitoring is an enterprise-class, server and application performance monitoring, alerting, and anomaly detection solution. It is available both in the cloud (SaaS) and On Premises. SPM also integrates with Logsene Log Management and Analytics to correlate metrics, alerts, anomalies, and events with application and server logs.
Get started
Basic setup steps are required to hook up both services:
- In PagerDuty: Get an API Key
- In SPM: Enter the API Key in SPM alert settings
1) In PagerDuty:
Create a new service:
- In your account, under the Services tab, click “Add New Service”.
- Select an Escalation Policy (e.g. default)
- Start typing “Sematext” for the Integration Type, which will narrow your filtering.
- Click the Add Service button
- Once the service is created, you’ll be taken to the Service page. On this page, you’ll see the “Service API key,” which you will need when you configure Sematext products to send events to PagerDuty. Copy the “Service API Key“ to the clipboard.
2) In SPM
1) Navigate to SPM Application Settings of your SPM App by clicking the App Settings button in the top right when you’re in the SPM UI.
2) Navigate to Alerts / PagerDuty“
3) Enter the API key from PagerDuty in the field Service API key
4) Press the Save button
Done. Every alert from your SPM app will be forwarded to PagerDuty, where you can manage escalation policies and configure notifications to other services like HipChat, Slack, Zapier, Flowdock, and more.
If you’ve got some feedback on this post or ideas for similar posts please let us know!