Effective and timely monitoring of your service is crucial to providing a highly available service. Early warnings delivered quickly to the right people who can collaborate on a solution is key to delivering the most number of 9s of uptime.
AWS Cloudwatch in conjunction with its Simple Notification Service (SNS) is great for creating alarms for your service and sending notifications to the right people. However, in order to facilitate collaboration in the spirit of chatops, sending these notifications to a chat app like Slack is much more effective than email or SMS alone.
Why you should use Slack to monitor AWS Services
Alarm notifications sent to a Slack channel is much more effective than sending alarms to email and/or SMS in several ways.
First, through the Slack channel, you can see who is online to respond to the issue. Team members can quickly post a message that they see the alarm and are available to investigate. The response can quickly be coordinated with available team members.
Second, the Slack app on the mobile phone combined with a devops chatbot means that not only can team members receive alarms and collaborate from anywhere, they can also take action on the alarms from anywhere.
Third, a Slack channel provides easy controls on notifications. You no longer have to manage members of an email alias, you can simply invite people to the channel and people can leave when they are not on call. Individuals can control how they want to be notified via Slack’s notification settings.
Finally, the Slack channel provides a convenient timeline for root cause and response analysis. You can easily see when alarms fired, who investigated, and what they did when assuming people communicate in the channel.
How to create a Slack alarm channel
Let’s explore how you can create a Slack alarm channel for your service using AWS Cloudwatch, SNS, and the Slack email integration.
The high level outline is as follows:
- Create channel in Slack for alarms
- Add Slack email integration which assigns an email address to the alarm channel
- Create an SNS topic for your alarms
- Create a subscription to the SNS topic that sends email to the email alias from step 2
- Set notification for alarms to go to the appropriate SNS topic
1. Create Slack alarm channel
The first step is to create a channel for your service’s alarms in Slack. Make a public channel, give it a name and purpose. Press the Create Channel button. For this example, I am creating an acmesite-alarms channel.
2. Add Slack email integration
Next step is to add the Slack email integration to get an email alias for our Slack alarm channel. This needs to be done by someone who has admin privileges for your Slack account. In the Slack app, click on the chevron in the upper left (to the right of your Slack account name) and select Apps & Integrations.
In the search box, type “email” and then select the Email addon.
This integration generates an email address for your Slack channel. Emails sent to this address will show up in your channel of choice. This integration is available to teams on the Slack Standard Plan and above. Press Add Configuration button.
Type in the name of the Slack alarm channel you created above and press the Add Email Integration button.
You can customize the name and icon used when an email is posted to your alarm channel.
You will receive an email from the slack email integration with the email address assigned to your channel. You will need this email address for subsequent steps. For the acmesite-alarms channel, the following email address was generated: [email protected]
Test that the integration is working by sending an email to the generated email address for your channel. You should see the email show up in the alarm channel you created similar to the one I sent to the acmesite-alarms channel below.
3. Create an SNS topic for your alarms
Amazon Simple Notification Service (Amazon SNS) is a managed push notification service that lets you send individual messages or to fan-out messages to large numbers of recipients. You create topics and one or more subscriptions for notifications whenever something is posted to that topic.
For this exercise, we are going to create an alarm topic for our site and a subscription with the email address from the Slack email integration above. Technically, we could set the email address for each Cloudwatch alarm we create, however, that is not very manageable. With SNS, we can manage it in one place. Moreover, with SNS we can publish to this topic through other monitoring tools besides Cloudwatch.
First create your SNS topic by going to AWS Services, SNS, and pressing the Create Topic link. Type in a topic name and an optional display name. Click the Create Topic button.
Next, create a subscription to the just created topic (acme-alarms for this example) by going to AWS Services, SNS, and clicking Topics on the left hand side. Check the topic you just created and press the Create New Topic button. Change the protocol to email and set the endpoint to the email assigned to your slack channel from above. Press the Create subscription button.
4. Create alarm that notifies to SNS topic
Now that we have an SNS topic for our service’s alarms that sends an email to our Slack alarm channel, we need to create one or more alarms that publish to this SNS topic. What alarms to create is going to be dependent on your service. However, since most services will have one or more EC2 instances, we’ll walk through the steps of creating an alarm based on EC2 metrics. Go to Services, CloudWatch, click on Alarms, create Alarm, then click on EC2 Metrics. Search for “cpu” and then check an EC2 instance in your service. Your screen will look similar to the one below except with your instances of course.
Click Next.
- Set name to myHighCpuAlarm
- Put in a description “CPU exceeds 60%”
- Click the “is:” drop down and select “>”
- Type in 60 in the edit box
- Set Period to 1 minute
- Under Actions,
- set Whenever this alarm: to “State is ALARM”
- select your SNS topic (for this example “acme-alarms”) for Send notification to: field
- Press the +Notification button
- set Whenever this alarm: to “State is INSUFFICIENT”
- select “acme-alarms” for Send notification to: field
Your screen should look similar to the one below
Click Create Alarm
You will get a message in the channel to confirm the subscription. Expand the message and click on the link to confirm the subscription.
Important: You will not receive notifications from alarms until you confirm the subscription.
5. Test that alarms will be sent to your Slack alarm channel
Now we want to verify we have everything wired up correctly and that alarms will indeed go to our Slack alarm channel. We can do this by using the AWS command line interface (see here).
For the configuration used in this example, the following AWS command will put the CloudWatch alarm called myHighCpuAlarm into the alarm state.
prompt> aws cloudwatch set-alarm-state –alarm-name myHighCpuAlarm –state-reason “testing alarm” –state-value ALARM
This will result in an alarm being sent to the alarm channel.
Now that you have successfully created an AWS Cloudwatch alarm that notifies a Slack channel, you can create more alarms and invite appropriate team members to your Slack alarm channel to be notified when alarms fired. You now have the beginnings of a chatops environment that will help your team to more quickly respond and collaborate on alarms fired by your service.
Add in a chatbot built using an existing devops open source frameworks like the NodeJS Opkit (https://github.com/Bandwidth/opkit) to automate the common response tasks and your team will be devops ninjas.
