sending icinga 1.x notifications to slack

i had some fun dealing with this. a good lot of my time is alloted to monitoring systems, so when i was told that a new team was going to play ping monkey i jumped on the opportunity to get them set up with everything they need to do the job coherently.

previously we would send monitoring alarms to our hipchat server via hipsaint which works beautifully. we've moved to slack, so therefore our integration needs to be updated.

step 1: you'll need to set up a new integration for your chat room via the slack web ui. you'll get a url hook as a result, which starts with https://hooks.slack.com/services/.

step 2: you'll need a shell script that will parse the alarms and send them to slack. i save this to /usr/local/bin/slack.sh. you're welcome to put it wherever, as long as the icinga user has execute access to it. this is heavily based off of this script, just heavily modified and updated to work with slack's webhooks and using icinga instead of nagios.

#!/bin/bash

WEBHOST_ICINGA="icinga.yourdomain.com"
SLACK_TEAM="yourslackteam"
SLACK_CHANNEL="#yourchannel"
SLACK_BOTNAME="icinga-bot"
WEBHOOK_URL="https://hooks.slack.com/services/your/hook/url"

if [ "$ICINGA_SERVICESTATE" = "OK" ]
then
    ICON_EMOJI=":thumbsup:"
elif [ "$ICINGA_SERVICESTATE" = "WARNING" ]
then
    ICON_EMOJI=":loudspeaker:"
elif [ "$ICINGA_SERVICESTATE" = "CRITICAL" ]
then
    ICON_EMOJI=":broken_heart:"
elif [ "$ICINGA_SERVICESTATE" = "UNKNOWN" ]
then
    ICON_EMOJI=":troll:"
else
    ICON_EMOJI=":octocat:"
fi

curl -X POST --data "payload={\"channel\": \"${SLACK_CHANNEL}\", \"username\": \"${SLACK_USERNAME}\", \"icon_emoji\": \":vertical_traffic_light:\", \"text\": \"${ICON_EMOJI} host: $ICINGA_HOSTNAME\nservice: $ICINGA_SERVICEDISPLAYNAME\nstate: $ICINGA_SERVICESTATE\nmessage: $ICINGA_SERVICEOUTPUT\nrunbook: $ICINGA_SERVICENOTESURL\nlink: \"}" ${WEBHOOK_URL}

step 3: modify your icinga instance to use your new script. this is going in the commands.cfg file but really it can go in any config file under objects.

    # 'notify-service-by-slack' command definition
    define command {
           command_name     notify-service-by-slack
           command_line      /usr/local/bin/slack.sh
           }

    # 'notify-host-by-slack' command definition
    define command {
           command_name     notify-host-by-slack
           command_line      /usr/local/bin/slack.sh
           }

we also need to modify our contact so that it also sends the message over slack. find your contact that is currently getting alarms, and just add our new commands in separated by a comma.

service_notification_commands   notify-by-email,notify-service-by-slack
host_notification_commands      host-notify-by-email,notify-host-by-slack

step 4: reload icinga.

step 5 (optional): customize.

the curl inside the script is utilizing icinga's macros to turn the variables into actual values. you can view the whole list here. please note that the format in the script prefixes ICINGA_ before each variable. i'm unsure why this is needed, the macros page doesn't talk about it.

hopefully there is another admin out that that will find this useful.

comments powered by Disqus