Real-time system alerts using Twitter OAuth (implementation)

Note: In order to understand this document it is strongly recommended you already read the theory article.
Also the author suppose the reader have a good base about GNU Linux and security.
English is not my first language, do not hesitate to point our any "bad writing".
License : Copyright Emeric Nasi, some rights reserved
This work is licensed under a Creative Commons Attribution 4.0 International License.

Introduction

I really like Twitter. I opened my own account a few days ago.
I was looking at the Twitter features. I just tough that Twitter could be used for more than public communications.
A problem you may face when dealing with monitoring is the reaction time. And having real-time alerts is the big issue.
I found out Twitter’s API could be used for that, in a more secure way then classic emails.
I implemented this solution and I will present it to you using two articles.
This is the second article, in which we are now going to implement the solution we discussed in the first one.

I Choose the tools

So what do we need?

• An entity that watches logs and that has real-time actions enabled.
• An entity that can use the Twitter over OAuth

I.1 Swatch for the real-time alert

I choose swatch because it is simple to understand and it is enough for what we are going to do.
The basic Swatch functioning is simple. It watches a file for any new added text. If the new text contains a certain string or pattern,
it will action an alert mechanism. This alert can be an echo, sending a mail, executing another program, etc.
Install swatch :
apt-get install swatch # for Debian based distrib
yum install swatch # for RedHat based distrib

I.2 Perl for the OAuth/Twitter API

Among the possible languages, I choose perl because it is fast and easy to write a program that does just what we need with it.
Perl has also a Twitter module that is well documented and easy to use.

First install perl if it isn’t already installed on your system (that is quite unprobable).
apt-get install perl # for Debian based distrib
yum install perl # for RedHat based distrib
After that install the OAuth CPAN perl module.
perl -MCPAN -e "install Net::OAuth"
When you have done that, you can implement OAuth calls and write your own
Twitter over OAuth code.
However why reinvent the wheel when a nice Twitter over OAuth module already exists?
Install this module :
perl -MCPAN -e "install Net::Twitter"

II Configure the tools

II.1 Create a special monitoring group.

groupadd -f --system thyl_g_monitor

II.2 Create the system directories.

MONITOR_PATH="/etc/sevagas/monitor" # Our application path
(umask 026 && mkdir -p "$MONITOR_PATH") # Create path
chown  root:thyl_g_monitor "$MONITOR_PATH" # Change group ownership
chmod  2750 "$MONITOR_PATH" # Setgid bit and prevent "Others" access 
mkdir  "$MONITOR_PATH/tmp" # Create temporary folder
chmod 3770 "$MONITOR_PATH/tmp" # Members of monitor group can add files but only owner of files can delete them
mkdir "$MONITOR_PATH/twitter"
chmod 750 "$MONITOR_PATH/twitter"

II.3 Create perl script.

Perl’s Net::Twitter module allows easy access and manipulation of the Twitter API, includind OAuth authentication.
In our case, after authentication, we are going to use the private message (direct message) call.
What is great about private message is that if you have activated the mobile features, you will also receive the alert by sms.
This script must be called with one parameter (the system alert)
and will send our system alert via private messages to our own Twitter account.

Create the file :
touch /etc/sevagas/monitor/twitter/the_bird_sentinel.pl
chmod 750 /etc/sevagas/monitor/twitter/the_bird_sentinel.pl
Now open it with your favorite text editor and paste the next code

#!/usr/bin/perl
# Adapted from http://search.cpan.org/~mmims/Net-Twitter-3.13003/lib/Net/Twitter/Role/OAuth.pm
use Net::Twitter;
 
# Save access token function
sub save_access_tokens # if necessary
{
	my ($at,$ats) = @_;
	# Grab keys
	$file = '/etc/sevagas/monitor/twitter/access-token';
	open(INFO, ">$file") or die("open: $!");           # open in write-only mode
	print INFO "$at"; # Save access token
	close(INFO);
	$file = '/etc/sevagas/monitor/twitter/access-token-secret';
	open(INFO, ">$file");           # open in write-only mode
	print INFO "$ats"; # Save accecc token secret
	close(INFO);
	return 0;
}
# Restore access token function
sub restore_access_tokens
{
	# Grab keys
	$file = '/etc/sevagas/monitor/twitter/access-token';
	open(INFO, "<$file")  or return 0;           # open in read-only mode
	$at = <INFO>;
	chomp $at; # remove '\n' at end of line
	close(INFO);
	$file = '/etc/sevagas/monitor/twitter/access-token-secret';
	open(INFO, "<$file")  or return 0;           # open in read-only mode
	$ats = <INFO>;
	chomp $ats; # remove '\n' at end of line
	close(INFO);
	return ($at,$ats);
}
 
# get first arg (direct message)
$direct_message = $ARGV[0];
if ( ! $direct_message ){
	print "This script requires one argument (the private message you wish to send).\n";
	exit(1);
}
# Grab consumer keys
$file = '/etc/sevagas/monitor/twitter/consumer-key';
open(INFO, "<$file") or die("open: $!");           # open in read-only mode
$consumerKey = <INFO>;
chomp $consumerKey; # remove '\n' at end of line
close(INFO);
$file = '/etc/sevagas/monitor/twitter/consumer-secret';
open(INFO, "<$file") or die("open: $!");           # open in read-only mode
$consumerSecret = <INFO>;
chomp $consumerSecret; # remove '\n' at end of line
close(INFO);
#print "Consumer key : $consumerKey\n";
#print "Consumer secret : $consumerSecret\n";
 
my $nt = Net::Twitter->new(
	traits          => ['API::REST', 'OAuth'],
	consumer_key    => $consumerKey,
	consumer_secret => $consumerSecret,
);
 
my($access_token, $access_token_secret) = restore_access_tokens();
 
if ($access_token && $access_token_secret) {
	#print "Access token : $access_token \n";
	#print "Acces token secret : $access_token_secret \n";
	$nt->access_token($access_token);
	$nt->access_token_secret($access_token_secret);
}
 
unless ( $nt->authorized ) {
	# The nt is not yet authorized: Do it now
	print "Authorize this app at ", $nt->get_authorization_url, " and enter the PIN#\n";
 
	my $pin = <STDIN>; # wait for input
	chomp $pin;
 
	my($access_token, $access_token_secret, $user_id, $screen_name) = $nt->request_access_token(verifier => $pin);
	save_access_tokens($access_token, $access_token_secret); # if necessary
}
 
# Everything's ready!
# Send a direct message to myself
$nt->new_direct_message("< your Twitter account >", $direct_message );
 
exit(0);

II.4 Follow Twitter OAuth workflow

Open your favorite navigator and log into your Twitter account.
Register for OAuth applications at http://twitter.com/oauth_clients
Correctly fill in the form. We assume the application name is The bird sentinel.
Choose "Client" application type, choose "Read and write" default access type and check "Yes, use Twitter for login". Then validate the form.
>> You should see a "Success" message.
The page your are on now is very important, you should note down all informations.
Type the following instructions :

touch "$MONITOR_PATH/twitter/consumer-key"
chmod 640 "$MONITOR_PATH/twitter/consumer-key"
echo "< your consumer key >" > "$MONITOR_PATH/twitter/consumer-key"
touch "$MONITOR_PATH/twitter/consumer-secret"
chmod 640 "$MONITOR_PATH/twitter/consumer-secret"
echo "< your consumer secret >" > "$MONITOR_PATH/twitter/consumer-secret"

After that, launch our perl script manually for initial authentication.
Be sure you are still logged into your Twitter account.
/etc/sevagas/monitor/twitter/the_bird_sentinel.pl "Hello!"
The script should suggest you to authorize the application at a given twitter address and enter a pin code.
Open your navigator at that address and accept to authorize the bird sentinel application.
>> You’ve successfully granted access to the bird sentinel!
Type the pin code in the script console and press enter.
OK you are now authorized to use the twitter API !!!
To verify that it worked, go to your Twitter account, you should have received a new private message :-).
You should also observe that the access token files are created in /etc/sevagas/monitor/twitter/

II.5 Configure swatch.

Swatch real-time alert is nice but the default installation has some lacks we are going to correct.
First we are going to run swatch daemon as its own system user.
useradd --system --no-user-group --gid thyl_g_monitor --groups adm  -s /bin/false  --comment "Swatch System User" thyl_u_swatch

Now swatch will run as ’thyl_u_swatch’ and will be able to read any files whose group is ’thyl_g_monitor’ or ’adm’.

Swatch generates some temporary files we want to clean.
We are going to write a cron job for that :

(umask 027 && touch /etc/cron.weekly/clean-swatch)	
echo '#!/bin/sh' > /etc/cron.weekly/clean-swatch
echo "# This script removes old tmp files generated by swatch in /etc/sevagas/monitor/tmp" >> /etc/cron.weekly/clean-swatch
echo "find /etc/sevagas/monitor/tmp -name \"*swatch*\" -type f -atime +3 -delete" >> /etc/cron.weekly/clean-swatch
echo  >> /etc/cron.weekly/clean-swatch
chmod 700 /etc/cron.weekly/clean-swatch

Now we are ready to configure our swatch for twitter daemon!
We can do much more but for our example we are going to watch the file /var/log/messages for any new message containing the word "bird_alert".
We create the swatch configuration file :
(umask 037 &&  touch "/etc/sevagas/monitor/swatch-twitter.conf")
Now edit the file /etc/sevagas/monitor/swatch-twitter.conf :

watchfor /bird_alert/
exec /etc/sevagas/monitor/twitter/the_bird_sentinel.pl "$_"
threshold type=limit,count=1,seconds=30

Now create the init script to launch our daemon:
(umask 077 && touch /etc/init.d/swatch-twitter)  
Edit /etc/init.d/swatch-twitter and add next code :

#!/bin/bash
### BEGIN INIT INFO
# Provides:          swatch for twitter daemon
# Required-Start:    
# Required-Stop:     
# Default-Start:     2 3 4 5
# Default-Stop:      0 6
# Short-Description: Launches swatch daemon for twitter real-time alert
# Description:       
### END INIT INFO
 
# Author: Emeric Nasi
# Contact : emeric.nasi[at]sevagas.com 
 
displayHelp () {
	echo " Swatch for twitter , usage : /etc/init.d/swatch-twitter start|stop|status "
	echo " All config and tmp file are stored in /etc/sevagas/thylacine/monitor/"
	exit 3
}
 
if [ $# != 1 ] 
then
	displayHelp
fi
case "$1" in 
start)
	echo "Starting swatch for twitter..."
	startCommand="swatch --daemon --pid-file /etc/sevagas/monitor/tmp/swatch-twitter.pid --script-dir=/etc/sevagas/monitor/tmp -c /etc/sevagas/monitor/swatch-twitter.conf -t /var/log/messages"
	if which sudo &>/dev/null # We run swatch with its own user (with sudo or su)
	then
		sudo -u thyl_u_swatch $startCommand
	else
		su -c "$startCommand" thyl_u_swatch
	fi
	;; 
stop) 
	echo "Stopping swatch for twitter..."
	kill $( cat "/etc/sevagas/monitor/tmp/swatch-twitter.pid" )
	rm /etc/sevagas/monitor/tmp/swatch-twitter.pid
	;; 
status) 
	if [ -f /etc/sevagas/monitor/tmp/swatch-twitter.pid ]
	then
		echo "swatch for twitter is running, process $(cat "/etc/sevagas/monitor/tmp/swatch-twitter.pid")."
	else
		echo "swatch for twitter is not running." 
	fi
	;;
*)
	displayHelp
esac
exit 0

Make script executable :
chmod 700 /etc/init.d/swatch-twitter
Install rc.d :
update-rc.d  swatch-twitter defaults # Starts at runlevel 2,3,4,5

III Test and Adapt

III.1 A quick test!

Run the swatch daemon :
/etc/init.d/swatch-twitter start
>>You should have the tmp files in etc/sevagas/monitor/tmp/
Now lets test our system!
echo "bird_alert : this is an alert" >> /var/log/messages
You can log into your twitter account and check your private messages, you should find a nice message containing "bird_alert : this is an alert" !!!

Pfiew, we are arriving at the end of this article. You can now modify the previous scripts
to adapt them at your own needs! You can combine this system with an IDS a firewall or any other soft that writes logs.

However, before doing anything read the next section...

III.2 Caution!

Be careful not to spam or flood Twitter with your logs!
You should only send the really bad alert and do not send the same alert again and again
Maybe you should keep the "bird_watcher" string required so you are sure other logs wont flood your twitter account.
Just be sure critical logging has the string "bird_watcher" inside the log.
Another thing, twitter has maximum 140 characters messages, if you want to send bigger messages, implement a system that truncates big messages in smaller ones.
Finally remember that Twitter has implemented the next limits :

• Direct Messages: 250 per day
• API Requests: 150 per hour
• Updates: 1000 per day

However, I hope that you won’t need more than 250 messages a day... 250 different critical alerts in one day means you don’t sleep to much...
And you should do a general security review of your systems!
Read more about Twitter imposed limits at http://help.twitter.com/forums/10711/entries/15364