Sunday, February 23, 2014

Graphing Twitter Mentions with the Raspberry PI

Objective:
Use the Raspberry PI to monitor Twitter for specific words or hashtags that are tweeted.  Graphically display the results on a publicly viewable webpage (http://open.sen.se/).
-----
If you are not interested in the build details and just want to see the result then take a look at this short video.  We had the RasPI set up to monitor for a "Tektronix" or "Agilent" mention in any and every message Twitter receives.

The video shows an iPad tweeting the message "Tektronix and Agilent. RasPI HashVote Test".  Since this tweet contains both search words the RasPI is monitoring for, each counter gets incremented and the graph is updated.

Tektronix and Agilent are well know test and measurement companies.  In general, they both get tweeted about the same number of times each day.  However, take a look at Agilent's graph the day they released their (not so good) quarterly earnings.  The graph clearly shows the extra Twitter chatter on Agilent due their earnings announcement. 
-----
So how does it work?
You're going to need a Raspberry PI connected to the internet (duh?), a Twitter account, and an open.se account.

Then you are going to create a Python script that runs on the Raspberry PI to search for Tweets and update the counters and graph on your open.se account.  The Python code is listed below.  If you are going to track two terms (as in the example above) you will have a separate Python script running for each search term.  You are going to need API authorization/access tokens for your Python scripts.  Don't panic; that's easy.

- Twitter API Token: Go to https://dev.twitter.com/.  You will need to create API tokens for each Python script you have running.  In the example above I am tracking two words; "Tektronix" and "Agilent", so I need to set up two API tokens.  If you decide to change the search terms from "Tektronix" or "Agilent" to "Happy" and "Sad" you will not have to create new API tokens.  Just simple change the search terms in the RasPI Python scripts.

- open.se API Token: Go to http://open.sen.se/ and create a "Channel" for each search term you want to track.  Again, the example above tracks two terms being tweeted so two channels are created.  After you create a "Channel" you will get a "FEED ID" for each channel.  You will also get an API token that is assigned to your account.  This API token is private to you and the same for each channel.  Then, play around with the "apps" at open.se to created graphs, counters, gauges, and a ton of other cool things.  Their tutorials are good, so I wont explain how to do that here.

Now boot your Raspberry PI, open your favorite text editor, and copy/past in the code below.  Replace the 'xxxxxxxx' with your custom API token and FEED ID information.  Again, you will need a Python script running for each Twitter term you are searching for.  There is probably a way to do this with one Python script, but I'm not that smart.

Run the script and watch the data flow and the counters update.  During debug and test, I would suggest tracking a commonly tweeted word such as "retweet" or "ipad".  That will help with the debug.  If you track a term like "WhiskeyTangoHotel.com" you may be waiting a while to see a result.  ;)

Mostly, the system runs perfectly, but for reasons unknown to me the Python script will 'freeze' from time to time.  I have had them running for days and days at a time without issue, but from time to time the freeze just happens.  If you know why, please enter into comments section of the youtube demo shown above.
-----
#  RasPI Python script to search Twitter for a string and post to open.se
#  by WhiskeyTangoHotel.Com with special thanks to Sparkfun and twython
#  FEB 2014



import time
from twython import TwythonStreamer
import httplib
import json as simplejson

#
# Search term that you want to find and count
#
Search_Term = 'WhiskeyTangoHotel.Com'  # Not case sensitive. TweET = tweet.
global Search_Term_Counter 
Search_Term_Counter = 0   # counts the finds

localtime = time.asctime( time.localtime(time.time()) )
print localtime
print "START searching for: " + Search_Term

# Twitter application authentication
APP_KEY = 'xxxxxxxxxxxxxxxx'
APP_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
OAUTH_TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
OAUTH_TOKEN_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

# open.se application authentication
SENSE_API_KEY = "xxxxxxxxxxxxxxxxxxxxx"
FEED_ID1 = 12345  #FeedID for Hashvote_Counter2 on open.se

# def function to send Search_Term_Counter to open.se for processing
def send_to_opensense(data):
#    print  >> fout, "\t=> Sending to OpenSense: %s" % data
        try:    # error trap to continue run if crash during open.se postings
                # prepare data     
                datalist = [{"feed_id" : FEED_ID1, "value" :data['F']},]  #:data string not important for counting, but leave it
                headers = {"sense_key": SENSE_API_KEY,"content-type": "application/json"}
                conn = httplib.HTTPConnection("api.sen.se")
                # format a POST request with JSON content
                conn.request("POST", "/events/", simplejson.dumps(datalist), headers)
                response = conn.getresponse()
                # you may get interesting information here in case it fails
                #   print >> fout, response.status, response.reason
                #   print >> fout, response.read()
                conn.close()
        except:
                pass

# def Hash_Counter called from 'class' below.  Add custom code here.
def Hash_Counter ():  
        global Search_Term_Counter
        Search_Term_Counter = Search_Term_Counter + 1
        localtime = time.asctime( time.localtime(time.time()) )
        print localtime
        print "Search Term " + Search_Term + " found " + str(Search_Term_Counter) + " times."
        data = {'F' : Search_Term_Counter}
        send_to_opensense(data)
        
# Setup callbacks from Twython Streamer
try:  # error trap to continue run if crash due to offsite TwythonStreamer
class Search_Twitter (TwythonStreamer):
def on_success(self, data):
#Hash_Counter()         # for debug only
if 'text' in data:
Hash_Counter()  # found so call the Hash_Counter def
print data['text'].encode('utf-8')  # Typically REM'd unless for debug
print "-----"  # seperator to format the screen.
time.sleep(10)  #pause xx seconds just to keep from flooding sen.se with data
except:
pass
   
# Create streamer to search Twitter fot the Search_Term var
try:
        stream = Search_Twitter(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
        stream.statuses.filter(track=Search_Term)
except:  #  KeyboardInterrupt:  # helps during debug to exit more gracefully on CNTRL C
        pass  
-----
Good luck and thanks for the visit!