Sunday, February 23, 2014

Graphing Twitter Mentions with the Raspberry PI

Objective:
Use the Raspberry PI to monitor Twitter for specific words or hashtags that are tweeted.  Graphically display the results on a publicly viewable webpage (http://open.sen.se/).
-----
If you are not interested in the build details and just want to see the result then take a look at this short video.  We had the RasPI set up to monitor for a "Tektronix" or "Agilent" mention in any and every message Twitter receives.

The video shows an iPad tweeting the message "Tektronix and Agilent. RasPI HashVote Test".  Since this tweet contains both search words the RasPI is monitoring for, each counter gets incremented and the graph is updated.

Tektronix and Agilent are well know test and measurement companies.  In general, they both get tweeted about the same number of times each day.  However, take a look at Agilent's graph the day they released their (not so good) quarterly earnings.  The graph clearly shows the extra Twitter chatter on Agilent due their earnings announcement. 
-----
So how does it work?
You're going to need a Raspberry PI connected to the internet (duh?), a Twitter account, and an open.se account.

Then you are going to create a Python script that runs on the Raspberry PI to search for Tweets and update the counters and graph on your open.se account.  The Python code is listed below.  If you are going to track two terms (as in the example above) you will have a separate Python script running for each search term.  You are going to need API authorization/access tokens for your Python scripts.  Don't panic; that's easy.

- Twitter API Token: Go to https://dev.twitter.com/.  You will need to create API tokens for each Python script you have running.  In the example above I am tracking two words; "Tektronix" and "Agilent", so I need to set up two API tokens.  If you decide to change the search terms from "Tektronix" or "Agilent" to "Happy" and "Sad" you will not have to create new API tokens.  Just simple change the search terms in the RasPI Python scripts.

- open.se API Token: Go to http://open.sen.se/ and create a "Channel" for each search term you want to track.  Again, the example above tracks two terms being tweeted so two channels are created.  After you create a "Channel" you will get a "FEED ID" for each channel.  You will also get an API token that is assigned to your account.  This API token is private to you and the same for each channel.  Then, play around with the "apps" at open.se to created graphs, counters, gauges, and a ton of other cool things.  Their tutorials are good, so I wont explain how to do that here.

Now boot your Raspberry PI, open your favorite text editor, and copy/past in the code below.  Replace the 'xxxxxxxx' with your custom API token and FEED ID information.  Again, you will need a Python script running for each Twitter term you are searching for.  There is probably a way to do this with one Python script, but I'm not that smart.

Run the script and watch the data flow and the counters update.  During debug and test, I would suggest tracking a commonly tweeted word such as "retweet" or "ipad".  That will help with the debug.  If you track a term like "WhiskeyTangoHotel.com" you may be waiting a while to see a result.  ;)

Mostly, the system runs perfectly, but for reasons unknown to me the Python script will 'freeze' from time to time.  I have had them running for days and days at a time without issue, but from time to time the freeze just happens.  If you know why, please enter into comments section of the youtube demo shown above.
-----
#  RasPI Python script to search Twitter for a string and post to open.se
#  by WhiskeyTangoHotel.Com with special thanks to Sparkfun and twython
#  FEB 2014



import time
from twython import TwythonStreamer
import httplib
import json as simplejson

#
# Search term that you want to find and count
#
Search_Term = 'WhiskeyTangoHotel.Com'  # Not case sensitive. TweET = tweet.
global Search_Term_Counter 
Search_Term_Counter = 0   # counts the finds

localtime = time.asctime( time.localtime(time.time()) )
print localtime
print "START searching for: " + Search_Term

# Twitter application authentication
APP_KEY = 'xxxxxxxxxxxxxxxx'
APP_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
OAUTH_TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
OAUTH_TOKEN_SECRET = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

# open.se application authentication
SENSE_API_KEY = "xxxxxxxxxxxxxxxxxxxxx"
FEED_ID1 = 12345  #FeedID for Hashvote_Counter2 on open.se

# def function to send Search_Term_Counter to open.se for processing
def send_to_opensense(data):
#    print  >> fout, "\t=> Sending to OpenSense: %s" % data
        try:    # error trap to continue run if crash during open.se postings
                # prepare data     
                datalist = [{"feed_id" : FEED_ID1, "value" :data['F']},]  #:data string not important for counting, but leave it
                headers = {"sense_key": SENSE_API_KEY,"content-type": "application/json"}
                conn = httplib.HTTPConnection("api.sen.se")
                # format a POST request with JSON content
                conn.request("POST", "/events/", simplejson.dumps(datalist), headers)
                response = conn.getresponse()
                # you may get interesting information here in case it fails
                #   print >> fout, response.status, response.reason
                #   print >> fout, response.read()
                conn.close()
        except:
                pass

# def Hash_Counter called from 'class' below.  Add custom code here.
def Hash_Counter ():  
        global Search_Term_Counter
        Search_Term_Counter = Search_Term_Counter + 1
        localtime = time.asctime( time.localtime(time.time()) )
        print localtime
        print "Search Term " + Search_Term + " found " + str(Search_Term_Counter) + " times."
        data = {'F' : Search_Term_Counter}
        send_to_opensense(data)
        
# Setup callbacks from Twython Streamer
try:  # error trap to continue run if crash due to offsite TwythonStreamer
class Search_Twitter (TwythonStreamer):
def on_success(self, data):
#Hash_Counter()         # for debug only
if 'text' in data:
Hash_Counter()  # found so call the Hash_Counter def
print data['text'].encode('utf-8')  # Typically REM'd unless for debug
print "-----"  # seperator to format the screen.
time.sleep(10)  #pause xx seconds just to keep from flooding sen.se with data
except:
pass
   
# Create streamer to search Twitter fot the Search_Term var
try:
        stream = Search_Twitter(APP_KEY, APP_SECRET, OAUTH_TOKEN, OAUTH_TOKEN_SECRET)
        stream.statuses.filter(track=Search_Term)
except:  #  KeyboardInterrupt:  # helps during debug to exit more gracefully on CNTRL C
        pass  
-----
Good luck and thanks for the visit!

Sunday, February 16, 2014

DS Nano v2 Oscilloscope Performance

The DS Nano v2 is a very low cost 1-channel oscilloscope that, for the ~$80US price, has served me pretty well for my hobby electronics.  The scope came with a nice soft case and two sets of "probes".  It has other features as well.

 ----
This page is not intended to put down the little DS Nano v2.  The scope behaves as advertised.  For example, I had a project that required a piezo sensor input.  Things were not working according to plan and the DS Nano v2 was able to capture the low frequency signal from the piezo and suggest that a diode was needed to tame the signal for use as a microcontroller input.


Here is another example of the DS Nano v2 doing a great job showing variable pulse width from a simple PICAXE18 program:

----
That said, the DS Nano v2 does have its limits and I always wanted to compare it to a "real" oscilloscope.  I was able to get my hands on a Tektronix MSO2042 Oscilloscope and Tektronix AFG3252 Signal Generator.  The AFG3252 can produce waveforms up to 240MHz extremely accurately.  Well in the range for our purposes as an input source.
-----
The table below compares some key specs of the DS Nano v2 and the MSO2042:
Again, for our test purposes the Tektronix MSO2042 Oscilloscope is way more than adequate to test the DS Nano v2. 
-----
So what about the results?  Below are some screen shots of signals with increasing frequency.  The DS Nano v2 is the smaller screen in the foreground.  The Tektronix MSO2024 is the larger screen in the background.  Both  oscilloscopes are connected to the same signal source in parallel with no concern for impedance matching (note the ringing).   You can immediately see that DS Nano fails miserable at detecting the  high frequency ringing.  But let's be fair; the DS Nano spec is 200KHz for analog bandwidth, so this result is not a surprise.

One important thing to look at in the pics below are the voltage level cursor lines on the DS Nano v2.  You will notice that there is really no attenuation in the signal the DS Nano v2 displays as we test to its 200KHz bandwidth; that's good.  However, the limit of the 1MS/s sample rate does start to show.  The 100KHz signal pic really shows how the DS Nano v2 sample rate effects the displayed output.

All that said, it you are looking for a <<$100US scope that will fit in your pocket, the DS Nano v2 is a handy choice.  If you use it within it's specs it is a great value!  If you need more call Tektronix!
----
-----

----







----