Tools of Change twitter visualization

by Liza Daly

Geeks love raw data, so I couldn’t resist a text dump of all #toc tweets that occurred during the conference.

Here’s the quick visualization I threw together (warning, it loads very slowly):

TOC tweet visualization

It skips any images that are broken or any that were listed as having the Twitter default icon. For some reason the raw data included a lot of default icons it shouldn’t have — for example, all of my tweets were listed with the default instead of my profile photo. Those are missing. I recommend starting with the XML instead (posted after I wrote this) and getting the profile icons directly via Twitter.

Source code that generates this from the raw text (not from the XML) follows:

#!/usr/bin/python
import re, copy, urllib, sys

# Pattern-match the twitter username
p = re.compile('^([^:]+):(.*)$')

data = []
tweet = None

for line in open('toc-tweets.txt'):

    # If we're not currently in a tweet, start a new record
    if not tweet:
        tweet = {}

    # Skip the useless relative date values
    if line.endswith('ago'):
        continue

    # This is an end-of-tweet marker, so push it on the stack
    # and reset the temporary datastore
    if line.startswith('*'):
        data.append(copy.deepcopy(tweet))
        tweet = None
        continue

    # Grab the user's image
    if line.startswith('http://'):
        tweet['img'] = line
        continue

    m = p.search(line)
    if m:
        # If we matched the regexps then we have a line containing a valid tweet
        tweet['name'] = m.group(1)
        tweet['message'] = m.group(2).replace('&', '&')

# Put in chronological order
data.reverse()

print '''<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html;charset=UTF-8" />
    <title>#toc timeline</title>
    <script src="http://www.google.com/jsapi" type="text/javascript"></script>
    <script type="text/javascript">
      google.load("jquery", "1.3");
    </script>
    <script src="jqueryui/jquery-ui-personalized-1.6rc6.min.js" type="text/javascript"></script>
    <style type="text/css">
       .tweet span { display: none; width: 300px; height:200px;  }
       .tweet { display: inline; z-index: -1;}
       .drop { width: 240px; height: 135px; padding: 0.4em; position:absolute;background:white;z-index:99;}
       .drop h3 { margin: 0; padding: 0.4em; text-align: center; }

    </style>
    <link rel="stylesheet" type="text/css" href="jqueryui/theme/ui.all.css"></link>
    <script type="text/javascript">
    jQuery(document).ready(function() {
      $('.tweet').click(function() {
        $(this).find('span').toggle('fast');
      });
    });
    </script>
   </head>
<body>
'''

images = {}

for t in data:
    if 'default_profile' in t['img']:
        continue
    image = t['img'].replace('mini', 'bigger')
    if image not in images:
        sys.stderr.write("Checking %s\n" % image)
        i = urllib.urlopen(image).read()
        if 'Error' in i:
            sys.stderr.write("Skipping...")
            continue
        images[image] = 1

    print '''
<span class="tweet">
  <img src="%s" alt="%s" width="73" height="73" />
  <span class="ui-widget-content ui-corner-all drop">
    <h3 class="ui-widget-header ui-corner-all"><a href="http://twitter.com/%s">%s</a></h3>
    <blockquote>%s</blockquote>
  </span>
</span>
''' % ( t['img'].replace('mini', 'bigger'),
       t['name'],
       t['name'],
       t['name'],
       t['message'])

print '''
</body>
</html>
'''