Using small data to illustrate what Big Data can do

I'm using the Twitter API to scrape some data for a few projects I'm working on. From the scrape of my own account it's interesting to see the relationships that pop out - even in a rudimentary analysis of my tweets. The data I've pulled together is from:

  • All of my own tweets (and who is mentioned in those tweets).
  • All tweets where I'm mentioned (and everyone who is mentioned in those tweets).

Ok. So what?

Below is a network graph of all "mentions" by people in the above data - omitting mentions by me. I excluded myself because the data is biased towards my own tweets, making a graph that looks like a hub-spoke with me in the middle. With me excluded it offers a glimpse into who mentions me most (and who else they might be mentioning).

Now - apply this logic to the CSEC wifi snooping and the NSA. Add in WiFi location data, SMS messages, and who knows what else. "Metadata" (which is an intentional misnomer in this context) very quickly becomes useful data.

For the record, I'm not a tinfoil hat wearer on this. But the next time a security official says "we're not recording the conversation, only the interaction" think about how quickly I'm able to do something like this with very limited data.

There is no such thing as "limited surveillance" in the age of big data.


Photo & Video Sharing by SmugMug