Skip navigation

Hey guys,

I’ve been playing around with using yahoo pipes to scrape data from the internet and have processing visulize that data looking for emergent patterns. here’s a little how to if anyones interested:


THE INTERNET -> YahooPipes ->Processing

This is a quick how-to on how you can write a custom yahoo pipes to scrape data from various websites and rss feeds then dynamically load them into a processing application for representation.

Basically you can use yahoo pipes to do what it’s good at, collect, scrape  and collate data from around the internet then return have it return only what data your interested in to a processing application in a nice standardised re-usable way though a xml file.

First example (simple):
We have yahoo pipes collect all known earthquakes over the last 7 days from a rss feed provided by usgs, we then use the “location extractor” provided by yahoo pipes to extract the earthquakes location to lat and longs and we also use a simple regex command to extract the earthquakes magnitude from the earthquakes description. All of this is then returned to a processing script where the location and magnitude of each earthquake is visualised.

The Yahoo Pipe: here

The processing Patch: here

Example 2: the average twit
This example searches for conversations about specific topics on twitter and builds up the average icon of those users talking about that topic. because we are not including icons more than once these faces tend towards individuals that twitter about a specific topic a lot, this means that more obscure topics tend to be among specific individuals resulting in a icon heavily weighted to the icons of a few users and more ubiquitous topics produce greater less defined icons.

The Yahoo Pipe: here

The processing Patch: here

requires the proXML library for processing from here


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: