Yahoo Pipes and Social Media Monitoring

Circuit Diagram ImageI have been exploring Yahoo Pipes as an effective tool for Social Media Monitoring – to aggregate data from key sources and begin to monitor social media mentions for key topics around industries, companies, brands and products.

The following article highlights my understanding of this technology and my progress to date, including my experience in building Yahoo Pipes for this purpose.

For those that don’t know about Yahoo Pipes, it is an application for creating “mash-ups”. In Yahoo’s own words:

Pipes is a powerful composition tool to aggregate, manipulate, and mashup content from around the web.

As easily as it lends itself to aggregating rss feed-based information it can also be used for the following: mapping data e.g. Flickr photos or live tweets; extracting information from web pages and databases; and manipulating and re-presenting information.

If you so wish you can create an application or “pipe” that extracts the dates of birth of popular actors from the IMDB website, calculates their current age and only shows those over fifty.

On one level Yahoo Pipes is simple and intuitive to use, on another it  makes use of regular expressions “regex” and can quickly get pretty complex.

This said, I am surprised that so few of us use it or even know much about it.

Here is my understanding and use to date, broken down into 5 key advice steps:

1. Understand the Platform

Probably good advice for any new technology. My first step was to create a Yahoo Pipes account and ‘get up-to-speed’ with this platform. Fortunately, the online tutorials are simple to understand and really help.

I also began to review other relevant pipes around Social Media Monitoring, looking for anything I could use or learn. Yahoo Pipes is all about building on what others have done and seeing something working in practice was as useful as any of the tutorials.

2. Learn by Doing

I familiarised myself with the functions through trial and error. When you create a Pipe you are faced with a user interface which is intuitive. It is designed for you to drag and drop any of a range of functions onto a working area. You then configure them and connect them up (like connecting “pipework”). A very simple process for what is ultimately a pretty sophisticated task – creating your own application or mash-up.

The functions are grouped under the following categories: Sources, User Inputs, Operators, Url builders, String, Date, Location and Number functions. There are too many individual functions to mention in this article but try to understand broadly what each does – also see the online tutorials for more detail.

3. Plan your Pipe (Application)

I began to design my Pipe logically and based on what some other Pipe Developers had done – “standing on the shoulders of giants” so to speak. It is easy to search and find the most popular and relevant Pipes and then view, copy and edit.

I also created a basic requirements  checklist before I started any work. My “Pipe” had to fulfill the following criteria:

  1. Create as an output an RSS feed of results that I can drop into a Feedreader and monitor on a daily basis.
  2. Make use of the following Social Media Search sources:
    • SocialMention
    • IceRocket Blog, News and Web
    • Google News, Blog Search
    • Bing Search, MSN Search
    • Yahoo Search
    • Twitter
    • YouTube
    • Flicker
    • BackType
    • BoardReader
    • Delicious
  3. Allow control over the search terms and date filters
  4. Provide the ability to add feeds directly, such as TripAdvisor or others that don’t use “structured” parameters e.g. q=”text”

4. Build Your Pipe

I got to work. On to my design area I began to drop the following key pieces of functionality:

(a) A “search” text input box: where I could add my search term

(b) A string replace: that took care of the spaces in my search term making them link friendly

(c) URL builders: I wanted my search term to create bespoke RSS feeds ‘on-the-fly’ for each of the sources above. This involved understanding how they created their RSS search results feeds.

(d) URL inputs: I wanted to enter some data directly (those feeds that are difficult or impossible to build). This function allowed me to do that.

(e) Feed Fetch: this tells Yahoo Pipes to go fetch the RSS data from each of the URL builders and the URL inputs.

(f) Union: this is the point where we merge our sources – it is the “unfiltered” River of News.

(g) Filters: these were used to filter my river of news based on date input, only show items after date x. Filters can also be used to permit or block entries that contain “keywords” in the title or description fields.

(h) Loop: This is important and allows all items to be amended structurally. I could add additional data to each individual feed but I used this function to standardise my date fields.

(i) Sort: I wanted to sort my results from latest to earliest results.

(j) Unique: With so many sources I wanted to remove any identical results and I used the item title to be the matching field.

(k) Pipe Output: the simplest but most important part. This is the output from my Social Media Monitoring system.

I connected the functions as I went, debugging and constantly referring to the tutorials and other pipes with each new piece of functionality applied. The screenshot in Figure 1 below, highlights the finished Design:

Figure 1: Finished Yahoo Pipes Design (Click image to enlarge)

Yahoo Pipes Design Screen

Source: The Author

5. Run and Test

With the Pipe design complete, the Pipe can now be used to create our output feed. Figure 2 presents a typical Yahoo Pipes user interface:

Figure 2: Yahoo Pipes User Interface (Click image to enlarge)

Yahoo Pipes User Interface
Source: The Author

Creating the output feed involves adding a Search Phrase, inputting the earliest date for results and adding any other feed urls directly. The pipe when run produces a series of results which can be viewed as an RSS feed.

This feed can then be added to any RSS feedreader and monitored. Figure 3 shows the feed output in Google Reader.

Figure 3: Output in an RSS Feedreader (Click image to enlarge)

Outputs feed in Google Reader
Source: The Author

And Finally…Review Performance

My opinion is that Yahoo Pipes is pretty powerful and free (although not free in terms of the time to create, maintain and of course monitor). We have been running this tool for the last few months and the results are promising.

This approach might be considered by many  as “good enough”. It offers more results than any one free source (it appears) and where there are key sites for your industry, like TripAdvisor, it can build in these results making “monitoring” life easier (also offering an advantage over some “paid” SMM tools that don’t index these sources).

Some search providers make life more difficult for the “Pipe Developer”, removing searchable RSS feeds – perhaps seeing them as a threat to their business. Others change their settings or remove their feeds. Feeds need to be managed and maintained in this sense.

For more information on free versus paid Social Media Monitoring tools, please also check out Social Media Monitoring Tools Demystified: Analysing Sources – the next installment is coming soon.

As always your comments are most welcome.



About Alan Stevenson

Over the last fifteen years I have split my time almost equally between Developing Digital strategies for public and private sector clients and helping organisations visualise, specify, plan and optimise technology-based solutions within their organisation.
This entry was posted in Social Media Performance and tagged , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s