Using Pipes

As I mentioned earlier, the main reason I wanted to check out pipes and come back to play with it was the user interface, it's drag'n'drop at it's very best for a web site design.

Simple 'hello world' Pipe

The concept is nothing new, you drag modules or components onto your work area and link them together (with pipes I guess ;) This visual form of programming hides the nuts&bolts involved from the user making it more accessible to non-programmers. It's design reminds me mostly of Microsoft's Filter Graph Editor:

Microsoft Graph Editor

...and the principle is the same. There is some talk of a Flash interface or maybe they meant flash in the more traditional sense, The interface is actually DHTML, Canvas, SVG and possibly based on some of the work done for the Yahoo! User Interface Library?. The 'hello world' pipe show above is as simple as it gets, so simple in fact that it does nothing :) It takes as the source the RSS feed (latest news) from our site and sends it to the output unchanged. As you can see the first module is called 'Fetch Feed' and accepts a URL for the RSS source; I've joined this directly to the second module called 'Pipe Output' (the debugger output previews what is sent to the output module). It would seam an example as simple as this actually has some advantage still, if the source feed was from a low bandwidth/slow site then receiving the feed from this simple pipe would sometimes retrieve the data from Yahoo! pipes local cache rather than it being fetched from the source site each time. The best way to learn about how the pipes work is to open up an existing pipe and just play around with the interface, if you get lost or break anything you can simply close the browser window and try again; You only ever work with a copy (or clone) of the pipe you view/edit, so plonk away!... I found pipe's visual editor very intuitive to use but here are a few tips and pointers to look out for:

More complex Yahoo! pipe

The above pipe is a little more complex, it uses multiple sources that are combined and actions taken on them to sort into date order and manipulate the data in some way to produce the desired output (to see the output from this pipe click the link at the top of this page). Compare this to Microsoft Graph Editor and you should begin to see some similarities; The Filter Graph takes a .mpg video file as the source, uses a module to split the data into a audio and video stream before sending the data to individual rendering modules to displaying the video and playing the audio.

Jumping in...

If you haven't clicked the link at the top of this page yet 'WackyB + Yahoo! Messenger & More... news feed (created with Yahoo! pipes)' check it out now... This new feed was created with pipes to combine all the news feeds from our sites along with any relevant information regarding Yahoo! Messenger from numerous sources around the web (full details later).

WackyB + Yahoo! Messenger pipe

As you can see we begin with a list of new feeds to be fetched, these are all output at the bottom of the 'Fetch Feed' as a new RSS feed, at this point the items are in to particular order and may consist of the total of each feed joined onto the end of the next.

One of the cool features of pipes is the ability to add sub pipes or previous pipes to your design. In the above example I've highlighted these sub pipes in purple, for now the output of these pipes are joined together with the site feeds by using a 'Union' module. I could have used multiple 'Fetch Feed' modules and combined them all with a union module but the 'Fetch Feed' module allows you to specify as many feed sources as you like which are joined internally. The union module then outputs a RSS feed of all of the above but in no order; This is the job of the next module 'Sort', here I have selected that the published date is used to sort all the news items beginning with the most recent at the top. This is really quite a powerful utility as it stands... with just a few simple drag'n'drop of modules and some simple search terms I've been able to combine ALL the information from across all our sites and relevant news items from across the world and present them in one list detailing all the latest changes! if there have been no updates to some sites then I don't have to bother checking them individually. The last block in this example uses the 'Regex' module or Regular Expression; This is where things can really start to happen with pipes (although not with this example! ;) Regular expressions allow for writing simple or complex rules for handling any part of the data coming in; Here I've used the 'Regex' module to tidy up the formatting of one of the news feeds by simple find and replace of text.

Sub Modules/Pipes

As highlighted in purple above, sub modules are complete pipes in themselves that are used within another pipe. As all pipes output their data as a RSS stream you could simply include the output of one pipe into another by entering the RSS URL into the 'Fetch Feed' module and this would work just as well. The great thing about using other pipes is the ability to modularize your pipes and reuse sub pipes, it can also help to keep complex layouts tidy by nesting complex parts away. For our WackyB example I used sub pipes to simplify the main pipe layout. I have two sub pipes called 'Yahoo Messenger News' and 'Merging some official blogs with filter messenger'. Sub pipe modules have a link to [open] them from within the layout (something you can not do with just the RSS feed). When you click the [open] link the sub pipes open in a separate tab within the editor. The expanded pipes are shown below:

Yahoo Messenger News pipe

The first pipe is the Yahoo! Messenger news, It gets news feeds from Goggle and Yahoo for anything related to Yahoo! Messenger, because the news feeds tend to pick up on any vague link to the search terms, the output from the 'Fetch Feed' goes through a filter module that only allows items containing the entire text "Yahoo Messenger". The next module is quite cleaver (although details of how it achieves it's results would be useful); 'Content Analysis' module takes an input RSS feed and analyses each item adding a new attribute to the news item output listing it's determination of the items content with a few words. Because news items tend to be very similar (all based on the same press release), I find this module very useful in filtering out duplicate news stories. Filtering by the content analysis output will tend to filter out more than it lets through but for my intentions for this pipe it's just what I want. With the feed then sorted and cut short to the latest 10 news items with the help of the 'Truncate' module, my first sub pipe is ready to go...

Merge of some official Yahoo! blog feeds filtered on messenger pipe

This second sub pipe (Merging Some Official Yahoo! Blogs with filter messenger) is used to pick up on any insider info regarding Yahoo! Messenger from various Yahoo! blogs. You can see I am simply fetching the feeds, filtering them for the search word "Messenger" in both the title and description before sorting them by date. Performing the filter first before the sort is more efficient as the pipe does not have to sort all the news items but only the very few filtered out. Again we cut the results down to the most recent 10.

Yahoo! Answers, Regex & Running our pipe

Our pipe is almost complete, the output from the sub pipes join the output from all our site feeds. The 'Regex' module is used to tidy up the output from one of our feeds. Bee gives up a lot of her time in helping others on Yahoo! Answers with Yahoo! Messenger related problems giving solutions, help and advice. She has accumulated a number of best answers for her advice and I wanted these to be included in our pipe. Searching around the Answers site for a RSS feed of this data proved futile (there are feeds for questions etc, but none I could find for best answers!?), So I hit the Answers developers site where I was able to form a search URL that would return what I wanted! :) ...well almost... The output was in the form of:

Resolved Question: how do i keep my real name from appearing on screen when I am instant messaging? (Answers: 1) (Comments: 0)

But I didn't want it to stand out as much from the rest of the content. Using the regular expression module I was able to modify this to display:

Answers: how do i keep my real name from appearing on screen when I am instant messaging?

using a simple replace for the words "Resolved Question:" (I should have expanded this to only look for this at the start of the title) and replacing "\(Answers: .\) \(Comments: .\)" with nothing to remove the additional text at the end. Regular expressions are definitely an advanced user module of Yahoo! pipes, used very simply in this example they can perform very complex operations. For more info about regular expressions see the wikipedia entry and this info site.

Output from running our pipe

The screen shot above is an example of the top few items returned from running our pipe. I've highlighted the source of each item so you can see the mix that has been compiled.

WackyB + Yahoo! Messenger & More pipe overview

Our pipe combines the news feeds from all our sites along with relevant news articles relating to Yahoo! Messenger. WackyB sites cover most of the instant messengers out there along with general PC help and advice and Mac news and info. WackyB.com also includes a lot more general current information along with the social side of WackyB at our forum's cafe. The main feeds of our pipe listed in order are:

WackyB.co.nz (Site feed)
WackyB.Com (Forums)
WackyB.co.nz (Forums)
bee.geek.nz (Site feed)
bee's answers on Yahoo Answers site (related to Yahoo Messenger problems)
WackyB.info (Site feed)
My sometimes answers on Yahoo Answers site (related to Yahoo Messenger problems)
Yahoo! Messenger blog
StickerCam (Forums)

They are combined with two other pipes:

1)

News feeds from Google and Yahoo news
filtered for "Yahoo Messenger"
filtered for similar Content analysis
filtered for same title
sorted by published date (chop off older news)
truncate at 5 (news is updated most often and I only want the most relevant otherwise our pipe will often list news items at the top)

2)

News postings from yahoo sources
a number of relevant feeds (not the yahoo messenger blog) filtered for messenger
truncate the latest 10

Using your pipe

Once you have created a pipe you can publish it! What this means is anyone can find your pipe from your pipes page or from searching.

WackyB's pipes page

You can also subscribe to your pipe with a number of predefined options available from the 'Subscribe' drop down list on the preview page:

Subscription options for pipe feed

Here you can grab the RSS feed to use in your favorite new reader, add to you Yahoo! page, or get updates and alerts over E-mail and messenger to name a few. The My Yahoo! option lets you add your pipe to your Yahoo! page, This is a great tool for personalizing your page just the way you want it, you could always pick which news feeds to display, now you can customise and transform the feeds you view any way you want:

My Yahoo! page with WackyB pipe feed added

 

One option that I really like is the E-mail Alert which lets you receive updates of new items almost when they happen or a daily summery:

If you run your own site or some blog sites (360/myspace/bebo/etc) you can insert the RSS feed to show up as part of your page. This is great for gathering news feeds keeping your site updated and active but also with relevant content. You can also add live news data to your site without having to run any server scripts, check out our live data feed [here...] with a link on how you can add this to your own site!

There are a couple of RSS news feed plugins available for Yahoo! Messenger that will keep you up to date with all the latest additions to your subscribed news feeds, I like the simplistic layout of the Yahoo! News plugin which has a few display options you can configure. You can also check out Basil plugin which has a few more options and alerts, both are shown below:

Yahoo! Messenger Plugins

You can also check out some of the many RSS converter sites out there that will transform a RSS feed into anything from HTML to a PDF file! Maybe we will see Yahoo! pipes adding some of these to their output options.

Problems, annoyances & beta versions
"Our Pipes are clogged! We’ve called the plumbers!"

Yahoo! Pipes is not without it's problems, there is certainly also room for some improvements but I'll come to that a bit later. At the time of writing this pipes is in beta release, I'm reasonably sure that it will stay this way for some time to come (years?) it appears to be a common practice with software/web companies these days to keep their products and services in beta release for as long as possible (Yahoo! Gallery, Yahoo! Mail, numerous Google services) I guess to excuse any problems or errors that may still be present and so avoid having to provide any help/support "It's still a beta product so you should expect some issues" ;)

There are some compatibility issues with older versions of some web browsers, but that can almost be expected considering the impressive interface they have constructed, basically you need the latest version of Internet Explorer (v7) but works well in Firefox (v1.5).

The biggest problem I've come across is incompatibility of the RSS feed generated with pipes; Maybe it's an issue with the pipes I've created, there is some discussion on the pipes forums where they are investigating some of these issues. I've had problems incorporating my pipe's feed into Yahoo's own 360 pages and the RSS2HTML site to name two...

I'm hoping a lot of the problems I'm seeing are simply my misunderstanding (took me ages to figure out how to add a description to my pipe!). Number of runs is still at 0 despite having run my pipe numerous times. Not sure how the search works? After publishing my pipe and despite adding all the appropriate tags I could think up at the time, I'm still unable to find my pipe when I search for it :(

One of my biggest annoyances with the editor is the open/close handles for the debug and modules library windows; I use them to resize the windows but of course as soon as I release the mouse button it closes that window pane as I should have dragged the window border....

There is no warning or notice about failed feeds, The Yahoo Answers feed appears to be the most fickle and will come and go as and when it pleases (It's a shame the Answers developers group appears to be dead).

Would be nice to be able to add comments to parts of the pipe; Even for the feeds I keep forgetting which source URL contains what news feed. When you are sharing your pipe for others to clone and modify it's also helpful to be able to explain the purpose of some modules (especially for the 'Regex' module). I've seen some pipes have an about box when you first open them but I've been unable to figure out how to put one of these in your own projects?

You can only add your own pipes as sub pipes... I can understand this as you can not edit other people's pipes, but it would be useful to be able to add other pipes as sub pipes in a read only state. This way when they update their pipe you will get the updates also. You could add their pipe output as a RSS feed but having the ability to open a sub pipe that does not belong to you as read only gives you a much greater understanding about how that feed is derived and builds a much better modular view.

High up on my wish list would be the ability to make your own modules, much in the same way you can add sub pipes. Being able to make modules that perform a certain function that you could just slot in each time you want it would be great. Sub pipes are different as they are still only a data source and do not allow other modules to link into them.

Cache refresh times... there is little reference to how often pipes will pull the live source feed, there is details on how to change the download rate of your own feeds but I think it would be great to be able to define this to some extent in your own pipes. Example is I would only like my Google & Yahoo news feeds to be download once a day as these feeds tend to change on a regular basis and I only want to be notified of new news each day.

Recursion & Circular reference

The ability to add arbitrary feeds as the source to your pipe introduces some interesting possibilities and undeterminable outcomes... For example, each pipe you create outputs it's own RSS feed; What if you were to use the output RSS feed of your pipe as it's own input? Or what about a longer chain of pipes each linked by RSS output feeds into each other's input and so eventually going full circle or as it's more commonly referred to: circular reference. This is not so difficult to imagine as Yahoo! pipes usage becomes more common, you could include a custom pipe feed into your own pipe which is then included as one of the sources for the original pipe feed you started with!

pipe with circular reference

The example pipe above illustrates this problem. The pipe takes two feeds as it's source; The first feed is the output of the pipe itself and the second a yahoo news feed which contains 10 items. Running this pipe produces an output of between 20 and 30 (apparently random as to if it's 20 or 30) I've highlighted in the screen shot above the repeated data.

What is apparent is that you don't get an infinite number of results returned! :) As to what mechanism Yahoo! pipes uses to control this is unknown to me; Maybe they use a loop counter.. maybe it's simply a maximum pipe execution time? maybe it's cache related in this simple example? Interestingly pipes allows you to add it's own pipe as a sub pipe... replacing the pipes own RSS feed input with it's self as a sub pipe produced between the original 10 and 20 items returned. If you then open this sub pipe you of course get the exact same pipe.. it appears you can continue doing this recursively for as long as you like or until boredom prevails :)

Recursive pipes

Summary

Pipes is still in early beta, but not that you would be able to tell that from the visual editor. New features, improvements and cool modules are being added all the time and the development team appear quite active the fact they are currently hiring is a good sign for its immediate future! (fancy a job as a plumber on your CV? ;) I'm happy with my pipe feed... it sits in my Yahoo! Messenger keeping me up to date with all the sites I would otherwise have to hop along to each time. True I could have done most of what I've achieved on my PC but I would never have gotten around to it. Yahoo! pipes is simple and accessible to use and it's visual editor invites you to play around without having to write a single line of code. I'm looking forward to it working correctly with more of the RSS readers and sites out there.

While playing around with pipes and hunting down more information about it I stumbled across a few other really cool sites that either have a little in common with pipes or work well with it. One such site that deserves a mention here is dapper "RSSify any website". The best way to check out what it does is to sit through the demo, it basically lets you extract content from sites that do not have a RSS or similar feed available.

Both pipes and dapper are both currently free to use... There are hints on both sites that this will change for high volume users. I wonder if in future versions we will start to see advertisement items inserted into your pipe's output? ;)

So... you can Put that in your pipe and smoke it! :)

- Andy :o,

Live output

In the table below should appear live data from our WackyB pipe (latest 10 items). Thanks to Hunlock.com for this script demonstrating the use of JSON output from your pipes!