Images in this post missing? We recently lost them in a site migration. We're working to restore these as you read this. Should you need an image in an emergency, please contact us at imagehelp@codebetter.com
Make Your Own RSS Feeds With Yahoo Pipes

When setting up the DDD series on http://dddstepbystep.com I wanted to bring it various information from across the web, to try and consolidate a lot of Domain Driven Design info all in one place.

My first thought was I would setup a blog, and whenever I found something that I thought was a good fit, I would copy a summary to the new site, and link back to the original piece. This was a great idea, apart from the fact that it was going to turn out to be horribly time consuming.

The next step was finding a better way of doing this, hence one of the reasons I hose to host the new site in Community Server – it has a “Mirrors” setting for each blog, where you can add any number of RSS feeds, and it will bring in a summary for you every time the job runs.

After a lot of messing around getting CS set up, I found one of the major limitations of CS is the “Mirrors” functionality … although it is good at actually bringing in the feeds, it has no way of filtering, combining or correlating feeds. It also has an arbitrary way of detecting duplicate entries, which appears to be based solely on the title – which is less than simplistic.

After a bit of searching and trying various options, I came across Yahoo Pipes – this has got to be one of the coolest tools on the web right now.

With Pipes I was able to set up a complex set of feeds, followed by a series of transformations that allowed me to sort out all the discrepancies that CS was unable to deal with.

The “DDD on the Web” Blog Feed

Firstly I was able to combine half a dozen feeds of blogs I know regularly have great information on DDD subjects. This was a simple matter of giving the Fetch Feeds source the URL of each RSS.

Then I needed to filter out any blog posts by myself – as one of the feeds is coming from devlicio.us I didn’t want to duplicate my posts on each site, as I post the full blog directly to the new site as well as here. With a quick drag and drop of the Filter operator, my posts were gone.

Next I wanted to remove all the posts that didn’t mention DDD or Domain Driven Design – another Filter in place and I was closer to what I needed.

The Unique operator allowed me to then remove any duplicate posts based on their content – the CS one would have allowed duplicated content if the title was different.

Next I wanted to add the original author name into the title … to turn “This is my Blog Post” into “This is my Blog Post by Dave D Spatch”. Things then got a little tricky. As I was bringing in feeds from various sites, and they were inconsistent in their naming, the author fields were often coming through in the wrong place. Using a combination of the Rename and Regex operators allowed me to sort this out and provided me with a much better title. A last Regex allowed me to replace a number of “username” type fields with the full names of those I knew – so for example I replace “colinjack” with “Colin Jack”.

And finally – we get a single feed that I pass on to Community Server for it to import. The last step was a simple cosmetic change, I changed the URL from some horrible id string, a nice human readable one: http://pipes.yahoo.com/dddstepbystep/ontheweb

All of which is now brought in correctly to the blog at: http://dddstepbystep.com/blogs/ontheweb/ where I can still manually add anything of interest the feed has missed.

The “DDD at Yahoo” Blog Feed

I wanted to also bring in the feed from the domaindrivendesign group at Yahoo – where a lot of general discussion around DDD takes place.

This had some similar elements to the other Pipe, but also required me to work around the limitation of Community Server I mentioned earlier – it detects a duplicate item based on the title alone – and skips it if it finds an existing match.

As the DDD Yahoo group has a lot of “Re: Your Subject” titles, CS discards all but the first it finds if you use the RSS directly.

Pipes again allowed me to resolve this, by appending the author to the beginning of the title, and adding the time to the end to the title, I can get almost uniqueness. The title looks a bit ugly, but I couldn’t find a better way to fool CS into doing it right.

In Conclusion

Yahoo Pipes is a superb tool – it can be a bit tricky to understand at first, and some things still seem convoluted to me … but it has allowed me to work around some tricky limitations with relative ease.

If you find a need to make your RSS feeds just a little better – then take a look. Better still, it can work with data from all over the web, and turn that into an RSS feed for you.

del.icio.us Tags: ,,

Posted 03-06-2009 3:11 PM by Jak Charlton
Filed under: ,

[Advertisement]

Comments

MisterPi wrote re: Make Your Own RSS Feeds With Yahoo Pipes
on 03-07-2009 4:19 AM

I use Yahoo Pipes for a long time now and I can say that it's very useful and powerful !

Jan Van Ryswyck wrote re: Make Your Own RSS Feeds With Yahoo Pipes
on 03-07-2009 7:34 AM

IF you're aggregating different blog feeds, then you probably alos want to include the blog of Yves Goeleven:

www.goeleven.com/default.aspx

He is a frequent blogger about DDD.

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)