When setting up the DDD series on http://dddstepbystep.com I wanted to bring it various information from across the web, to try and consolidate a lot of Domain Driven Design info all in one place.
My first thought was I would setup a blog, and whenever I found something that I thought was a good fit, I would copy a summary to the new site, and link back to the original piece. This was a great idea, apart from the fact that it was going to turn out to be horribly time consuming.
The next step was finding a better way of doing this, hence one of the reasons I hose to host the new site in Community Server – it has a “Mirrors” setting for each blog, where you can add any number of RSS feeds, and it will bring in a summary for you every time the job runs.
After a lot of messing around getting CS set up, I found one of the major limitations of CS is the “Mirrors” functionality … although it is good at actually bringing in the feeds, it has no way of filtering, combining or correlating feeds. It also has an arbitrary way of detecting duplicate entries, which appears to be based solely on the title – which is less than simplistic.
After a bit of searching and trying various options, I came across Yahoo Pipes – this has got to be one of the coolest tools on the web right now.
With Pipes I was able to set up a complex set of feeds, followed by a series of transformations that allowed me to sort out all the discrepancies that CS was unable to deal with.
Firstly I was able to combine half a dozen feeds of blogs I know regularly have great information on DDD subjects. This was a simple matter of giving the Fetch Feeds source the URL of each RSS.
Then I needed to filter out any blog posts by myself – as one of the feeds is coming from devlicio.us I didn’t want to duplicate my posts on each site, as I post the full blog directly to the new site as well as here. With a quick drag and drop of the Filter operator, my posts were gone.
Next I wanted to remove all the posts that didn’t mention DDD or Domain Driven Design – another Filter in place and I was closer to what I needed.
The Unique operator allowed me to then remove any duplicate posts based on their content – the CS one would have allowed duplicated content if the title was different.
Next I wanted to add the original author name into the title … to turn “This is my Blog Post” into “This is my Blog Post by Dave D Spatch”. Things then got a little tricky. As I was bringing in feeds from various sites, and they were inconsistent in their naming, the author fields were often coming through in the wrong place. Using a combination of the Rename and Regex operators allowed me to sort this out and provided me with a much better title. A last Regex allowed me to replace a number of “username” type fields with the full names of those I knew – so for example I replace “colinjack” with “Colin Jack”.
And finally – we get a single feed that I pass on to Community Server for it to import. The last step was a simple cosmetic change, I changed the URL from some horrible id string, a nice human readable one: http://pipes.yahoo.com/dddstepbystep/ontheweb
All of which is now brought in correctly to the blog at: http://dddstepbystep.com/blogs/ontheweb/ where I can still manually add anything of interest the feed has missed.
I wanted to also bring in the feed from the domaindrivendesign group at Yahoo – where a lot of general discussion around DDD takes place.
This had some similar elements to the other Pipe, but also required me to work around the limitation of Community Server I mentioned earlier – it detects a duplicate item based on the title alone – and skips it if it finds an existing match.
As the DDD Yahoo group has a lot of “Re: Your Subject” titles, CS discards all but the first it finds if you use the RSS directly.
Pipes again allowed me to resolve this, by appending the author to the beginning of the title, and adding the time to the end to the title, I can get almost uniqueness. The title looks a bit ugly, but I couldn’t find a better way to fool CS into doing it right.
Yahoo Pipes is a superb tool – it can be a bit tricky to understand at first, and some things still seem convoluted to me … but it has allowed me to work around some tricky limitations with relative ease.
If you find a need to make your RSS feeds just a little better – then take a look. Better still, it can work with data from all over the web, and turn that into an RSS feed for you.
del.icio.us Tags: DDD
03-06-2009 3:11 PM