Images in this post missing? We recently lost them in a site migration. We're working to restore these as you read this. Should you need an image in an emergency, please contact us at imagehelp@codebetter.com
Project Failures - Blame the Process, or the People?

Prompted partially by some comments yesterday on my post on How to Make Late Software, Even Later, and partially by a discussion on the altdotnet Yahoo list, I wrote this long email. As it became an epic in its own right, I thought it deserved a blog.

We Are Doing Agile!

I can claim to be doing Agile, and following with some degree of conformance, a known and understood Agile methodology. If my project fails, we must blame the Agile process, and our practices in some way for this

I can claim to be doing Agile, but make up my own variation of a known methodology, in which case a failure of the project is largely encumbant upon me for deciding to sacrifice the practices we chose to ignore, and the new ones we added.

I can claim to be doing Agile while just coding hell for leather and following no methodology, just cherry picking a couple of practices here and there, and picking the principles I like - in which case any failure is mine to bear.

All of these approaches should be highlighted in a risk analysis to management to make the decisions *before* you decide to pursue any of these paths. It is the decision of management to weigh risk vs reward and to make that decision. If I didn't write that report, or I wrote it badly, then again the fault is mine to bear. If I wrote it clearly expressing the dangers of a hack and see approach, and management chose that path, then it is their cross to bear.

You Cannot Blame A Process For Producing Bad Results

If all participants in a waterfall methodology like RUP followed the practice rigourously, then I have little doubt the project would have a high chance of success (as defined by the process, a close match to the intial requirements)
 
If all participants in an Agile process like XP followed the practice rigourously, then I have little doubt the project would have a high chance of success (as defined by by the process, a close match to the evolving requirements)

Bad people on any project can stuff any methodology, and bad application of good practices will always result in failure. Agile just gives you much earlier feedback that you are not getting it right - how you choose to interpret that feedback, or how management choose to, is the fault of those communicating or interpreting the information - not of the process.

How To Fail Under Agile

The worst, and I mean absolute worst project failure I ever worked on was done using Agile, using Extreme Programming, and the practices were fairly closely followed. It should have been a success. And for a few small but critical mistakes it would have been.
 
The first big mistake was not evolving the architecture first. The application had been developed under XP for over 9 months when I was brought in the provide the "Enterprise component" of the application. It became rapidly apparent that the application had been written as a desktop applciation, and was too tightly coupled to allow any extension points for an evolution of the functionality into a different UI and service layer.

The second big mistake was a large amount of code that had been written to allow the client applciation to operate on domain objects, using an ORM, persisted across a web service layer, relying on massive amounts of code generation. This was a technical mistake that should have been picked up by prototyping and getting the architecture right first.

The third big mistake was a very poor application of the XP process. The bug backlog for each week was continual, and continally increasing. It was seen as acceptable that stuff didn;t work, and in each stand up it was just refered to as a bug and would go on the backlog. Nobody ever stopped to take a grip of this, and realise that this was a warning sign. As complexity was increaing, the number of interactions in the software was going up rapidly, and people spent more time bug fixing. Out of around 5000 unit tests (afair) that existed when I got there, around 2000+ were failing - this was explained as changes to the code and hence they couldn't get them working. This invalidated pretty much the entire test suite.

These were not problems with the process (XP), but clear problems with no up front design or prototyping (point one), no technical understanding of the challenges involved (point two), and people badly applying the process (point three).

Was Agile or Extreme Programming to Blame?

Yeah, we know loads of it is in flux, and it is a bit 'out there' but this is Agile man, we live by the seat of our pants, we ride fast and hard, this is EXTREME!!!!!

In the respect that it allowed developers to go to senior management and say this, hell yes Agile and XP were to blame. It gave the excuse to developers that they could excuse stuff being chaotic, as being all part of the way this stuff works.

The reality was the project failed due to a manager that for all his ability, had never done Agile or XP before, and had taken to quoting Martin Fowler PoEAA so often when we tried to raise problems, that his copy was starting to fall apart. And the business owner took the Agile/XP no requirements thing to the Nth degree and would change his mind on features on a daily basis, so the team could never get a handle on the problem. And lastly the team, that had gone into a terminal march towards inevitable failure, but had never spoken up loudly enough.

Oddly enough, that was the single team I have worked on that had by far the highest technical ability of any team as a whole I have ever worked with. They had some absolutely superb developers - but sometimes even that isn't enough.


Posted 05-09-2008 8:15 AM by Jak Charlton

[Advertisement]

Comments

Zdeslav Vojkovic wrote re: Project Failures - Blame the Process, or the People?
on 05-09-2008 5:12 AM

"and had taken to quoting Martin Fowler PoEAA so often..."

This reminds me a little of how many discussions on DDD group end up pointing to what Evans has said in The Book, sometimes even beyond the common sense or without taking into account the specifics of the original problem being discussed. Like that scene from 'Finding Nemo':

- Nemo: How old are sea turtles?

- Marlin: Sea turtles? I don't know.

- Nemo: Sandy Plankton from next door, he says they live to be a hundred years old.

Kasper wrote re: Project Failures - Blame the Process, or the People?
on 05-09-2008 6:37 AM

Very good post. But I'd like you to expand on the following:

"...large amount of code that had been written to allow the client applciation to operate on domain objects.."

Shouldn't the client application work with  a domain model?

Wojciech Gebczyk wrote re: Project Failures - Blame the Process, or the People?
on 05-09-2008 7:02 AM

Hmm...

In one sentence "And for a few small but critical mistakes" in the next one "The first big mistake ". So the mistakes were big or small? How small mistake can be critical? strange...

Jak Charlton wrote re: Project Failures - Blame the Process, or the People?
on 05-09-2008 7:03 AM

@Kasper

In this particular case, the ORM tool created and maintained it's own domain objects. these had properties for persistence, and so a decision had been taken at some point to use codde generation as part of the build process to create duplicate classes for all these entities, and to rehydrate them from a web service call. The web service call had then been expanded to be a unit of work, containing these domain entities.

Essentially a lot of code existed to keep two copies of the domain model - one on the client (in a multi user system), and one on the server. Then there was a lot of code to reconcile these.

The design was technically nuts. In hindsight it was nuts, at the time I was brought in it was more than highly suspect. Perhaps when it was first conceived (which was when alternatives like NHibernate were pretty immature) this was a "good idea" but it rapidly turned out to be a bad decision.

We started towards a new reference implementation that passed data transfer objects, but in the end we made a decision to terminate the project as too much money had been spent, with little return.

Jak Charlton wrote re: Project Failures - Blame the Process, or the People?
on 05-09-2008 7:17 AM

@Wojciech

You can make a small mistake that turns out to be critical ... I could forget to lock my car when I leave it (small mistake) resulting in my car being stolen, containing my wallet, resulting in my identity being stolen, resulting in my bankruptcy (critical failure)

So a small but critical mistake, turns out to be a big mistake :)

Dew Drop - May 9, 2008 | Alvin Ashcraft's Morning Dew wrote Dew Drop - May 9, 2008 | Alvin Ashcraft's Morning Dew
on 05-09-2008 8:50 AM

Pingback from  Dew Drop - May 9, 2008 | Alvin Ashcraft's Morning Dew

Grimbeorn wrote re: Project Failures - Blame the Process, or the People?
on 05-09-2008 11:04 AM

In my experience, it's the people that make good software, not the process.  One can't cookie-cutter software development into a process that anyone can use to write software (though I've seen it tried in more places than I'd care to admit).

A process can only streamline (or hamper) software development.  It usually doesn't make it any better or worse.

Neil Weber wrote re: Project Failures - Blame the Process, or the People?
on 05-09-2008 12:34 PM

RUP is not a waterfall process!  It's an iterative process.

In your last post, you mentioned that you do your architecture up front.  In this post, you advocate doing a design up front.  Neither is Agile.  The whole point of Agile is to get a working application to the customer early for feedback.

SG wrote re: Project Failures - Blame the Process, or the People?
on 05-09-2008 6:35 PM

99% Most the time the problem is communication

A failure to communicate & lack of revealing the truth problems of the project to the customer.

Processes don't replace good honest leaders.

Agile is the new waterfall.  You MUST test, you MUST do this, you MUST do that.

As soon as Agile goes mainstream it loses it's agile nature and becomes a new bail and chain.

Jak Charlton wrote re: Project Failures - Blame the Process, or the People?
on 05-10-2008 12:40 AM

@Neil

>>>RUP is not a waterfall process!  It's an iterative process.<<<

No, it really is waterfall, it certainly isn't Agile, and it's iterative nature does not provide the proper feedback loop that iterative implies. In fact, RUP has it's own anti pattern, mini-waterfall.

>>>In your last post, you mentioned that you do your architecture up front.  In this post, you advocate doing a design up front.  Neither is Agile.  The whole point of Agile is to get a working application to the customer early for feedback.<<<

Wrong, wrong WRONG!

Agile does not in any methodology dictate you don't design up front. You don't design up front, you *will* fail - in any methodology. This is where I suspect you haven't encountered any properly run Agile projects (and I agree most are very badly run)

Agile design up front is different ... a large proportion of the design is done in code (prototypes, spikes), a large amount is done in unit tests, and a large amount is done on whiteboards. There is some written design too.

Waterfall design (read RUP here) is done on paper, *before* coding commences, any coding. The RUP concept of prototyping here does not have any feedback loop to those doing the design, it si only for technical evaluations. The design is fixed before the developers get to code it.

Agile design is done while coding, and allows business involvement directly back to the developers, not to a layer of analysts writing UML

Regardless, as the title of the post implies, and the post clearly says... Waterfall done right would succeed in its objectives, as would Agile done right ... the only difference is the level at which they provide feedback, and allow evolution of the requirements.

Jak Charlton wrote re: Project Failures - Blame the Process, or the People?
on 05-10-2008 12:42 AM

@SG

Bang on the mark - people are really the problem, communication is a massive problem.

Agile is far from the new watwerfall, but if it is imposed the way you describe, then it certainly has a good chance of failure too.  I would say that these days, Agile is the new mainstream, but that doesn't mean the people doing it are doing it right, or well - mostly as the great majority of them have no experience running Agile projects.

redgreenrefactor wrote re: Project Failures - Blame the Process, or the People?
on 05-11-2008 4:04 PM

Sorry but RUP is not waterfall. There is no BUF  The use cases are only identified during inception and elaboration. In the construction the use cases are further designed in detail, coded, tested and delivered to the client. If you think RUP is waterfall you didn't investegate it enough and don't understand it. If you don't let rup intimidate you rup is agile. Otherwise you're just covering you're bud.

Neil Weber wrote re: Project Failures - Blame the Process, or the People?
on 05-12-2008 3:17 PM

RUP's prototypes certainly do have a feedback loop.  The prototypes are shown to the business representatives before coding starts.  How could that not be feedback?

A number of people have called you on your claim that RUP is not waterfall.  Me thinks you either need to revise your thinking or back up your claim with a clear argument.

In addition, you've wisely stated that some projects should not use Agile.  Wouldn't you agree that RUP is a suitable methodology for those projects?

Jak Charlton wrote re: Project Failures - Blame the Process, or the People?
on 05-13-2008 11:19 AM

@redgreenrefactor

We will have to disagree then ... RUP is in my mind a waterfall approach, it has every quality required to be very anti-Agile. Notably you run in mini waterfall processes, that I have only ever seen done in iterations of upwards of 3 months.

Agile is a mental attitude more than a written specification.

You could probably be Agile while doing RUP, but the process will not support you. IN an Agile process such as XP or SCRUM, the feedback is nearly instant.

Jak Charlton wrote re: Project Failures - Blame the Process, or the People?
on 05-13-2008 11:25 AM

@Neil

>>>RUP's prototypes certainly do have a feedback loop.  The prototypes are shown to the business representatives before coding starts.  How could that not be feedback?<<<

In Agile you don;t only show them the prototypes, you show them the real working code, as you are working on it. You don;t get an agreement on prototypes and use cases, then go off on a flight of fancy for a while to present some finished product. RUP with an iteration of 2 weeks or less, could claim to be Agile, anything more and it is not.

>>>A number of people have called you on your claim that RUP is not waterfall.  Me thinks you either need to revise your thinking or back up your claim with a clear argument.<<<

We will have to agree to disagree. I have only worked on two RUP projects, and been close to maybe 3 or 4 more ... but all were very much waterfall, and none had the mental attitude that comes from an Agile process.

>>>In addition, you've wisely stated that some projects should not use Agile.  Wouldn't you agree that RUP is a suitable methodology for those projects?<<<

Well either RUP is Agile or it isn't ... :)

Failing that, the kind of projects Agile shouldn;t be involved in, to be honest shouldn't even get off the starting block - under Agile, Waterfall, or an in between process like RUP

Not having user involvement, not having clear project ownership, not having clear goals, not having a good quality team, etc etc etc are recipees for disaster regardless of your process.

To repeat myself - Agile processes just let you fail faster and get feedback earlier ... I think the point of my post was that actually, you can do Agile well or Waterfall well - or you can do both badly. Or to answer the title of the post - the problem with software development is people, not processes.

redgreenrefactor wrote re: Project Failures - Blame the Process, or the People?
on 05-13-2008 12:25 PM

Casey,

I agree with you on processes stand or fall with the people following them (not). Funny thing is that in all kinds of production processes people follow guidelines and processes. Somehow in our business you have a lot of introvert people who don't give a toss about processes, estimates, deadlines etc. as long as they can program, program, program and ............be left alone.

Jak Charlton wrote re: Project Failures - Blame the Process, or the People?
on 05-13-2008 2:55 PM

>>>etc. as long as they can program, program, program and ............be left alone.<<<

Many developers became developers because they are by nature insular ... in a former time they would likely have become academics where they could stay locked away from the world ... not so long ago, being a developer WAS a solo activity, now it is very firmly a team sport. These are the people that find it hard to adapt to a changing world.

Paul Rayner wrote re: Project Failures - Blame the Process, or the People?
on 05-14-2008 12:34 AM

Is anyone here familiar with Scott Ambler's agile extensions to RUP (see www.enterpriseunifiedprocess.com and www.ambysoft.com/.../agileUP.html)? The AUP and EUP variations specifically break the construction phase into iterations and promote iterative releases over time.

E K G wrote re: Project Failures - Blame the Process, or the People?
on 09-30-2009 7:58 PM

This sounds EXACTLY like the same aweful project I just left. Couple an "Agile" implementation with serious budget and resource constraints and I wanted to jump off a cliff! All in all this was one of the most amazing learning experiences for me - it was like being in the school of EXTREME project failure. We joked about it because it wasn't a process at all - it was a mob mentality! And a lot of things happen when projects turn into chaotic mobs like downward political pressure. Towards the end, no one cared to speak up anymore because there was a tendency for management to put intense pressure on people and renetlessly question them on the standups and then assign them to do more of the work that they couldnt even handle. The testing lead was so strapped that he would just lie to everyone and say testing passed. We all (new graduates) had these WHOPPING outrageous tasks like "onboard all departments in the organization." It was pure madness. I'm in absolute shock that I survived that.

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)