Monday, January 5, 2009

Some Personal Thoughts on Agile Development, Part 2

In Part 1, I talked about how I learned that the concepts of agile development could be effective, while I was at CCA as a programmer. But how does agile development achieve consistently improved results, as I found in the survey I mentioned? Well, I found some clues when I went to work for Prime Computer, right after CCA.

Prime was the first time I got to deal with project leadership in anything but a one-person project (except a 3-week stint at CCA). Prime was a very different company – thousands instead of 40-100 people, and therefore 40 software-development-related people in my own corner of the company out of more than a thousand; plus, Prime was really still a company where the software was aimed at selling the hardware. So I not only learned about development from a much broader and managerial perspective; I also learned about large organizations’ overall attitude towards development.

Insight # 1: Theoretically, doing a Gantt chart based on your experience and measuring yourself on being on time, within budget, and doing all the functionality is the way to go; in the real world, it has little relation to success. Here’s my story; I set out a schedule and sent it to my boss; in order to fund it, he figured out how to reschedule, and cut the time and costs by 25%; he sent it to his boss, who in order to sell it to the company head, did the same thing. So my best guess of how long it would take was sliced by almost ½. When I pointed out that this was probably not going to happen, my bosses reassured me that they understood that the numbers were optimistic – and then expressed concern during the project when things fell behind their schedule. I have seen this frequently enough since that I have cockily created Kernochan’s Law: Multiply the estimated time for a project by 4/3 ** (number of levels of management review) to get a realistic number.

Insight #2: There is no developer career path. In Prime’s case, after a certain level you could either be a marketer or a “Principal Software Engineer”. From that point on, if you stayed on the developer hierarchy, you would either be an advisor thinking grand theoretical thoughts or a manager; but as an advisor you would not expand your say in the non-technical aspects of the product, and as a manager you would move out of the developer career path. Again, this is a pattern I frequently see elsewhere: doctors doctor and also manage the hospital; physicists run biotech shops and also direct the research; software developers rarely are allowed to combine the technical and marketing aspects of product design as part of their career path, nor are they trained to do so.

What does this have to do with getting consistent results from agile development? Well, if you think about it, the number of levels of managerial review in agile development is effectively zero: it’s just you the developer and the end user for every change. Moreover, you’ve opened yourself to changes in the spec; so who knows what the realistic estimate for project end should be? And yet, when you do this, you look back at the end of the project, and by any realistic measure, you’ve actually done things faster – even though each change should mean you’re falling further and further behind your original budget. Agile will do consistently better in all realistic metrics, like development cost compared to what the same product should have cost by the old process, because it does not manage nor measure by flawed numbers and incentives.

Moreover, agile will also do consistently better because it implicitly gives good developers more of a professional career path. At the same time you think you are madly asking for intense programming spurts, you are actually asking for lots of programmer input into the non-technical design of the product, using the end user as an excuse. The reason the user is more satisfied is not just because he or she is listened to more; it is also that the programmer is combining technical and non-technical aspects in program design, making a better combined product. That doesn’t affect costs, except costs to retrofit; it does affect return on investment, customer satisfaction, and the “agility” of the resulting product (because some of the non-technical reasons for a new version have already been dealt with).

All right, so my experience tells me that agile development is a great idea and will deliver consistent results. Is it enough? As the British say, not by a long chalk. In my Finale, I explore what my personal story tells me is missing.

No comments: