Food for Thought: Agile Methods - Beyond the Hype 
An e-newsletter published by Software Quality Consulting, Inc.
July/August 2005, Vol. 2 No. 7

To view a web version of this newsletter, click on the following link:
http://www.swqual.com/newsletter/vol2/no7/vol2no7.html.

--------------------------------------------------------------------------------

Welcome to Food for Thought(TM), an e-newsletter from Software Quality 
Consulting (http://www.swqual.com/index.html?Intro). I've created free
subscriptions for my valued business contacts. If you find this newsletter
informative, I encourage you to continue reading. Feel free to pass this
newsletter along to colleagues by clicking th Forward Email link at the bottom
of this email. If you�ve received this newsletter from a colleague and would
like to subscribe, please click this Enter New Subscription link
(http://www.swqual.com/newsletter/Subscribe.htm). If you don't wish to receive
this newsletter, click the SafeUnSubscribe(TM) link at the bottom of this
newsletter, and you won�t be bothered again.

Your continued feedback on this newsletter is most welcome. Please send 
your comments and suggestions to [email protected].

--------------------------------------------------------------------------------

In This Months� Topic, I discuss the raging debate over Agile Methods...

Regular features to look for each month are:

- Monthly Morsels
  Hints, tips, techniques and reference info related to this month�s topic 

- Calendar
  Conferences, workshops, and meetings of interest to software engineers, 
  QA engineers and anyone interested in software development 

--------------------------------------------------------------------------------

*** Agile Methods - Beyond the Hype***

Agile Methods came to the forefront with the publication of the so-called 
Agile Manifesto [1] in 2001. Not since Dijkstra�s [2] classic paper �Go To 
Statement Considered Harmful� in 1968 where he stated �... the go to 
statement should be abolished from all �higher level� languages...� has 
there been such an impassioned debate about software development 
methodology. There has been so much hoopla about the perceived benefits of 
�lightweight� vs. �heavyweight� development processes that camps have 
formed on both sides of this issue, each with their own zealots and their 
own books...

I distinctly remember reading the �manifesto� in 2001. It was the first 
technical article I ever read that elicited a visceral reaction. It made 
me mad. My reaction was something like this:

  �What were these guys doing? We have been working so hard to improve the 
  perception of software engineering and instill discipline in developers 
  and managers. And then this... Are they nuts?

  And why would anyone use the word manifesto in an article on software 
  development?� 

The term �manifesto� evoked visions of the �Communist Manifesto� [3] 
written in 1848 by Karl Marx and Frederick Engels � the acknowledged 
beginning of the Communist Movement. Here is a short snippet from an 
introduction by G. S. Jones [3]:

  �... the Manifesto sketches a vision of reality that, at the start of a 
  new millennium and against a backdrop of endless chatter about 
  globalization and deregulation, looks as powerful and contemporary a 
  picture of our own world as it might have appeared to those reading it 
  in 1848.�

Wow. I immediately started writing a letter to the editor of IEEE 
Computer. Something I had never done before. My frustration was evident in 
my letter:

  �Please excuse my cynicism. But I�ve been waiting a long time for 
  software engineering to become a respected engineering discipline. While 
  progress has been made, it has been painfully slow and frequently 
  impeded by people within the software community.� [4] 

When the December 2001 issue of IEEE Computer, I was surprised to see my 
letter was selected for publication. Good, I thought, someone needs to 
speak up for process! I received a few e-mails in response to the letter, 
all supporting my views.

THE FOUR VALUES

Agile Methods are a collection of several methodologies that include:

- Extreme Programming (XP) 
- Crystal 
- Scrum 
- Dynamic Systems Development 
- Adaptive Software Development 
- Lean Software Development 

While each is different, they all share common values as expressed by the 
�manifesto�. To refresh your memory, the essence of the �manifesto� are 
the four values: 

  �We are uncovering better ways of developing software by doing it and 
  helping others do it. Through this work we have come to value:

    - individuals and interactions over processes and tools, 
    - working software over comprehensive documentation, 
    - customer collaboration over contract negotiation, 
    - responding to change over following a plan. 

  That is, while there is value in the items on the right, we value the 
  items on the left more.� [1] 

In the subsequent �endless chatter� about Agile Methods, I unwittingly 
became the poster boy for the process camp. I was mentioned in an IEEE 
Computer article written by Barry Boehm and Tom DeMarco: [6]

  �Those on the pro side�Jim Highsmith and others�argued that the current 
  approaches are broken and should be replaced. Those on the con 
  side�Steven Rakitin, for example�said that agile approaches are just 
  hacking by another name and that we shouldn�t abandon our disciplined 
  processes now that we�re finally getting them right.�

And a sidebar to another IEEE Computer article written by Barry Boehm [5] 
stated that:

  �Although their advocates have used each of these agile methods 
  successfully in practice, critics who prefer process-based methods 
  remain skeptical. In a letter [4] to Computer, Steven Rakitin indicates 
  that, in his experience, the items on the right are essential, while 
  those on the left serve only as easy excuses for hackers to keep on 
  irresponsibly throwing code together with no regard for engineering 
  discipline. He provides �hacker interpretations� that turn agile value 
  propositions such as �responding to change over following a plan� into 
  chaos generators. Rakitin�s hacker interpretation of �responding to 
  change over following a plan� is roughly �Great! Now I have a reason to 
  avoid planning and to just code up whatever comes next.� �

So much for my five minutes of fame.

A SHORT HISTORY LESSON...

The publication of Dijkstra�s article on go to statements [2] in 1968 led 
to what has come be known in software engineering lore as the Methodology 
Wars. (See [7]). People like Ed Coad, James Martin, Ed Yourdon, Tom 
DeMarco, Michael Jackson (no, not that Michael Jackson), and others 
developed and then began pushing their own software development 
�methodology�. Each of these gurus had their own cult following and 
detractors.

  They all promised, �if only those lazy, good-for-nothing programmers 
  would adopt their methodology, all of our problems would be solved�! 
  (Does any of this sound familiar?)

The Three Amigos  The advent of �structured methodologies� gradually 
evolved into Object-Oriented (OO) methods, mostly due to the work of the 
infamous Three Amigos (Grady Booch, Ivar Jacobson, and James Rumbaugh).

So, to continue our brief history, OO begat UML which begat RUP (Rational 
Unified Process), which begat lots of money for Rational from books, Case 
Tools, and training. For a more complete view of the history of software 
development, visit Wikipedia
(http://en.wikipedia.org/wiki/Software_engineering).

The timeline below is my attempt at putting the Methodology Wars in 
perspective...

Software Development Methodology Timeline

[for timeline image, see the HTML version � 
http://www.swqual.com/newsletter/vol2/no7/vol2no7.html]

As you can see, many methodologies have come and gone over the past three 
decades. Having lived through all of these, I�ve become a bit cynical when 
yet another methodology is purported to be the Silver Bullet because we 
should remember what Fred Brooks said � there is no Silver Bullet. [8]
 
In looking back over the past three decades, I believe there have been 
cases where each methodology shown above has performed well and cases 
where each has failed miserably. This is true for every methodology from 
Agile Methods to CMM and everything in between. 

WHY HAS THERE BEEN SO MUCH HYPE? 

One of the things I believe the Agile zealots learned quickly was that 
there�s a lot of money to be made as a result of publishing books on 
methodologies. Today, amazon.com lists at least 27 books with �Agile 
Methods� in the title! Anyone remotely affiliated with an Agile project it 
seems has written a book on the virtues of Agile Methods.

The C3 project at Chrysler
(http://c2.com/cgi/wiki?ChryslerComprehensiveCompensation) was the first large
project where many Agile principles and methods were applied collectively and
intentionally. The small team working on C3 included Kent Beck, Ron Jeffries,
Chet Hendrickson and others. What is not so well known is that the project was 
a failure (http://c2.com/cgi/wiki?CthreeProjectTerminated).

In spite of the fact that the project failed to deliver what the customer 
expected...

  �By the time C3 was cancelled, XP was well on its way to the phenomenon 
  that it is today... Beck, Jeffries, Hendrickson and others had book 
  contracts in hand.� [9]

I am constantly amazed by the ability of those of us in the software 
industry to put positive spins on projects that would otherwise be viewed 
as dismal failures.

Consumer advocates constantly remind us that:

  If it sounds too good to be true, it probably is.

This applies to telephone solicitation schemes as well as software 
development methodologies.

MY BOTTOM LINE...

Agile Methods can work given a specific set of circumstances. In my 
opinion, those circumstances are projects where:

  - software is being developed for use solely within the company 
  - a real, live �customer� (end user) is available to the project team 24x7 
  - the risk of failure is low 
  - the project team is highly motivated and led by an excellent leader 
  - long term maintenance of software is not a concern 

Under these circumstances, I believe Agile Methods may work well. But so 
will other methods. And this is the source of my frustration with all of 
the hype about Agile Methods. Agile Methods can be effective but so can 
other methods. 

For projects that don�t meet the circumstances above, read what Roger 
Pressman observed:

  �I contend that software engineering principles always work. It�s never 
  inappropriate to stress solid problem solving, good design, and thorough 
  testing (not to mention the control of change, an emphasis on 
  quality,...). A specific software process might fail because it is 
  overkill, or the work products it requires are unnecessary or 
  burdensome, or a person or team becomes overly dogmatic in the 
  application of the process. But history is on the side of a solid 
  engineering approach.� [10]

And for projects that are high-risk or life-threatening, read what Mark 
Paulk said:

  �Should organizations use XP, as published, for life-critical or 
  high-reliability systems? Probably not. XP�s lack of design 
  documentation and de-emphasis on architecture are risky.� [11]

Those Agile Methods like XP that rely on verbal communications rather than 
written documents appeals to some people. The people this doesn�t appeal 
to are sustaining engineers. I pity the poor slobs who have to maintain 
applications developed using Agile Methods. With no design documentation 
or requirements to refer to, you�re left with only the code. I contend 
that even incomplete or outdated design documentation is better than no 
documentation. How many times have you seen cases where maintenance is so 
difficult because of a lack of design documentation that we decide it 
would be easier to start over than to figure out how the existing code 
works?

Like the CMM, many Agile Methods include practices that can be applied 
regardless of methodology. For example, test first design is an XP 
practice. Test first design is definitely a good thing to do whether you 
are using an Agile Method or not.

Deciding what methodology to use is a difficult task. Before committing to 
a specific method, become as informed as you can on both the advantages 
and disadvantages. Remember, there is no silver bullet. Developing complex 
software is a difficult and challenging task. There are many pitfalls 
along the way. Use what will work best for the situation you are in. 
Leverage the skills of the people that will be working on the project. And 
most of all, do what makes sense.

That�s my story and I�m sticking to it...

This issue was inspired by a spirited debate titled �Agile Methods 
Considered Harmful?� held at a monthly meeting of the Software Quality 
Group of New England between myself and my friend and colleague Scott 
Andersen.

PAY IT FORWARD 

If you find this newsletter of value, please consider the following: 

  Norm Kerth, a highly respected consultant, suffered a disabling brain 
  injury in an automobile accident. As a result, he cannot work and lives 
  on a limited income. You can help him by sending a donation. Checks can 
  be made payable to Norm Kerth Benefit Fund and sent to Norm Kerth 
  Benefit Fund c/o Process Impact, 11491 SE 119th Drive, Clackamas, OR 
  97015-8778. You can also visit Karl Weiger�s website (Process Impact
  http://www.processimpact.com/goodies.shtml)for more details about contributing
  to the fund. Thanks. 

More information about the Pay It Forward foundation...
(http://www.payitforwardfoundation.org/)

Some unfinished business...

In the June newsletter on Project Retrospectives, I neglected to 
acknowledge my colleague and bike-riding friend, Mike McGonagle, who 
introduced me to Norman Kerth and the project retrospective process. My 
humble apologies Mike.

Please note: There will be no issue in August � see you again in 
September.

--------------------------------------------------------------------------------

***Monthly Morsels***

Every month in this space you�ll find additional information related to 
this month�s topic.

- References:

  [1] Highsmith, J. and Cockburn, A., �Agile Software Development: The 
  Business of Innovation�, IEEE Computer, September 2001, p. 120-122

  [2] Dijkstra, E., �Go To Statement Considered Harmful�, Comm. ACM, Vol. 
  11, No. 3, March 1968, p. 147-148.

  [3] Marks, K. and Engels, F., The Communist Manifesto, Penguin Classics, 
  2002.

  [4] Rakitin, S., �Manifesto Elicits Cynicism�, Letters to the Editor, 
  IEEE Computer, December 2001, p. 4.

  [5] DeMarco, T. and Boehm, B., �The Agile Methods Fray�, IEEE Computer, 
  June 2002, p. 90-92.

  [6] Boehm, B., �Get Ready for Agile Methods, With Care�, IEEE Computer, 
  January 2002, p. 64-69.

  [7] Software Methodologies: The Battle of the Gurus, Info-tech Research 
  Group, London, Ontario, Canada, 2002

  [8] Brooks, F. P., "No Silver Bullet: Essence and Accidents of Software 
  Engineering," IEEE Computer, Vol. 20, No. 4 (April 1987) pp. 10-19.

  [9] Stephens, M. and Rosenberg, D., Extreme Programming Refactored: The 
  Case Against XP, Apress, 2003.

  [10] Pressman, R., �What A Tangled Web We Weave�, IEEE Software, Jan/Feb 
  2000, pp. 18-21.

  [11] Paulk, M., �Extreme Programming from a CMM Perspective�, IEEE 
  Software Nov/Dec 2001, p. 19-26. 
  
- Books:

  There is a plethora of books on Agile Methods. Just visit your favorite 
  on-line bookstore and search on Agile Methods. Here�s a sampling:

  Books on Agile Methods: 

  - Extreme Programming (XP)

    Beck, K., Extreme Programming Explained, Addison-Wesley, 2000. 

  - Crystal

    Cockburn, A., Crystal Clear : A Human-Powered Methodology for Small 
    Teams, Addison-Wesley, 2004. 

  - Scrum

    Schwaber, K. and Beedle, M., Agile Software Development with SCRUM, 
    Prentice-Hall, 2001 

  - Dynamic Systems Development

    Coleman and Verbruggen, A quality software process for rapid 
    application development, Software Quality Journal 7, p. 107-1222 
    (1998) 

  - Adaptive Software Development

    Highsmith, J., Adapive Software Development, Dorset House, 2000 

  - Lean Software Development

    Poppendick, T. and Poppendick, M., Lean Software Development: An Agile 
    Toolkit, Addison-Wesley Professional, 2003.

  Books critical of Agile Methods: 

  - Boehm, B. and Turner, R., Balancing Agility and Discipline: A Guide 
    for the Perplexed, Addison-Wesley, 2003 

  - Stephens, M. and Rosenberg , D., Extreme Programming Refactored: The 
    Case Against XP, Apress, 2003 

  - McBreen, P. Questioning Extreme Programming. Addison-Wesley, Boston. 
    2003. 

- On-line Resources: 

  - Unbiased information from Wikipedia on Agile Development Methods...
    (http://en.wikipedia.org/wiki/Agile_software_development#References)
  
  Chrysler Comprehensive Compensation (C3) Project Wiki Sites 
  
  - This site provides interesting commentary on the project...
    (http://c2.com/cgi/wiki?ChryslerComprehensiveCompensation)

  - And, after the project was terminated... 
    (http://c2.com/cgi/wiki?CthreeProjectTerminated)

  Sites supportive of Agile Methods: 
  
  - Agile Alliance (http://www.agilealliance.org/)

  - Extreme Programming (http://www.extremeprogramming.org/)
  
  Sites critical of Agile Methods: 
  
  - Software Reality (http://www.softwarereality.com/ExtremeProgramming.jsp)

--------------------------------------------------------------------------------

***Calendar***

Every month, you�ll find news here about local and national events that 
are of interest to the software community ...

- Software Quality Calendar

  There are many organizations that sponsor monthly meetings, workshops, 
  and conferences of interest to software professionals. Find out what�s 
  happening... (http://www.swqual.com/links/upcoming.html)

- Workshops Offered by Software Quality Consulting

  Software Quality Consulting offers workshops in many topics related to 
  software process improvement. Get more info... 
  (http://www.swqual.com/training/on_site.html)

--------------------------------------------------------------------------------

***About SQC***

Software Quality Consulting provides consulting, training, and auditing 
services tailored to meet the specific needs of clients. We help clients 
fine-tune their software development processes and improve the quality of 
their software products. The overall goal is to help clients achieve 
Predictable Software Development(TM) � so that organizations can consistently 
deliver quality software with promised features in the promised timeframe.
To learn more about how we can help your organization, visit our web site 
(http://www.swqual.com/index.html?AboutSQC) or send us an email ([email protected]).

--------------------------------------------------------------------------------

I hope this newsletter has been informative and helpful. Your comments and 
feedback are most welcome. Send me your feedback... ([email protected])

Thanks,

Steve Rakitin
[email protected]

Food for Thought and Predictable Software Development are trademarks of Software 
Quality Consulting, Inc. 

Copyright � 2005. Software Quality Consulting, Inc. All rights reserved. Graphic 
design by Sage Studio ([email protected])