Food for Thought: An E-Newsletter Published by Software Quality Consulting, Inc.
December 2005, Vol. 2 No. 11 - All Software is Defective

To view a web version of this newsletter, click on the following link:
http://www.swqual.com/newsletter/vol2/no11/vol2no11.html.

--------------------------------------------------------------------------------

Welcome to Food for Thought(TM), an e-newsletter from Software Quality 
Consulting (http://www.swqual.com/index.html?Intro). I've created free
subscriptions for my valued business contacts. If you find this newsletter
informative, I encourage you to continue reading. Feel free to pass this
newsletter along to colleagues by clicking this Forward Email link
(http://ui.constantcontact.com/roving/sa/fp.jsp?plat=i&p=f&m=sctz69n6). If
you�ve received this newsletter from a colleague and would like to subscribe,
please click this Enter New Subscription link
(http://www.swqual.com/newsletter/Subscribe.htm?Newsletter). If you don't wish
to receive this newsletter, click the SafeUnSubscribe(TM) link at the bottom of
this newsletter, and you won�t be bothered again.

Your continued feedback on this newsletter is most welcome. Please send 
your comments and suggestions to [email protected].

----------------------

***In This Issue***

In This Months� Topic, I discuss implications of defective software... 

Regular features to look for each month are:

- Monthly Morsels
  Hints, tips, techniques and reference info related to this month�s topic 

- Calendar
  Conferences, workshops, and meetings of interest to software engineers, 
  QA engineers and anyone interested in software development 

--------------------------------------------------------------------------------

***This Month�s Topic***

All Software is Defective

Implications for the Software Industry... 

  Your alarm clock rings and switches to soothing nature sounds that ease 
  you into your morning. As you hit the shower, the thermostat has already 
  raised the temperature setting so that the kitchen will be warm when you 
  come down for breakfast. The coffee maker starts brewing your favorite 
  caffeinated beverage at just the right time. After breakfast, you head 
  out to work. Your car�s on-board computer monitors everything from 
  anti-lock brakes, engine, emissions, airbags, seat belts, tire pressure, 
  audio system, and of course, the GPS navigation system. On the way to 
  work, you stop at an ATM kiosk. You get a call on your cell phone and 
  respond by sending a text message. On the highway, you pay tolls using a 
  transponder that bills your credit card. The on-board navigation system 
  receives updates on traffic conditions and recommends alternate routes 
  so you arrive at your office in the shortest possible time...
  By the time you arrive at work, you have already used many millions of 
  lines of code, without even realizing it. 

Unless you live under a rock, it is almost impossible to go through a day 
without coming in contact with software. Often, it is not obvious that we 
are using software when doing some task. Software is everywhere and 
affects our lives on a daily basis. And yet, all software is defective.

After over five decades of software development experiences, we do not 
have the ability to develop software that has zero defects. Nor do we have 
the ability to prove that software has zero defects. 

To be fair, we don�t have the ability to produce anything of similar 
complexity that has zero defects. Cars, airplanes, medical devices, 
weapons systems, ATM machines, etc., all have defects. So why should we 
expect more from software? Well, with many products we can often 
demonstrate �that we have taken all practical steps and used all available 
methods to make the product as close to defect free as possible.� [1] We 
frequently can�t do this for software products. 

Just like other complex products, we are not expected to prove that 
software is defect-free. Unlike other products, however, most customers 
expect software to have defects. 

  �The imperfect software that provides computers with intelligence and 
  functionality is purchased because it is affordable and readily 
  available, and has enjoyed the �forgiveness� that comes with being a 
  relatively new technology.� [5] 

When it comes to software, we should be expected to take advantage of all 
that we know about software development in order to produce software that, 
when delivered, has as few defects as possible. A problem that pervades 
the software industry is that: 

  �The current common view that it is impossible to produce defect free 
  software is an excuse for not making the effort to do quality work.� [1] 

WHY DOESN�T MANAGEMENT SEEM TO CARE ABOUT QUALITY? 

When customers demand better quality, Management generally responds. Their 
actions may not always lead to improvement, but they do take action. The 
automotive industry is a prime example. 

In the software industry, a recent study by the US National Institute of 
Science and Technology (NIST) [3] estimates that software defects cost the 
US economy almost $60 billion a year! Given the increasing trends in 
application complexity and the lack of improvements in software 
development and testing, this number is certainly likely to increase... 

WHY DON�T CUSTOMERS SEEM TO CARE ABOUT QUALITY? 

�Because defective software works.� [1] It may not work in every situation 
or for every user, but it usually works as long as users stay within 
defined use models. These defined use models represent the best guess of 
developers and testers as to how users will use the software. 

Watts Humphrey calls these defined use models the �testing footprint� [2] 
as shown below.

  [See the graphic in the HTML version of this newsletter � 
  http://www.swqual.com/newsletter/vol2/no11/vol2no11.html]

The green area represents the use model that the people who developed the 
application had in mind. Users who work within the green area will 
encounter a few problems. As these problems are resolved, the software, 
while far from defect-free, works well enough to satisfy customers whose 
use model is within the green area. 

However, when users start thinking of new ways to use the software, 
serious problems can arise. These new uses represent the yellow areas that 
surround the green shaded middle. The yellow area represents uses that 
were not anticipated by the developers and testers. Since these new uses 
were not anticipated, the software was not designed to support these uses 
and was certainly not tested in the context of these new uses. So, it 
should come as no surprise that there will be many serious problems...

Furthermore, as observed by Humphrey,

  �When coupled with the explosive growth of the Internet and the 
  resulting exposure to hackers, criminals, and terrorists, the need for 
  reliable, dependable, and secure software systems will steadily 
  increase. If experience is any guide, as these systems are used to 
  perform more critical functions, they will get more complex and less 
  reliable. Unfortunately, this probably means that it will take a severe, 
  disruptive, and highly public software failure to get people concerned 
  about software quality.� [2] 

A �...severe, disruptive, and highly pubic software failure...� � haven�t we 
already had several of those? In last month�s newsletter
(http://www.swqual.com/newsletter/vol2/no10/vol2no10.html), I identified 
several famous bugs that certainly were severe, disruptive and highly 
public. But there is no outrage, not yet anyway. In the not too distant 
future, customers will be demanding that software companies improve 
quality. As customers have done with other products, they will seek those 
companies that can and are doing better...

So, given the ubiquitous nature of software and the fact that all software 
is defective, what are the implications for developers, testers, 
management, and users...

IMPLICATIONS FOR DEVELOPERS...

A problem with current software development methods is that developers 
have come to rely on tools rather than their own skills to find problems 
with their code. In the old days, programmers (as they were called then) 
were more scientific, since most were trained in the discipline of 
mathematics or physics. Back then, when you developed a program, you spent 
a considerable amount of time desk-checking your work. The end result was 
usually programs that not only compiled correctly, but that generally 
worked the first time.

  At a software engineering conference many years ago, an electrical 
  engineer was giving a talk about a circuit analysis program that he had 
  developed that was widely used. No one had ever found a defect in this 
  program and he was asked how did he achieve such an accomplishment? His 
  reply was very insightful. He said, �I didn�t think defects were 
  allowed.�

Today, not only are defects allowed they are expected. Many developers 
have resigned themselves to the fact that there is little they can do to 
reduce the number of defects. Today, many applications are released with 
hundreds or even thousands of known defects and an unknown number of 
defects that were not found. How big is this unknown number of defects 
that are not found? Let�s look at the Defect Injection Rate for a sample 
of 810 experienced developers: [1]

  See the table in the HTML version of this newsletter � 
  http://www.swqual.com/newsletter/vol2/no11/vol2no11.html]

  Please try this at work... 

  The average defect injection rate for all developers in this group was 
  120, or about 1 defect for every 8 lines of code. Think about the size 
  of your applications � now do the math... 

  See the table in the HTML version of this newsletter � 
  http://www.swqual.com/newsletter/vol2/no11/vol2no11.html]

  This number probably isn�t something you�ll want to brag about. 

The frustrating thing about this is that we have proven methods developers can use to reduce the Defect Injection Rate. These techniques include the Personal Software Process (PSP - http://www.sei.cmu.edu/tsp/psp.html), Formal Inspections (http://www.methodsandtools.com/archive/archive.php?id=29), and Joint Application Development (JAD - http://en.wikipedia.org/wiki/Joint_application_development) just to name a few. We have these techniques yet we choose not to use them.

As a developer, you have the choice to do good work. The quality of your 
work, be it design or code, should be something that you are passionate 
about and proud of.

        You need to be passionate about quality
       if you want to produce a quality product.

IMPLICATIONS FOR TESTERS...

Most current software testing models are based on a premise that may no 
longer be valid. The premise is that we (developers and testers) 
understand how users are going to use an application. If this premise is 
true, we can use many testing techniques to come up with a reasonably good 
set of tests. In the right environment, we can find many defects. Even so, 
testing is an expensive way to find bugs. The only more expensive way is 
to let customers find them...
  
  Please try this at work...
  The find/fix cycle represents the time required to find and fix one 
  defect. Every software development company should know how long their 
  find/fix cycle is. Use this table to help you estimate your find/fix 
  cycle cost

  See the table in the HTML version of this newsletter � 
  http://www.swqual.com/newsletter/vol2/no11/vol2no11.html]

In reality though, applications have been and will continue to be used in 
ways not anticipated by developers and testers as illustrated by the 
testing footprint figure above. Several high profile defects described in 
last month�s newsletter (http://www.swqual.com/newsletter/vol2/no10/vol2no10.html) are prime examples. So what can testers do about this?


Testers depend on information.

- There is information that is known (requirements and domain knowledge, 
  for example). 

- There is information testers don�t know (ways individual customers might 
  choose to perform an operation using the application, for example) 

- There is also information that you don�t know you don�t know. This is 
  the killer. When customers push the envelope and move that yellow area 
  shown in the testing footprint above � bad things can happen and there�s 
  little that testers can do to anticipate this. 

What can be done though is to minimize the impact of what you don�t know. 
This can be done by using a combination of testing techniques including:

- Exploratory Testing (ET)

  James Bach defines exploratory testing as being similar to doing a jig 
  saw puzzle. The techniques and approaches people use to do a jig saw 
  puzzle often change as the puzzle begins to take shape. He says that the 
  �puzzle changes the puzzling.� [6]

  Read more about Exploratory Testing... 
  (http://www.satisfice.com/articles/what_is_et.htm)

- Act Like A Customer Testing (ALAC)

  Testers need domain knowledge to be effective. Armed with domain 
  knowledge, testers can put themselves in their customers shoes and do 
  what customers do. This is what I call Act Like a Customer Testing.

  Read more about ALAC... 
  (http://www.swqual.com/newsletter/vol2/no4/vol2no4.html)

- Storytelling

  One very effective technique to help testers do ALAC testing is to use 
  storytelling to help identify specific scenarios that need to be tested.

  Read more about storytelling...
  (http://www.swqual.com/newsletter/vol2/no9/vol2no9.html) 

- Requirements-based testing

  Requirement-based testing is testing against the requirements. This is 
  just as important as these other techniques. 

IMPLICATIONS FOR MANAGEMENT...

Until customers demand higher quality, management will not likely pay much 
attention to this issue. But just like the automobile industry, by the 
time management decides to finally address this issue, it will be too 
late. Customers will have found higher quality products elsewhere. 
Capers Jones [4] offers a list of things management can do now to be 
proactive rather than reactive:

- Recognize that no single method is adequate. 

- Testing alone is insufficient. 

- Formal inspections and tests combined give high efficiency, low costs 
  and short schedules. 

- Defect prevention plus inspections and tests give highest cumulative 
  efficiency and best economics. 

- �Bad fix� injection needs special solutions. 

- Database errors need special solutions. 

- Web �content� needs special solutions. 

Lastly, many organizations offer incentives to get products released. 
However, there are often no incentives in to produce code that has fewer 
defects, This sends the message to developers that getting product 
released is more important than quality. There needs to be a balance. 
There needs to be incentives for both � quality and time to market.
 
IMPLICATIONS FOR USERS... 

Users need to demand that software suppliers deliver software with fewer 
defects. User need to find those suppliers that have a reputation for 
quality. For those suppliers that deliver shoddy products, users need to 
threaten legal action if their quality doesn�t improve. Often, that�s the 
only way to get their attention... 

SUMMARY 

Everyone involved in the production and consumption of software needs to 
recognize that all software is defective. Given the pervasive nature of 
software in business and society, we need to find and use techniques that 
have a demonstrated impact on reducing defects in order to develop 
applications that are more robust, safer, and less likely to fail.
 
Till next time, happy holidays!

p.s. If there is a topic that is of particular interest to you, please let 
me know and maybe I can address it in a future newsletter.

PAY IT FORWARD

If you haven�t already done so, please consider donating to your favorite 
charity during this holiday season. There are many people in our 
neighborhoods who need help all year round but especially at this time of 
year.

Read more about the Pay It Forward Foundation...
(http://www.payitforwardfoundation.org/)

--------------------------------------------------------------------------------

***Monthly Morsels***

Every month in this space you�ll find additional information related to 
this month�s topic.

- References:

  [1] Humphrey, W., �The Quality Attitude�,
  news@sei newsletter, Number 3, 2004.
  (http://www.sei.cmu.edu/news-at-sei/columns/watts_new/2004/3/watts-new-2004-3.pdf)

  [2] Humphrey, W., �Defective Software Works�,
  news@sei newsletter, Number 1, 2004.
  (http://www.sei.cmu.edu/news-at-sei/columns/watts_new/2004/1/watts-new-2004-1.htm)

  [3] �The Economic Impacts of Inadequate Infrastructure for Software 
  Testing�, NIST Planning Report 02-3, May 2002. 

  [4] Jones, C., � Software Quality in 2002: A Survey of the State of the 
  Art,� Software Productivity Research, 2002. 

  [5] Kowala, A. and Coffee, K., �Why Is Error Prevention Important?�, 
  Stickyminds.com, 2002. 

  [6] Bach, J., �Exploratory Testing Explained�, 2003. 

- Books

  For information on formal inspections:

  Rakitin, S., Software Verification and Validation for Practitioners and 
  Managers, Artech House, 2001. 

  Wiegers, K., Peer Reviews in Software, Addison-Wesley, 2002. 

- On-line Resources

  Read more about Personal Software Process (PSP)
  (http://www.sei.cmu.edu/tsp/psp.html)

  Read more about formal inspections
  (http://www.methodsandtools.com/archive/archive.php?id=29)

  Formal inspection training
  (http://www.swqual.com/training/peer_reviews.html)

  Read more about Joint Application Development 
  (http://en.wikipedia.org/wiki/Joint_application_development)

--------------------------------------------------------------------------------

***About SQC***

Every month you�ll find news here about local and national events that 
are of interest to the software community...

- Stay tuned for details of an upcoming panel discussion on Offshore 
  Outsourcing sponsored by the Boston SPIN and the Software Quality Group 
  of New England (SQGNE)... 

- Software Quality Calendar

  There are many organizations that sponsor monthly meetings, workshops, 
  and conferences of interest to software professionals. Find out what�s 
  happening... (http://www.swqual.com/links/upcoming.html)

- Workshops Offered by Software Quality Consulting

  Software Quality Consulting offers workshops in many topics related to 
  software process improvement. Get more info...
  (http://www.swqual.com/seminars/courses.html)

--------------------------------------------------------------------------------

***About SQC***

Software Quality Consulting provides consulting, training, and auditing 
services tailored to meet the specific needs of clients. We help clients 
fine-tune their software development processes and improve the quality of 
their software products. The overall goal is to help clients achieve 
Predictable Software Development(TM) � so that organizations can consistently 
deliver quality software with promised features in the promised timeframe. 

To learn more about how we can help your organization, visit our web site
(http://www.swqual.com/index.html?AboutSQC) or send us an email
([email protected]).

--------------------------------------------------------------------------------

I hope this newsletter has been informative and helpful. Your comments and 
feedback are most welcome. Send me your feedback...([email protected])

Thanks,

Steve Rakitin
[email protected]

Food for Thought, Predictable Software Development, Act Like a Customer,
and ALAC are trademarks of Software Quality Consulting, Inc.
Copyright 2005. Software Quality Consulting, Inc. All rights reserved.
Graphic design by Sage Studio