Food for Thought: What�s Your Testing ROI?
An e-newsletter published by Software Quality Consulting, Inc.
April 2005, Vol. 2 No. 4 

To view a web version of this newsletter, click on the following link:
http://www.swqual.com/newsletter/vol2/no4/vol2no4.html.

--------------------------------------------------------------------------------

Welcome to Food for Thought(TM), an e-newsletter from Software Quality 
Consulting (http://www.swqual.com/?Intro). I've created free subscriptions for
my valued business contacts. If you find this newsletter informative, I
encourage you to continue reading. Feel free to pass this newsletter along to
colleagues by clicking this Forward Email link
(http://ui.constantcontact.com/roving/sa/fp.jsp?plat=i&p=f&m=sctz69n6). If
you�ve received this newsletter from a colleague and would like to subscribe,
please click this Enter New Subscription link
(http://www.swqual.com/newsletter/Subscribe.htm). If you don't wish to receive
this newsletter, click the SafeUnSubscribe(TM) link at the bottom of this
newsletter, and you won�t be bothered again.

Your continued feedback on this newsletter is most welcome. Please send 
your comments and suggestions to [email protected]. 

--------------------------------------------------------------------------------

In This Months� Topic, we we discuss the notion of Testing ROI... 

Regular features to look for each month are:

  Monthly Morsels
  - Hints, tips, techniques and reference info related to this month�s topic 

  Calendar
  - Conferences, workshops, and meetings of interest to software engineers, 
    QA engineers and anyone interested in software development 

--------------------------------------------------------------------------------

***This Month�s Topic***

WHAT�S YOUR TESTING ROI?

Have you ever thought about how expensive software testing is? Even if 
your testing is outsourced, testing is an expensive, time-consuming 
activity. Like coding, testing is labor-intensive. Tests have to be 
planned, developed, debugged, maintained and executed. Defects, once 
found, need to be tracked and triaged. Decisions need to be made about 
which defects should be fixed. When fixed, defects need to be verified and 
regression testing needs to be performed to determine if fixes have 
introduced new defects. 

Like most other expensive, time-consuming business processes, it�s 
important to know what the Testing Return on Investment (ROI) is and how 
it could be improved. Many software companies have no idea how much 
testing costs, much less their Testing ROI. You�d think that an activity 
so critical to the business and so expensive would be constantly measured, 
monitored, and managed...

WHAT IS TESTING ROI?

The most basic definition of ROI is:

  ROI = [(Payback - Investment)/Investment)]*100
  where: Payback is the amount earned from your investment 
  Investment is the cost of resources required to generate the payback

To calculate Testing ROI, we need to understand both the investment and 
the payback.

THE INVESTMENT

Testing is performed by developers (unit testing) and QA (functional, 
system, validation testing are several terms used for testing performed by 
QA). For this discussion, we�re focusing only on testing performed by QA. 
The testing investment can be determined by looking at testing costs, such 
as:

- Cost of resources (labor) spent on all testing and testing-related 
  activities

- Cost of resources (equipment and infrastructure) to support testing 
  activities

- Costs of resources (labor) required to deal with defects found from 
  testing 

Companies may have additional costs that may be specific to their 
business. Quantifying these costs is certainly feasible.

THE PAYBACK

The payback from testing is determined by looking at:

- Customer satisfaction is higher as a result of encountering far fewer 
  defects

- Employee satisfaction is higher as a result of greater pride in 
  producing higher quality products

- Sales and revenue are higher because higher quality products compete 
  more successfully against products perceived to be of lesser quality

- Development and support costs are lower because higher quality products 
  are less expensive to develop and support

- Maintenance costs are lower because higher quality products are easier 
  to maintain and enhance

- etc... 

Capers Jones [1] surveyed hundreds of software development companies. Of 
those that produce higher quality software, he found that they have:

- shorter development schedules 
- lower development and maintenance costs 
- better reliability and longer mean time to failure 
- larger market share 
- better user satisfaction 
- better employee morale 
- less chance of ending up in court 

Quantifying testing payback is difficult, not because we can�t identify 
what it is, but because it is difficult to measure. We all know that 
testing identifies defects and that there�s a strong correlation between 
defects and quality. When defects are fixed properly, quality improves. 

As a result, it is difficult to measure the Testing ROI because many of 
the Paybacks are not conducive to measurement or represent intangibles. 
For example, if you are among the few software companies who measure 
customer satisfaction, can you determine what portion of customer 
satisfaction is attributable to product quality? Can you determine what 
portion of product quality is attributable to testing (as opposed to using 
good engineering practices)? This is hard to do...

So, while we all agree that testing is a good thing to do, we can�t 
realistically measure the Testing ROI. But, since we all agree testing is 
a good thing to do, we can identify ways to make testing more effective. 
Taking steps to make testing more effective will clearly improve the 
Testing ROI, even though we may not be able to directly measure it. 

Here are some examples of what I mean when I say we can improve testing 
effectiveness:

- Are you testing the most important features? How do you know? 
- Are there features you�re not testing? I bet you are... 
- Are you testing the same features many times? I bet you are... 
- Are you testing against vague and ambiguous requirements? I bet you are... 
- Does your regression suite have tests that are not relevant? I bet it 
  does... 

THE PROBLEM

Testing requires a significant investment in time and effort. Each year, 
companies spend hundreds of thousands of hours testing software. Typical 
test suites often number into the thousands of tests, many of which 
require hundreds of hours to develop, maintain, and execute. Often, tests 
are written against requirements that are vague and ambiguous. Regression 
test suites evolve over time and often include large numbers of what I 
call �non-productive� tests. These tests often are looking for problems in 
areas where there aren�t problems. Couple this with the fact that testers 
are inclined to develop more tests in the areas of the application they 
are most familiar with, leaving other areas under-tested or not tested at 
all.

We need ways to assess testing effectiveness. And, we need to assess 
effectiveness BEFORE we start testing so that we increase the Testing ROI.

MEASURING TESTING EFFECTIVENESS

Very few techniques have been developed to measure testing effectiveness. 
Here are some examples of what has been done:

Chernak [2] defined a measure called test case effectiveness (TCE) as:
 
  TCE = Ntc / Ntot * 100%
  
  where:

  Ntc = defects found by test cases
  Ntot = sum of defects found by test cases and found by other means

Huber [3] at Hewlett-Packard developed a set of metrics
(http://www.swqual.com/articles/HPMetricDescription.pdf) to measure test 
effectiveness. The H-P metrics illustrate the importance of measuring the 
effectiveness of the testing activity. 

FACTORS THAT IMPACT TESTING EFFECTIVENESS

In looking at the testing effectiveness measures from Huber and Chernak, I 
think we need to focus on productive vs. non-productive testing as one way 
to improve testing effectiveness.
 
  Productive tests are tests that have a much higher probability of 
  finding defects. Since the whole purpose of testing is to find defects, 
  we would obviously want to develop as many productive tests as possible. 

  Non-productive tests are tests that will likely never find a defect, no 
  matter how many times the tests are executed.

If you accept the premise that we need to focus on developing as many 
productive tests as possible and that having more productive tests will 
surely improve testing effectiveness, we can then focus on identifying 
factors that affect our ability to write productive tests. My list 
includes:

- Testing against Poorly written requirements 
- Unrealistic development and testing schedules 
- Lack of measurement capability 
- Lack of testing infrastructure (staff, hardware, systems, automation 
  tools) 
- Lack of domain knowledge 
- Lack of training in basic testing techniques 

Given these factors, we can now look at ways to assess Testing 
Effectiveness.

TESTING EFFECTIVENESS ASSESSMENT

A Testing Effectiveness Assessment is a process that can improve the 
Testing ROI and includes the following five steps:

1 Scrub the Requirements

  Tests can only be as good as the requirements they are intended to test. 
  If requirements are vague and ambiguous, the tests will not be very 
  effective. Therefore, the first step in the assessment process is a 
  thorough review of the requirements.

  Based on this review, a Requirements Scrub may be necessary. For more 
  information on writing requirements that are unambiguous, check out the 
  Sept 2004 Food for Thought
  (http://www.swqual.com/newsletter/vol1/no1/vol1no1.html).

2 Update the Requirements Trace Matrix (RTM)

  Once the requirements have been scrubbed, the next step is to review and 
  update the RTM. The RTM is an extremely important document. In its 
  simplest form, it provides a way to determine if all requirements are 
  tested. However, the RTM can do much more.

  In this example, the RTM is used as a test planning tool to help 
  determine how many tests are required, what types of tests are required, 
  whether tests can be automated or manual, and if any existing tests can 
  be re-used. Using the RTM in this way helps ensure that the resulting 
  tests are most effective.

  The RTM can serve many purposes over the course of a development 
  project. Initially, it can be used as a planning tool (as illustrated 
  above). Once the tests are developed and Validation Testing has begun, 
  the RTM can be used to help determine the extent of regression test 
  required based on the relationship be requirements, design, code, and 
  tests as illustrated below.

  When it becomes necessary to perform regression testing, the accurate 
  information included in the RTM will be invaluable in helping to select 
  a reasonable set of tests to run.

3 Analyze the Existing Test Suite

  The RTM can also be used to help analyze the existing test suite. For 
  example, an analysis of test types can be performed to determine if a 
  variety of test types have been associated with each requirement. By 
  using a wide variety of types of tests, the tests are likely to be more 
  productive.

  Examples of Test Types 
  Functional
  Algorithmic
  Positive
  Negative
  Usability
  Boundary
  Act Like a Customer
  Startup
  Shutdown
  Configuration
  Platform
  Load/Stress
  Security Performance
  Data Flow
  Safety-related
  Compatibility
  Documentation
  Timing
  Error Checking
  Power Failure
  Resources
  Installation
  Upgradeability
  Volume Scalability
  Throughput

  In addition to looking for a variety of test types, there are specific 
  test strategies that are also important to identify. For example: 

  - Data Flow Testing

    Data flow testing is a strategy based on known patterns of data usage 
    and data flow. Data flow tests are often derived from data flow 
    diagrams.

  - Equivalence Classes

    A group of tests can form an equivalence class if:

       - they all test the same thing 
       - if one test passes, all tests will likely pass 
       - if one test fails, all tests will likely fail 
    
    Identifying potential equivalence classes allows redundant, 
    non-productive tests to be identified.
    
  - Boundary Analysis

    By identifying ranges and boundaries, tests can be created that are:

    - well within boundary or range 
    - one less than boundary 
    - equal to boundary 
    - one greater than boundary 
    - well beyond boundary 
    
    Boundary analysis ensures that all boundary conditions are covered. 
    Again, this increases the likelihood that these tests are productive.

  - Act Like a Customer Testing

    Act Like a Customer (ALAC) testing is based on how users actually use 
    your product. These tests are critical in helping to uncover the kinds 
    of defects that users are likely to encounter. ALAC tests can be 
    easily created from Use Case/Workflow diagrams.

4 Scrub the Test Suite

  The next step in the Testing Effectiveness process is to remove 
  redundant, non-productive tests from the test suite. 

5 Identify Additional Training for Testers

  Training for testers can help improve skills in areas such as: 

  - Test Planning 
  - Act Like a Customer Testing 
  - Equivalence Classes 
  - Data Flow Testing 
  - Boundary Analysis 
  - Requirements Management 
  - Test Automation 

THE BOTTOM LINE

So, while it may not be practical to measure Testing ROI, improving 
testing effectiveness will certainly improve the Testing ROI whether we 
can measure it or not.
Till next time...

--------------------------------------------------------------------------------

***Monthly Morsel***

Every month in this space you�ll find additional information related to 
this month�s topic.

  References:

  [1] Jones, C., Software Quality: Analysis and Guidelines for Success, 
  ITP, 1997

  [2] Chernak, Y., �Validating and Improving Test-Case Effectiveness�, 
  IEEE Software, January-February 2001.

  [3] Huber, J., �Efficiency and Effectiveness Measures To Help Guide the 
  Business of Software Testing�, SM/ASM Conference, 1999. 
  
  On-line Resources:


  - StickyMinds: Articles of interest to testing professionals
    (http://www.stickyminds.com/)

  - Software Quality Professional: ASQ Software Division�s quarterly journal
    (http://www.asq.org/pub/sqp/)

  - Better Software Magazine
    (http://www.stickyminds.com/BetterSoftware)

  Testing Resources:

  Some of my favorite books on testing: 

  - Kaner, C., et. al, Testing Computer Software, 2nd ed, Thomson Computer 
    Press, 1993

  - Marick, B., The Craft of Software Testing, Prentice-Hall PTR, 1995

  - Copeland, L., A Practitioner�s Guide to Software Test Design, Artech 
    House, 2003

  - Kit, E., Software Testing in the Real World, Addison-Wesley, 1995 

--------------------------------------------------------------------------------

***Calendar***

Every month, you�ll find news here about local and national events that 
are of interest to the software community ...

- Software Quality Calendar

  There are many organizations that sponsor monthly meetings, workshops, 
  and conferences of interest to software professionals. Find out what�s 
  happening... (http://www.swqual.com/links/upcoming.html)

- Workshops Offered by Software Quality Consulting

  Software Quality Consulting offers workshops in many topics related to 
  software process improvement. Get more info... 
  http://www.swqual.com/seminars/courses.html)

--------------------------------------------------------------------------------

***About SQC***

Software Quality Consulting provides consulting, training, and auditing 
services tailored to meet the specific needs of clients. We help clients 
fine-tune their software development processes and improve the quality of 
their software products. The overall goal is to help clients achieve 
Predictable Software Development(TM) � so that organizations can consistently 
deliver quality software with promised features in the promised timeframe. 

To learn more about how we can help your organization, visit our web site
(http://www.swqual.com/?AboutSQC) or send us an email ([email protected]).

--------------------------------------------------------------------------------

I hope this newsletter has been informative and helpful. Your comments and 
feedback are most welcome. Send me your feedback... ([email protected])
Thanks,

Steve Rakitin
[email protected]

Food for Thought and Predictable Software Development are trademarks of Software 
Quality Consulting, Inc.

Copyright � 2005. Software Quality Consulting, Inc. All rights reserved. Graphic 
design by Sage Studio