Food for Thought: How Do You Define Software Quality?
An e-newsletter published by Software Quality Consulting, Inc.
December 2004, Vol. 1 No. 4 

To view a web version of this newsletter, click on the following link:
http://www.swqual.com/newsletter/vol1/no4/vol1no4.html.

---------------------------------------------------------------------------------

Welcome to Food for Thought(TM), an e-newsletter from Software Quality 
Consulting (http://www.swqual.com/). I've created free subscriptions for my
valued business contacts. If you find this newsletter informative, I encourage you
to continue reading. Feel free to pass this newsletter along to colleagues by 
clicking this Forward to a Friend link. If you've received this newsletter 
from a colleague and would like to subscribe, please click this Enter New 
Subscription link. If you don't wish to receive this newsletter, click the 
SafeUnSubscribe(TM) link at the bottom of this newsletter, and you won't be 
bothered again. 

Your continued feedback on this newsletter is most welcome. Please send 
your comments and suggestions to [email protected].

Wishing you and your family a safe and joyous holiday and a prosperous New 
Year...

---------------------------------------------------------------------------------

In This Months' Topic, we discuss the difficulty of defining and 
measuring a most important term -- software quality. 

Regular features to look for each month are: 

- Monthly Morsel: Hints, tips, techniques and reference info related to 
  this month's topic. 
- Calendar: Conferences, workshops, and meetings of interest to software 
  engineers, QA engineers and anyone interested in software development. 

----------------------------------------------------------------------------------

***How Do You Define Software Quality?***

Software quality has been a constant topic of concern in the software 
industry. Everyday, in almost every part of the world, company's large and 
small struggle with issues related to software quality. Think about the 
situation at your company: 

- How many discussions have you had with your Development Team, your QA 
  Team, with Sr. Management, with the Board of Directors, etc., about the 
  quality of your software? 
- Does your company have a definition of software quality and if so, is it 
  measurable? 
- How satisfied are your customers with the quality of your software? When 
  was the last time you measured customer satisfaction? 

Most software companies spend considerable time and money - directly and 
indirectly - dealing with this issue. Software quality is often a crucial 
factor in many critical business decisions made on a daily basis. Yet, we 
have a hard time defining exactly what software quality means. We often 
use release criteria as a substitute for a good definition. 

YOU CAN'T IMPROVE WHAT YOU CAN'T MEASURE 

We talk about issues related to software quality all the time but yet we 
don't have a definition of precisely what it means. Especially since we 
should all know by now that we can't improve what we can't measure. 
Without a clear, concise, measurable definition of software quality, how 
can we make good business decisions regarding the use of limited 
engineering resources, areas of focus for quality improvement, tools and 
techniques that might be appropriate for quality improvement, etc. Well, 
we can't. 

This leads me to state what I believe is a basic principle of software 
quality definitions: 

Regardless of how software quality is defined, for the definition to be 
meaningful, it must be measurable. 

There are many different viewpoints on the issue of defining software 
quality. And they seem to fall into a few different groupings. Here are a 
sampling of several definitions from highly respected software engineering 
gurus:

Conformance to Requirements Definition:

  Roger Pressman defines software quality as:

  - "Conformance to explicitly defined functional and performance 
    requirements, explicitly documented development standards, and 
    implicit characteristics that are expected of professionally developed 
    software. 

  - When we examine an item based on its measurable characteristics, two 
    kinds of quality may be encountered: quality of design and quality of 
    conformance. 

Quality of design refers to characteristics designers specify for 
items. Grade of materials, tolerances, performance specifications 
all contribute to the quality of design. 

Quality of conformance is degree to which design specs are followed 
during manufacturing. Again, greater degree of conformance, higher 
level of quality of conformance." [1] 

Customer-centric Definitions:

  Watts Humphrey says: 

    "The principal focus of any software quality definition should be the 
    users' needs. Crosby [6] defines quality as 'conformance to 
    requirements.' While one can debate the distinction between 
    requirements, needs, and wants, quality definitions must consider the 
    users' perspectives. The key questions then are, who are the users, 
    what is important to them, and how do their priorities relate to the 
    way you build, package, and support your products?" [2] 

  Al Davis defines quality as: 

    "Quality isn't about zero defects or measurable improvements in defect 
    rates, nor is it about meeting documented requirements. It's no more 
    and no less than satisfying customer needs (whether or not the needs 
    are properly documented)." [5]

Conformance to Requirements AND Customer-centric Definitions:

  Capers Jones defines quality as:

  "The term quality will be used to mean software that has these six 
  attributes: 


  - Low levels of defects when deployed, ideally approaching zero defects 

  - High reliability, or capability of running without crashes or strange 
    results 

  - A majority of users expressing satisfaction with software when 
    surveyed 

  - A structure that minimizes bad fixes or insertion of new defects 
    during repairs 

  - Effective customer support when problems do occur 

    Rapid repairs for defects, especially for high-severity defects" [3]

  The IEEE Glossary of Software Engineering Terminology [4] defines 
  quality as: 

    (1) The degree to which a system, component, or process meets specified 
    requirements. 

    (2) The degree to which a system, component, or process meets customer or 
    user needs or expectations. 

Touchy-feely Definitions:

  Jerry Weinberg defines quality as: 

    "Quality is value to some person." [9] 
  
     James Bach, Ed Yourdon and others have pushed the notion that software 
     quality needs to be "Good Enough". To claim that any given thing is Good 
     Enough is to agree with the following propositions: 

     - It has sufficient benefits. 

     - It has no critical problems. 

     - The benefits sufficiently outweigh the problems.

     - In the present situation, and all things considered, further 
       improvement would be more harmful than helpful." [7] 

"-ilities" Definitions: 

  Robert Glass says: 

    "Quality is a collection of 'ilities':  
    
    - reliability - the ability to operate error free

    - modifiability - the ability to have enhancement changes made easily 

    - understandability - the ability to understand the software readily, in 
      order to change/fix it  

    - efficiency - the speed and compactness of the software

    - usability - the ability to use the software easily  

    - testability - the ability to construct and execute test cases easily  

    - portability - the ability to move the software easily from one 
      environment to another" [8] 

Are you totally confused by all of these different definitions? I sure am... 

GIVE ME A DEFINITION I CAN USE! 

How is it possible that so many smart people can have such different views 
of such a simple term? Maybe quality isn't so simple after all. Or, have 
we made it more complicated than it needs to be? 
As part of research I'm doing for a book, I recently asked some of my 
fellow consultants the following questions: 

Are definitions that are not measurable worthwhile?

  Here's what Karl Wiegers had to say:
 
    "I think definitions that aren't measurable are still worthwhile if 
    they help create a shared focus of a project team on its technical and 
    business objectives so they can work together toward a valuable 
    outcome. Even if not quantitatively measurable, there still needs to 
    be some way to assess whether a product or project has achieved its 
    quality objectives. Therefore, a non-measurable definition can be 
    worthwhile but is not adequate for making a definitive evaluation of 
    whether a suitable level of quality is achieved."

Should there be only one definition or many definitions depending upon 
the situation and context?

  Karl Wiegers said: 

    "I doubt we'll ever have a single definition that fits all situations. 
    A key aspect is that there are multiple dimensions of quality that 
    need to be defined for any product. That's why vague definitions like 
    Weinberg's 'quality is value to some person' aren't sufficiently 
    helpful. Conformance to requirements (Crosby) assumes that the 
    requirements are correct and complete, which isn't something I see 
    very often!" 

What's a definition of software quality that makes the most sense to 
you? 

  Here's what Robin Goldsmith said: 

    "A system's quality is the extent to which it meets weighted stated 
    and implied exterior, interior, and future REAL business requirements 
    of all affected internal and external stakeholders consistent with 
    standards of design, workmanship, and performance." 

  Johanna Rothman told me: 

    "I use Weinberg's 'Quality is value to someone'. I find that while it 
    seems vague, it allows me to have a richer discussion about who all 
    the someones are and what they find valuable. This definition also 
    helps me define the context for quality in a given project. Once you 
    start identifying your someones and what they find valuable, you can 
    determine how to quantify that into release criteria, so you meet a 
    minimum standard of project quality for your specific project." 

Johanna raises an interesting point - many organizations confuse quality 
and release criteria. (An example of release criteria - some number of P1 
bugs). Release criteria should be thought of as a measurable objective 
derived from your definition of quality. 

A discussion with my colleague Bill Eventoff helped confirm a thought that 
I had about the definition. While there are those who believe conformance 
to requirements is not sufficient (because the requirements are usually 
flawed and don't reflect user needs), it is important to be able to 
measure the ability of an organization to build a product that conforms to 
requirements - even if those requirements are flawed. Getting the 
requirements right can be measured separately from the organization's 
ability to build a product that conforms to requirements. 

QUALITY OF BUILDING VS. QUALITY OF SPECIFYING 

Maybe the reason we don't agree on definitions of software quality is 
because we have made the term so all-encompassing that a meaningful 
definition has become difficult if not impossible. 

The ability to consistently develop software that is traceable to written 
requirements is good. If the requirements are flawed but the software 
matches the requirements, is that not a quality product? I would argue it 
is. 

What we need to do is separate the specifying from the developing. Let's 
measure the quality of developing while we work on improving the quality 
of specifying. And besides, if we had good quality of developing, then 
wouldn't the quality of specifying be captured in customer satisfaction 
surveys?
 
THE BOTTOM LINE 

Every organization that develops software needs a working definition of 
software quality. How can you create a good definition? You need to take 
into consideration many factors such as customer's needs, industry 
expectations, regulatory requirements, business objectives, etc. With a 
good understanding of these factors, you can then develop your own 
definition as follows: 
First, your definition needs to meet the principle stated above - to be 
useful, it must be measurable.

Second, recognize that, as James Bach observed, "quality is situational". 
In some situations, quality may be more important to the organization and 
to customers than in other situations. (Caveat - While this is generally 
the case for most software companies, it often isn't the case when 
software is life-supporting or life-threatening). 

Third, use the following distinctions to separate specification and 
development quality: 

- Development Quality is the ability to develop software based on and 
  traceable to documented requirements. 

- Specification Quality is the ability to specify product requirements 
  that represent real user needs and expectations and are consistent with 
  overall business objectives. 

Then, using these distinctions: 

  Formulate your definition for Development Quality based on using pieces 
  of the definitions from Pressman and Jones and include as many of the 
  "-ilities" as you deem appropriate. 

  - Use activities such as Design Reviews, Code Inspections, requirements 
    traceability, test results, and number of P1 bugs, etc., as measures. 

  Formulate your definition for Specification Quality by using information 
  from Karl Wiegers [10], Robin Goldsmith [11] and others. 

  - Use activities such as Focus Groups, Usability Studies, and Customer 
    Satisfaction surveys as measures.

By defining and then measuring development quality and specification 
quality, we can focus process improvement activities on those areas where 
the problems really are - as indicated by these measures. Isn't this 
better than the meaningless definitions we often have? Using this 
paradigm, every software company would have different definitions. When it 
comes to software quality, one size clearly doesn't fit all. 

Well, that's my story and I'm sticking to it. 

Happy holidays! 

-------------------------------------------------------------------------------

***Monthly Morsel***

Every month in this space you'll find additional information related to 
this month's topic. 

- Internet references:

  Click here to view a Wiki page on the subject of what is quality
  (http://c2.com/cgi/wiki?WhatIsQuality)

  ASQ Software Division Website
  (http://www.asq.org/perl/index.pl?g=softwareforum)

  Software Quality Group of New England Website
  (http://www.sqadvice.com/SQGNE_Home.htm)

- Books and articles:

  [1] Pressman, R., Software Engineering: A Practitioner's Approach, 
  McGraw-Hill, 4th edition, 1997.

  [2] Humphrey, W., A Discipline for Software Engineering, Addison-Wesley, 
  1995.

  [3] Jones, C., Software Quality: Analysis and Guidelines for Success, 
  ITP, 1997.

  [4] IEEE Standard Glossary of Software Engineering Terminology, IEEE Std 
  610.12-1990, 1990.

  [5] Eickelman, N. and Hayes, J. eds., "New Year's Resolutions for 
  Software Quality", IEEE Software, Jan-Feb 2004, pp. 12-13.

  [6] Crosby, P., Quality Without Tears, McGraw-Hill Book Company, 1984, 
  p.64.

  [7] Bach. J., "Good Enough Quality - Beyond the Buzzword", IEEE 
  Computer, August 1997.

  [8] Glass, R., "Revisiting the definition of software quality", 
  StickyMinds Weekly Column From 10/08/01.

  [9] Weinberg, G., Quality Software Management - Systems Thinking, Dorset 
  House, 1991.

  [10] Wiegers, K., Software Requirements, Microsoft Press, 1999.

  [11] Goldsmith, R., Discovering the REAL Business Requirements for 
  Software Project Success, Artech House, 2004. 

---------------------------------------------------------------------------------

***Calendar***

Every month, you'll find news here about local and national events that 
are of interest to the software community ... 

- Software Quality Calendar

  There are many organizations that sponsor monthly meetings, workshops, 
  and conferences of interest to software professionals. Find out what's 
  happening (http://www.swqual.com/links/upcoming.html)
  
- Workshops Offered by Software Quality Consulting

  Software Quality Consulting offers workshops in many topics related to 
  software process improvement. Get more info
  (http://www.swqual.com/seminars/courses.html)

---------------------------------------------------------------------------------

***About SQC***

Software Quality Consulting provides consulting, training, and auditing 
services tailored to meet the specific needs of clients. We help clients 
fine-tune their software development processes and improve the quality of 
their software products. The overall goal is to help clients achieve 
Predictable Software Development(TM) - so that organizations can consistently 
deliver quality software with promised features in the promised timeframe. 

To learn more about us and how we can help your organization, visit our 
web site (http://www.swqual.com/) or send us an email ([email protected]).

I hope this newsletter has been informative and helpful. Your comments and 
feedback are most welcome. Send me your feedback...

Happy Holidays,

Steve Rakitin
[email protected]


Food for Thought and Predictable Software Development are trademarks of Software 
Quality Consulting, Inc.
 
Copyright (c) 2004. Software Quality Consulting, Inc. All rights reserved. Graphic 
design by Sage Studio