Food for Thought: What is Software Quality Assurance?
An e-newsletter published by Software Quality Consulting, Inc.
January 2005, Vol. 2 No. 1 

To view a web version of this newsletter, click on the following link:
http://www.swqual.com/newsletter/vol2/no1/vol2no1.html.

--------------------------------------------------------------------------------

Welcome to Food for Thought(TM), an e-newsletter from Software Quality 
Consulting (http://www.swqual.com/). I've created free subscriptions for my
valued business contacts. If you find this newsletter informative, I encourage
you to continue reading. Feel free to pass this newsletter along to colleagues
by clicking the Forward Email link at the bottom of this page. If you�ve
received this newsletter from a colleague and would like to subscribe, please
click this Enter New Subscription link
(http://www.swqual.com/newsletter/Subscribe.htm).
If you don't wish to receive this newsletter, click the SafeUnSubscribe(TM) at
the bottom of this newsletter, and you won�t be bothered again.

Your continued feedback on this newsletter is most welcome. Please send 
your comments and suggestions to [email protected].

--------------------------------------------------------------------------------

***In This Issue***

In This Months� Topic, we continue the discussion of software quality
(http://www.swqual.com/newsletter/vol1/no4/vol1no4.html) 
begun last month with a discussion of software quality assurance.

Regular features to look for each month are:
 
- Monthly Morsel: Hints, tips, techniques and reference info related to 
  this month�s topic. 
- Calendar: Conferences, workshops, and meetings of interest to software 
  engineers, QA engineers and anyone interested in software development. 

--------------------------------------------------------------------------------

***What is Software Quality Assurance?***

Ever wonder where Software Quality Assurance (SQA) came from? The activity 
we call SQA evolved from Independent Verification & Validation (IV&V) � a 
collection of activities associated with large, mission-critical projects. 
Understanding IV&V is important because it represents the �heritage of 
SQA�. Many of the tasks that we typically associate with SQA originated 
with IV&V. This month, we�ll explore the evolution of SQA from IV&V as a 
backdrop for a discussion on �What is SQA?�

HISTORICAL PERSPECTIVE

Back in the late 1950's, software began to find its way into systems 
procured by the US Dept. of Defense (DoD). Not surprisingly, these 
projects were consistently behind schedule, over budget, and had many 
technical problems. Frequently, software never worked as intended and many 
projects were cancelled before anything was delivered. Throughout this 
period, software development contractors often gave overly optimistic 
assessments of the software development status to the DoD. The DoD was 
frequently unaware of schedule, budget, and technical problems until late 
into the program � when they were often unable to understand them and 
assess their impact.

The IV&V role was established to address this problem. The first program 
to use IV&V was the Atlas Missile Program in the late 1950's. An 
�independent software tester� was hired to �perform additional, unbiased 
testing of the software�. [1] By employing someone who was totally 
separate from the software development contractor, the DoD hoped to get a 
more accurate and objective technical assessment of the project's status.

Over time, the role of the �IV&V contractor� became critically important. 
IV&V has been and still is used on many large, mission-critical projects 
for DoD, NASA, FAA, HUD, and DEA. Today, the set of tasks performed by 
IV&V contractors is quite comprehensive. See the draft IEEE Standard 1012 
Software Verification & Validation for specifics
(http://www.swqual.com/draftstandard/P1012D12.pdf). (Note this is a draft 
standard and has not yet been approved.)

Since Atlas, much data has been collected to support the assertion that 
projects with IV&V perform much better than similar projects without IV&V. 
[2], [3] As a result of this data, NASA now requires IV&V to be applied on 
applicable NASA projects. [13]

Much of the success of IV&V is attributable to the fact that IV&V 
contractors are completely independent of the software development 
organization. Working for and reporting to the procuring entity, IV&V 
contractors provide an unbiased, objective technical and managerial 
assessment of a project. As a result, the procuring entity is in a much 
better position to identify and resolve issues that could otherwise easily 
be overlooked (intentionally or unintentionally) by the software 
development contractor. By raising these issues in a timely manner, they 
are more likely to be resolved in a way that does not negatively affect 
the project.

EMERGENCE OF SQA

During the 1970�s, as software development activity began to expand, 
software development companies were experiencing the same poor results as 
did government agencies a decade earlier. Companies had difficulty 
delivering software within the constraints of schedule, budget, and 
quality. Many projects undertaken in the 1980�s and 90�s were disasters � 
either because they failed to deliver anything, they grossly exceeded 
budget and schedule parameters, or they delivered software of such poor 
quality that it was unusable.

During the 1980�s, we experienced what became known as the �software 
crisis� � the point in time when spending on software maintenance exceeded 
spending on creating new software products. The advent of the �software 
crisis� brought with it a host of changes - not the least of which was the 
emergence of SQA.

Drawing on its roots in IV&V, SQA evolved into an effective tool that 
software development companies have used to help identify quality problems 
earlier in the development process. While SQA was viewed as the �poor 
stepchild� of software development, many enlightened managers of the day 
saw measurable benefit from integrating SQA into the software development 
process.

By the 1990�s, many software companies had SQA functions within their 
organizations. Yet, high profile software failures continued to occur. 
(see [4, 5, 14]) Was SQA not living up to expectations? Hard to say. But 
there were several differences in the nature of software being developed 
during this time that are worth noting:

- Complexity of software developed during the 90�s increased 
  significantly. 
- Competitive business pressures also increased significantly. 
- Software was being used in many new areas � especially areas that were 
  life-threatening. 
- Many people working in SQA received little formal training in SQA. SQA 
  engineers were expected to learn their craft primarily from on-the-job 
  training. 
- Universities failed to recognize that SQA is a legitimate discipline 
  unto itself and that it requires specialized training. 

As observed by Watts Humphrey:
 
  �SQA is a valid discipline in its own right and people can be SQA 
  experts without being software design experts. This SQA expertise is 
  what is required to establish a strong quality program. It includes 
  knowledge of statistical methods, quality control principles, the 
  software process, and an ability to deal effectively with people in 
  contentious situations.� [6] 

WHAT IS SQA?

Just like quality (http://www.swqual.com/newsletter/vol1/no4/vol1no4.html),
there are many definitions of SQA:

  The �confidence� definitions:

  The IEEE defines quality assurance (QA) as:
 
    �A planned and systematic pattern of all actions necessary to provide 
    adequate confidence that an item or product conforms to established 
    technical requirements.

    A set of activities designed to evaluate the process by which products 
    are developed or manufactured�. [7]

  Daniel Galin defines SQA as:
 
    �A systematic, planned set of actions necessary to provide adequate 
    confidence that the software development process or the maintenance 
    process of a software system product conforms to established 
    functional technical requirements as well as with the managerial 
    requirements of keeping the schedule and operating within budgetary 
    confines�. [8]
 
  The �visibility� definition:

  The Software Engineering Institute (SEI) defines SQA as:
 
    �Software Quality Assurance provides management with appropriate 
    visibility into the process being used by the software project and of 
    the products being built�. [9]
 
  The �assurance� definition:

  Don Reifer says:
 
    �Software quality assurance is the system of methods and procedures 
    used to assure that the software product meets its requirements. The 
    system involves planning, measuring and monitoring developmental 
    activities performed by others.� [10]
 
  The �fitness for use� definition:

  Schulmeyer and McManus define SQA as:
 
    �The systematic activities providing evidence of the fitness for use 
    of the total software product�. [11]

 The �umbrella� definition:

  Pressman defines SQA as an �umbrella activity that is applied throughout 
  the software process�. [12] He further states that �SQA encompasses: 

  1.  A quality management approach 
  2.  Effective software engineering technology (methods and tools) 
  3.  Formal technical reviews applied throughout the software process 
  4.  A multi-tiered testing strategy 
  5.  Control of software documentation and the changes made to it 
  6.  A procedure to measure compliance with software development standards 
  7.  Measurement and reporting mechanisms� [12]

  The �management tool� definition:

  Watts Humphrey defines SQA as a �management tool that must be properly 
  used to be effective.� [6]
 
THE MANY FACES OF SQA

Before I give you my definition of SQA, I first want to share with you 
some of the many different roles that SQA often plays in organizations.

 The �Process Police� 

    SQA�s motto might be: 

�Our job is to make sure Development follows the process.� 

    SQA�s role:

- Audits work products to identify deficiencies 
- Determines compliance with project development plans and software 
  development process

Focus on process assessment rather than product assessment

    What often happens: 

- SQA is perceived as nitpickers obsessed with minutia 
- SQA insists on inordinate amounts of mostly unproductive testing 
- SQA becomes obsessed with finding the �last defect� 

  The �Customer Advocate� 

    SQA�s motto might be: 

�You must please us because we represent the customer.�

    SQA�s role: 

- Identifies issues customers would likely find 
- Helps organization become sensitive to customer needs 
- Actively involved in performing �Act Like a Customer Testing� which 
  results in higher customer satisfaction

    What often happens:

- SQA may not have sufficient �domain knowledge� to accurately 
  represent customers 
- SQA may inject their own preferences or biases in the name of 
  customers 

 The �Analyst�

    SQA�s motto might be: 

�The more data we collect the more ammunition we have.�

    SQA�s role:

- Collects lots of data on all aspects of product and process 
- Good data can be used to help improve processes and products 
- Examples: defect data, test coverage data, requirements traceability 
  data, etc...

    What often happens: 

- SQA can�t see the forest for the data � analysis paralysis sets in 
- SQA becomes obsessed with meaningless metrics 
- Customers are more concerned with not finding defects than with 
  metrics 

Confused by all these different definitions?

OK, SO WHAT IS SQA?

First, some points to consider:

- To be effective, SQA must be independent of the development 
  organization.

  IV&V is effective because IV&V contractors are independent 
  (managerially, technically, and financially) of the software development 
  contractor. For SQA to be most effective, a similar degree of 
  independence is needed. The SQA function should report to the same 
  executive that the most senior development manager reports to. The SQA 
  budget should not be contingent upon development�s budget. And SQA 
  should attempt to collect as much technical information as possible 
  independently of the development organization.

- SQA provides Management with timely, factual information.

  The key here is that SQA provides Management with objective, technical 
  information. Management needs to view SQA as a source of meaningful 
  information that can help them make difficult decisions. SQA should not 
  make the release decision. That decision rightly belongs with 
  Management.

- Management uses this information to make appropriate business decisions.

  SQA should provide information on risk, potential customer impact, etc. 
  By using information from SQA, development, and other sources Management 
  can then make informed decisions about releasing products. The 
  information SQA provides needs to be considered in this decision, but 
  ultimately, the decision rests with Management. Sometimes SQA may not 
  agree with the decisions Management makes. (Get over it � it�s their job 
  not yours!). If Management never agrees with SQA, there�s a problem... 

Ok, here�s my definition of SQA:

SQA AS A �PROVIDER OF INFORMATION�

The most effective role that SQA can play is as a provider of useful, 
timely information.

In this role: 
  
  SQA�s motto would be: 

    �We review what was done and provide an objective technical assessment 
    supported by facts so that management can make better business 
    decisions.�


  SQA�s role would include:


    Providing objective technical information Management can use to make 
    better business decisions

    Providing information appropriate for the nature of products and risks 
    associated with those products

    Being more concerned with reducing risks than insisting on 100% 
    compliance with process 

WHAT ROLE IS BEST FOR YOUR ORGANIZATION?

The role SQA plays clearly needs to be commensurate with the business risk 
associated with product development as well as consumer risk in using the 
product. To find the SQA role suitable for your organization:
 
  Define SQA Goals and Objectives

  Align these goals and objectives with overall business goals and 
  objectives. Fctor in business risk and consumer risk. Remember that:
 
    �It is a mistake to assume that SQA people themselves can do anything 
    about quality.� [6]
 
  Define SQA�s Role Within the Organization

  Based on the goals and objectives, define the specific role SQA will 
  play. Remember, that:
 
    �If SQA fulfills its responsibilities and if senior management refuses 
    to allow line management to ship products until the SQA issues have 
    been addressed, then SQA can help management improve product quality.� 
    [6]
 
  Staff SQA With Talented People

  Management needs to ensure that the best people are found to staff the 
  SQA function. These people require training and support just like any 
  other critical function within the organization.
 
    �Getting good people into SQA is one of the most difficult problems 
    software managers face.� [6]
 
  Establish Effective SQA Processes

  Just like development, to be effective SQA needs good processes. The 
  processes developed by SQA should be focused on achieving the following 
  broad goals: 

    To help development improve software quality by assessing the quality 
    of software products as well as the quality of the processes used to 
    produce those products.

    To determine the level of compliance with established development 
    processes and procedures.

    To bring to Management�s attention any deficiencies in either the 
    product or the process so that Management can take appropriate 
    corrective action. 

I�d like to leave you with the following thoughts:
 
  �When software projects fail, it�s usually because a manager didn�t 
  insist that the work be done the right way.� [15]


  SQA isn�t responsible for the quality of your software � Development is.


  To change the quality of your software, you need to change the process 
  used to create it.
 
Till next time, adios amigos!

--------------------------------------------------------------------------------

***Monthly Morsels***

Every month in this space you�ll find additional information related to 
this month�s topic. 

- Software Quality Assurance Organizations:

  ASQ Software Division Website
  (http://www.asq.org/perl/index.pl?g=softwareforum)
  Software Quality Group of New England Website
  (http://www.sqadvice.com/SQGNE_Home.htm) 
  SEI Software Process Improvement Network (SPIN)
  (http://www.sei.cmu.edu/collaborating/spins/spins.html)
  Boston SPIN 
  (http://www.boston-spin.org/)
  Quality Assurance Institute (QAI)
  (http://www.qaiusa.com/) 
  And on a lighter note... finally, a good reason for document reviews
  (http://www.testingstuff.com/reviews.html)

- Books and articles: 

  [1] Nelson, J. G., "Software Testing in Computer-Driven Systems", in 
  Software Quality Management, ed. Fisher, Matthew J., and Cooper, John 
  D., Petrocelli Books, 1979 
  [2] Arthur, J. D. and Nance, R. E., �Verification and Validation Without 
  Independence: A Recipe for Failure�, Proc. 2000 Winter Simulation 
  Conference, Orlando FL, 2000. 
  [3] Wallace, D. R., and Fuji, R. U., �Software Verification and 
  Validation: Its Role in Computer Assurance and Its Relationship with 
  Software Project Management Standards�, National Institute of Standards 
  and Technology, Special Publication 500-165, May 1989 
  [4] Glass, R., Software Runaways: Lessons Learned from Massive Software 
  Project Failures, Prentice-Hall PTR, 1997 
  [5] Weiner, L., Digital Woes: Why We Should Not Depend On Software, 
  Addison-Wesley, 1993. 
  [6] Humphrey, W. S., Managing the Software Process, Addison-Wesley, 1989 
  [7] IEEE Glossary of Software Engineering Terminology, IEEE Standard 
  610.12-1990, IEEE, NY. 
  [8] Galin, D., Software Quality Assurance � From Theory to 
  Implementation, Pearson Education Limited, 2004. 
  [9] Paulk, M., et. al., The Capability Maturity Model: Guidelines for 
  Improving the Software Process, Addison-Wesley, 1995 
  [10] Reifer, D., State of the Art in Software Quality Management, Reifer 
  Consultants, 1985. 
  [11] Schulmeyer, G. G. and McManus, J. I., eds. Handbook of Software 
  Quality Assurance, 2nd ed, Van Nostrand Rheinhold, 1992 
  [12] Pressman, R. S., Software Engineering: A Practitioner's Approach, 
  McGraw-Hill, 4th edition, 1997. 
  [13] NASA Policy Directive NPD 8730.4A, Effective August 1, 2001. 
  [14] Johnson, J., �Chaos: The Dollar Drain of IT Project Failures�, 
  Application Development Trends, January 1995, p. 41-47. 
  [15] Humphrey, W., Winning with Software � An Executive Strategy, 
  Addison Wesley, 2002. 

--------------------------------------------------------------------------------

***Calendar***

Every month, you�ll find news here about local and national events that 
are of interest to the software community...

- Software Quality Calendar

  There are many organizations that sponsor monthly meetings, workshops, 
  and conferences of interest to software professionals. Find out what�s 
  happening... (http://www.swqual.com/links/upcoming.html)

- Workshops Offered by Software Quality Consulting

  Software Quality Consulting offers workshops in many topics related to 
  software process improvement. Get more info...
  (http://www.swqual.com/seminars/courses.html)

--------------------------------------------------------------------------------

***About SQC***

Software Quality Consulting provides consulting, training, and auditing 
services tailored to meet the specific needs of clients. We help clients 
fine-tune their software development processes and improve the quality of 
their software products. The overall goal is to help clients achieve 
Predictable Software Development(TM) � so that organizations can consistently 
deliver quality software with promised features in the promised timeframe.

To learn more about us and how we can help your organization, visit our 
web site (http://www.swqual.com/) or send us an email ([email protected]).

--------------------------------------------------------------------------------

I hope this newsletter has been informative and helpful. Your comments and 
feedback are most welcome. Send me your feedback... (mailto:[email protected])

Thanks,
Steve Rakitin
[email protected]


Food for Thought and Predictable Software Development are trademarks of Software 
Quality Consulting, Inc. 

Copyright � 2005. Software Quality Consulting, Inc. All rights reserved. Graphic 
design by Sage Studio