|
Server : Apache/2.4.62 System : FreeBSD fbsdweb2.web.rcn.net 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64 User : www ( 80) PHP Version : 8.3.8 Disable Function : NONE Directory : /domains/srakitin/OLD/newsletter/vol6/no5/ |
Upload File : |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>Food for Thought: Software Quality Assurance turns 50 - Part 2</title>
<link href="/newsletter/StyleSheet.css" rel="stylesheet" type="text/css">
</head>
<OpenTracking/>
<!-- Do NOT delete previous line if you want to get statistics on the number of opened emails -->
<body>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0">
<tr align="center" valign="top">
<td colspan="2"><img src="/newsletter/images/FoodForThoughtLogo.gif" alt="Food for Thought" width="600" height="105"></td>
</tr>
<tr class="Reference">
<td align="left" valign="top"><p>An e-newsletter published by<br>
Software Quality Consulting, Inc. </p>
</td>
<td align="right" valign="top"><p>September 2009 , Vol. 6 No. 5 <br>
[<a href="/newsletter/vol6/no5/vol6no5.txt" target="_blank">Text-only Version</a>]</p>
</td>
</tr>
</table>
<br>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td align="left" valign="top">
<p>Welcome to <em><strong>Food for Thought™</strong></em>, an e-newsletter from <strong><a href="/index.html?Intro" target="_blank">Software Quality Consulting</a></strong>. I've created free subscriptions for my valued business contacts. If you find this newsletter informative, I encourage you to continue reading. Feel free to pass this newsletter along to colleagues by clicking this <strong><a href="http://ui.constantcontact.com/sa/fwtf.jsp?m=1011294719058&ea=info%40swqual.com&a=1102611577848&id">Forward Email</a></strong> link. If you’ve received this newsletter from a colleague and would like to subscribe, please click this <strong><a href="/newsletter/Subscribe.htm?Newsletter" target="_blank">Enter New Subscription</a></strong> link. If you don't wish to receive this newsletter, click the <strong><a href="#bottom">SafeUnSubscribe</a></strong>™ link at the bottom of this newsletter, and you won’t be bothered again.</p>
<p>Your continued feedback on this newsletter is most welcome. Please send your comments and suggestions to <strong><a href="mailto:[email protected]">[email protected]</a></strong>.</p></td>
</tr>
</table>
<br>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="right" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/InThisIssue.gif" alt="In This Issue" width="114" height="37"></td>
<td width="15"> </td>
<td align="left" valign="top"><p>In <a href="#article"><strong>This Months’ Topic</strong></a>, I begin a discussion on the state of the software quality assurance profession...<br>
<br>
Regular features to look for each month are:</p>
<ul>
<li> <a href="#morsel"><strong>Monthly Morsels</strong></a><br>
Hints, tips, techniques and reference info related to this month’s topic</li>
</ul>
<ul>
<li> <a href="#calendar"><strong>Calendar</strong></a><br>
Conferences, workshops, and meetings of interest to software engineers, QA engineers and anyone interested in software development</li>
</ul>
</td>
</tr>
</table>
<br>
<br>
<a name="article"></a>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="left" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/ThisMonthsTopic.gif" alt="This Month's Topic" width="114" height="37"></td>
<td width="15"> </td>
<td width="471" align="left" valign="top" class="BodyText"><p align="center" class="Headline"><strong>Software Quality Assurance turns 50<br>
A critical look at the state of the profession<br>
<br>
Part 2 -
<strong> Present State of the Profession</strong> </strong> </p>
<p> Software Quality Assurance (SQA) as we know it was first applied to software development projects about 50 years ago. To recognize this important milestone, the state of the SQA profession is the topic for this issue of my newsletter. In my <strong><a href="/newsletter/vol6/no4/vol6no4.html" target="_blank">June newsletter</a></strong>, I discussed the history and evolution of SQA. In this month’s installment, I discuss some SQA successes and failures. The last installment will focus on the future of SQA.</p>
<p><strong> Present Situation</strong></p>
<p> Today the best most highly skilled software developers inject an average of one defect for every 8 lines of code they write. [2] The single most common reason software engineers inject defects has been and is still poorly written requirements.</p>
<p> We also have anecdotal information suggesting that through verification activities such as peer reviews and static analysis and validation testing, we typically find about 95% of these injected defects. The end result is:</p>
<blockquote>
<p><strong> released software has, on average, a defect density in the range of 5-6 defects per thousand lines of code (KLOC).</strong></p>
</blockquote> <p> So if we look at a typical software-based system that has one million lines of code, here’s what we’d expect to find:</p>
<blockquote>
<p><strong> Defects injected:</strong> using 1 defect /8 lines of code = ~120,000 </p>
<p><strong> Defects removed:</strong> assuming 95% found = 114,000 </p>
<p><strong> Defects remaining:</strong> (defects injected - defects removed) = <strong> 6,000 </strong></p>
</blockquote> <p> To put this in perspective, 2010 model-year cars will have about 100 million LOC. Given the above, there could be as many as <strong> 600,000</strong> defects that remain in the software that controls your car.</p>
<ul>
<li><strong><a href="/newsletter/vol4/no6/vol4no6.html" target="_blank">Read more about safety-critical software...</a></strong></li>
</ul>
<p>Recently, a prestigious group of researchers from the <strong><a href="http://sites.nationalacademies.org/nrc/index.htm" target="_blank">National Research Council</a></strong> published a book on the issues related to developing software for dependable systems. Here’s their sobering assessment of the present situation:</p>
<blockquote>
<p> “Society is increasingly dependent on software. Software failures can cause or contribute to serious accidents that result in death, injury, significant environmental damage, or major financial loss. Such accidents have already occurred and without intervention, the increasingly pervasive use of software - especially in arenas such as transportation, heath care, and the broader infrastructure - may make them more frequent and more serious.” [1]</p>
</blockquote></td>
</tr>
<tr>
<td align="left" valign="top" background="/newsletter/images/RedSpacer.gif"><p><img width="110" height="143" src="/newsletter/vol6/no5/vol6no5_clip_image002.jpg"></p>
<p align="center" class="Reference"> Complexity is increasing so rapidly that software<br>
engineers cannot possibly<br>
understand everything about how software works... </p></td>
<td> </td>
<td align="left" valign="top" class="BodyText"><p>The problem is exacerbated by a pervasive <strong>lack of evidence</strong> about both the incidence and severity of software failures. This lack of evidence has led the National Research Council researchers to the following conclusions: [1]</p>
<ul>
<li> Better <strong>evidence</strong> is needed so that approaches aimed at improving the dependability of software can be objectively assessed. </li>
</ul>
<ul>
<li> For now, the pursuit of dependability in software systems should focus on the <strong>construction and evaluation of evidence.</strong></li>
</ul>
<p>What we must recognize is that software engineering and SQA best practices are <strong>necessary but not sufficient</strong> to ensure that software systems are dependable and safe. This has become painfully clear as most agree that current software development methods have failed to keep pace with the exponential increase in software complexity.</p>
<blockquote>
<p> “An awareness of the need for simplicity comes only with bitter experience and humility gained from years of practice. There is no alternative to simplicity. Advances in technology or development methods will not make simplicity redundant; on the contrary, they will give it greater leverage.” [1]</p>
</blockquote>
<p><strong> While there have been some successes...</strong></p>
<p> While the outlook is generally bleak, there are a few positive results that are noteworthy:</p>
<ul>
<li><strong> Testing</strong></li>
</ul>
<blockquote>
<p>As observed by Tony Hoare: </p>
<blockquote>
<p>“The real value of tests is not that they detect bugs in the code but that they detect inadequacies in the methods, concentration, and skills of those who design and produce the code.” [4]</p>
</blockquote>
<p>The software industry has finally recognized that testing is an indispensable part of the software development process. Today, most every software development organization has some form of testing group. We have also recognized that in some cases, testing can find defects more easily than other methods. </p>
The fact that anecdotal evidence indicates that we are capable of finding up to 95% of injected defects is certainly encouraging.</blockquote>
<ul>
<li><strong> Test Automation</strong></li>
</ul>
<blockquote>
<p>Test automation is another bright spot. Today, the array of test automation tools available is staggering. More and more software development organizations are turning to test automation as a way to perform more rigorous testing more often and as a result, enable testers to focus on testing the more challenging aspects of complex applications. More and more organizations are using test automation in conjunction with nightly builds as a way to catch problems at the point they are introduced.</p>
<ul>
<li><strong><a href="http://en.wikipedia.org/wiki/List_of_GUI_testing_tools" target="_blank">Review a list of test automation tools...</a></strong></li>
</ul>
</blockquote>
<ul>
<li><strong> Independent Verification and Validation (IV&V)</strong></li>
</ul>
<blockquote>
<p>Large government organizations have recognized that for their software development programs to succeed, they need to include an IV&V function. Independence is one of the reasons IV&V has been so successful. As a result, organizations like NASA now require IV&V on all programs that involve software.</p>
</blockquote>
<ul>
<li><strong> Static Analysis and Formal Methods</strong></li>
</ul>
<blockquote>
<p>A new crop of static analyzers has emerged in the past few years to help catch problems before testing begins. These tools can find things like memory leaks and improper use of pointers that often can result in bugs that are difficult to find. </p>
In addition, there has been renewed interest in applying <strong><a href="http://en.wikipedia.org/wiki/Formal_methods" target="_blank">formal methods</a></strong> to complex, safety-critical applications. These methods can also be very effective in finding problems that would otherwise be very difficult to find.
<ul>
<li><strong><a href="http://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis" target="_blank">Review a list of static analysis tools...</a></strong></li>
</ul>
</blockquote>
<ul>
<li><strong> Training</strong></li>
</ul>
<blockquote>
<p><strong><a href="http://www.cs.fit.edu/" target="_blank">Florida Institute of Technology</a></strong> has been offering advanced courses on <strong><a href="http://testingeducation.org/" target="_blank">Software Testing</a></strong> developed under the guidance of <strong><a href="https://services.fit.edu/profiles/profile.php?value=135" target="_blank">Cem Kaner...</a></strong></p>
<p>The IEEE has published a <strong><a href="http://www.swebok.org/" target="_blank">Software Engineering Body of Knowledge (SWEBOK) </a></strong>that includes both software quality and software testing skills.</p>
<p>And lastly, amazon.com currently lists almost 14,000 books with the words <strong>software</strong> and <strong>quality</strong> in the title.</p>
<ul>
<li><strong><a href="/training/on_site.html" target="_blank">Review a list of additional training courses...</a></strong></li>
</ul>
</blockquote>
<p><strong> There have been many more failures...</strong></p>
<p> Software complexity has increased exponentially over the last five decades. The engineering discipline we use to develop and test software hasn’t kept pace with this increase in complexity. As a result, we still have problems developing large, complex systems.</p>
<ul>
<li><strong><a href="/newsletter/vol2/no10/vol2no10.html" target="_blank">Read about some high profile software failures</a></strong><strong></strong></li>
</ul>
<p>The <strong>Standish Group’s CHAOS 2009</strong><strong> Report</strong> shows a marked <strong>decrease</strong> in project success rates, with</p>
<ul>
<li><strong> 32%</strong> of all projects surveyed were <strong>successful</strong> - defined as projects delivered on time, on budget, with required features and functions. </li>
</ul>
<ul>
<li><strong> 44%</strong> of all projects surveyed were <strong>challenged</strong> - defined as late, over budget, and/or with less than the required features and functions </li>
</ul>
<ul>
<li><strong> 24%</strong> of all projects surveyed <strong>failed</strong> - defined as cancelled prior to completion or delivered and never used. </li>
</ul>
<p>The trend over the past 15 years is also very revealing:</p>
<table width="471" border="1" cellpadding="5" cellspacing="0" bordercolor="#000000" class="BodyText">
<tr bgcolor="#FFFF99">
<td align="left" valign="middle"><p><strong> Year </strong></p></td>
<td align="center" valign="middle"><p align="center"><strong> ‘09 </strong></p></td>
<td align="center" valign="middle"><p align="center"><strong> ‘06 </strong></p></td>
<td align="center" valign="middle"><p align="center"><strong> ‘04 </strong></p></td>
<td align="center" valign="middle"><p align="center"><strong> ‘02 </strong></p></td>
<td align="center" valign="middle"><p align="center"><strong> ‘00 </strong></p></td>
<td align="center" valign="middle"><p align="center"><strong> ‘98 </strong></p></td>
<td align="center" valign="middle"><p align="center"><strong> ‘96 </strong></p></td>
<td align="center" valign="middle"><p align="center"><strong> ‘94 </strong></p></td>
</tr>
<tr>
<td align="left" valign="middle"><p><strong> Successful Projects </strong></p></td>
<td align="center" valign="middle"><p align="center"> 32% </p></td>
<td align="center" valign="middle"><p align="center"> 35% </p></td>
<td align="center" valign="middle"><p align="center"> 29% </p></td>
<td align="center" valign="middle"><p align="center"> 34% </p></td>
<td align="center" valign="middle"><p align="center"> 28% </p></td>
<td align="center" valign="middle"><p align="center"> 26% </p></td>
<td align="center" valign="middle"><p align="center"> 27% </p></td>
<td align="center" valign="middle"><p align="center"> 16% </p></td>
</tr>
<tr>
<td align="left" valign="middle"><p><strong> Challenged </strong><strong>Projects</strong></p></td>
<td align="center" valign="middle"><p align="center"> 44% </p></td>
<td align="center" valign="middle"><p align="center"> 19% </p></td>
<td align="center" valign="middle"><p align="center"> 53% </p></td>
<td align="center" valign="middle"><p align="center"> 15% </p></td>
<td align="center" valign="middle"><p align="center"> 23% </p></td>
<td align="center" valign="middle"><p align="center"> 28% </p></td>
<td align="center" valign="middle"><p align="center"> 40% </p></td>
<td align="center" valign="middle"><p align="center"> 31% </p></td>
</tr>
<tr>
<td align="left" valign="middle"><p><strong> Failed </strong><strong>Projects</strong></p></td>
<td align="center" valign="middle"><p align="center"> 24% </p></td>
<td align="center" valign="middle"><p align="center"> 46% </p></td>
<td align="center" valign="middle"><p align="center"> 18% </p></td>
<td align="center" valign="middle"><p align="center"> 51% </p></td>
<td align="center" valign="middle"><p align="center"> 49% </p></td>
<td align="center" valign="middle"><p align="center"> 46% </p></td>
<td align="center" valign="middle"><p align="center"> 33% </p></td>
<td align="center" valign="middle"><p align="center"> 53% </p></td>
</tr>
</table>
<p>Note that some people have <strong><a href="http://www.infoq.com/news/Standish-Chaos-Report-Questioned" target="_blank">questioned</a></strong> the methods used by the Standish Group in their surveys. </p>
<p> Let’s look at other areas where we still have work to do...</p>
<ul>
<li><strong> Software Quality Assurance</strong></li>
</ul>
<blockquote>
<p>Many software development organizations have yet to recognize that SQA is much more than testing. As Roger Pressman [5] has stated, SQA is an umbrella for many value-added activities that can significantly improve product quality.</p>
</blockquote>
<p><img width="444" height="386" src="/newsletter/vol6/no5/vol6no5_clip_image001.jpg"></p>
<blockquote>
<p> SQA is still not recognized as a legitimate and important discipline. There are no universities that offer any kind of undergraduate degree in Software Quality. As a result, most people who are in an SQA role have few choices when seeking training in their field. This is a sad situation and needs to be rectified...</p>
<p>As observed by Watts Humphrey:</p>
</blockquote>
<ul>
<blockquote>
<p>“SQA is a valid discipline in its own right and people can be SQA experts without being software design experts. This SQA expertise is what is required to establish a strong quality program. It includes knowledge of statistical methods, quality control principles, the software process, and an ability to deal effectively with people in contentious situations.” [3]</p>
</blockquote>
<li><strong> Testing and Test Tools</strong></li>
</ul>
<blockquote>
<p>While there have been positive improvements in these key areas, there have still been many failures attributable to ineffective testing. This can be partly due to the: </p>
<ul>
<li> lack of education and training resources available to testers </li>
<li> increasing complexity of software systems </li>
<li> lack of domain knowledge</li>
</ul>
<p>Test automation tools also have a way to go as well. While there have been many significant improvements in tools, many test automation tools require programming experience and are far too complicated and expensive for many organizations.</p>
</blockquote>
<ul>
<li><strong> Requirements</strong></li>
</ul>
<blockquote>
<p>After 50 years of experience, you’d think that we would realize just how important clear, unambiguous requirements are. Sadly, many organizations just don’t get it. They rush into writing code without understanding what it is they are supposed to develop. Developers guess - and as you could imagine - they only guess right half of the time.</p>
</blockquote>
<ul>
<li><strong> Metrics</strong></li>
</ul>
<blockquote>
<p>We still have difficulty defining metrics in ways that can lead to meaningful comparisons across projects - even within the same organization. We need to agree on a small number of basic measures that can be applied across a wide variety of applications and organizations so that we can determine which projects are more effective at meeting project quality goals.</p>
</blockquote>
<ul>
<li><strong> Evidence of Dependability</strong></li>
</ul>
<blockquote>
<p>One of the key findings of the <strong>National Research Council’s</strong> study is that we need to focus on providing evidence that software is dependable. This evidence needs to reflect specific claims for dependability made by the developing organization. </p>
</blockquote>
<p> The list of areas for improvement is very long - the items I’ve described here are ones I believe are most important... </p>
<p><strong> The Bottom Line....</strong></p>
<p> There have certainly been some notable improvements in <strong>software quality assurance</strong> over the past 50 years. Unfortunately, these improvements have been overshadowed by several high-profile failures. To me, this indicates that the improvements in process and practice have not kept pace with the increases in complexity.</p>
<p> Next month, we’ll discuss the future of SQA...</p>
‘Til next time... </td>
</tr>
</table>
<br>
<br>
<a name="morsel"></a>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="right" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/MonthlyMorsels.gif" alt="Monthly Morsels" width="114" height="37"></td>
<td width="15"> </td>
<td align="left" valign="top"><p> Every month in this space, you’ll find additional information related to this month’s topic.</p>
<p><strong> References</strong></p>
<ol>
<li> Jackson, D., <em>et. al.</em>, <em>Software for Dependable Systems - Sufficient Evidence?</em> National Research Council, National Academies Press, 2007.<br>
<br>
</li>
<li>Humphrey, W., “The Quality Attitude”, <em>news@seinewsletter</em>, Number 3, 2004.<br>
<br>
</li>
<li>Humphrey, W. S., <em>Managing the Software Process</em>, Addison-Wesley, 1989.<br>
<br>
</li>
<li>Hoare, C. A. R., “How did software get so reliable without proof?”, <em>Lecture Notes in Computer Science</em>, 1051:1-17, 1996.<br>
<br>
</li>
<li>Pressman, R., <em>Software Engineering – A Practitioners Approach</em>, 4th edition, McGraw-Hill, 1997.</li>
</ol></td>
</tr>
</table>
<br>
<br>
<a name="calendar"></a>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="right" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/Calendar.gif" alt="Calendar" width="114" height="37"></td>
<td width="15"> </td>
<td align="left" valign="top"><p> Every month you’ll find news here about local and national events that are of interest to the software community…</p>
<ul>
<li><strong> Software Quality Calendar</strong></li>
</ul>
<blockquote>
<p>There are many organizations that sponsor monthly meetings, workshops, and conferences of interest to software professionals. <strong><a href="/links/upcoming.html" target="_blank">Find out what’s happening…</a></strong></p>
</blockquote>
<ul>
<li><strong> Workshops Offered by Software Quality Consulting</strong></li>
</ul>
<blockquote>
<p>Software Quality Consulting offers workshops in many topics related to software process improvement. <strong><a href="/seminars/courses.html" target="_blank">Get more info…</a></strong></p>
</blockquote></td>
</tr>
</table>
<br>
<br>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="right" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/AboutSQC.gif" alt="About SQC" width="114" height="37"></td>
<td width="15"> </td>
<td align="left" valign="top"><p> Software Quality Consulting provides consulting, training, and auditing services tailored to meet the specific needs of clients. We help clients fine-tune their software development processes and improve the quality of their software products. The overall goal is to help clients achieve Predictable Software Development™ – so that organizations can consistently deliver quality software with promised features in the promised timeframe. </p>
To learn more about how we can help your organization, <strong><a href="/index.html?AboutSQC" target="_blank">visit our web site</a></strong> or <strong><a href="mailto:[email protected]">send us an email</a></strong>.</td>
</tr>
</table>
<br>
<br>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td align="left" valign="top"><p> I hope this newsletter has been informative and helpful. Your comments and feedback are most welcome. <strong><a href="mailto:[email protected]">Send me your feedback…</a></strong></p>
<p>Thanks,</p>
<p> <img src="/newsletter/images/BusinessCard.gif" width="270" height="121" align="right"><img src="/newsletter/images/Signature.gif" width="90" height="68"><br>
Steve Rakitin<br>
<br>
<strong><a href="mailto:[email protected]">[email protected]</a></strong></p></td>
</tr>
</table>
<div align="center"><br>
<FONT class="Reference">Food for Thought, Predictable Software Development, Act Like a Customer,<br>
and ALAC are trademarks of Software Quality Consulting, Inc.<br>
Copyright 2009. Software Quality Consulting, Inc. All rights reserved.<br>
Graphic design by <a href="http://www.sarahcoledesign.com/" target="_blank"><strong>Sarah Cole Design</strong></a>.</FONT></div>
<a name="bottom"> </a></body>
</html>