|
Server : Apache/2.4.62 System : FreeBSD fbsdweb2.web.rcn.net 14.1-RELEASE FreeBSD 14.1-RELEASE releng/14.1-n267679-10e31f0946d8 GENERIC amd64 User : www ( 80) PHP Version : 8.3.8 Disable Function : NONE Directory : /domains/srakitin/OLD/newsletter/vol6/no1/ |
Upload File : |
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<title>Food for Thought: Risk-based Testing</title>
<link href="/newsletter/StyleSheet.css" rel="stylesheet" type="text/css">
</head>
<OpenTracking/>
<!-- Do NOT delete previous line if you want to get statistics on the number of opened emails -->
<body>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0">
<tr align="center" valign="top">
<td colspan="2"><img src="/newsletter/images/FoodForThoughtLogo.gif" alt="Food for Thought" width="600" height="105"></td>
</tr>
<tr class="Reference">
<td align="left" valign="top"><p>An e-newsletter published by<br>
Software Quality Consulting, Inc. </p>
</td>
<td align="right" valign="top"><p>January 2009 , Vol. 6 No. 1 <br>
[<a href="/newsletter/vol6/no1/vol6no1.txt" target="_blank">Text-only Version</a>]</p>
</td>
</tr>
</table>
<br>
<br>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td align="left" valign="top">
<p>Welcome to <em><strong>Food for Thought™</strong></em>, an e-newsletter from <strong><a href="/index.html?Intro" target="_blank">Software Quality Consulting</a></strong>. I've created free subscriptions for my valued business contacts. If you find this newsletter informative, I encourage you to continue reading. Feel free to pass this newsletter along to colleagues by clicking this <strong><a href="http://ui.constantcontact.com/roving/sa/fp.jsp?plat=i&p=f&m=sctz69n6">Forward Email</a></strong> link. If you’ve received this newsletter from a colleague and would like to subscribe, please click this <strong><a href="/newsletter/Subscribe.htm?Newsletter" target="_blank">Enter New Subscription</a></strong> link. If you don't wish to receive this newsletter, click the <strong><a href="#bottom">SafeUnSubscribe</a></strong>™ link at the bottom of this newsletter, and you won’t be bothered again.</p>
<p>Your continued feedback on this newsletter is most welcome. Please send your comments and suggestions to <strong><a href="mailto:[email protected]">[email protected]</a></strong>.</p></td>
</tr>
</table>
<br>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="right" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/InThisIssue.gif" alt="In This Issue" width="114" height="37"></td>
<td width="15"> </td>
<td align="left" valign="top"><p>In <a href="#article"><strong>This Months’ Topic</strong></a>, I discuss techniques for risk-based testing...<br>
<br>
Regular features to look for each month are:</p>
<ul>
<li> <a href="#morsel"><strong>Monthly Morsels</strong></a><br>
Hints, tips, techniques and reference info related to this month’s topic</li>
</ul>
<ul>
<li> <a href="#calendar"><strong>Calendar</strong></a><br>
Conferences, workshops, and meetings of interest to software engineers, QA engineers and anyone interested in software development</li>
</ul>
</td>
</tr>
</table>
<br>
<br>
<a name="article"></a>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="left" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/ThisMonthsTopic.gif" alt="This Month's Topic" width="114" height="37"></td>
<td width="15"> </td>
<td width="471" align="left" valign="top" class="BodyText"><p align="center" class="Headline">Risk-based Testing</p>
<p> Managing risk is a challenge for every project team. Some project teams ignore risk and hope it won’t affect them - it always does. Other project teams take a more proactive approach by identifying and managing risks and coming up with possible mitigations - a much more realistic and effective approach.</p>
<p> My earlier discussions on risk (<strong><a href="/newsletter/vol5/no5/vol5no5.html" target="_blank">June 2008</a></strong> and <strong><a href="/newsletter/vol5/no6/vol5no6.html" target="_blank">Sept 2008</a></strong>) identified many different kinds of risk that project teams typically encounter. These risks may include both<strong> internal risks</strong> as well as <strong>external risks</strong>.</p>
<p><strong>Internal Risks</strong> [1] may include:</p>
<ul>
<li><strong> Schedule Risks</strong></li>
</ul>
<ul>
<ul>
<li> Is the schedule realistic? </li>
<li> What assumptions were made in developing the schedule? </li>
<li> Are all the resources identified in the schedule available from the start?</li>
</ul>
<blockquote>
<p><strong><a href="http://www.ieeeboston.org/edu/2009spring/course_estimating.htm" target="_blank">Learn more about Estimating and Scheduling Best Practices...</a></strong></p>
</blockquote>
</ul>
<ul>
<li><strong> Staffing Risks</strong><br>
<br>
</li>
<ul>
<li> Are the best people available? </li>
<li> Do they have the right skills for this project? </li>
<li> Are enough people with right skills available? </li>
<li> Are people committed for the duration of the project? </li>
<li> Have staff members received necessary training? </li>
<li> Will turnover likely affect the project? </li>
</ul>
</ul>
<ul>
<li><strong> Process Risks</strong>
<br>
<br>
<ul>
<li> Are requirements well defined and unambiguous? </li>
<li> Is there a documented development process? </li>
<li> Does Management support following the development process? </li>
<li> Is the development process followed? </li>
<li> Are project software standards followed? </li>
<li> Are peer reviews part of the development process? </li>
<li> Has everyone been trained in peer reviews? </li>
<li> Are CM tools, procedures, and training in place? </li>
<li> Is there a process for changing requirements?</li>
</ul>
<blockquote>
<p><strong><a href="/training/requirements.html" target="_blank">Learn more about Writing Requirements...</a></strong></p>
<p><strong><a href="/training/peer_reviews.html" target="_blank">Learn more about Peer Reviews and Inspections...</a></strong></p>
</blockquote>
</li>
</ul>
<ul>
<li><strong> Technology Risks</strong>
<br>
<br>
<ul>
<li> Is technology new to your organization? </li>
<li> Are new algorithms required? </li>
<li> Does software interface with new or unproven hardware? </li>
<li> Does software interface with unproven 3rd party software? </li>
<li> Are there unreasonable performance requirements?</li>
</ul>
</li>
</ul>
<p><strong>External risks</strong> include:</p>
<ul>
<li><strong> Risks to society</strong></li>
</ul>
<blockquote>
<p>In 2004, the US Food and Drug Administration (FDA) recalled an infusion pump - a device used in hospitals to regulate the dosage of intravenous medication. The recall was initiated after several patients died as a result of receiving over doses of medication. An investigation revealed that defective software in the device allowed the dosage time information (hours and minutes) to be interchanged thus leading to the over dosage.</p>
<p><strong><a href="http://www.ieeeboston.org/edu/2009spring/course_med_devices.htm" target="_blank">Learn more about reducing risks of medical device software...</a></strong></p>
</blockquote>
<ul>
<li><strong> Financial or economic risks</strong></li>
</ul>
<blockquote>
<p>In 2003, defective software was a major contributor to the Northeast power blackout, the worst power system failure in North America. Over 50 million customers lost power as 100 power plants were shut down. Financial losses from this failure were estimated at <strong>$6 billion</strong>. [3]</p>
<p><strong><a href="/training/swvvmissioncritical.html" target="_blank">Learn more about Software Verification and Validation for Mission-critical Systems...</a></strong></p>
</blockquote>
<ul>
<li><strong> Political risks</strong></li>
</ul>
<blockquote>
<p>Software has even been involved in politics. The public’s confidence in electronic voting machines has been tainted due to poor design and ignored risks. A recent controversy in California highlights this issue: </p>
<blockquote>
<p>“California Secretary of State Debra Bowen announced on Friday that the state hopes to recertify and continue using electronic voting machines produced by Diebold, Sequoia, and Hart, even though the machines have known security vulnerabilities and severe flaws. The state government decided that the machines can still be used as long as the vendors adhere to a lengthy list of requirements that aim to limit the potential for security breaches and machine failure.</p>
<p>This announcement from the state follows extensive red team security audits that illuminated profound security failings in all of the electronic voting machines that were subjected to scrutiny. The security researchers, who analyzed the voting machines found ways to modify firmware, gain root access, trivially circumvent voting machine physical security mechanisms, install self-propagating trojan horses, and manipulate mock elections. On Diebold’s voting machine, which uses the Windows operating system, researchers even found a remotely-accessible administrative account that wasn’t protected by a password.” [4] </p>
</blockquote>
</blockquote>
<p>Risk-based testing is yet another attempt to try to focus the testing activity in areas that provide the most value to customers and development organizations.</p>
<p><strong> What kinds of risks are we talking about?</strong></p>
<p> In the context of risk-based testing, we want to identify areas of <strong>risk to your customers</strong>. Some examples:</p>
<ul>
<li><strong> Customers </strong> have problems installing your newly-released application </li>
</ul>
<ul>
<li><strong> Customers</strong> report problems with using new features - they don’t seem to work as they had expected them to </li>
</ul>
<ul>
<li><strong> Customers</strong> report data integrity problems </li>
</ul>
<ul>
<li><strong> Customers </strong> have performance problems </li>
</ul>
<p> Risk-based testing is all about identifying and managing risks that could negatively affect your customers when they use your software. </p>
<p> How can project teams identify these risks? There are a couple of tools and techniques available to help...</p>
<p><strong> What do t-shirts have to do with risk?</strong></p>
<p> Given that risk-based testing is focused on risks that could negatively affect your customers, there is likely much discussion and disagreement within the project team as to what exactly those risks might be... here’s where the t-shirts come in.</p>
<p><strong> T-shirt sizing</strong> is a very effective estimating technique developed by Steve McConnell [2]. We can use this technique to help identify risks as well. Here’s how...</p>
<blockquote>
<p> Key members of the Project Team - representing Marketing, Development, QA and Customer Support identify the most critical risks they see from the customer’s perspective. These risks are put into a table and each group is asked to rank all of them using t-shirt sizes - small, medium and large as illustrated below. Once the risks are ranked, the group convenes and discusses them. At the end of the discussion, new risks may be identified and changes to the rankings may be made. The process iterates until the team agrees on the risks and their rankings.</p>
<p> The most critical risks are identified and these are then the risks that testers focus on in their testing. </p>
</blockquote> <table width="471" border="1" cellpadding="3" cellspacing="0" bordercolor="#000000" class="BodyText">
<tr align="center" valign="middle" bgcolor="#FFFF99">
<td width="73"><p align="center"><strong> Risk</strong></p></td>
<td width="96"><p align="center"><strong> Marketing</strong></p></td>
<td width="120"><p align="center"><strong> Development</strong></p></td>
<td width="84"><p align="center"><strong> QA </strong></p></td>
<td width="96"><p align="center"><strong> Customer Support </strong></p></td>
</tr>
<tr align="center" valign="middle">
<td width="73" bgcolor="#FFCC99"><p align="center"> Risk A </p></td>
<td width="96"><p align="center"><strong> Large</strong></p></td>
<td width="120"><p align="center"><strong> Small</strong></p></td>
<td width="84"><p align="center"><strong> Medium</strong></p></td>
<td width="96"><p align="center"><strong> Medium</strong></p></td>
</tr>
<tr align="center" valign="middle">
<td width="73" bgcolor="#FFCC99"><p align="center"> Risk B </p></td>
<td width="96"><p align="center"><strong> Small</strong></p></td>
<td width="120"><p align="center"><strong> Large</strong></p></td>
<td width="84"><p align="center"><strong> Large</strong></p></td>
<td width="96"><p align="center"><strong> Large</strong></p></td>
</tr>
<tr align="center" valign="middle">
<td width="73" bgcolor="#FFCC99"><p align="center"> Risk C </p></td>
<td width="96"><p align="center"><strong> Large</strong></p></td>
<td width="120"><p align="center"><strong> Small</strong></p></td>
<td width="84"><p align="center"><strong> Medium</strong></p></td>
<td width="96"><p align="center"><strong> Medium</strong></p></td>
</tr>
<tr align="center" valign="middle">
<td width="73" bgcolor="#FFCC99"><p align="center"> Risk D </p></td>
<td width="96"><p align="center"><strong> Medium</strong></p></td>
<td width="120"><p align="center"><strong> Medium</strong></p></td>
<td width="84"><p align="center"><strong> Small</strong></p></td>
<td width="96"><p align="center"><strong> Small</strong></p></td>
</tr>
<tr align="center" valign="middle">
<td width="73" bgcolor="#FFCC99"><p align="center"> Risk E </p></td>
<td width="96"><p align="center"><strong> Small</strong></p></td>
<td width="120"><p align="center"><strong> Small</strong></p></td>
<td width="84"><p align="center"><strong> Medium</strong></p></td>
<td width="96"><p align="center"><strong> Medium</strong></p></td>
</tr>
</table>
<p><strong> Fault Tree Analysis (FTA)</strong></p>
<p> A fault tree is a tool that can help uncover potential customer risks in your applications before they are released to customers. To use FTA you will need a small team of people - ideally one developer, one tester, one customer service person and a facilitator (a.k.a. Project Manager). Here’s what happens:</p>
<p> The FTA Team comes up with a list of say 5 or 10 of the <strong>worst possible things that could happen when your customers try to use your new application. </strong>These could be things like:</p>
<ul>
<li> The install doesn’t work </li>
<li> The migration process from the previous release doesn’t work </li>
<li> Customers are not happy with performance </li>
<li> Once installed, the system crashes often... </li>
<li> etc. </li>
</ul>
<p> The team then creates a Fault Tree for each item on the list. Each item is placed at the top of the fault tree and then the team asks - How can this happen? Then they identify ways that the item at the top can happen. Here’s a simple example:</p>
<p><img width="464" height="154" src="/newsletter/vol6/no1/vol6no1_clip_image002.jpg"></p>
<p> In this example, the team said that there are four ways this could happen. Once they are satisfied that they haven’t overlooked anything, they pick one of these and decompose it further, by asking again “How can this happen” as shown below...</p>
<p align="center"><img width="464" height="281" src="/newsletter/vol6/no1/vol6no1_clip_image004.jpg"></p>
<p> The team determined there are two ways that could cause an Install Script Problem - one being the wrong version of the script was included in the release and the other being that some of the assumptions used in creating the install script were wrong. You will note that Wrong Version and Used wrong assumptions appear in a circle. This means that they are basic events - in other words, they can’t be further decomposed. Often there may be a few intermediate levels between the top row and the basic events... </p>
<p> This process is repeated until all of the branches of the tree are decomposed down to their basic events. Once the tree is completed, the testers now have several things they should test to help reduce the risk that the Install Doesn’t Work...</p>
<p align="center"><strong><a href="/training/risk2.html" target="_blank">Learn more about Risk Management using Fault Tree Analysis...</a></strong></p>
<p><strong> The Bottom Line</strong></p>
<p> Risk-based testing can add value to the testing activity by dramatically reducing the likelihood that customers will encounter the risks you identify. Clearly, the team identifying these risks needs to have enough domain knowledge to really understand what your customers do with your software.</p>
<p> Risk-based testing should be one of several different testing methods used on projects. These include:</p>
<ul>
<li> Requirements-based testing </li>
</ul>
<ul>
<li> Scenario testing </li>
</ul>
<ul>
<li> Act Like a Customer Testing™</li>
</ul>
<ul>
<li> Risk-based testing </li>
</ul>
<ul>
<li> Exploratory testing </li>
</ul>
<p>The wider the variety of test methods used, the more effective your testing will be.</p>
‘Til next time...</td>
</tr>
</table>
<br>
<br>
<a name="morsel"></a>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="right" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/MonthlyMorsels.gif" alt="Monthly Morsels" width="114" height="37"></td>
<td width="15"> </td>
<td align="left" valign="top"><p> Every month in this space, you’ll find additional information related to this month’s topic.</p>
<p><strong> References</strong></p>
<ol>
<li> Pressman, R., <em>Software Engineering: A Practitioner’s Approach</em>, McGraw-Hill, 1997, 4th ed. <br>
<br>
</li>
<li> McConnell, S., <em>Software Estimation – Demystifying the Black Art</em>, Microsoft Press, 2006 <br>
<br>
</li>
<li><strong><a href="http://en.wikipedia.org/wiki/2003_North_America_blackout" target="_blank"> 2003 Northeast Power Blackout affects 50 million people</a><br>
<br>
</strong></li>
<li> Paul, R., “<strong><a href="http://arstechnica.com/news.ars/post/20070806-california-to-recertify-insecure-voting-machines.html" target="_blank">California to recertify insecure voting machines</a></strong>”, <em>ars technica</em>, August 2007. </li>
</ol>
<p><strong>Additional Resources</strong></p>
<ol>
<li><strong><a href="http://en.wikipedia.org/wiki/Fault_tree_analysis" target="_blank">Fault Tree Analysis basics</a><br>
<br>
</strong></li>
<li>Lister, T. and DeMarco, T., <em>Waltzing With Bears: Managing Risk on Software Projects</em>, Dorset House, 2003. </li>
</ol></td>
</tr>
</table>
<br>
<br>
<a name="calendar"></a>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="right" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/Calendar.gif" alt="Calendar" width="114" height="37"></td>
<td width="15"> </td>
<td align="left" valign="top"><p> Every month you’ll find news here about local and national events that are of interest to the software community…</p>
<ul>
<li><strong> Software Quality Calendar</strong></li>
</ul>
<blockquote>
<p>There are many organizations that sponsor monthly meetings, workshops, and conferences of interest to software professionals. <strong><a href="/links/upcoming.html" target="_blank">Find out what’s happening…</a></strong></p>
</blockquote>
<ul>
<li><strong> Workshops Offered by Software Quality Consulting</strong></li>
</ul>
<blockquote>
<p>Software Quality Consulting offers workshops in many topics related to software process improvement. <strong><a href="/seminars/courses.html" target="_blank">Get more info…</a></strong></p>
</blockquote></td>
</tr>
</table>
<br>
<br>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td width="114" align="right" valign="top" background="/newsletter/images/RedSpacer.gif"><img src="/newsletter/images/AboutSQC.gif" alt="About SQC" width="114" height="37"></td>
<td width="15"> </td>
<td align="left" valign="top"><p> Software Quality Consulting provides consulting, training, and auditing services tailored to meet the specific needs of clients. We help clients fine-tune their software development processes and improve the quality of their software products. The overall goal is to help clients achieve Predictable Software Development™ – so that organizations can consistently deliver quality software with promised features in the promised timeframe. </p>
To learn more about how we can help your organization, <strong><a href="/index.html?AboutSQC" target="_blank">visit our web site</a></strong> or <strong><a href="mailto:[email protected]">send us an email</a></strong>.</td>
</tr>
</table>
<br>
<br>
<table width="600" border="0" align="center" cellpadding="0" cellspacing="0" class="BodyText">
<tr>
<td align="left" valign="top"><p> I hope this newsletter has been informative and helpful. Your comments and feedback are most welcome. <strong><a href="mailto:[email protected]">Send me your feedback…</a></strong></p>
<p>Thanks,</p>
<p> <img src="/newsletter/images/BusinessCard.gif" width="270" height="121" align="right"><img src="/newsletter/images/Signature.gif" width="90" height="68"><br>
Steve Rakitin<br>
<br>
<strong><a href="mailto:[email protected]">[email protected]</a></strong></p></td>
</tr>
</table>
<div align="center"><br>
<FONT class="Reference">Food for Thought, Predictable Software Development, Act Like a Customer,<br>
and ALAC are trademarks of Software Quality Consulting, Inc.<br>
Copyright 2009. Software Quality Consulting, Inc. All rights reserved.<br>
Graphic design by <a href="http://www.sarahcoledesign.com/" target="_blank"><strong>Sarah Cole Design</strong></a>.</FONT></div>
<a name="bottom"> </a></body>
</html>