Food for Thought: An e-newsletter published by Software Quality Consulting, Inc.
October 2005, Vol. 2 No. 9
What�s Your Story? - Storytelling as a Critical Testing Skill

To view a web version of this newsletter, click on the following link:
http://www.swqual.com/newsletter/vol2/no9/vol2no9.html.

--------------------------------------------------------------------------------

Welcome to Food for Thought(TM), an e-newsletter from Software Quality 
Consulting (http://www.swqual.com/index.html?Intro). I've created free
subscriptions for my valued business contacts. If you find this newsletter
informative, I encourage you to continue reading. Feel free to pass this
newsletter along to colleagues by clicking this Forward Email link. If you�ve
received this newsletter from a colleague and would like to subscribe, please
click this Enter New Subscription link
(http://www.swqual.com/newsletter/Subscribe.htm?Newsletter). If you don't wish
to receive this newsletter, click the SafeUnSubscribe(TM) link at the bottom of
this newsletter, and you won�t be bothered again.

Your continued feedback on this newsletter is most welcome. Please send 
your comments and suggestions to [email protected].

--------------------------------------------------------------------------------

In This Months� Topic, I discuss using storytelling as a testing 
technique. 

Regular features to look for each month are:

- Monthly Morsels
  Hints, tips, techniques and reference info related to this month�s topic 

- Calendar
  Conferences, workshops, and meetings of interest to software engineers, 
  QA engineers and anyone interested in software development 

--------------------------------------------------------------------------------

***What�s Your Story? - Storytelling as a Critical Testing Skill***

Once upon a time, there was a king who instructed his most trusted 
developers to build a new system for tracking taxes the king collected 
from the those who lived in his kingdom. The king had his Tax Collector 
prepare a Software Requirements Specification and hold a roundtable 
review. Questions were discussed and the developers started coding. To 
save some gold, the king decided to outsource the testing. He found a 
group of testers in a neighboring realm and issued a proclamation allowing 
these testers to work on this project from their realm. Testers would use 
carrier pigeons to report bugs back to the developers. 

The developers worked hard and eventually delivered the application to the 
testers several fortnights later than planned. Testers started testing and 
a few problems were found. The carrier pigeons were bored most of the 
time. Finally, when the moon was full, the testers were instructed to stop 
testing and deliver the application to the Tax Collector. The Tax 
Collector was very demanding.

The Tax Collector tried to import existing tax rolls from his Excelsior 
97 Spreadsheet into the new application � that didn�t work. He tried 
changing the number of goats owned by one of the peasants who lived in the 
realm � that didn�t work either. Needless to say, he found many problems 
and was not pleased.

The Tax Collector spoke to the king about the situation. The king was 
irate and asked the testers why they missed so many of these problems. The 
king soon discovered that the realm the testers were from (New Hamp Shire, 
I believe) had no taxes. The testers, the king surmised, were not aware of 
his tax rules:

- The king exacted additional taxes on Knights and Lords who lived within 
  the realm but who owned land in other realms. 

- Peasants were taxed at one rate on their land and at a different rates 
  on the number of cows and goats they owned, the number of wagons they 
  owned, and the amount of money they made selling their meager wares. 

- Senior peasants could grovel before the king and ask for tax abatements. 
  If the king was in a good mood that day, they would pay far less in 
  taxes. 

All of this was foreign to the testers, since in their realm, there were 
no such taxes. The testers were thus not able to imagine ways to test this 
software since they lacked realm-knowledge. All they could do was test 
against the Tax Collector�s incomplete requirements specification. 
Clearly, this was not sufficient.

Luckily, the king was able to find new testers from within his realm who 
were all too familiar with the vagaries of the king�s tax system. The new 
testers had first-hand knowledge about many different situations the Tax 
Collector would face. They even knew of some peasants who grazed their 
goats in New Hamp Shire to avoid the king�s taxes. Their knowledge was 
turned into tests and as a result, many more bugs were found. Eventually, 
the bugs were repaired and the Tax Collector was happy. The new testers 
however, were not, since the new Tax Collection software they just tested 
requires them to hand over more of their meager income to the king. 

The moral -- in addition to basic testing skills, testers need the 
following:

- domain knowledge � testers need to be intimately familiar with how 
  customers use the software they are testing... 

- the ability to tell stories � stories that illustrate specific scenarios 
  that need to be tested... 

Let�s look at how domain knowledge and storytelling can help add value to 
your testing. 

DOMAIN KNOWLEDGE

To be effective as a tester, you need to have domain knowledge. To further 
illustrate this point, Brian Marick [1] tells the following (probably 
true) story:

  A customer rents a car for a three-day business trip. Midway through the 
  rental, he extends the rental for another week. This, by the way, gives 
  him enough rental points to reach Preferred status. Several days later, 
  he calls to report that the car has been stolen. He insists that the 
  Preferred benefit of on-site replacement applies, even though he was not 
  a Preferred customer at the start of the rental. The company agrees and 
  a new car is delivered to him. Two days after that, he calls to report 
  that the �stolen� car has been found. It turns out he�d forgotten where 
  he�d parked it. He wants one of the cars picked up and the appropriate 
  transaction closed. Oh, and one other thing, the way he discovered the 
  missing car was by backing into it with its replacement, so they�re both 
  damaged. 

This story illustrates the importance of domain knowledge. Imagine if you 
had never rented a car in your life and were asked to test a rental car 
reservation system. You couldn�t possibly anticipate the kinds of problems 
in Brian�s story. As a result, those situations would not be tested and 
would only be found when a customer like Brian came along.

Testers who have had experiences like Brian�s, can express their domain 
knowledge in the form of realistic stories � stories that illustrate 
specific scenarios that need to be tested...

STORYTELLING

Brian�s car rental experience is an example of a �story� that can be used 
to identifyconditions in the rental car application that need to be tested 
[1]:

- Upgrading to Preferred status (during rental period) 
- Extending a rental period 
- Handling a �stolen� car 
- On-site replacement 
- Undoing an on-site replacement 
- Undoing handling of reported �stolen� car 
- Return of a damaged car 

Using stories as the basis for developing tests is not a new idea. It�s 
been around for some time and has many different names. It�s been called 
scenario testing, Act Like a Customer (ALAC) testing, and most recently, 
soap operatesting [1].

So what are scenario-based testing, ALAC testing, and soap opera testing? 
And how are these methods different from the traditional approach of 
testing against the spec?

SCENARIO-BASED TESTING

Cem Kaner [2] defines a scenario as:

  �... a hypothetical story, used to help a person think through a complex 
  problem or system. A scenario test is a test based on a scenario. The 
  ideal scenario test has several characteristics:

  - The test is based on a story about how the program is used, including 
    information about the motivations of the people involved. 

  - The story is motivating . A stakeholder with influence would push to 
    fix a program that failed this test. (Anyone affected by a program is 
    a stakeholder. A person who can influence development decisions is a 
    stakeholder with influence.) 
  
  - The story is credible . It not only could happen in the real world; 
    stakeholders believe that something like it probably will happen. 
  
  - The story involves a complex use of the program or a complex 
    environment or a complex set of data. 
    
  - The test results are easy to evaluate . This is valuable for all 
    tests, but is especially important for scenarios because they are 
    complex.� 

Read more about ways to create good scenarios...
(http://www.swqual.com/newsletter/vol2/no9/scenarios.pdf)

ACT LIKE A CUSTOMER TESTING(TM) (ALAC(TM))

Act Like a Customer Testing(TM) is based upon the following simple 
principles:

- All software has defects 

- When customers use your software, they typically only find a very small 
  percentage of the total number defects that are present 

- Finding and fixing those defects customers are likely to find and 
  ignoring the rest will result in very satisfied customers 

One caveat, ALAC(TM) Testing is not appropriate for software that is 
life-threatening or life-supporting... 
  
In addition to the three principles, ALAC(TM) requires testers to have domain 
knowledge and to have the ability to translate their domain knowledge into 
test cases. Often, when I�m helping testing groups improve effectiveness, 
the biggest obstacle I encounter is the lack of domain knowledge.
So how does one acquire domain knowledge? This is not easy and can�t be 
done in a short period of time. Here are some suggestions:

- Visit customers

  Many testers I�ve met have never visited or even talked to a real 
  customer. Testers need to be given the opportunity to spend a day in 
  your customer�s shoes. Testers need to experience first hand, your 
  customer�s pain, to observe what they do with your software, and how 
  that differs from the way in which they�ve been testing... 
  Spend time (a lot of time) with your technical support team

- Testers need to become best friends with technical support. Testers 

  should make it a point to take a support person to lunch at least once a 
  month. This way, testers can find out what wacky things customers are 
  doing with your software so that they can create stories, scenarios, and 
  tests for them. I encourage testers to spend time working in support 
  answering calls from customers. Again, this serves to help gain 
  important first-hand experiences... 

- Recruit testers from Technical Support

  It is easier to train someone who has extensive domain knowledge in good 
  testing practices than it is to provide domain knowledge to someone who 
  has good testing skills. The QA Group should be a possible career path 
  for your Technical Support staff. Both groups should work closely with 
  each other. 

- Attend industry-specific conferences

  Industry-specific conferences are a good way to learn more about your 
  industry and the people who work in the industry, your potential 
  customers. 

Like I said, acquiring domain knowledge is not easy and can�t happen 
overnight. But it is definitely worth the effort if you are serious about 
improving your effectiveness as a tester and if your company is serious 
about reducing the number of defects your customers find...

SOAP OPERA TESTING

Hans Buwalda defines Soap Opera Testing as follows:

  �As many readers know, soap operas are dramatic daytime television shows 
  that were originally sponsored by soap vendors. They depict life in a 
  way that viewers can relate to, but the situations portrayed are 
  typically condensed and exaggerated. In one episode, more things happen 
  to the characters than most of us will experience in a lifetime. 
  Opinions may differ about whether soap operas are fun to watch, but it 
  must be great fun to write them. Soap opera testing is similar to a soap 
  opera in that tests are based on real life, exaggerated, and condensed.� 
  [1] 

Buwalda devised this approach by inviting people who had extensive domain 
knowledge to work with testers in small teams to create stories based on 
extreme or exaggerate situations. By working together, the testers were 
able to create several stories, which were then turned into tests using 
the scenario-based testing approach described above.

TESTING AGAINST THE SPEC...

Testing against the spec is an important kind of testing to do. However, 
for most complex applications, it�s never sufficient. You always need to 
do scenario testing, Act Like a Customer Testing(TM), Soap Opera Testing and 
even some lightly scripted exploratory testing if you are truly concerned 
with delivering value to your customers.

Buwalda [1] illustrates the major difference between what he calls 
�mechanical testing� (testing against the spec) and Soap Opera Testing 
(using stories, scenarios, and ALAC(TM)).

The mechanical tests are often derived solely from one requirement in the 
Software Requirements Spec (SRS) or Functional Spec. This results in a 
one-to-one mapping between requirements and tests. This is good because it 
represents the place to start. And, for those test teams that have testers 
with little or no domain knowledge, this is the kind of testing that those 
folks can do. Figure 1 below illustrates the mapping between requirements 
(or objectives) and tests when testers are focused on �testing against the 
spec�. 

When testers use stories, soap operas, scenarios, ALAC(TM), and exploratory 
testing techniques, they create many-to-many relationships between 
requirements and tests, as shown in Figure 2. This helps uncover problems 
that would clearly be missed if we only did the mechanical type of testing 
against the spec.

AT THE END OF THE DAY, TESTERS NEED TO WRITE TEST CASES...

Whether you�re testing against the spec or doing scenario-based testing 
using stories, ALAC(TM) tests, or soap operas, at the end of the day, testers 
need to write test cases. So what is a test case? Well, like most things 
in software engineering, there is no universally accepted definition of 
what constitutes a �test case�. 

Cem Kaner has a very insightful definition. He says:

  �In my view, a test case is a question that you ask of the program. The 
  point of running the test is to gain information, for example whether 
  the program will pass or fail the test.

  An important implication of defining a test case as a question is that a 
  test case must be reasonably capable of revealing information .� [3] 

Brian Marick uses another term to describe a �lightly documented test 
case.� He calls this a test idea . 

  �A test idea is a brief statement of something that should be tested. 
  For example, if you're testing a square root function, one idea for a 
  test would be �test a number less than zero�. The idea is to check if 
  the code handles an error case.� [4] 

Read more about test case definitions...
(http://www.swqual.com/newsletter/vol2/no9/testcase.pdf)

We also need a definition of what constitutes a �good� test case. I often 
refer to test cases as being either �productive� or �non-productive�. I 
define these terms as follows:

- a �productive� test case is a test that has a high probability of 
  finding defects 

- a �non-productive� test case has a very low probability of finding 
  defects 

Given these definitions, we would want testers to write as many 
�productive� test cases as possible. Since test suites are rarely 
reviewed, many companies have very large test suites that contain mostly 
non-productive tests.

How can you tell if your scenario tests are productive or not? One way is 
to review them with your peers and ask the question, is this test likely 
to find a defect that a customer would find?

Another way is to use Caper Jones� defect removal efficiency metric. This 
metric measures how good a job your testers are doing at Acting Like a Customer when they test...

Read more about defect-removal efficiency metric...
(http://www.swqual.com/newsletter/vol2/no9/Capers.pdf)

The higher the defect removal efficiency metric, the better your test 
cases are at finding those defects that customers are likely to find. 
So what�s your story?

To improve effectiveness, testers need domain knowledge and storytelling 
skills. Testers can acquire domain knowledge in several ways. Storytelling 
skills, well that�s another story...

Have you heard any good stories lately? More importantly, have you told 
any good stories lately?

PAY IT FORWARD

If you find this newsletter of value, please consider the following:

  Norm Kerth is a highly respected consultant who developed the Project 
  Retrospective techniques discussed in the July-Aug newsletter
  (http://www.swqual.com/newsletter/vol2/no7/vol2no7.html). He was in 
  a serious car accident and suffered a disabling brain injury. As a 
  result, he cannot work and lives on a very limited income. You can help 
  recognize his contribution to our industry by sending a small donation. 
  Checks can be made payable to Norm Kerth Benefit Fund and sent to Norm 
  Kerth Benefit Fund c/o Process Impact, 11491 SE 119th Drive, Clackamas, 
  OR 97015-8778. You can also visit Karl Weiger�s website (Process Impact �
  http://www.processimpact.com/goodies.shtml) for more details about
  contributing to the fund. Thanks.

Read more about the Pay It Forward foundation...
(http://www.payitforwardfoundation.org/)

--------------------------------------------------------------------------------

***Monthly Morsels***

Every month in this space, you�ll find additional information related to 
this month�s topic.

- References:

  [1] Buwalda, H., �Soap Opera Testing�, Better Software, February 2004.

  [2] Kaner, C., �Scenario Testing�, June 2003.

  [3] Kaner, C., �What is a Good Test Case?�, presented at STAR East, May 
  2003.

  [4] Marick, B., �Testing for Programmers�, Half-day workshop, 2000. 

- Books:

  Kaner, C. et. al., Testing Computer Software, 2nd edition, Thompson 
  Computer Press, 1993.

  Marick, B., The Craft of Software Testing, Prentice-Hall, 1995.

  Kaner, C., et. al., Lessons Learned in Software Testing, Wiley, 2002.

  Kit, E., Software Testing in the Real World, Addison-Wesley, 1995.

  Beizer, B., Black-Box Testing: Techniques for Functional Testing of 
  Software and Systems, Wiley, 1995.

  Copeland, L., A Practitioners Guide to Software Test Design, Artech 
  House, 2004.

  Myers, G., The Art of Software Testing, Wiley Interscience, 1979. 

- On-line Resources:

  Better Software Magazine
  (http://www.stickyminds.com/BetterSoftware/magazine.asp)

  This magazine always has interesting articles on current testing 
  practices...

  Software Engineering Institute
  (http://www.sei.cmu.edu/)

  Here you can search for articles on testing � there are many...

  IEEE Computer Society Digital Library
  (http://www.computer.org/portal/site/ieeecs/index.jsp)

  Search the vast library of the IEEE Computer Society for testing 
  resources and standards... 

--------------------------------------------------------------------------------

***Calendar***

Every month, you�ll find news here about local and national events that 
are of interest to the software community ...

- Software Quality Calendar

  There are many organizations that sponsor monthly meetings, workshops, 
  and conferences of interest to software professionals. Find out what�s 
  happening...
  (http://www.swqual.com/links/upcoming.html)

- Workshops Offered by Software Quality Consulting

  Software Quality Consulting offers workshops in many topics related to 
  software process improvement. Get more info...
  (http://www.swqual.com/seminars/courses.html)

--------------------------------------------------------------------------------

***About SQC**

Software Quality Consulting provides consulting, training, and auditing 
services tailored to meet the specific needs of clients. We help clients 
fine-tune their software development processes and improve the quality of 
their software products. The overall goal is to help clients achieve 
Predictable Software Development(TM) � so that organizations can consistently 
deliver quality software with promised features in the promised timeframe.

To learn more about how we can help your organization, visit our web site
(http://www.swqual.com/index.html?AboutSQC) or send us an email
([email protected]).

--------------------------------------------------------------------------------

I hope this newsletter has been informative and helpful. Your comments and 
feedback are most welcome. Send me your feedback... ([email protected])

Thanks,

Steve Rakitin
[email protected]


Food for Thought, Predictable Software Development, Act Like a Customer,
and ALAC are trademarks of Software Quality Consulting, Inc.
Copyright 2005. Software Quality Consulting, Inc. All rights reserved.
Graphic design by Sage Studio