Testing Complex and Embedded Systems

Saunder November 10, 2010 10:35 K11134˙C000

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

CRC Press

Taylor & Francis Group

6000 Broken Sound Parkway NW, Suite 300

Boca Raton, FL 33487-2742

© 2011 by Taylor and Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Printed in the United States of America on acid-free paper

10 9 8 7 6 5 4 3 2 1

International Standard Book Number: 978-1-4398-2140-4 (Hardback)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts have been

made to publish reliable data and information, but the author and publisher cannot assume responsibility for the

validity of all materials or the consequences of their use. The authors and publishers have attempted to trace the

copyright holders of all material reproduced in this publication and apologize to copyright holders if permission to

publish in this form has not been obtained. If any copyright material has not been acknowledged please write and let

us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmitted,

or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented, includ￾ing photocopying, microfilming, and recording, or in any information storage or retrieval system, without written

permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.com

(http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood Drive, Danvers,

MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and registration for a variety

of users. For organizations that have been granted a photocopy license by the CCC, a separate system of payment

has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for

identification and explanation without intent to infringe.

Library of Congress Cataloging-in-Publication Data

Pries, Kim H., 1955-

Testing complex and embedded systems / Kim H. Pries, Jon M. Quigley.

p. cm.

Summary: “Using combinatorial approaches, this book aims to motivate testers and testing

organizations to perform meaningful testing. The text details planning activities prior to

testing, how to scope the work, and how to achieve a successful conclusion. Rather than

presenting the entire continuum of testing for a particular product or design attribute,

this volume focuses on boundary conditions. The authors provide various techniques that

can be used to streamline testing and help identify problems before they occur, including

turbocharge testing methods from Six Sigma. Coverage includes testing, simulation, and

emulation”-- Provided by publisher.

Includes bibliographical references and index.

ISBN 978-1-4398-2140-4 (hardback)

1. Embedded computer systems--Testing. I. Quigley, Jon M. II. Title.

TK7895.E42P738 2010

004.16--dc22 2010043713

Visit the Taylor & Francis Web site at

http://www.taylorandfrancis.com

and the CRC Press Web site at

http://www.crcpress.com

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

Contents

List of Figures ................................................................................... xix

List of Tables ...................................................................................xxiii

Preface.............................................................................................. xxv

Acknowledgment........................................................................... xxvii

About the Authors .......................................................................... xxix

1 Does Your Testing Look Like This? ............................................1

1.1 Last-Minute Flailing .........................................................................1

1.2 Fiascos Uncovered Weeks before Launch ..........................................2

1.3 Huge Warranty Problems .................................................................2

1.4 Customer Dissatisfaction ..................................................................4

2 Benefits of Improved Testing......................................................5

2.1 Product Problems Revealed Early......................................................6

2.2 Improved Reliability = Lower Cost ..................................................8

2.3 Happy Customers .............................................................................8

2.4 Confidence in a Fine Product ...........................................................8

2.5 Cost-Effective Testing Solutions Not Waiting Until

the Last Minute ................................................................................9

3 Overview.....................................................................................11

3.1 Goals of Testing..............................................................................11

3.1.1 Discover Product Defects.................................................11

3.1.2 Prevent Defects ................................................................13

3.1.3 Contain Defects to a Single Release..................................14

3.1.4 Statistical Analysis of Release Readiness............................15

3.1.5 Discover Product Failure Limits.......................................15

3.2 Types of Testing .............................................................................17

3.2.1 Qualitative .......................................................................18

3.2.2 Quantitative.....................................................................19

v

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

vi
Contents

3.2.3 Semi-Quantitative (Count Data)......................................20

3.2.4 The Four-Phase Approach ...............................................21

3.3 Levels of Testing .............................................................................25

3.3.1 Scale-Up, Scale-Down—The Reductive Approach...........25

3.3.2 Levels of Testing ..............................................................28

3.3.3 Holistic, Counter-Reductive Approach ............................28

4 Basic Principles..........................................................................33

4.1 Looking at the Evidence..................................................................33

5 The Question..............................................................................37

5.1 Not Phrases.....................................................................................37

5.1.1 “We Tested and No Trouble Found”...............................37

5.2 Instead ............................................................................................38

5.2.1 Assume Customer Observation Correct but

May Be Poorly Stated (Erroneous Correlation) ................38

5.2.2 “What Set of Conditions Could Cause This

Event to Occur?” .............................................................40

5.2.3 If Issue Not Seen, Assume You Have Not

Found the Trigger............................................................41

6 Contradictory Perspectives of Testing .....................................43

6.1 Organic/Inorganic...........................................................................43

6.1.1 Organic............................................................................43

6.1.2 Inorganic..........................................................................44

6.1.3 Conclusion.......................................................................44

6.2 Quantitative/Qualitative .................................................................45

6.2.1 Qualitative Testing ..........................................................46

6.2.2 Semi-Quantitative Testing ...............................................46

6.2.3 Quantitative Testing ........................................................46

6.2.4 Conclusion.......................................................................47

6.3 Objective/Subjective .......................................................................47

6.3.1 Subjective.........................................................................47

6.3.2 Objective .........................................................................47

6.3.3 Conclusion.......................................................................48

6.4 Deterministic/Probabilistic .............................................................48

6.4.1 Deterministic Is an Assumption .......................................48

6.4.2 Probabilistic .....................................................................49

6.4.3 Conclusion.......................................................................50

6.5 Variable/Attribute ...........................................................................50

6.5.1 Attributes.........................................................................50

6.5.2 Variable............................................................................51

6.5.3 Conclusion.......................................................................52

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

Contents
vii

6.6 Continuous/Discrete.......................................................................52

6.6.1 Discrete............................................................................52

6.6.2 Continuous......................................................................53

6.6.3 Conclusion.......................................................................53

6.7 Wide-Ranging/Focused...................................................................54

6.7.1 Wide-Ranging..................................................................54

6.7.2 Focused............................................................................55

6.7.3 Conclusion.......................................................................55

6.8 Many/Few.......................................................................................56

6.8.1 Law of Large Numbers.....................................................56

6.8.2 Correct Identification

of Distribution.................................................................56

6.8.3 How Small Can We Go?..................................................58

6.8.4 t-Statistics ........................................................................59

6.8.5 White Box........................................................................59

6.8.6 Black Box.........................................................................59

6.8.7 Conclusion.......................................................................60

6.9 Structure/Unstructured...................................................................60

6.9.1 Structure ..........................................................................60

6.9.2 Unstructured....................................................................61

6.9.3 Conclusion.......................................................................62

6.10 Ordered/Random..........................................................................63

6.10.1 Ordered .........................................................................63

6.10.2 Random.........................................................................64

6.10.3 Conclusion.....................................................................64

6.11 Nominal/Overstress ......................................................................65

6.11.1 Nominal ........................................................................65

6.11.2 Overstress.......................................................................65

6.11.3 Conclusion.....................................................................66

6.12 Single Environment/Multiple Environments.................................66

6.12.1 Single Environment .......................................................66

6.12.2 Multiple Environments..................................................67

6.12.3 Conclusion.....................................................................67

6.13 Compliance/Characterization........................................................67

6.13.1 Compliance Testing.......................................................68

6.13.2 Characterization.............................................................69

6.13.3 Conclusion.....................................................................69

6.14 High-Level/Detailed .....................................................................70

6.14.1 High-Level.....................................................................70

6.14.2 Detailed .........................................................................71

6.14.3 Conclusion.....................................................................71

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

viii
Contents

6.15 Growing/Imposing........................................................................72

6.15.1 Growing.........................................................................72

6.15.2 Imposing........................................................................72

6.15.3 Conclusion.....................................................................73

6.16 Bench/Field ..................................................................................73

6.16.1 Bench.............................................................................73

6.16.2 Field...............................................................................76

6.16.3 Mixture..........................................................................78

6.16.4 Conclusion.....................................................................79

6.17 Abstraction/Verisimilitude ............................................................79

6.17.1 Abstraction.....................................................................80

6.17.2 Verisimilitude ................................................................81

6.17.3 Conclusion.....................................................................82

6.18 Reproducible/Nonreproducible.....................................................82

6.18.1 Reproducible..................................................................82

6.18.2 Nonreproducible............................................................83

6.18.3 Conclusion.....................................................................83

6.19 Repeatable/Nonrepeatable ............................................................84

6.19.1 Automated Testing Allows for Repeatability

When the Failure Is Truly Repeatable ............................84

6.19.2 Nonrepeatable Testing Is a Serious Problem ..................84

6.19.3 Conclusion.....................................................................86

6.20 Linear/Nonlinear ..........................................................................86

6.20.1 Linear.............................................................................86

6.20.2 Nonlinear.......................................................................86

6.20.3 Conclusion.....................................................................87

6.21 Fine/Coarse...................................................................................88

6.21.1 Coarse............................................................................88

6.21.2 Fine................................................................................88

6.21.3 Conclusion.....................................................................89

6.22 Combinatorial/Exhaustive/Stochastic............................................89

6.22.1 Exhaustive......................................................................89

6.22.2 Stochastic/Exploratory (Random) ..................................90

6.22.3 Combinatorial................................................................90

6.22.4 Foreground/Background: Foreground............................92

6.22.5 Foreground/Background: Background ...........................93

6.22.6 Conclusion.....................................................................94

6.23 Focused/Fuzzy ..............................................................................94

6.23.1 Focused..........................................................................94

6.23.2 Fuzzy .............................................................................94

6.23.3 Conclusion.....................................................................95

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

Contents
ix

6.24 Instant/Enduring ..........................................................................95

6.24.1 Instant ...........................................................................95

6.24.2 Enduring........................................................................95

6.24.3 Conclusion.....................................................................96

6.25 Inside System/Outside System ......................................................96

6.25.1 Inside System .................................................................96

6.25.2 Outside System ..............................................................97

6.25.3 Conclusion.....................................................................98

6.26 Ambiguity/Clarity.........................................................................98

6.26.1 Ambiguity......................................................................98

6.26.2 Clarity............................................................................99

6.26.3 Conclusion.....................................................................99

6.27 Sensitive/Insensitive ......................................................................99

6.27.1 Sensitive.........................................................................99

6.27.2 Insensitive ....................................................................101

6.27.3 Conclusion...................................................................101

6.28 Long-Range/Short-Range............................................................102

6.28.1 Short-Range.................................................................102

6.28.2 Long-Range .................................................................103

6.28.3 Conclusion...................................................................103

6.29 Costly/Cheap..............................................................................104

6.29.1 Cheap (Less Expensive)................................................104

6.29.2 Costly (Expensive) .......................................................104

6.29.3 Conclusion...................................................................105

6.30 Flexible/Inflexible .......................................................................105

6.30.1 Inflexible......................................................................105

6.30.2 Flexible ........................................................................106

6.30.3 Conclusion...................................................................107

6.31 Parameter/Tolerance ...................................................................107

6.31.1 Parameter Approach.....................................................107

6.31.2 Tolerance Approach.....................................................108

6.31.3 Conclusion...................................................................109

6.32 Standard/Failure .........................................................................109

6.32.1 Test to Standard...........................................................109

6.32.2 Test to Failure..............................................................110

6.32.3 Conclusion...................................................................112

6.32.4 Small Sample/Large Sample: Small Sample ..................112

6.32.5 Small Sample/Large Sample: Large Sample ..................113

6.32.6 Conclusion...................................................................114

6.33 “Good”/“Bad” Testing................................................................114

6.33.1 Bad ..............................................................................114

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

x
Contents

6.33.2 Confirm Fixes from Previous Testing...........................116

6.33.3 Good ...........................................................................116

6.33.4 Conclusion...................................................................117

6.34 Parallel/Sequential.......................................................................117

6.34.1 Parallel .........................................................................117

6.34.2 Sequential ....................................................................118

6.34.3 Conclusion...................................................................119

6.35 Fractal/Smooth/Ordered.............................................................119

6.35.1 Fractal Stimulation Is Effectively “Noise”.....................119

6.35.2 Smooth Stimulation Is Not Necessarily

“Not-Noise” but Rather Invariant Stimulation.............120

6.35.3 Ordered Stimulation Is Definitely “Not-Noise”...........121

6.35.4 Conclusion...................................................................121

7 The Use of Noise ......................................................................123

7.1 Realistic ........................................................................................123

7.2 Can Use Taguchi Approach ..........................................................123

7.3 Sometimes Difficult to Simulate ...................................................125

7.4 Where to Apply Noise? .................................................................125

7.5 Exogenous Shocks.........................................................................125

7.6 Self-Generated Noise ....................................................................126

8 How to Perform “Bad’’ Tests ...................................................129

8.1 Do Not.........................................................................................129

8.1.1 Add Stress ......................................................................129

8.1.2 Go Beyond the Specified “Limits”..................................129

8.1.3 Use Unusual Combinations of Events............................131

8.1.4 Check All Inputs and Outputs .......................................131

8.1.5 Follow Up......................................................................131

8.2 Do ................................................................................................131

8.2.1 Let the Designers Create the Test Plan...........................131

8.2.2 Test Only for Nominal Input Values .............................132

8.2.3 Make Test Plans that Are Not Consistent

with Historical Evidence ................................................132

8.2.4 Provide a Benevolent, Air-Conditioned

Environment..................................................................132

8.2.5 Forget About Extreme Situations ...................................132

8.2.6 Ignore the FMEA or Fault Tree .....................................133

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

Contents
xi

9 Documenting the Testing........................................................135

9.1 Components of a Test Plan...........................................................136

9.1.1 Title .............................................................................136

9.1.2 Purpose ........................................................................136

9.1.3 Tester and Supervisor...................................................137

9.1.4 Contact Information ....................................................137

9.1.5 Contents ......................................................................137

9.1.6 Mechanical Tests..........................................................137

9.1.7 Environmental Tests ....................................................138

9.1.8 Electronic Tests............................................................138

9.1.9 Software Tests..............................................................139

9.1.10 Measurement Uncertainty............................................139

9.1.11 Equipment Calibration Dates ......................................139

9.1.12 Equipment Identification.............................................140

9.2 Components of a Test Report .......................................................141

9.2.1 Same as Test Plan + Description of Outcome ...............141

9.3 The DVP&R Format....................................................................142

9.4 Failures .........................................................................................142

9.4.1 Relevant Failures (MIL-STD-2074)...............................142

9.4.2 Irrelevant Failures (MIL-STD-2074) .............................151

10 Test Administration.................................................................153

10.1 Test Management .......................................................................153

10.1.1 Test Process .................................................................156

10.1.2 Test Marketing ............................................................157

10.2 Test Scheduling ..........................................................................158

10.2.1 Schedule the Tests as Early as Possible .........................158

10.2.2 Effort Is Not Labor Intensive .......................................159

10.2.3 Effort Is Labor Intensive ..............................................159

10.2.4 Do Not Wait Until Product Release

to Schedule Testing......................................................159

10.3 Test Human Resources ...............................................................160

10.3.1 Technicians versus Engineers .......................................160

10.4 Test Device Resources.................................................................161

10.4.1 Whenever Possible, Provide Device Backup .................161

10.4.2 Keep the Devices Calibrated ........................................161

10.4.3 A Slower Device May Work Well as a Backup .............161

10.4.4 Consider Alternative Methods for Testing ...................165

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

xii
Contents

10.5 Test Quality................................................................................165

10.5.1 Issues Discovered Post-Testing.....................................165

10.5.2 Deliberately Adding Faults to the Unit

under Test ...................................................................166

10.5.3 Testing Organization Should Always Be Prepared

to Present Their Quality Data......................................166

10.6 Test Costing ...............................................................................167

10.6.1 Calculate the Cost of Testing—The Amount

of Time It Takes ..........................................................167

10.6.2 Laboratory Costing ......................................................167

10.6.3 Test Cost Quotation ....................................................167

10.7 Test Risk.....................................................................................169

10.7.1 Type 1 Failures ............................................................169

10.7.2 Type 2 Failures ............................................................169

10.8 Calibration Issues........................................................................169

11 Advanced Concepts .................................................................171

11.1 Test Impacts ...............................................................................171

11.1.1 Measurement Uncertainty............................................171

11.1.2 Factor Analysis.............................................................171

11.1.3 Bias by Test Engineer...................................................172

11.1.4 “Passing”......................................................................172

11.1.5 Sampling from Dependent Variable.............................173

11.1.6 “No Evidence of Cancer” or “Evidence

of No Cancer”? ............................................................173

11.2 Construct Validity ......................................................................173

11.2.1 Threats to Construct Validity.......................................173

11.2.2 Translating Hypotheses into Test

Scenarios......................................................................174

11.2.3 Why Is This Important?...............................................174

11.3 Types of Bias...............................................................................175

11.3.1 Causal Relationship......................................................175

11.3.2 Causal Experimentation—Experimental

Relationship.................................................................175

11.3.3 Validity........................................................................175

11.3.4 Threats to Statistical Validity .......................................177

11.3.5 Threats to Internal Validity..........................................178

11.3.6 Threats to External Validity .........................................179

11.3.7 In Specifying and Selecting the Study Sample ..............180

11.3.8 In Measuring Exposures and Outcomes .......................182

11.3.9 In Analyzing the Data ..................................................184

11.3.10 In Interpreting the Analysis..........................................184

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

Contents
xiii

11.4 Reliability and Confidence..........................................................185

11.4.1 Test Results Reliability.................................................185

11.4.2 Test Results Confidence ..............................................186

11.5 Life Testing.................................................................................186

11.5.1 Determine Stimuli the Product Will Be

Subjected to in Real Life (Including

Frequency and Duration).............................................187

11.5.2 Apply Multiple Stimuli Simultaneously .......................188

11.5.3 Sometimes Hard to Correlate with Other Tests ...........189

11.5.4 Highly Accelerated Life Testing ...................................189

11.6 Testing Team Early Involvement ................................................190

11.6.1 Enables Past “Finds” in Past Testing to Be

Considered during the New Design .............................191

11.6.2 Early Development of Test Plan and Review

Ensures Plan Suitability ...............................................191

11.6.3 Provides Time to Create Any Specialized Tools

or Fixtures for the Testing............................................191

12 Software Test Documentation.................................................193

12.1 IEEE 829....................................................................................193

12.2 Defect Life Cycle ........................................................................195

13 Configuration Management ....................................................197

13.1 Configuration Management Areas...............................................198

13.2 Planning .....................................................................................200

13.3 Elements of Configuration Management ....................................200

13.3.1 Configuration Identification .......................................203

13.3.2 Configuration Control ................................................203

13.3.3 Configuration Tracking (Status Accounting) ..............204

13.3.4 Configuration Auditing................................................206

13.4 Importance of Configuration Management.................................207

13.4.1 Release Notes...............................................................207

13.4.2 Identify HW/SW and System Configurations

to Test Up Front..........................................................208

13.4.3 Test Plan Grows with Feature/Function

Growth and Configuration ..........................................208

13.4.4 Traceability..................................................................208

13.5 Links to Testing..........................................................................209

13.6 Risks ...........................................................................................209

13.6.1 Configuration Management Failures............................209

13.6.2 Deviations....................................................................210

13.6.3 Configuration Management and Outsourcing..............210

© 2011 by Taylor and Francis Group, LLC

Saunder November 10, 2010 10:35 K11134˙C000

xiv
Contents

14 Software Testing ......................................................................211

14.1 Overview.....................................................................................211

14.1.1 Qualification Testing ...................................................211

14.1.2 Exploratory or Stochastic Testing.................................212

14.1.3 Combinatorial Testing.................................................212

14.2 Software Testing—The Problem.................................................213

14.2.1 Astronomical Numbers of Test Cases to Get

Complete Coverage......................................................213

14.2.2 Development Philosophy Can Affect ...........................213

14.3 Test Metrics................................................................................216

14.3.1 Cost of Test ................................................................216

14.3.2 Failures Found .............................................................219

14.3.3 Average Cost per Failure Found ...................................221

14.3.4 Test Cases Documented...............................................222

14.3.5 Test Cases Executed.....................................................222

14.3.6 Test Coverage (Test Cases to Requirements)................225

14.4 Software Boundary Testing.........................................................225

14.4.1 Maximum and Minimum Values.................................226

14.5 Static Code Analysis....................................................................227

14.5.1 Test without Testing....................................................227

14.6 Dynamic Code Analysis ..............................................................228

14.6.1 Logic Analyzers Can Perform This Function................228

14.6.2 In-Circuit Emulators Also Do This..............................229

14.6.3 Debuggers Have Dynamic Behavior.............................229

14.6.4 The Capability Allows for in Situ Analysis

of Code Behavior .........................................................229

15 Systems Testing........................................................................231

15.1 End-to-End Testing ....................................................................231

15.2 Big-Bang Testing ........................................................................232

15.3 Top-Down Testing.....................................................................232

15.4 Bottom-Up Testing ....................................................................233

15.5 Managing Multiple Test Hardware

and Software Deliveries ...............................................................233

15.6 Test Configuration Management ................................................234

15.7 System Variations and Test Demands .........................................234

15.8 Functional Testing......................................................................236

15.9 System Response to Multiple System Failures .............................238

15.10 Ranges of System Performance....................................................238

© 2011 by Taylor and Francis Group, LLC