, Cédric. Next Generation Java Testing

Next Generation

Java™ Testing

This page intentionally left blank

Next Generation

Java™ Testing

TestNG and Advanced Concepts

Cédric Beust

Hani Suleiman

Upper Saddle River, NJ • Boston • Indianapolis • San Francisco

New York • Toronto • Montreal • London • Munich • Paris • Madrid

Capetown • Sydney • Tokyo • Singapore • Mexico City

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks.

Where those designations appear in this book, and the publisher was aware of a trademark claim, the designations have

been printed with initial capital letters or in all capitals.

The authors and publisher have taken care in the preparation of this book, but make no expressed or implied warranty

of any kind and assume no responsibility for errors or omissions. No liability is assumed for incidental or consequential

damages in connection with or arising out of the use of the information or programs contained herein.

The publisher offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales,

which may include electronic versions and/or custom covers and content particular to your business, training goals,

marketing focus, and branding interests. For more information, please contact:

U.S. Corporate and Government Sales

(800) 382-3419

[email protected]

For sales outside the United States please contact:

International Sales

[email protected]

Visit us on the Web: www.awprofessional.com

Library of Congress Cataloging-in-Publication Data

Beust, Cédric.

Next generation Java testing : TestNG and advanced concepts / Cédric

Beust, Hani Suleiman.

p. cm.

Includes bibliographical references and index.

ISBN 0-321-50310-4 (pbk. : alk. paper)

1. Java (Computer program language) 2. Computer software—Testing. I.

Suleiman, Hani. II. Title.

QA76.73.J3B49 2007

005.13'3—dc22 2007031939

Copyright © 2008 Pearson Education, Inc.

All rights reserved. Printed in the United States of America. This publication is protected by copyright, and permission

must be obtained from the publisher prior to any prohibited reproduction, storage in a retrieval system, or transmission

in any form or by any means, electronic, mechanical, photocopying, recording, or likewise. For information regarding

permissions, write to:

Pearson Education, Inc.

Rights and Contracts Department

501 Boylston Street, Suite 900

Boston, MA 02116

Fax: (617) 671-3447

ISBN-13: 978-0-321-50310-7

ISBN-10: 0-321-50310-4

Text printed in the United States on recycled paper at Courier in Stoughton, Massachusetts.

First printing, October 2007

v

Contents

Foreword xiii

Preface xv

Acknowledgments xxi

About the Authors xxiii

Chapter 1 Getting Started 1

Beyond JUnit 3 3

Stateful Classes 3

Parameters 4

Base Classes 4

Exceptions Are Not That Exceptional 4

Running Tests 5

Real-World Testing 6

Configuration Methods 6

Dependencies 6

Epiphanies 7

JUnit 4 7

Designing for Testability 8

Object-Oriented Programming and Encapsulation 8

The Design Patterns Revolution 9

Identifying the Enemy 10

Recommendations 16

TestNG 17

Annotations 17

Tests, Suites, and Configuration Annotations 18

Groups 20

testng.xml 21

Conclusion 21

Chapter 2 Testing Design Patterns 23

Testing for Failures 23

Reporting Errors 24

Runtime and Checked Exceptions 25

vi Contents

Testing Whether Your Code Handles Failures Gracefully 27

When Not to Use expectedExceptions 31

testng-failed.xml 32

Factories 34

@Factory 35

org.testng.ITest 38

Data-Driven Testing 39

Parameters and Test Methods 42

Passing Parameters with testng.xml 44

Passing Parameters with @DataProvider 47

Parameters for Data Providers 50

The Method Parameter 50

The ITestContext Parameter 52

Lazy Data Providers 54

Pros and Cons of Both Approaches 59

Supplying the Data 60

Data Provider or Factory? 62

Tying It All Together 63

Asynchronous Testing 67

Testing Multithreaded Code 71

Concurrent Testing 72

threadPoolSize, invocationCount, and timeOut 75

Concurrent Running 79

Turning on the Parallel Bit 82

Performance Testing 83

Algorithm Complexity 84

Testing Complexity 87

Mocks and Stubs 90

Mocks versus Stubs 90

Designing for Mockability 95

Mock Libraries 96

Selecting the Right Strategy 99

Mock Pitfalls 100

Dependent Testing 103

Dependent Code 104

Dependent Testing with TestNG 105

Deciding Whether to Depend on Groups or on Methods 106

Dependent Testing and Threads 110

Failures of Configuration Methods 110

Inheritance and Annotation Scopes 113

The Problem 113

Pitfalls of Inheritance 116

Test Groups 119

Syntax 120

Groups and Runtime 122

Running Groups 125

Using Groups Effectively 127

Contents vii

Code Coverage 132

A Coverage Example 133

Coverage Metrics 134

Coverage Tools 136

Implementation 146

Beware! 147

A Guide to Successful Coverage 147

Conclusion 150

Chapter 3 Enterprise Testing 153

A Typical Enterprise Scenario 154

Participants 155

Testing Methodology 155

Issues with the Current Approach 156

A Concrete Example 157

Goals 159

Nongoals 160

Test Implementation 160

Testing for Success 161

Building Test Data 163

Test Setup Issues 166

Error Handling 172

Emerging Unit Tests 175

Coping with In-Container Components 177

Putting It All Together 178

Exploring the Competing Consumers Pattern 182

The Pattern 182

The Test 184

The Role of Refactoring 186

A Concrete Example 187

An In-Container Approach 193

Conclusion 194

Chapter 4 Java EE Testing 197

In-Container versus Out-of-Container Testing 198

In-Container Testing 200

Creating a Test Environment 200

Identifying Tests 201

Registering Tests 203

Registering a Results Listener 204

Java Naming and Directory Interface (JNDI) 207

Understanding JNDI’s Bootstrapping 207

Spring’s SimpleNamingContextBuilder 209

Avoiding JNDI 210

viii Contents

Java Database Connectivity (JDBC) 210

c3p0 212

Commons DBCP 213

Spring 213

Java Transaction API (JTA) 215

Java Open Transaction Manager (JOTM) 217

Atomikos TransactionEssentials 218

Java Messaging Service (JMS) 219

Creating a Sender/Receiver Test 219

Using ActiveMQ for Tests 221

Java Persistence API (JPA) 225

Configuring the Database 227

Configuring the JPA Provider 227

Writing the Test 229

Simulating a Container 230

Using Spring as the Container 231

Enterprise Java Beans 3.0 (EJB3) 236

Message-Driven Beans 237

Session Beans 240

Another Spring Container 243

Disadvantages of a Full Container 244

Java API for XML Web Services (JAX-WS) 246

Recording Requests 248

Setting Up the Test Environment 248

Creating the Service Test 251

XPath Testing 253

Testing Remote Services 254

Servlets 255

In-Container Testing 255

Mock/Stub Objects 255

Refactoring 257

Embedded Container 257

In-Memory Invocation 260

XML 262

Using dom4j 263

Using XMLUnit 264

Conclusion 266

Chapter 5 Integration 269

Spring 270

Spring’s Test Package Features 271

Test Class Hierarchy 272

Guice 280

The Issue with Spring 280

Enter Guice 281

A Typical Dependency Scenario 282

Contents ix

The Object Factory 284

Guice Configuration 286

Guice-Based Test 290

Grouping Test Dependencies 291

Injecting Configuration 293

DbUnit 295

Configuration 295

Usage 297

Verifying Results 299

HtmlUnit 303

Configuration 304

Usage 305

Selenium 310

Swing UI Testing 312

Testing Approach 312

Configuration 313

Usage 314

Tests for Painting Code 316

Continuous Integration 320

Why Bother? 320

CI Server Features 320

TestNG Integration 321

Conclusion 322

Chapter 6 Extending TestNG 325

The TestNG API 325

org.testng.TestNG, ITestResult, ITestListener,

ITestNGMethod 325

A Concrete Example 328

The XML API 331

Synthetic XML Files 333

BeanShell 335

BeanShell Overview 335

TestNG and BeanShell 337

Interactive Execution 339

Method Selectors 341

Annotation Transformers 346

Annotation History 346

Pros and Cons 348

Using TestNG Annotation Transformers 348

Possible Uses of Annotation Transformers 353

Reports 355

Default Reports 355

The Reporter API 360

The Report Plug-in API 360

x Contents

Writing Custom Annotations 366

Implementation 367

Testing 371

Conclusion 375

Chapter 7 Digressions 377

Motivation 377

The TestNG Philosophy 378

The Care and Feeding of Exceptions 378

Stateful Tests 382

Immutable State 382

Mutable State 383

The Pitfalls of Test-Driven Development 385

TDD Promotes Microdesign over Macrodesign 385

TDD Is Hard to Apply 386

Extracting the Good from Test-Driven Development 388

Testing Private Methods 388

Testing versus Encapsulation 391

The Power of Debuggers 392

Logging Best Practices 394

The Value of Time 397

Conclusion 399

Appendix A IDE Integration 401

Eclipse 401

Installing the Plug-in 401

Verifying the Installation 404

Creating a Launch Configuration 404

Configuring Preferences 410

Converting JUnit Tests 410

IntelliJ IDEA 411

Installing the Plug-in 411

Running Tests 412

Running Shortcuts 417

Viewing Test Results 418

Running Plug-in Refactorings 419

Appendix B TestNG Javadocs 421

JDK 1.4 and JDK 5 421

Shortcut Syntax for JDK 5 Annotations 423

Contents xi

Annotation Javadocs 423

@DataProvider/@testng.data-provider 425

@Factory/@testng.factory 426

@Parameters/@testng.parameters 426

@Test/@testng.test 427

The org.testng.TestNG Class 428

The XML API 432

Appendix C testng.xml 435

Overview 436

Scopes 437

XML Tags 437

<suite> 437

<packages> and <package> 440

<parameter> 441

<suite-files> and <suite-file> 442

<method-selectors>, <method-selector>,

<selector-class>, and <script> 443

<test> 444

<groups>, <define>, and <run> 446

<classes> and <class> 446

<methods> 447

Appendix D Migrating from JUnit 449

JUnitConverter 449

From the Command Line 449

From ant 452

Integrated Development Environments 453

Eclipse 453

IDEA 454

Incremental Migration and JUnit Mode 455

Converting JUnit Code 456

Assertions 457

Running a Single Test 458

Maintaining State between Invocations 461

Suite-wide Initialization 463

Class-wide Initialization 463

The AllTests Pattern 463

Testing Exceptions 467

The Parameterized Test Case Pattern 469

Index 471

This page intentionally left blank

xiii

Foreword

Doing the right thing is rarely easy. Most of us should probably eat better,

exercise more, and spend more time with our families. But each day, when

confronted with the choice of being good or doing the easy thing, we often

choose the easy thing because, well, it’s easier.

It is the same way with software testing. We all know that we should

spend more effort on testing, that more testing will make our code more

reliable and maintainable, that our users will thank us if we do, that it will

help us better understand our own programs, but each day when we sit

down to the computer and choose between writing more tests and writing

more application code, it is very tempting to reach for the easy choice.

Today, it is largely accepted that unit testing is the responsibility of

developers, not QA, but this is a relatively recent development, one for

which we can largely thank the JUnit testing framework. It is notable that

JUnit had such an impact because there’s not really very much to it—it’s a

simple framework, with not a lot of code. What enabled JUnit to change

developer behavior where years of lecturing and guilt could not was that, for

the first time, the pain of writing unit tests was reduced to a bearable level,

making it practical for the merely responsible to include unit testing in our

daily coding. Rather than make testing more desirable, which is not such an

easy sell (eat those vegetables, they’re good for you!), JUnit simply made it

easier to do the right thing.

With all the righteousness of the newly converted, many developers

proclaimed their zeal for testing, proudly calling themselves “test-infected.”

This is all well and good—few could argue that software developers were

doing too much testing, so more is probably an improvement—but it is only

the first step. There’s more to testing than unit tests, and if we expect devel￾opers to take this next step we must provide testing tools that reduce the pain

of creating them—and demand testability as a fundamental design require￾ment. If software engineering is ever to become a true engineering discipline,

testing will form one of the critical pillars on which it will be built. (Perhaps

one day, writing code without tests will be considered as professionally irre￾sponsible as constructing a bridge without performing a structural analysis.)

xiv Foreword

This book is dedicated to the notion that we’ve only just begun our rela￾tionship with responsible testing. The TestNG project aims to help develop￾ers take the next step—the NG stands for “next generation”—enabling

broader and deeper test coverage that encompasses not only unit tests but

also acceptance, functional, and integration tests. Among other useful fea￾tures, it provides a rich mechanism for specifying and parameterizing test

suites, encompassing concurrent testing and a flexible mechanism for

decoupling test code from its data source. (And, as proof that TestNG is suc￾ceeding, a number of its features have been adopted in more recent ver￾sions of JUnit.)

One challenge to more effective developer testing, no matter what tools

are provided, is that writing effective tests requires different skills than writ￾ing effective code. But, like most skills, testing can be learned, and one of

the best ways to learn is to watch how more experienced hands might do it.

Throughout this book, Hani and Cédric share with you their favorite tech￾niques for effectively testing Java applications and for designing applications

and components for testability. (This last skill—designing for testability—is

probably one of the most valuable lessons from this book. Designing code

for testability forces you to think about the interactions and dependencies

between components earlier, thereby encouraging you to build cleaner,

more loosely coupled code.) Of course, the TestNG framework is used to

illustrate these techniques, but even if you are not a TestNG user (and not

interested in becoming one), the practical techniques presented here will

help you to be a better tester and, in turn, a better engineer.

Brian Goetz

Senior Staff Engineer, Sun Microsystems