Tài liệu Self-Service Linux: Mastering the Art of Problem Determination pptx

Self-Service Linux®

BRUCE PERENS’ OPEN SOURCE SERIES

www.phptr.com/perens

◆ Java™ Application Development on Linux®

Carl Albing and Michael Schwarz

◆ C++ GUI Programming with Qt 3

Jasmin Blanchette and Mark Summerfield

◆ Managing Linux Systems with Webmin: System Administration and Module Development

Jamie Cameron

◆ Understanding the Linux Virtual Memory Manager

Mel Gorman

◆ PHP 5 Power Programming

Andi Gutmans, Stig Bakken, and Derick Rethans

◆ Linux® Quick Fix Notebook

Peter Harrison

◆ Implementing CIFS: The Common Internet File System

Christopher Hertel

◆ Open Source Security Tools: A Practical Guide to Security Applications

Tony Howlett

◆ Apache Jakarta Commons: Reusable Java™ Components

Will Iverson

◆ Embedded Software Development with eCos

Anthony Massa

◆ Rapid Application Development with Mozilla

Nigel McFarlane

◆ Subversion Version Control: Using the Subversion Version Control System in Development

Projects

William Nagel

◆ Intrusion Detection with SNORT: Advanced IDS Techniques Using SNORT, Apache, MySQL,

PHP, and ACID

Rafeeq Ur Rehman

◆ Cross-Platform GUI Programming with wxWidgets

Julian Smart and Kevin Hock with Stefan Csomor

◆ Samba-3 by Example, Second Edition: Practical Exercises to Successful Deployment

John H. Terpstra

◆ The Official Samba-3 HOWTO and Reference Guide, Second Edition

John H. Terpstra and Jelmer R. Vernooij, Editors

◆ Self-Service Linux®: Mastering the Art of Problem Determination

Mark Wilding and Dan Behman

perens_series_7x9.25.fm Page 1 Tuesday, August 16, 2005 2:17 PM

Mark Wilding and Dan Behman

PRENTICE HALL

Professional Technical Reference

Upper Saddle River, NJ ● Boston ●

Indianapolis ● San Francisco ● New York ●

Toronto ● Montreal ● London ● Munich ●

Paris ● Madrid ● Capetown ● Sydney ●

Tokyo ● Singapore ● Mexico City

Self-Service Linux®

Mastering the Art of Problem

Determination

Many of the designations used by manufacturers and sellers to distinguish their products

are claimed as trademarks. Where those designations appear in this book, and the publisher

was aware of a trademark claim, the designations have been printed with initial capital

letters or in all capitals.

The authors and publisher have taken care in the preparation of this book, but make no

expressed or implied warranty of any kind and assume no responsibility for errors or omissions.

No liability is assumed for incidental or consequential damages in connection with or arising

out of the use of the information or programs contained herein.

The publisher offers excellent discounts on this book when ordered in quantity for bulk

purchases or special sales, which may include electronic versions and/or custom covers and

content particular to your business, training goals, marketing focus, and branding interests.

For more information, please contact:

U. S. Corporate and Government Sales

(800) 382-3419

[email protected]

For sales outside the U. S., please contact:

International Sales

[email protected]

Visit us on the Web: www.phptr.com

Library of Congress Number: 2005927150

Copyright © 2006 Pearson Education, Inc.

This material may be distributed only subject to the terms and conditions set forth in the Open

Publication License, v1.0 or later (the latest version is presently available at http://

www.opencontent.org/openpub/).

ISBN 0-13-147751-X

Text printed in the United States on recycled paper at R.R. Donnelley in Crawfordsville,

Indiana.

First printing, September, 2005

I would like to dedicate this book to my wife, Caryna, whose relentless nagging and

badgering forced me to continue working on this book when nothing else could. Just

kidding... Without Caryna’s support and understanding, I could never have written

this book. Not only did she help me find time to write, she also spent countless hours

formatting the entire book for production. I would also like to dedicate this book to

my two sons, Rhys and Dylan, whose boundless energy acted as inspiration

throughout the writing of this book.

Mark Wilding

Without the enduring love and patience of my wife Kim, this laborous

project would have halted long ago. I dedicate this book to her,

as well as to my beautiful son Nicholas, my family,

and all of the Botzangs and Mayos.

Dan Behman

Gutmans_Frontmatter Page vi Thursday, September 23, 2004 9:05 AM

Contents

Preface

Chapter 1: Best Practices and Initial Investigation

Chapter 2: strace and System Call Tracing Explained

Chapter 3: The /proc Filesystem

Chapter 4: Compiling

Chapter 5: The Stack

Chapter 6: The GNU Debugger (GDB)

Chapter 7: Linux System Crashes and Hangs

Chapter 8: Kernel Debugging with KDB

Chapter 9: ELF: Executable and Linking Format

A: The Toolbox

B: Data Collection Script

Index

Contents

Preface xvii

1 Best Practices and Initial Investigation 1

1.1 Introduction 1

1.2 Getting Your System(s) Ready for Effective Problem

Determination 2

1.3 The Four Phases of Investigation 3

1.3.1 Phase #1: Initial Investigation Using Your Own Skills 5

1.3.2 Phase #2: Searching the Internet Effectively 9

1.3.3 Phase #3: Begin Deeper Investigation (Good Problem

Investigation Practices) 12

1.3.4 Phase #4: Getting Help or New Ideas 21

1.4 Technical Investigation 28

1.4.1 Symptom Versus Cause 28

1.5 Troubleshooting Commercial Products 38

1.6 Conclusion 39

2 strace and System Call Tracing Explained 41

2.1 Introduction 41

2.2 What Is strace? 41

2.2.1 More Information from the Kernel Side 45

2.2.2 When to Use It 48

2.2.3 Simple Example 49

2.2.4 Same Program Built Statically 53

2.3 Important strace Options 54

2.3.1 Following Child Processes 54

2.3.2 Timing System Call Activity 55

○○○○○○○○○○○○○○○ ○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○

○○○○○○

○○○○○○○○○○○○○○○○○○

○○○○○○ ○○○○○

○○○○○○○○○○○○○○○○ ○○○○○○○

○○○○○○○○○○○○○ ○○○○○

○○○○○○○○○○○○○○○

○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○

○○○○○○○ ○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○

○○○○○○○○○ ○○○○○○○○

○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○ ○○○○○○○○○○○○○○○○○

2.3.3 Verbose Mode 57

2.3.4 Tracing a Running Process 59

2.4 Effects and Issues of Using strace 60

2.4.1 strace and EINTR 61

2.5 Real Debugging Examples 62

2.5.1 Reducing Start Up Time by Fixing

LD_LIBRARY_PATH 62

2.5.2 The PATH Environment Variable 65

2.5.3 stracing inetd or xinetd (the Super Server) 66

2.5.4 Communication Errors 68

2.5.5 Investigating a Hang Using strace 69

2.5.6 Reverse Engineering (How the strace Tool Itself Works) 71

2.6 System Call Tracing Examples 74

2.6.1 Sample Code 75

2.6.2 The System Call Tracing Code Explained 87

2.7 Conclusion 88

3 The /proc Filesystem 89

3.1 Introduction 89

3.2 Process Information 90

3.2.1 /proc/self 90

3.2.2 /proc/<pid> in More Detail 91

3.2.3 /proc/<pid>/cmdline 107

3.2.4 /proc/<pid>/environ 107

3.2.5 /proc/<pid>/mem 107

3.2.6 /proc/<pid>/fd 108

3.2.7 /proc/<pid>/mapped base 108

3.3 Kernel Information and Manipulation 109

3.3.1 /proc/cmdline 109

3.3.2 /proc/config.gz or /proc/sys/config.gz 109

3.3.3 /proc/cpufreq 109

3.3.4 /proc/cpuinfo 110

3.3.5 /proc/devices 110

3.3.6 /proc/kcore 111

3.3.7 /proc/locks 111

3.3.8 /proc/meminfo 111

3.3.9 /proc/mm 111

3.3.10 /proc/modules 112

3.3.11 /proc/net 112

3.3.12 /proc/partitions 112

3.3.13 /proc/pci 113

3.3.14 /proc/slabinfo 113

x Contents

○○○○○○○○○○○○○○ ○○○○○○○○○○

○○○○○ ○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○

○○○○○○○ ○○○○○○○○○○○○○○○

○○○○○○○ ○○○○○○○○○○○○○○○

○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○○○○○○

○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○ ○○○○○○

○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○ ○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○ ○○○○○○○○

○○○○○○○○○○ ○○○○○○○○○○○

○○○○○○○ ○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○ ○○○○○○

○○○○○○○○○○○○○○○○ ○○○○○○○○

○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○ ○○○○○○○

○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○ ○○○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○ ○○○○○

○○○○○○○○○ ○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○ ○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○ ○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○ ○○○○○○○○

3.4 System Information and Manipulation 113

3.4.1 /proc/sys/fs 113

3.4.2 /proc/sys/kernel 115

3.4.3 /proc/sys/vm 120

3.5 Conclusion 120

4 Compiling 121

4.1 Introduction 121

4.2 The GNU Compiler Collection 121

4.2.1 A Brief History of GCC 121

4.2.2 GCC Version Compatibility 122

4.3 Other Compilers 122

4.4 Compiling the Linux Kernel 123

4.4.1 Obtaining the Kernel Source 123

4.4.2 Architecture Specific Source 124

4.4.3 Working with Kernel Source Compile Errors 124

4.4.4 General Compilation Problems 128

4.5 Assembly Listings 133

4.5.1 Purpose of Assembly Listings 134

4.5.2 Generating Assembly Listings 135

4.5.3 Reading and Understanding an Assembly Listing 136

4.6 Compiler Optimizations 140

4.7 Conclusion 149

5 The Stack 151

5.1 Introduction 151

5.2 A Real-World Analogy 152

5.3 Stacks in x86 and x86-64 Architectures 153

5.4 What Is a Stack Frame? 157

5.5 How Does the Stack Work? 159

5.5.1 The BP and SP Registers 159

5.5.2 Function Calling Conventions 162

5.6 Referencing and Modifying Data on the Stack 171

5.7 Viewing the Raw Stack in a Debugger 173

5.8 Examining the Raw Stack in Detail 176

5.8.1 Homegrown Stack Traceback Function 180

5.9 Conclusion 191

6 The GNU Debugger (GDB) 193

6.1 Introduction 193

6.2 When to Use a Debugger 194

6.3 Command Line Editing 195

Contents xi

○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○ ○○○○○○○○○

○○○○○○ ○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○

○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○

○○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○○○ ○○○○○○

○○○○○○○○ ○○○○○○○○

○○○○○○○ ○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○

○○○○○○○ ○○○○○○○

○○○○○○○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○

○○○○

○○○○○○○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○○○

○○○○○○○○○○○○○○○○○○ ○○○○○○○○○○○○

○○○○○○○○○ ○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○

○○○○○○○○○○○○ ○○○○○○

○○○○○○○○ ○○○○○○○

○○○○○○○○○○

○○○○○○○○○○ ○○○○○

○○○○○○○○○ ○○○○○○○

○○○○○○○○○○

○○○○○○○○○○○○○○ ○○○○○○○○○○○○○○○○

○○○○○○○○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○○○○○○○○

6.4 Controlling a Process with GDB 196

6.4.1 Running a Program Off the Command Line with GDB 197

6.4.2 Attaching to a Running Process 199

6.4.3 Use a Core File 200

6.5 Examining Data, Memory, and Registers 204

6.5.1 Memory Map 204

6.5.2 Stack 206

6.5.3 Examining Memory and Variables 210

6.5.4 Register Dump 217

6.6 Execution 220

6.6.1 The Basic Commands 221

6.6.2 Settings for Execution Control Commands 223

6.6.3 Breakpoints 228

6.6.4 Watchpoints 230

6.6.5 Display Expression on Stop 234

6.6.6 Working with Shared Libraries 235

6.7 Source Code 238

6.8 Assembly Language 240

6.9 Tips and Tricks 241

6.9.1 Attaching to a Process—Revisited 241

6.9.2 Finding the Address of Variables and Functions 244

6.9.3 Viewing Structures in Executables without Debug

Symbols 246

6.9.4 Understanding and Dealing with Endian-ness 250

6.10 Working with C++ 252

6.10.1 Global Constructors and Destructors 252

6.10.2 Inline Functions 256

6.10.3 Exceptions 257

6.11 Threads 260

6.11.1 Running Out of Stack Space 265

6.12 Data Display Debugger (DDD) 266

6.12.1 The Data Display Window 268

6.12.2 Source Code Window 272

6.12.3 Machine Language Window 273

6.12.4 GDB Console Window 274

6.13 Conclusion 274

7 Linux System Crashes and Hangs 275

7.1 Introduction 275

7.2 Gathering Information 275

7.2.1 Syslog Explained 276

7.2.2 Setting up a Serial Console 277

xii Contents

○○○○○○○○○○○○○○○○○○

○○○○○○○○○ ○○○○○

○○○○○○○○○○○○ ○○○○○○○○○○○

○○○○○○○○○ ○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○

○○○○○○○○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○○○○○ ○○○○○○○○○○○○○○○

○○○○○○○ ○○○○○○○○○○○○○

○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○

○○○○ ○○○○○○○○○○

○○○○○○○○○○○○○○ ○○○○○○○○○○○○○○○○

○○○○○○○○○○○○ ○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○

○○○○○○○○ ○○○○○

○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○ ○○○○○

○○○○○○○○○ ○○○○○○○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○

○○○○○○ ○○○○○○○○○○○

○○○○○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○

○○○○○○ ○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○ ○○○○○○

○○○○○○○○○○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○

7.2.3 Connecting the Serial Null-Modem Cable 278

7.2.4 Enabling the Serial Console at Startup 279

7.2.5 Using SysRq Kernel Magic 281

7.2.6 Oops Reports 281

7.2.7 Adding a Manual Kernel Trap 281

7.2.8 Examining an Oops Report 284

7.2.9 Determining the Failing Line of Code 289

7.2.10 Kernel Oopses and Hardware 293

7.2.11 Setting up cscope to Index Kernel Sources 294

7.3 Conclusion 295

8 Kernel Debugging with KDB 297

8.1 Introduction 297

8.2 Enabling KDB 297

8.3 Using KDB 299

8.3.1 Activating KDB 299

8.3.2 Resuming Normal Execution 300

8.3.3 Basic Commands 300

8.4 Conclusion 305

9 ELF: Executable and Linking Format 307

9.1 Introduction 307

9.2 Concepts and Definitions 309

9.2.1 Symbol 309

9.2.2 Object Files, Shared Libraries, Executables, and Core

Files 311

9.2.3 Linking 314

9.2.4 Run Time Linking 318

9.2.5 Program Interpreter / Run Time Linker 318

9.3 ELF Header 318

9.4 Overview of Segments and Sections 324

9.5 Segments and the Program Header Table 325

9.5.1 Text and Data Segments 329

9.6 Sections and the Section Header Table 331

9.6.1 String Table Format 335

9.6.2 Symbol Table Format 335

9.6.3 Section Names and Types 338

9.7 Relocation and Position Independent Code (PIC) 362

9.7.1 PIC vs. non-PIC 363

9.7.2 Relocation and Position Independent Code 366

9.7.3 Relocation and Linking 367

9.8 Stripping an ELF Object 371

Contents xiii

○○○○○○○○

○○○○ ○○○○○○

○○○○○○○○○○○○○○○○○

○○○○○○○○○ ○○○○○○○○○○○○○○○○

○○○○○○ ○○○○○○○○○

○○○○○○ ○○○○○○○○○○○

○○○○ ○○○○○○○

○○○○○○○○○○○○○○○

○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○ ○○○○○

○○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○○

○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○

○○○○○○○○○○○○○ ○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○

○○○○○○○○○○○ ○○○○○○○○○○○○

○○○○○○○○○

○○○○○○○○○○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○ ○○○○○○○○

○○○○○ ○○○○○○○○

○○○○○○○○ ○○○○○○○○○○

○○○○○○○ ○○○○○○○

○○○○○○○○○○○ ○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○ ○○○○○

○○○○ ○○○○○

○○○○○○○○○○○○○○○○○○ ○○○○○

○○○○○○○○

○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○

9.9 Program Interpreter 372

9.9.1 Link Map 376

9.10 Symbol Resolution 377

9.11 Use of Weak Symbols for Problem Investigations 382

9.12 Advanced Interception Using Global Offset Table 386

9.13 Source Files 390

9.14 ELF APIs 392

9.15 Other Information 392

9.16 Conclusion 392

A The Toolbox 393

A.1 Introduction 393

A.2 Process Information and Debugging 393

A.2.1 Tool: GDB 393

A.2.2 Tool: ps 393

A.2.3 Tool: strace (system call tracer) 394

A.2.4 Tool: /proc filesystem 394

A.2.5 Tool: DDD (Data Display Debugger) 394

A.2.6 Tool: lsof (List Open Files) 394

A.2.7 Tool: ltrace (library call tracer) 395

A.2.8 Tool: time 395

A.2.9 Tool: top 395

A.2.10 Tool: pstree 396

A.3 Network 396

A.3.1 Tool: traceroute 396

A.3.2 File: /etc/hosts 396

A.3.3 File: /etc/services 396

A.3.4 Tool: netstat 397

A.3.5 Tool: ping 397

A.3.6 Tool: telnet 397

A.3.7 Tool: host/nslookup 397

A.3.8 Tool: ethtool 398

A.3.9 Tool: ethereal 398

A.3.10 File: /etc/nsswitch.conf 398

A.3.11 File: /etc/resolv.conf 398

A.4 System Information 399

A.4.1 Tool: vmstat 399

A.4.2 Tool: iostat 399

A.4.3 Tool: nfsstat 399

A.4.4 Tool: sar 400

A.4.5 Tool: syslogd 400

A.4.6 Tool: dmesg 400

xiv Contents

○○○○○○○○○○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○ ○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○○○

○○○○○○○○

○○○○○○○○○○○○○ ○○○○○○○○○○○○○○○○○

○○○○○○ ○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○ ○○○○

○○○○○○○○○○○ ○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○ ○○○○○○○○○○○○

○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○

○○○○○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○

○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○ ○○○○○○○○○

○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○ ○○○○○○○○○○○○○

○○○○○○○○○○○○○○ ○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○ ○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○○

○○○○○○ ○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○ ○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○ ○○○○○

○○○○○○○○○ ○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○

A.4.7 Tool: mpstat 400

A.4.8 Tool: procinfo 401

A.4.9 Tool: xosview 401

A.5 Files and Object Files 401

A.5.1 Tool: file 401

A.5.2 Tool: ldd 401

A.5.3 Tool: nm 402

A.5.4 Tool: objdump 402

A.5.5 Tool: od 402

A.5.6 Tool: stat 402

A.5.7 Tool: readelf 403

A.5.8 Tool: strings 403

A.6 Kernel 403

A.6.1 Tool: KDB 403

A.6.2 Tool: KGDB 403

A.6.3 Tool: ksymoops 404

A.7 Miscellaneous 404

A.7.1 Tool: VMWare Workstation 404

A.7.2 Tool: VNC Server 405

A.7.3 Tool: VNC Viewer 405

B Data Collection Script 407

B.1 Overview 407

B.1.1 -thorough 409

B.1.2 -perf, -hang <pid>, -trap, -error <cmd> 409

B.2 Running the Script 410

B.3 The Script Source 410

B.4 Disclaimer 419

Index 421

Contents xv

○○○○○○○○○○○○○○○○○○○○○ ○○○○

○○○○○○ ○○○○○○○○○○○○○○○○○○○

○○○○○○ ○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○ ○○○○○○○○○○○○○○○○○

○○○○○○○○○○○ ○○○○○○○○○○○○○○○○○

○○○○○○○○○○○ ○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○ ○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○ ○○○○○○○○○

○○○○○○○○○○○○○○○○○ ○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○○○ ○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○ ○○○○○

○○○○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○○○ ○○○○○○○○○○

○○○○○○○○○○ ○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○○○○○○○○○ ○○○○

○○○○○○ ○○○○○○○○○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○○

○○○○ ○○○○○○○○○○○○○○○○○○○○○○

○○○○○○○○○○○○○ ○○○○○○○○○○○○○○○○○○

○○○○○ ○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○○