Book The Grid Core Technologies

TEAM LinG

The Grid

The Grid

Core Technologies

Maozhen Li

Brunel University, UK

Mark Baker

University of Portsmouth, UK

Copyright © 2005 John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester,

West Sussex PO19 8SQ, England

Telephone (+44) 1243 779777

Email (for orders and customer service enquiries): [email protected]

Visit our Home Page on www.wiley.com

All Rights Reserved. No part of this publication may be reproduced, stored in a retrieval system or

transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or

otherwise, except under the terms of the Copyright, Designs and Patents Act 1988 or under the terms of

a licence issued by the Copyright Licensing Agency Ltd, 90 Tottenham Court Road, London W1T 4LP,

UK, without the permission in writing of the Publisher. Requests to the Publisher should be addressed

to the Permissions Department, John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester,

West Sussex PO19 8SQ, England, or emailed to [email protected], or faxed to
+44 1243 770620.

Designations used by companies to distinguish their products are often claimed as trademarks.

All brand names and product names used in this book are trade names, service marks, trademarks or

registered trademarks of their respective owners. The Publisher is not associated with any product or

vendor mentioned in this book.

This publication is designed to provide accurate and authoritative information in regard to the subject

matter covered. It is sold on the understanding that the Publisher is not engaged in rendering

professional services. If professional advice or other expert assistance is required, the services of a

competent professional should be sought.

Other Wiley Editorial Offices

John Wiley & Sons Inc., 111 River Street, Hoboken, NJ 07030, USA

Jossey-Bass, 989 Market Street, San Francisco, CA 94103-1741, USA

Wiley-VCH Verlag GmbH, Boschstr. 12, D-69469 Weinheim, Germany

John Wiley & Sons Australia Ltd, 33 Park Road, Milton, Queensland 4064, Australia

John Wiley & Sons (Asia) Pte Ltd, 2 Clementi Loop #02-01, Jin Xing Distripark, Singapore 129809

John Wiley & Sons Canada Ltd, 22 Worcester Road, Etobicoke, Ontario, Canada M9W 1L1

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print

may not be available in electronic books.

Library of Congress Cataloging in Publication Data

Li, Maozhen.

Core technologies / Maozhen Li, Mark Baker.

p. cm.

ISBN-13 978-0-470-09417-4 (PB)

ISBN-10 0-470-09417-6 (PB)

1. Computational grids (Computer systems) 2. Electronic data processing—Distributed processing.

I. Baker, Mark. II. Title.

QA76.9.C58L5 2005

005.3

6—dc22

2005002378

British Library Cataloguing in Publication Data

A catalogue record for this book is available from the British Library

ISBN-13 978-0-470-09417-4 (PB)

ISBN-10 0-470-09417-6 (PB)

Typeset in 11/13pt Palatino by Integra Software Services Pvt. Ltd, Pondicherry, India

Printed and bound in Great Britain by Antony Rowe Ltd, Chippenham, Wiltshire

This book is printed on acid-free paper responsibly manufactured from sustainable forestry in which

at least two trees are planted for each one used for paper production.

Contents

About the Authors xiii

Preface xv

Acknowledgements xix

List of Abbreviations xxi

1 An Introduction to the Grid 1

1.1 Introduction 1

1.2 Characterization of the Grid 1

1.3 Grid-Related Standards Bodies 4

1.4 The Architecture of the Grid 5

1.5 References 6

Part One System Infrastructure 9

2 OGSA and WSRF 11

Learning Objectives 11

Chapter Outline 11

2.1 Introduction 12

2.2 Traditional Paradigms for Distributed Computing 13

2.2.1 Socket programming 14

2.2.2 RPC 15

2.2.3 Java RMI 16

2.2.4 DCOM 18

2.2.5 CORBA 19

2.2.6 A summary on Java RMI, DCOM and CORBA 20

2.3 Web Services 21

2.3.1 SOAP 23

2.3.2 WSDL 24

2.3.3 UDDI 26

2.3.4 WS-Inspection 27

2.3.5 WS-Inspection and UDDI 28

2.3.6 Web services implementations 29

2.3.7 How Web services benefit the Grid 33

vi CONTENTS

2.4 OGSA 34

2.4.1 Service instance semantics 35

2.4.2 Service data semantics 37

2.4.3 OGSA portTypes 38

2.4.4 A further discussion on OGSA 40

2.5 The Globus Toolkit 3 (GT3) 40

2.5.1 Host environment 41

2.5.2 Web services engine 42

2.5.3 Grid services container 42

2.5.4 GT3 core services 43

2.5.5 GT3 base services 44

2.5.6 The GT3 programming model 50

2.6 OGSA-DAI 53

2.6.1 OGSA-DAI portTypes 54

2.6.2 OGSA-DAI functionality 56

2.6.3 Services interaction in the OGSA-DAI 58

2.6.4 OGSA-DAI and DAIS 59

2.7 WSRF 60

2.7.1 An introduction to WSRF 60

2.7.2 WSRF and OGSI/GT3 66

2.7.3 WSRF and OGSA 69

2.7.4 A summary of WSRF 70

2.8 Chapter Summary 70

2.9 Further Reading and Testing 72

2.10 Key Points 72

2.11 References 73

3 The Semantic Grid and Autonomic Computing 77

Learning Outcomes 77

Chapter Outline 77

3.1 Introduction 78

3.2 Metadata and Ontology in the Semantic Web 79

3.2.1 RDF 81

3.2.2 Ontology languages 83

3.2.3 Ontology editors 87

3.2.4 A summary of Web ontology languages 88

3.3 Semantic Web Services 88

3.3.1 DAML-S 89

3.3.2 OWL-S 90

3.4 A Layered Structure of the Semantic Grid 91

3.5 Semantic Grid Activities 92

3.5.1 Ontology-based Grid resource matching 93

3.5.2 Semantic workflow registration and discovery in myGrid 94

3.5.3 Semantic workflow enactment in Geodise 95

3.5.4 Semantic service annotation and adaptation in ICENI 98

3.5.5 PortalLab – A Semantic Grid portal toolkit 99

3.5.6 Data provenance on the Grid 106

3.5.7 A summary on the Semantic Grid 107

CONTENTS vii

3.6 Autonomic Computing 108

3.6.1 What is autonomic computing? 108

3.6.2 Features of autonomic computing systems 109

3.6.3 Autonomic computing projects 110

3.6.4 A vision of autonomic Grid services 113

3.7 Chapter Summary 114

3.8 Further Reading and Testing 115

3.9 Key Points 116

3.10 References 116

Part Two Basic Services 121

4 Grid Security 123

4.1 Introduction 123

4.2 A Brief Security Primer 124

4.3 Cryptography 127

4.3.1 Introduction 127

4.3.2 Symmetric cryptosystems 128

4.3.3 Asymmetric cryptosystems 129

4.3.4 Digital signatures 130

4.3.5 Public-key certificate 130

4.3.6 Certification Authority (CA) 132

4.3.7 Firewalls 133

4.4 Grid Security 134

4.4.1 The Grid Security Infrastructure (GSI) 134

4.4.2 Authorization modes in GSI 136

4.5 Putting it all Together 140

4.5.1 Getting an e-Science certificate 140

4.5.2 Managing credentials in Globus 146

4.5.3 Generate a client proxy 148

4.5.4 Firewall traversal 148

4.6 Possible Vulnerabilities 149

4.6.1 Authentication 149

4.6.2 Proxies 149

4.6.3 Authorization 150

4.7 Summary 151

4.8 Acknowledgements 151

4.9 Further Reading 151

4.10 References 152

5 Grid Monitoring 153

5.1 Introduction 153

5.2 Grid Monitoring Architecture (GMA) 154

5.2.1 Consumer 155

5.2.2 The Directory Service 156

5.2.3 Producers 157

5.2.4 Monitoring data 159

viii CONTENTS

5.3 Review Criteria 161

5.3.1 Scalable wide-area monitoring 161

5.3.2 Resource monitoring 161

5.3.3 Cross-API monitoring 161

5.3.4 Homogeneous data presentation 162

5.3.5 Information searching 162

5.3.6 Run-time extensibility 162

5.3.7 Filtering/fusing of data 163

5.3.8 Open and standard protocols 163

5.3.9 Security 163

5.3.10 Software availability and dependencies 163

5.3.11 Projects that are active and supported;

plus licensing 163

5.4 An Overview of Grid Monitoring Systems 164

5.4.1 Autopilot 164

5.4.2 Control and Observation in Distributed

Environments (CODE) 168

5.4.3 GridICE 172

5.4.4 Grid Portals Information Repository (GPIR) 176

5.4.5 GridRM 180

5.4.6 Hawkeye 185

5.4.7 Java Agents for Monitoring and Management (JAMM) 189

5.4.8 MapCenter 192

5.4.9 Monitoring and Discovery Service (MDS3) 196

5.4.10 Mercury 201

5.4.11 Network Weather Service 205

5.4.12 The Relational Grid Monitoring

Architecture (R-GMA) 209

5.4.13 visPerf 214

5.5 Other Monitoring Systems 217

5.5.1 Ganglia 217

5.5.2 GridMon 219

5.5.3 GRM/PROVE 220

5.5.4 Nagios 221

5.5.5 NetLogger 222

5.5.6 SCALEA-G 223

5.6 Summary 225

5.6.1 Resource categories 225

5.6.2 Native agents 225

5.6.3 Architecture 226

5.6.4 Interoperability 226

5.6.5 Homogeneous data presentation 226

5.6.6 Intrusiveness of monitoring 227

5.6.7 Information searching and retrieval 231

5.7 Chapter Summary 233

5.8 Further Reading and Testing 236

5.9 Key Points 236

5.10 References 236

CONTENTS ix

Part Three Job Management and User Interaction 241

6 Grid Scheduling and Resource Management 243

Learning Objectives 243

Chapter Outline 243

6.1 Introduction 244

6.2 Scheduling Paradigms 245

6.2.1 Centralized scheduling 245

6.2.2 Distributed scheduling 246

6.2.3 Hierarchical scheduling 248

6.3 How Scheduling Works 248

6.3.1 Resource discovery 248

6.3.2 Resource selection 251

6.3.3 Schedule generation 251

6.3.4 Job execution 254

6.4 A Review of Condor, SGE, PBS and LSF 254

6.4.1 Condor 254

6.4.2 Sun Grid Engine 269

6.4.3 The Portable Batch System (PBS) 274

6.4.4 LSF 279

6.4.5 A comparison of Condor, SGE, PBS and LSF 288

6.5 Grid Scheduling with QoS 290

6.5.1 AppLeS 291

6.5.2 Scheduling in GrADS 293

6.5.3 Nimrod/G 293

6.5.4 Rescheduling 295

6.5.5 Scheduling with heuristics 296

6.6 Chapter Summary 297

6.7 Further Reading and Testing 298

6.8 Key Points 298

6.9 References 299

7 Workflow Management for the Grid 301

Learning Outcomes 301

Chapter Outline 301

7.1 Introduction 302

7.2 The Workflow Management Coalition 303

7.2.1 The workflow enactment service 305

7.2.2 The workflow engine 306

7.2.3 WfMC interfaces 308

7.2.4 Other components in the WfMC reference model 309

7.2.5 A summary of WfMC reference model 310

7.3 Web Services-Oriented Flow Languages 310

7.3.1 XLANG 311

7.3.2 Web services flow language 311

7.3.3 WSCI 313

7.3.4 BPEL4WS 315

7.3.5 BPML 317

7.3.6 A summary of Web services flow languages 318

x CONTENTS

7.4 Grid Services-Oriented Flow Languages 318

7.4.1 GSFL 318

7.4.2 SWFL 321

7.4.3 GWEL 321

7.4.4 GALE 322

7.4.5 A summary of Grid services flow languages 323

7.5 Workflow Management for the Grid 323

7.5.1 Grid workflow management projects 323

7.5.2 A summary of Grid workflow management 329

7.6 Chapter Summary 330

7.7 Further Reading and Testing 331

7.8 Key Points 332

7.9 References 332

8 Grid Portals 335

Learning Outcomes 335

Chapter Outline 335

8.1 Introduction 336

8.2 First-Generation Grid Portals 337

8.2.1 A three-tiered architecture 337

8.2.2 Grid portal services 338

8.2.3 First-generation Grid portal implementations 339

8.2.4 First-generation Grid portal toolkits 341

8.2.5 A summary of the four portal tools 348

8.2.6 A summary of first-generation Grid portals 349

8.3 Second-Generation Grid Portals 350

8.3.1 An introduction to portlets 350

8.3.2 Portlet specifications 355

8.3.3 Portal frameworks supporting portlets 357

8.3.4 A Comparison of Jetspeed, WebSphere Portal

and GridSphere 368

8.3.5 The development of Grid portals with portlets 369

8.3.6 A summary on second-generation Grid portals 371

8.4 Chapter Summary 372

8.5 Further Reading and Testing 373

8.6 Key Points 373

8.7 References 374

Part Four Applications 377

9 Grid Applications – Case Studies 379

Learning Objectives 379

Chapter Outline 379

9.1 Introduction 380

9.2 GT3 Use Cases 380

9.2.1 GT3 in broadcasting 381

9.2.2 GT3 in software reuse 382

9.2.3 A GT3 bioinformatics application 387

CONTENTS xi

9.3 OGSA-DAI Use Cases 387

9.3.1 eDiaMoND 387

9.3.2 ODD-Genes 388

9.4 Resource Management Case Studies 388

9.4.1 The UCL Condor pool 388

9.4.2 SGE use cases 389

9.5 Grid Portal Use Cases 390

9.5.1 Chiron 390

9.5.2 GENIUS 390

9.6 Workflow Management – Discovery Net Use Cases 391

9.6.1 Genome annotation 391

9.6.2 SARS virus evolution analysis 391

9.6.3 Urban air pollution monitoring 392

9.6.4 Geo-hazard modelling 394

9.7 Semantic Grid – myGrid Use Case 394

9.8 Autonomic Computing – AutoMate Use Case 395

9.9 Conclusions 397

9.10 References 398

Glossary 401

Index 419

About the Authors

Dr Maozhen Li is currently Lecturer in Electronics and Computer

Engineering, in the School of Engineering and Design at Brunel

University, UK. From January 1999 to January 2002, he was

Research Associate in the Department of Computer Science,

Cardiff University, UK. Dr Li received his PhD degree in 1997, from

the Institute of Software, Chinese Academy of Sciences, Beijing,

China. His research interests are in the areas of Grid computing,

problem-solving environments for large-scale simulations, soft￾ware agents for semantic information retrieval, multi-modal user

interface design and computer support for cooperative work. Since

1997, Dr Li has published 30 research papers in prestigious inter￾national journals and conferences.

Dr Mark Baker is a hardworking Reader in Distributed Systems

at the University of Portsmouth. He also currently holds visiting

chairs at the universities of Reading and Westminster. Mark has

resided in the relative safety of academia since leaving the British

Merchant, where he was a navigating officer, in the early 1980s.

Mark has held posts at various universities, including Cardiff,

Edinburgh and Syracuse. He has a number of geek-like inter￾ests, which his research group at Portsmouth help him pursue.

These include wide-area resource monitoring, messaging systems

for parallel and wide-area applications, middleware such as infor￾mation and security services, as well as performance evaluation

and modelling of computer systems.

Mark’s non-academic interests include squash (getting too old),

DIY (he may one day finish his house off), reading (far too many

science fiction books), keeping the garden ship-shape and a beer

or two to reduce the pain of the aforementioned activities.