Evaluating democracy assistance

EVALUATING

Krishna Kumar

Evaluating

Democracy

Assistance

Evaluating

Democracy

Assistance

Krishna Kumar

&

GIFT OF THE ASIA FOUNDATION

NOT FOR RE-SALE

QUÀ TẶNG CỦA QUỸ CHÂU Á

KHồNG ĐƯỢC BÁN LẠI

LYNNE

RỈENNER

PUBLISHERS

BOULDER

LONDON

The views expressed in this book are the author’s and should not be attributed to either

the Department of State or the US Agency for International Development.

Published in the United States of America in 2013 by

Lynne Rienner Publishers, Inc.

1800 30th Street, Boulder, Colorado 80301

www.rienner.com

and in the United Kingdom by

Lynne Rienner Publishers, Inc.

3 Henrietta Street, Covent Garden, London WC2E 8LU

© 2013 by Lynne Rienner Publishers, Inc. All rights reserved

L ibrary of Congress C ataloging-in-Publication D ata

Kumar, Krishna, 1940-

Evaluating democracy assistance/Krishna Kumar.

p. cm. „ '

Includes bibliographical references and index.

ISBN 978-1-58826-858-7 (alk. paper)

ISBN 978-1-58826-883-9 (p b k .: alk. paper)

1. D em ocratization— International cooperation. 2. T echnical assistance. I. Title.

JC423.K85 2012

327.1—dc23

2012028227

/

B ritish C ataloguing in Publication D ata

A Cataloguing in Publication record for this book

is available from the British Library.

Printed and bound in the United States of America

The paper used in this publication meets the requirements

@ of the American National Standard for Permanence of

Paper for Printed Library Materials Z39.48-1992.

5 4 3 2 1

To the memory o f my late father, Mohan Murari,

who, through his lifelong struggle for freedom

and democracy, taught me to cherish them,

And to my late friend Ram C. Malhotra,

who sparked my interest in evaluation.

Contents

Acknowledgments

1 Why This Book?

2 The Why and What of Evaluating Democracy Assistance

Why Democracy Evaluations? 8

The Meaning of Evaluation 10

Types of Democracy Evaluations 13

Evaluation Modalities 22

Obstacles to Evaluating Democracy Interventions 28

3 Democracy Indicators: Their Use and Misuse in Evaluations

Micro-Level Indicators 32

Macro-Level Indices and Evaluations 38

Meso-Level Indices 43

Use of Meso-Level Indices in Albania 48

Disaggregating Macro-Level Indices SI

Conclusion 54

4 Designing Realistic Monitoring Systems

The Meaning of Monitoring 58

Guiding Principles 60

Designing a Monitoring System 62

Operating a Monitoring System 67

Collaborative Critical Reflections 68

5 Planning and Managing Evaluations

The Manager’s Role During the Planning Phase 72

The Manager's Role During Implementation 79

The Manager’s Role During Review and Follow-Up 84

viii Contents

6 Experimental and Quasi-Experimental Designs 87

The Logic Behind Experimental and

Quasi-Experimental Designs 89

Different Designs 91

Methodological and Practical Obstacles 96

The International Republican Institute’s Experience 102

The National Democratic Institute’s Experience 108

Conclusion 113

7 Nonexperimental Designs 117

Pre- and Postdesigns 118

Cross-Sectional Design 120

Case Study Evaluation Designs 121

Steps to Improve Methodological Rigor 130

8 Methods for Data Collection 137

Quantitative and Qualitative Methods 138

Sample Surveys and Censuses 140

Structured Direct Observation 148

Secondary Analysis of Data 151

Document Reviews 153

Key Informant Interviews 155

Focus Group Discussions 159

Group Interviews 162

Multimethod Approach 165

9 Communicating Findings and Recommendations 167

Guiding Principles for Effective Communication 168

Critical Elements of Evaluation Reports 173

W iillcn P ioducts 184

Verbal Communication 187

1 0 Concluding Observations 191

Appendix 1 : Acronyms 199

Appendix 2: Glossary of Evaluation Terms 201

Appendix 3: A Note on Democracy Assistance Programs 209

References 219

Index 231

About the Book 241

Acknowledgments

A lthough I cannot individually acknow ledge all the friends, col￾leagues, and dem ocracy experts who have helped me in this en￾deavor, I will always remain indebted to them. A few deserve special

mention, however.

Before deciding to w rite this book, I discussed my proposal

with m any leaders of dem ocracy nongovernm ental organizations

and dem ocracy experts, including K enneth W ollack, president of

the N ational Dem ocratic Institute (NDI); Tom Carothers, vice pres￾ident of the Carnegie Endow m ent for International Peace; Richard

Blue, vice president of Social Im pacts; and G erry Hyman, senior

adviser, C enter for Strategic and International Studies. Kenneth

W ollack invited me to present the proposal at an NDI d irecto rs’

m eetin g , and I alsn discusserl ideas for the b ook w ith R ebekah ITs￾atin, program officer for m onitoring and evaluation at the National

Endow m ent for Dem ocracy. David Black from the D em ocracy/

G overnance Office at the US Agency for International Development

(USA ID ) organized a small m eeting to discuss the proposal. They

all encouraged me to w rite this book and prom ised their full sup￾port. I am grateful for their encouragem ent and trust.

Many staff m embers at NDI, particularly Linda Stern, Peter

M anikas, and Sef Ashiagbor, shared their experiences and thoughts

and provided valuable evaluation reports. Linda also contributed two

sections to the book— one on N D I’s experience with experim ental

designs and a note on case study design. My friends at the Interna￾tional Republican Institute (IR I)— Jeffrey Lilley, Liz Ruedy, and

x Acknowledgments

Jonathan Jones— were also very generous with their time. Jonathan

contributed to Chapter 6 with a note on IR I’s experience with exper￾imental designs.

I am grateful to Lom e Craner, president of IRI, and Larry G ar￾ber, deputy assistant adm inistrator at USAID, for reading the entire

m anuscript. Tom Carothers, Peter M anikas, Linda Stern, Jonathan

Jones, Liz Ruedy, Sef A shiagbor, M ichael Patton, and Harry Blair

read earlier drafts of the m anuscript and gave valuable suggestions

about its contents and organization. There are not sufficient words to

thank them.

I wish to acknow ledge the continual support o f M elissa Schild

and Peter Davis in the Office of the D irector o f Foreign Assistance

Resources at the State Department. 1 am also thankful to the staff of

the departm ent’s library, particularly Linda Schw eizer, for their

timely assistance in getting me needed books and articles for writing

the book.

I would be remiss if I did not thank my wife, Parizad, for allow￾ing me to work on the book m ost w eekends and holidays during

2011, and also my daughters, Sonia and Sanaz, for their support.

Finally, I would also like to acknow ledge Lynne Rienner, who

has published my previous five books and has taken yet another gam ­

ble on this one. Senior project editor Lesli Brooks A thanasoulis and

copyeditor Diane Foose did a superb job in editing the manuscript.

— Krishna Kumar

Evaluating

Democracy

Assistance

1

Why This Book?

Unlike developm en t assistance, results chain of dem ocracy as￾sistance is non-linear and can be obscured by multiple factors

and actors at play in the dynamic contexts where our programs

take place.

— Kenneth D. Wollack, president of the NDI

I am not convinced if you can evaluate dem ocracy programs.

The types of evaluations you do are of little help to us. You

should trust the judgm ent of those who are struggling to pro￾mote dem ocracy at great personal risk.

—A Russian democracy activist

During the past two decades, international aid to promote democracy

in developing and transition countries has significantly increased.'

W hile precise figures are not available, it is estim ated to be more

than $3 billion.2 The international com m unity increasingly treats

dem ocracy assistance on a par with assistance to improve health, ed￾ucation, or econom ic growth. It now provides assistance to promote

hum an rights, the rule of law, and gender equality. It also assists in

developing electoral institutions for com petitive elections and gives

support for holding free and fair elections. In addition, it gives tech￾nical and financial assistance to strengthen civil society, political par￾ties, and independent media, which are essential for the functioning

of a dem ocratic society. Finally, the international com munity assists

in im proving governance, as it is now recognized that the institution￾alization of dem ocracy also depends on the functioning of efficient

1