The evidence debate continues: Chris Whitty and Stefan Dercon respond from DFID

Yesterday Chris Roche and Rosalind Eyben set out their concerns over the results agenda. Today Chris Whitty (left), DFID’s Director of Research and Evidence and Chief Scientific Adviser and Stefan Dercon (right), its Chief Economist, respond.

It is common ground that “No-one really believes that it is feasible for external development assistance to consist purely of ‘technical’ interventions.” Neither would anyone argue that power, politics and ideology are not central to policy and indeed day-to-day decisions. Much of the rest of yesterday’s passionate blog by Rosalind Eyben and Chris Roche sets up a series of straw men, presenting a supposed case for evidence-based approaches that is far removed from reality and in places borders on the sinister, with its implication that this is some coming together of scientists in laboratories experimenting on Africans, 1930s colonialism, and money-pinching government truth-junkies. Whilst this may work as polemic, the logical and factual base of the blog is less strong.

Rosalind and Chris start with evidence-based medicine, so let’s start in the same place. One of us (CW) started training as the last senior doctors to oppose evidence-based medicine were nearing retirement. ‘My boy’ they would say, generally with a slightly patronising pat on the arm, ‘this evidence-based medicine fad won’t last. Every patient is different, every family situation is unique; how can you generalise from a mass of data to the complexity of the human situation.” Fortunately they lost that argument. As evidence-informed approaches supplanted expert opinion the likelihood of dying from a heart attack dropped by 40% over 10 years, and the research tools which achieved this (of which randomised trials are only one) are now being used to address the problems of health and poverty in Africa and Asia.

The consequences of moving from expert (ie opinion-based, seniority-based and anecdote-based) to evidence-based healthcare policy, far from being some sinister neocolonial experiment, have been spectacular. To quote a recent Economist headline, ‘Africa is currently experiencing some of the fastest falls in childhood mortality ever seen, anywhere’. It is a great example of the positive side to modern Africa the current excellent Oxfam publicity campaign (right) is all about. This success is based on many small bits of evidence, from many disciplines, leading to multiple incrementally better interventions. Critically, it also involves stopping doing things which the expert consensus agreed should work, but which when tested do not. It is no accident that one of the most evidence-based parts of development is also one where development efforts have had some of their greatest successes.

Proper evidence empowers the decision-maker to be able to make better choices. This is a good thing. In every discipline, in every country, where rigorous testing of the solutions of experts has started, many ways of doing things promoted by serious and intelligent people with years of experience have been shown not to work. International development is no different, except that the communities we seek to assist are more vulnerable, including to our bad choices.

Much of what we all do in international development has very limited evidence that it does any good  (in this it is no different from many other policy areas) – which is not the same as saying it is pointless. Rather we don’t know what is pointless. Some of our actions will work better than we think, much of it will work much less well than we hope, and some of it will be damaging the poorest without us realising it. In the evidence-light areas we just don’t know which are which.

We must have the humility to accept that we are all often wrong, however reflexive the practitioner, however deep their reading and experience and passion to do good. Evidence-based approaches are not about imposing a particular theory or view of the world. It is simply about taking any opportunity to test our own solutions in the best way available, using evidence honestly when it is available to inform (note the word) decisions, and when the facts change, changing our minds.

This honesty includes saying to decision-makers when evidence is methodologically weak, mixed or missing so they know they are on their own, unable to rely on (or make a claim on) the evidence. The worst possible solution, which we know Chris and Ros would also deplore, is using the social power of the ‘expert’ to imply we know the answer when we actually have no solid evidential basis for our opinion or prejudice.

A few false assumptions about evidence-based decision making

Some of those who express unease about evidence-based policy and practice seem to assume that it is always based on randomised trials and quantitative methodologies: not so. Methods from all disciplines, qualitative and quantitative, are needed, with the mix depending on the context. Randomised trials are one tool amongst very many, although a good one in the right setting. The argument that evidence-based approaches can “only apply in cases of individual treatment and not the wider community level” ignores over 30 years of methodology which has done exactly that, with very convincing results.

A sterile argument  between people who are on the one side believe that a  randomised trial can answer any question (they can’t), and people who do not appear to be aware of any  methodological advances since the 1970s except in their own narrow field is a depressingly familiar experience. We know this does not apply to Rosalind and Chris, but listening to people passionately critiquing methodologies they have not taken the trouble to understand does no good to anyone. This applies both to a randomista who seems to believe that all there is to social research is a few focus groups and in-depth interviews, and to people from a more qualitative social science background who would have trouble explaining the difference between cluster randomised and step-wedge design but assume both are irrelevant to social research anyway (both can be used to measure societal rather than individual effects).

It is tempting to take every point the authors make where we have concerns about their factual basis and logical framework but we will take just three.

“Evidence-based approaches are pre-occupied with avoiding bias and increasing the precision of estimates of effect”. On less bias – generally true. Please complete the sentence ‘More biased research is better because…’. On precision – no, incorrect, the range of situations where a more precise answer is a better answer is small.

One statement we would like to address head-on starts “Evidence-based approaches became linked to value for money concerns to deliver ‘results’…”. We agree- and this is a good thing. Doing a pointless thing, professionally delivered and passionately believed in, is always going to be poor value for money. Testing what works and what does not therefore is essential to value for money. More importantly, doing pointless things diverts very limited human and financial resources, in an ocean of need, away from those who could best use them- not what any of us are in international development to do.

Is it “technical approaches” on the one hand, and “power, political economy” analysis on the other?

Rosalind and Chris’ key criticism is that evidence-based approaches “deflect attention from the centrality of power [and] politics […] in shaping society”, and they offer “power analyses” as an apparent alternative to assessing rigorously what works. This creates a false dichotomy, as if a choice has to be made between a “technical, rational and scientific approach to development” and an approach that recognises politics and the role of power. It is easy rhetoric, but troubling and, if taken much further, even dangerous. Understanding power and politics and how to assist in social change also require careful and rigorous evidence, and again, results are not simply what experts would have expected a priori. Recent studies on the positive impacts of female leadership quotas in rural India are for many of us rather surprisingly good news, even if one can fairly worry about its applicability in other settings, while the struggle to find systematically a positive impact of decentralisation and community-driven development programmes is important to internalise in our actions for change, and highlights the importance of understanding contexts and politics. In these cases, it is not a matter of just RCTs, but of rigour, and of combining appropriate methods, including more qualitative and political economy analysis.

Strong analysis of politics and power without offering much in terms of what can be acted upon is similarly unhelpful. They criticise an evidence-focused agenda by stating that “to act ‘technically’ in a politically complex context can make external actors pawns of more powerful vested interests and therefore by default makes them, albeit unintentionally, political actors.” But all actions by external actors will interact with political forces and vested interests. In many of the settings where development actors want to make a difference, power and political institutions are biased against the poor. Being able to act on strong evidence of what works in constrained political settings is crucial.

A reductionist and misinformed view of evidence as purely ‘technical’ or as being only about “what works” is unhelpful – it is also about generating evidence and understanding (and learning) on why interventions and approaches may work, including understanding the social, political, and economic factors that may enable or constrain success of different approaches. Far from the search for evidence pushing us in a ‘technical’, apolitical direction it has reinforced the importance of understanding and trying to tackle the underlying causes of poverty and conflict. There is agreement on the importance of politics and institutions in shaping growth, security and human development. However, the ability of external actors to influence institutions is much less clear and this is where DFID research is now focussed. Ros and Chris have misread the context – the commitment to evidence has opened up the space fundamentally to challenge conventional, technical approaches to aid.

Why it matters for international development

There are large areas of international development where decision-makers are largely flying blind – forced to make decisions purely on gut feeling and ideology not because they wish to because they have no option. Try making difficult decisions in education policy compared to health policy and the difference in usable evidence is dramatic – yet both are complex, social and context-dependent parts of human life. It is always puzzling when people say airily ‘health is easy’- it is not, and is an intensely political and social subject requiring interventions at societal level.

Today we can eradicate rinderpest in cattle and build bridges over the Zambezi based on rock-solid evidence from many disciplines, but do not have anywhere near as clear an idea how to reduce violence against women or tackle police corruption. All are great challenges with social dimensions but in two of them people have set about finding and testing solutions in a systematic way over many decades.

Having robustly tested evidence-based solutions certainly does not eliminate politics: the decision whether to build a bridge, what sort and where, is an intensely political choice – but at least those making the choice now have a fair assumption it will stand up- based on hundreds of years of incremental evidence. The evidence-barren areas in development are a collective, and in our view shameful, failure by us all in the academic and practitioner community. We should never excuse them with the feeble assertion that it is too difficult or complicated. Development is difficult and complicated – but the bases for making decisions will gradually improve if we are serious about improving it.

In conclusion, we collectively have the capacity to be able to give to our successors in every continent a far better basis on which to make their decisions for their lives than our generation have. To imply it is not worth trying to provide the best and most rigorous evidence to those who need to make difficult decisions because they will have other influences as well is like saying to someone going for a walk in dangerous mountains that they do not need a map because there will be many other factors that will determine where they go. That is true – but they are still less likely to fall off the cliff if they have one.

Where evidence is clear-cut we should be making that plain to decision makers – and where it is not we should say that as well, be honest about what is there and try to get better evidence for the future. That, in essence, is what evidence-based decision making is about – and all it is about. If the academic community is serious about trying to assist those working in the field (including in Oxfam), and above all empowering the most vulnerable communities to make the most informed possible decisions available for their own development, we should be putting our greatest efforts into supporting decision-makers to use the best evidence, and finding better methodologies in areas where we currently have very weak evidence. There are many, and this should be tackled as a matter of urgency.

Tomorrow, Chris Roche and Rosalind Eyben respond

The political implications of evidence-based approaches (aka start of this week’s wonkwar on the results agenda)

The debate on evidence and results continues to rage. Rosalind Eyben (left) and Chris Roche (right, dressed for battle), two of the organisers of April’s Big Push Forward conference on the Politics of  Evidence, kick off a discussion. Tomorrow Chris Whitty, DFID’s Director of Research and Evidence and Chief Scientific Adviser, and Stefan Dercon, its Chief Economist, respond

Distinct from its more general usage of what is observed or experienced, ‘evidence’ has acquired a particular meaning relating to proof about ‘what works’, particularly through robust evidence from rigorous experimental trials. But no-one really believes that it is feasible for external development assistance to consist purely of ‘technical’ interventions. Most development workers do not see themselves as scientists in a laboratory, but more as reflective practitioners seeking to learn how to support locally generated transformative processes for greater equity and social justice. Where have these experimental approaches come from and what is at stake?

The origins and critiques of evidence-based approaches

Evidence-based approaches are pre-occupied with avoiding bias and increasing the precision of estimates of effect. In the UK they spread beyond clinical practice when the government elected in 1997 was keen to demonstrate that its decisions would not be driven by political ideology but rather by objective evidence. Evidence-based approaches became linked to value for money concerns to deliver ‘results’ as efficiently and effectively as possible, by a government recently described as ‘truth junkies’.

Yet, even within medicine, the leap from evidence-based clinical practice into evidence-based policy was challenged. A British Medical Journal article by Nick Black in 2001 drew on an extensive body of contemporary literature on policy processes to argue that policy was shaped by institutional arrangements, values and beliefs and a variety of different sources of information.

Opponents of evidence-based education critiqued its positivist assumptions; its linear cause-effect thinking; and the poor understanding of the tensions between scientific and democratic control of educational practice. An OECD report in 2007 on the reasons for the uptake of evidence-based education in Sweden, the UK, Canada, the USA and Australia, noted the increasing pressure for greater accountability of expenditure and effectiveness, an explosion in the search for measurable outcomes, and demands that impacts and effectiveness be given a monetary value.

The same report noted that evidence-based approaches were largely absent in OECD countries ‘less used to empirical and quantitative methodologies in the social sciences.’ The de-politicization of policy making is one of the reasons given by development researchers for its neglect in France.

In UK social policy, evidence-based approaches with their ‘gold standard’ of experimental or quasi-experimental design, have been criticised as being inapplicable to complex issues. In What Works, Tony Harrison argued that evidence-based approaches can only apply in cases of individual treatment and not at the wider community level where multiple perspectives come into play and no agreement exists about the nature of the problem.This of course is the case with most development programmes, and in particular those that seek transformational change.

Evidence based approaches in development: an anti-politics firewall?

Arguably evidence-based approaches build an anti-politics firewall. Development assistance becomes a ‘technical’ best practice intervention based on rigorous objective evidence, delivering best value for money to domestic taxpayers and recipient country citizens mostly without interfering in that country’s politics. They are the latest manifestation of  a certain long-standing approach to development that as Timothy Mitchell wrote in Power of Development, speaks to the sector’s ‘need to overlook its internal involvement in the places and problems it analyses and present itself instead as an external intelligence that stands outside the objects it describes’.

In the 1930s Africa was seen as ‘a living laboratory’ to achieve improvements in the welfare of the populations. Evidence-based approaches are reviving the development-as-laboratory idea. In 2012 the World Bank established a Gender Innovation Lab to design ‘innovative interventions to address gender inequality and to develop rigorous research projects in order to produce evidence on what works and what does not’.  Jeffrey Sachs’ Millennium Villages have been framed as  ‘laboratories to lift people out of poverty’. The most well-known is the J-Pal Poverty Action Lab whose mission is to reduce poverty ‘by ensuring that policy is based on scientific evidence’.

In the absence of political debate, this approach can exacerbate the tendency to see people as subjects requiring treatment, rather than as citizens with political voice. Power silences any challenges to the technical framing of ‘the problem’, foreclosing discussion of the structural causes and consequences of inequity and how these should be tackled. To act ‘technically’ in a politically complex context can make external actors pawns of more powerful vested interests and therefore by default makes them, albeit unintentionally, political actors.

High stakes

Evidence-based technical approaches can therefore deflect attention from the centrality of power, politics and ideology in shaping society. We agree with the view of the Developmental Leadership Program that recent research suggests that  the development sector should be ‘at the frontier of a narrative shift between a technical, rational, and scientific approach to development, and a recognition that politics matters; that poverty reduction is not a technical problem but requires significant social change, and that this social change is, and must be, both political and locally led.’

However this has some significant implications for external actors. We need to be self-aware to avoid disempowering others.  This requires undertaking power analyses with ourselves factored in – as organisations and individuals who can make a positive or negative contribution, often inadvertently.  It means engaging with a wider and more diverse group of policy actors in the state, civil society and the private sector; whenever possible, supporting debate, locally-driven problem solving, and independent research. It means avoiding overly linear project-based aid modalities that demand omniscience before they have even begun.

As Michael Sandel has recently argued in his book about the moral limits to markets, how we put values (and prices) on things can change their meaning, as well as change the relationship between economic actors.  More information, (dare we say ‘evidence’), is needed to draw some firmer conclusions about the consequences of evidence-based approaches to designing projects and assessing results. This is why the Big Push Forward is currently seeking to crowd-source more information from development practitioners about how they actually experience the ‘results’ agenda, and why we believe this issue needs more debate.

And make sure you come back tomorrow for DFID’s counterblast

Lant Pritchett v the Randomistas on the nature of evidence – is a wonkwar brewing?

Last week I had a lot conversations about evidence. First, one of the periodic retreats of Oxfam senior managers reviewed our work on livelihoods, humanitarian partnership and gender rights. The talk combined some quantitative work (for example the findings of our new ‘effectiveness reviews’), case studies, and the accumulated wisdom of our big cheeses. But the tacit hierarchy of these different kinds of knowledge worried me – anything with a number attached had a privileged position, however partial the number or questionable the process for arriving at it. In contrast, decades of experience were not even credited as ‘evidence’, but often written off as ‘opinion’. It felt like we were in danger of discounting our richest source of insight – gut feeling.

In this state of discomfort, I went off for lunch with Lant Pritchett (right – he seems to have forgiven me for my screw-up of a couple of years ago). He’s a brilliant and original thinker and speaker on any number of development issues, but I was most struck by the vehemence of his critique of the RCT randomistas and the quest for experimental certainty. Don’t get me (or him) wrong, he thinks the results agenda is crucial in ‘moving from an input orientation to a performance orientation’ and set out his views as long ago as 2002 in a paper called ‘It pays to be ignorant’, but he sees the current emphasis on RCTs as an example of the failings of ‘thin accountability’ compared to the thick version.

In a forthcoming paper (which I will definitely link to when it’s published), Lant defines thick accountability as ‘an “account” in the sense of a justificatory narrative of my actions, the story of my actions I tell to those whose opinion of me is important (including myself, but including family and kinsmen, friends, co-workers, co-religionists, people I respect and desire admiration from) that explains why my actions are in accord with, and deserving of, a positive view of myself.    In contrast, thin accountability is “accounting”, which is that small part of the account about which objective facts can be established.’  He sketched out the inevitable 2×2 matrix for me

The challenge in most development work is to move from top left to bottom right. There are occasions when thin accountability/high performance works – typically routine functions like delivering mail or building roads. But anything involving the messiness of people and institutions requires thick accountability, involving deep bonds of trust and reciprocal relationships that are likely to be defined by a setting’s unique history and geography – what he calls ‘folk practices, from which formal organizations can (re)emerge’.

He argues that the randomistas just don’t get this. His critique of RCT culture ranged pretty wide:

• The politics of RCTs: ‘RCTs are a tool to cut funding, not to increase learning.’  ‘Randomization is a weapon of the weak’ – a sign of how politically vulnerable the argument for aid has become since the end of the Cold War. ‘Henry Kissinger wouldn’t have demanded an RCT before approving aid to some country.’ And I can’t see the military running RCTs to assess the value for money of new weaponry before asking for more cash (mind you, if they did, that might at least save some money on Trident….).
• The lack of interest in theory: ‘the randomistas are going back to alchemy – atheoretic experimentation’.
• RCTs test at most a few project variants using ‘project vs non-project’, whereas interventions are typically multiple, overlapping and synergistic (i.e. the whole cannot be reduced to a sum of parts).
• No-one evaluates the evaluators. At the very least, given how much RCTs cost, you need to know that the findings are useful elsewhere (so-called ‘external validity’). But once you have multiple RCTs on the same issue (and their spread is starting to produce such comparable studies), you find very little external validity – the results of an RCT in one country and time are not replicated elsewhere (with the possible exception of deworming in schools, but even that iconic RCT story is contested). This is the big contrast with real science, where replicability is a key condition of validity.

Patronising? Overpromising? Nah....

In another recent paper, he argues instead for ‘structured experiential learning’, which involves rigorous and intelligent conversation, rather than the illusory certainty of numbers. Get people in a room, agree what the problem is, agree to try out some experiments to solve the problem, and set up rapid feedback to identify failure and/or build on success. In another recent paper, he calls this ‘Problem Driven Iterative Adaptation (PDIA)’. It sounds very similar to the conclusions of the Africa Power and Politics Programme, which I reviewed recently. In yet another paper (he’s horribly prolific), he also draws a neat distinction between experiments and experimentation:

‘Perhaps surprisingly, the experimentation and experiments approaches are not at all the same. I argue that experiments, while a terrific method for generating PhD dissertations and published papers, will have impact on development and development practice only insofar as they are embedded in an experimentation approach (which they are often not).’

The feeling I got from these conversations was of two tribes encamped and preparing for battle. That line from Henry V comes to mind: ‘from camp to camp, through the foul womb of Boston night, the hum of either army stilly sounds.’ On one side are the ‘best fit’ institutionalists and complexity people, with their focus on path dependence, evolution and trial and error. On the other are the ‘universal law’ experimentalists, offering the illusory certainty of numbers, and (crucially) comfort to the political paymasters seeking to prove to sceptical publics that aid works. It’s hard to see how they can both be right, or happily coexist for long. Time for a wonkwar on this blog, I think…..

Poor Economics – a rich new book from Abhijit Banerjee and Esther Duflo

Just finished Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty, the latest Big Book on development. Like all good books, it has its own website, full of background papers etc. It’s from the doyennes of the new focus on measurement in general and randomized control trials (RCTs) in particular, Abhijit Banerjee and Esther Duflo from the Abdul Latif Jameel Poverty Action Lab (J-PAL) at MIT. Given that provenance, I was half-expecting a fairly reductionist form of ‘if it can’t be measured, forget it’ thinking, but like many gurus, they are much more intelligent and nuanced than their followers.

Their overall approach is Bottom of the Pyramid meets Freakonomics – ‘leave the large questions aside and focus on the lives and choices of poor people’ and why interventions by governments or NGOs do/don’t work. They draw heavily on J-PAL’s 240 experiments in forty countries. It’s all very beguiling, with RCTs presented as the ‘cleanest way to answer questions’. They are unashamedly, indeed belligerently, micro, small-is-beautiful technocrats, with little time for the ‘big questions and grand theories’ of Jeff Sachs or Bill Easterly (who both get a polite kicking, characterized respectively as ‘supply wallah’ and ‘demand wallah’).

Instead, Banerjee and Duflo prefer the benign paternalism of the ‘nudge’ approach – providing information, changing default options, introducing incentives to encourage (but not coerce) poor people to do what we think is best for them. As they point out ‘Aren’t we, those who live in the rich world, the constant beneficiaries of paternalism now so thoroughly embedded into the system (e.g. chlorinated water, routine vaccinations) that we hardly notice it?’

Chapters cover themes (hunger, population, health, education) and institutions (risk management, microfinance, entrepreneurialism, politics v policies). Each asks heretical questions (why don’t poor people do what’s best for them – spend more on food, and less on treats? Why do they insist on expensive and useless injections from quack doctors, rather than cheaper and more effective prevention?) and is crammed with fascinating research and surprising conclusions (the most cost effective ways to delay teenage pregnancy is to give out free school uniforms – page 115 for the explanation).

The authors identify five key lessons in their final chapter (‘In Place of Sweeping Conclusions’….):

“1. The poor often lack critical pieces of information and believe things that are not true (e.g. on immunization, or benefits of education)

2. The poor bear responsibility for too many aspects of their lives. The richer you are, the more the ‘right’ decisions are made for you. [hence the need for nudges]

3. There are good reasons that some markets are missing for the poor, or that the poor face unfavourable prices in them….in some cases technological or institutional innovation may allow a market to develop where it was missing, [but in others] governments should step in.

4. Poor countries are not doomed to failure because they are poor, or because they have had an unfortunate history….. failures have less to do with some grand conspiracy of the elites to maintain their hold on the economy and more to do with some avoidable flaw in the detailed  design of policies and the ubiquitous three Is: ignorance, ideology and inertia…. It is possible to improve governance and policy without changing the existing social and political structures

5. Expectations about what people are able or unable to do all too often end up turning into self-fulfilling prophecies. Changing expections is not easy, but it is not impossible”

Beyond the specifics, I found myself wrestling with the underlying narrative – how do Banerjee and Duflo actually see poor people? On one hand, they are genuinely and unpaternalistically fascinated by them – the book is studded with detailed conversations with poor people as the authors try and uncover the logic behind their choices and predicaments. Poor people are definitely not ‘the other’, but just like the rest of us, albeit living lives of greater risk and effort. The approach seems to be

a) talk to poor people
b) on the basis of these conversations and observations, come up with some hypotheses
c) then design an RCT or other data-driven exercise to test them. Discard the bad ones, and keep the ones that fit the data.
e) Then speculate on ‘likely explanations’ for the relationships you have established

What they don’t seem to do (rather than that last stage) is go back and talk to the poor people again to check if their explanations are the right ones – ‘the poor’ as the authors call them, seem to be important as informants, but not the final arbiter – instead, ultimate truth lies in what they call ‘the verdict of the data’. So for example, when they discuss the finding in Peru that ‘when former squatters were handed out property rights, fertility declined, but only if the woman’s name was included on the title’ they speculate on the reasons for this (women acquiring bargaining power) instead of going to ask some women. I would have liked more of a ‘Voices of the Poor’ exercise to reality check their conclusions and discussions with poor people themselves, but at times (e.g. in the chapter on health), ‘the poor’ started to resemble little more than lab rats.

A common critique of this kind of work is that it ignores issues of power and politics, but at least for this book, that isn’t really fair. The authors are well aware of issues of power in terms of patriarchy or caste. But they are incrementalist and disagree with the structural focus of the ‘political economy’ approach or sweeping calls for revolution – they argue that positive changes can be achieved with less pain, often on a massive scale, even in hostile political environments, simply by minor tweaks to policies and institutions.

And they aren’t scared to follow the logic of their own arguments, even when it takes them in some surprising directions, not least concluding that government intervention is widely needed, that giving out stuff for free and social safety nets are often a good idea, that poor people would usually prefer a government job to the hassle and risk of being an entrepreneur, or that industrial policy can make a lot of sense. Their treatment of microfinance, despite it’s close association with RCTs, echoes many of the misgivings in the recent discussion on this blog. So it’s hard to pigeon-hole the work as particularly right or left wing.

But is it a Big Book? Yes in terms of the approach – I think it will leave a lasting impact on its readers in showing the merits of a bottom-up, evidence-driven approach. But not, I think, in terms of content – lots of interesting, surprising facts and analyses, but no one big message. Given their suspicion of grand narratives, I’m sure the authors would be quite happy with that.

And here (again) is Esther Duflo, queen of the randomistas and all-round rising star, strutting her stuff on TED

Randomized Controlled Trials: panacea or mirage?

Randomized Controlled Trials (RCTs) are all the rage among development wonks at the moment. Imported from medical research, they offer the tantalizing allure to social scientists of finally overcoming the Achilles’ heel of real-world research – the counterfactual (aka ‘how do we know what would have happened if we hadn’t lobbied the government/ employed the teachers/ built the road etc?). Here, one of the RCT gurus (and recent winner of the almost-as-good-as-a-Nobel John Bates Clark medal for young economists), Esther Duflo, sets out the case for the technique [h/t Chris Blattman]. She is the Abdul Latif Jameel Professor of Poverty Alleviation and Development Economics in the Department of Economics at MIT and a founder and director of the Jameel Poverty Action Lab (J-PAL) (must take a look at her business card sometime).

Not everyone is convinced – in a recent debate in the Enterprise Development and Microfinance journal James Copestake (anti) slugged it out with Dean Karlan and Nathanael Goldberg (pro). Copestake criticizes RCTs on four grounds:

“Problem selection bias. I am worried by reports of bright young “randomistas” narrowing the research agenda by selecting issues for research to fit their preferred tool, rather than finding the best tool to fit the most important issues. For example, use of RCTs to test product design changes should not divert attention from other influences on impact that may be harder to randomise: geographical targeting methods and organisational culture, for example.

External validity. RCTs require a fixed investment and generate evidence at the end of a discrete period of time, rather than continuously. This accentuates the difficulty of choosing which few among many possible ‘treatments’ should be studied, where and when. The value of findings then depends upon their transferability.

Cost effectiveness. I’m very much in favour of experimentation and testing, but remain to be convinced that RCTs are necessarily the most cost-effective way for managers and policy makers operating in complex, diverse and uncertain contexts to evaluate them, compared to triangulating routine monitoring data against focus group discussions and individual satisfaction surveys, for example.

Fourth, there may well be other more technical problems with RC studies. For example, it will not always be possible to ensure that treatment and control groups are not contaminated through spillover effects between them: response to not having a treatment being affected by my knowledge that others are having it, for example.”

If you want to read Karlan and Goldberg’s replies to these criticisms to make your own mind up who’s right, I’m afraid you have to pay (or go to the library). Anyone know of an ungated version? Or read the wikipedia entry linked to at the top of this post, which I thought gave a rather good summary of the pros and cons.