FP2P greatest hits: top posts & comments by theme (ag, inequality, results, education etc) now available

Oh boy, I have a greatest hits album. This blog was launched back in 2008 to help promote the first edition of From Poverty to Power, but rapidly

Easily confused.....

acquired an identity and readership of its own. I reckon there’s about half a million words or so up there now, and some of them have to be worth reading (monkeys, typewriters etc). So my long-suffering colleagues Sarah Minty read the lot, and grouped together relevant posts under categories such as Agriculture, Climate Change, Inequality, or Monitoring, Evaluation & Learning. The links take you through both to posts and (just as important) comments.

Pdf with links to posts in all categories here.

Some categories are not currently included, but probably should be:

• Posts that stoked most controversy within Oxfam (example)
• Videos that my nephews and nieces enjoyed (example)
• Funnies (example)

The pubs team made a special effort to get this out for next week’s Politics of Evidence conference, as January’s big debate between Ros Eyben, Chris Roche, Chris Whitty and Stefan Dercon (plus dozens of highly qualified commenters) on this topic has proved particularly popular with students, academics and practitioners.

Any suggestions for further refining the categories are of course, welcome.

So What do I take Away from The Great Evidence Debate? Final thoughts (for now)

The trouble with hosting a massive argument, as this blog recently did on the results agenda (the most-read debate ever on this blog) is that I then have to make sense of it all, if only for my own peace of mind. So I’ve spent a happy few hours digesting 10 pages of original posts and 20 pages of top quality comments (I couldn’t face adding the twitter traffic).

(For those of you that missed the wonk-war, we had an initial critique of the results agenda from Chris Roche and Rosalind Eyben, a take-no-prisoners response from Chris Whitty and Stefan Dercon, then a final salvo from Roche and Eyben + lots of comments and an online poll. Epic.)

On the debate itself, I had a strong sense that it was unhelpfully entrenched throughout – the two sides were largely talking past each other,  accusing each other of ‘straw manism’ (with some justification) and lobbing in the odd cheap shot (my favourite, from Chris and Stefan ‘Please complete the sentence ‘More biased research is better because…’ – debaters take note). Commenter Marcus Jenal summed it up perfectly:

‘The points of critique focus on the partly absurd effects of the current way the results agenda is implemented, while the proponents run a basic argument to whether we want to see if our interventions are effective or not. I really think the discussion should be much less around whether we want to see results (of course we do) and much more around how we can obtain these results without the adverse effects.’

There were some interesting convergences though, particularly Whitty and Dercon’s  striking acknowledgement of the importance of power and politics, which are often assumed to be excluded from the results agenda. But what they actually said was

‘Understanding power and politics and how to assist in social change also require careful and rigorous evidence.’

True, but what about reversing the equation? Does understanding the role of evidence in development also require a careful and rigorous understanding of power and politics? They never fully address that crucial point, which is at the heart of Roche and Eyben’s critique.

Both sides (rather oddly, as acknowledged experts in their fields) decried the role of experts. Whitty and Dercon called for ‘moving from expert (i.e. opinion-based, seniority-based and anecdote-based) to evidence-based policy’. Ah, turns out that what is actually being suggested is a move from one kind of expert (practitioners) to another (evidence/evaluation).

As a non number-cruncher I also took exception to their apparent belief that only those who understand the methodological intricacies of different evaluation techniques are eligible to pass judgement. On that basis politicians would be out of a job, and only rocket scientists would get to pronounce on Trident.

There was also a really confusing exchange on the hierarchy of evidence. Whitty and Dercon show a surprising (to me at least) commitment to multi-disciplinarity: ‘Methods from all disciplines, qualitative and quantitative, are needed, with the mix depending on the context….. it is not a matter of just RCTs, but of rigour, and of combining appropriate methods, including more qualitative and political economy analysis.’

Music to the ears of the critics, but is it actually, you know, true? Everything I hear from evaluation bods is that DFID does actually see RCTs as the gold standard, and other forms of evidence as inferior. Roche and Eyben returned to the attack on this in their response, arguing that what Whitty and Dercon call the ‘evidence-barren areas in development’ are only barren if you discount sociology and anthropology, among others, as credible sources of evidence. By the way, Ed Carr has a brilliant new post on the (closely linked) clash between quants and quals, arguing that while quants can establish causation, only quals can explain how that causation occurs.

But the exchange did provide me with one important (I think) lightbulb moment. It was about failure. Whitty and Dercon were particularly convincing on this: the evidence agenda ‘involves stopping doing things which the expert consensus agreed should work, but which when tested do not’. This is a nice Popperian twist – the role of evidence is not to prove that things work, but to prove they don’t, forcing us to challenge received wisdom and standard approaches. This is indeed what I noticed about Oxfam’s recent ‘effectiveness reviews’ – if you find no or negative impact, then you (rightly) start to re-examine all your assumptions. But if this is the proper role for the evidence agenda, is it politically possible? By coincidence I have just read Ed Carr’s forceful critique of Bill Gates’ approach to evaluation, arguing that failure is often airbrushed out in order to safeguard funding and credibility. That seems a pretty fundamental contradiction.

The comments were just as thought-provoking. One of the key messages that emerged is the gulf between these debates and what those in charge of gathering results in aid agencies actually face – highly constrained resources, crazy time pressure, and the need to deliver some (any!) results to feed the MEL machine. Oxfam’s Jennie Richmond reflected on the gap between theory and practice yesterday.

Commenter Enrique Mendizabal asked whether we are demanding a different role for evidence in poor countries than in our own.

‘In the UK, health policy is decided by a great many number of factors or appeals (evidence, sure, but also values, tradition, biases, political calculations, etc). We may complain about it but we accept that it is a system that works. But health policy for Malawi (or other heavily Aid dependent countries) is decided mainly by evidence (or what often passes as evidence at the time) and usually by foreign experts…. would we be happy with USAID funding a large evidence-based campaign to reform the NHS or our education policy?’

But he took his argument a step further – if the final decision should be left to the interplay of evidence (of different sorts), politics and negotiation, then DFID and other donors would be better advised to boost the ‘enabling environment’ for such debates and decisions by investing in tertiary education in developing countries:

‘strengthening economic policy debate is a more adequate objective than achieving policy change (even if it is evidence based).’

Commenter David highlighted a fundamental point that rather went missing in the initial exchange – how the results agenda does or doesn’t work in complex systems:

‘The results agenda approach tends, by presenting development as objectively knowable if broken down into discrete and small bits, todrive attention toward small, more easily measurable interventions to test, particular those that are suited to situations that are simple or complicated rather than complex. Current processes around evidence-based results fail to grapple with complex systems, interaction effects, and emergent properties that dominate most aid project landscapes.

A fundamental critique of the evidence-based revolution is that it actually diminishes efforts to get rigorous evidence about addressing complex challenges. We all want evidence, it’s a question of whether the current framing of “evidence-based” is distorting what types of evidence we gather and value. For those who think that the current emphases on methods to test what works are distorting how we value the evidence coming in (RCT=gold, qualitative methods=junk), this offers little other than platitudes about lots of other methods existing.

Personally, I would be a bigger proponent of the evidence-based revolution if it was coming to folks interested in power, politics, and development, and asking them what their questions are and what evidence might contribute to their work. Absent a learning agenda set to fit complex space and concern itself with power, it will continue to seem to me to be an instance of methods leading research – or searching for keys under the light rather than inventing a flashlight.’

To be fair, Roche and Eyben explicitly chose to focus on the politics of evidence, rather than the implications of complex systems (for example, the question of external validity in complex systems – or lack of it – raised by Lant Pritchett in our recent conversation.)

Final thoughts? After about 500 votes, the poll went narrowly to Whitty and Dercon (34% v 31% for Roche and Eyben, with a pleasing late rally for the ‘totally confused’ camp – my natural habitat). I think Chris Roche and Rosalind Eyben need to work on their communication style (more punchy, less abstract, more propositional). Chris Whitty and Stefan Dercon should give some examples of gold standard anthropological or sociological evidence to allay the doubts over their true commitment to multi-disciplinarity, and take the complex systems question more seriously.

A massive thankyou to all who took part, and please can you come back for another go in a year or so? This one isn’t going away.

Theory’s fine, but what about practice? Oxfam’s MEL chief on the evidence agenda

Two Oxfam responses to the evidence debate. First Jennie Richmond, (right) our results czarina (aka Head of Programme Performance and Accountability) wonders what it all means in for the daily grind of NGO MEL (monitoring, evaluation and learning). Tomorrow I attempt to wrap up.

The results wonkwar of last week was compelling intellectual ping-pong. The bloggers were heavy-hitters and the quality of the comments provided lots of food for thought. However, I was left wondering what it all meant for those of us who work in NGOs, trying to generate and learn from ‘evidence’ on a daily basis. I found myself unable to simply vote, so instead I blog….

The results and evidence agendas have brought some real benefits to NGOs in my view. First and foremost, it is important and right that those of us who claim to work in the interests of the poorest people in the world and are stewards of other people’s money, should set ourselves high standards for our own impact. In its simplest form the results agenda asks us to justify the trust others have placed in us, by demonstrating whether we are actually bringing about positive change. In Oxfam GB, accountability has long been held as a core organisational value. It is not the results and agenda that has got us thinking about how to capture and communicate our effectiveness, but it has provided a helpful additional push.

A further positive is that space has been created both within our own organisations and in the wider sector, to stop, listen and learn. MEL-istas (as Duncan calls us) 5 years ago struggled to get the ear of senior managers (let alone Ministers). But the results agenda has increased the stakes around MEL – encouraging organisations not only to increase investment, but also to listen to the findings coming from our own data gathering and analysis.

However, it has also increased the demand and the expectation, which are not easily met by all NGOs. In Oxfam GB the investment in MEL has increased over the last couple of years, undoubtedly, but still it is a real stretch to deliver the ever-more ambitious demands from donors, to develop tools to tell the story of our broader organisational impact, and to ensure that we are developing innovative ways of measuring cutting-edge programming areas, such as resilience, enterprise development and influencing.

And we are one of the largest international development NGOs in the UK. How much more difficult for the smaller and niche NGOs, or those who lack the flexible financing that permits investment in MEL and innovation? We are conscious in Oxfam that we and other large NGOs need to guard against distorting the NGO market place by pushing the boundaries on MEL and impact too far, and thereby creating expectations that cannot be met by everyone. Somehow we all need to keep our sights on a proportionate approach.

It is not just important to generate evidence, but also to use it properly. There is increased demand for serious, evidence-based conversations about what works.  None of us can get away with decisions made purely on gut instinct, force of habit or ideological leaning. We are challenged by the ‘evidence’ question to collate and distil from the broad knowledge base we have at our disposal. And this has in some cases led to surprises. Rigorous studies, whether based on qualitative or quantitative methods, can challenge our preconceptions – showing us impact where we were not optimistic, or the opposite. The test, of course, comes when new programmes are designed. Will the body of evidence be applied – will we be able to find it for starters (in our often not-so-state-of-the-art knowledge management systems), and will it be politically acceptable in our own organisations to apply it to practice?

So, how can we use the results and evidence agendas and make them useful to us as NGOs?  We need to do this in a way that a) is true to the actual work we do (which in the case of Oxfam includes a great deal of work that drives for political change and influencing) and b) does not distort decision-making away from the right decisions (i.e. what most suits the specific needs and opportunities of each context) in our efforts to be able to measure and communicate what we are doing.

One of the concerns raised in last week’s blog was that in some institutions, evidence becomes synonymous with impact evaluations, and even specifically with Randomised Control Trials. As all the bloggers agreed, the default use of one research method for interventions of all types is simply nonsensical. You only have to look at the enormous variety of the things we do in international development (from campaigning for policy change to delivery of bed-nets, from building of bridges to raising awareness of the rights of citizens) to realise that one approach is just not going to cut it.

Another challenge is that so much of what we do in international development is extremely hard to measure. How can we trace the input through to impact chain and clearly demonstrate the ‘on the ground’ changes we have brought about in people’s lives when the investment is in budget support or core funding?  How can we reduce the process of a community standing up against acts of violence against women to a Value for Money calculation? The ethical dilemmas and practical difficulties wrapped up in measuring and ‘evidencing’ many of the processes we are involved in are huge. And, as Eyben and Roche point out, much of what we engage with in international development is messy and political. We need to make sure that the tools we have at our disposal for evidence generation are sophisticated and nuanced enough to acknowledge this messy political reality, and that we are sharing ideas on how to do this in a practical and affordable way.

The push for evidence should go hand in hand with a more entrepreneurial approach to development, opening up space for honest

MEL that - US military mindmap of Afghanistan

reflection on both success and failure. That is the theory. But, of course, there are obstacles to this becoming a reality. Our systems in large institutions, including NGOs, are designed to demonstrate success. We all have our logframes and our KPIs, and we want to be able to put a tick in the box. No-one wants their project to be the one famous for not achieving what it set out to do, even if the real story is that it helped enormously to generate learning for future projects. Complexity thinking is having some influence right now, which helps to raise the right questions about process and incentives. However, we have a long way to go before even in the most reflexive learners in NGOs and other development institutions want their project to be hailed as the great failure.

So, we proceed with caution – welcoming the increased space the Results Agenda provides to consider ‘what seems to work’, and the profile it gives to the need to take a thorough and transparent look at the information coming out of our programmes. But, wary of the dangers of distorting what we do in order to make it measurable; of placing the MEL ‘bar’ for NGOs too high to reach; of the over-emphasis of certain methodologies; and of the danger of ignoring political realities in the work that we do. It is certainly helpful to keep reflecting and questioning, however, from all sides of the debate – so the wonkwar of last week was welcome.

Evidence and results wonkwar final salvo (for now): Eyben and Roche respond to Whitty and Dercon + your chance to vote

In this final post (Chris Whitty and Stefan Dercon have opted not to write a second installment), Rosalind Eyben and

Chris Roche reply to their critics. And now is your chance to vote (right) – but only if you’ve read all three posts, please. The comments on this have been brilliant, and I may well repost some next week, when I’ve had a chance to process.

Let’s start with what we seem to agree upon:

• Unhappiness with ‘experts’ – or at least the kind that pat you patronizingly on the arm,
• The importance of understanding context and politics,
• Power and political institutions are generally biased against the poor,
• We don’t know much about the ability of aid agencies to influence transformational change,
• Mixed methods approaches to producing ‘evidence’ are important. And, importantly,
• We are all often wrong!

We suggest the principal difference between us seems to concern our assumptions about: how different kinds of change happen; what we can know about change processes; if how and when evidence from one intervention can practically be taken and sensibly used in another; and how institutional and political contexts then determine how evidence is then used in practice. This set of assumptions has fundamental importance for international development practice.

Firstly, we understand social change to be emergent and messy. Organised efforts to direct change confront the impossibility of any of us ever having a total understanding of all the sets of societal relationships and contested meanings that generate change and are in constant flux. New inter-relational processes are constantly being generated that in turn affect and change those already in existence. Complexity theory privileges a concern for process as much as goals and supports an approach that seeks to make a difference by working through relationships rather than focusing on narrowly defined pre-set projects and outcomes. It encourages being explicit about values and a concern for how an organisation’s intervention is judged by others, in particular by those that are meant to ultimately benefit, and the creation of effective feedback mechanisms – including, but not limited to, those produced by high quality research.

At their best, development practitioners often have to surf the unpredictable realities of national politics, spotting opportunities supporting interesting new initiatives, acting like entrepreneurs or searchers, rather than planners. They are keeping their eye on processes and looking to ride those waves that appear to be heading in the direction that matches their own agencies’ mission and values, and which can support local coalitions for change.  On the contrary, assuming that development practitioners are in control and that change is predictable – as expressed through some of the demands of evidence-based planning approaches – prevent them from responding effectively to feedback in an often unpredictable and dynamic policy environment, and can, if badly managed, chain them to a desk. Ben Ramalingam’s blog site – Aid on the Edge of Chaos – offers current insights on complexity thinking in development.

That it is relatively easier to eradicate rinderpest in cattle and build bridges than tackle police corruption or reduce violence against women is because the first are examples of what Dave Snowden describes as complicated problems and the latter are complex – an effect of there being so many collaborators involved in non-routine interventions with absence of consensus among them.  Such issues can’t be ‘solved’ like a Sudoku puzzle. In that respect, we were puzzled by Chris and Stefan’s two examples of what we would describe as complex issues. We found the first – the effect of political quotas for women in rural India – to be somewhat superficial and wondered why so little reference was made to the considerable number of studies from political sociology on the same topic that ask more probing questions and arguably provide more insightful understanding of what has been learnt in different contexts.  The World Bank study on whether top-down large scale interventions can stimulate bottom-up participation was on the other hand  puzzling for exposing myths that perhaps only World Bank staff had previously believed in, while ignoring the very considerable body of sociological and anthropological knowledge on this topic. It led us to wondering whether you need economists to find something out for it to be accepted as evidence.  Perhaps that explains some of ‘the evidence-barren areas in development’………

Which brings us to the second set of assumptions about how we know and therefore what is judged as evidence.  This is about more than pluralism and mixed methods, though we recognise that recent advances, in this case funded by DFID, are important.  Let’s start by insisting that a criterion for rigorous research is that it should be explicit about its assumptions or world-view. We suggest that a weakness in many studies is that they usually focus solely on the methodological and procedural and render invisible their ‘philosophical plumbing’. The evidence-based approaches that Stefan and Chris advocate are imposing a certain view of the world, just as our approaches do. Their claims to the contrary foreclose any possible discussion about the different intellectual traditions in interpreting reality.  Theory invites argument and debate.

An interesting paper by Greenhalg and Russell on evaluating health programmes notes how experimental approaches often ignore thetricky philosophical and political questions. Like the authors of that article, we take an approach that recognizes the partial (in both senses of the word) nature of our knowledge. How does this approach try to deal with unavoidable bias?  Through seeking to use dialogic, democratic methods in which multiple perspectives and understandings of what is at stake are explored, and the use of multiple and hybrid approaches.  The implications for practice are to be involved in mutual single and double-loop learning and adaptation as you go along. This does not preclude specific studies commissioned from ‘experts’, but it is not they alone who should define the problem nor should they assume that only their kind of knowledge has validity for collective efforts to try to secure greater equity and social justice.  Knowledge and power are bed-mates.  Our critique of ‘expertise’ – the laboratory references are an extreme example of the trend – is that expertise often uses its power to ignore other ways of knowing and doing, something Chris and Stefan would seem to agree with. Might it be that some of these ways might prove to be pretty good at tackling police corruption or reducing violence against women?

This is where reflexivity comes in.  Those of us working as practitioners, bureaucrats and scholar activists in international development cannot escape the contradiction that we are strategizing for social transformation from a position in a global institution – international development – that can and does sustain inequitable power relations, as much as it succeeds in changing them. Reflexive practice seeks to address these power inequities by recognizing that (a) many problems we seek to address are the products of human interaction – and some very important problems for people with less voice go ignored for that reason, and  (b) even if people are in agreement about there being a problem, they will often offer multiple diagnoses for its existence, and thus of course (c) multiple solutions, which need to be debated democratically with different kinds of evidence, based on alternative ways of knowing, and having the space to be heard.

We are heartened to note that Chris and Stefan believe “that all actions by external actors will interact with political forces and vested interests” and that “in many of the settings where development actors want to make a difference, power and political institutions are biased against the poor”. We would therefore assume that a reflexive donor would recognise that their power and agenda need examination as much as anyone else’s.

Chris and Stefan suggest ‘the commitment to evidence has opened up the space fundamentally to challenge conventional, technical approaches to aid.’ We would agree, but it would seem that the exception to this is when it comes to addressing the power of donors such as DFID, being honest about the domestic political pressures they are under, and assessing the possibility that their behaviour (including how evidence-based approaches are managerialised) may on occasions be undermining processes of development and social transformation. Is DFID drawing upon anthropologists or ethnographic researchers, as the Police in the UK have recently done, to understand how its policies on, for example, results or value for money change behaviour in the agency, and its relationships with others?

To imply that we are suggesting that ‘it is not worth trying to provide the best and most rigorous evidence to those who need to make difficult decisions’ is simply a wilful mis-stating of our position. On the contrary we are arguing there is more ‘evidence’ out there than some seem to admit because their world view precludes seeing this as such. Where we in particular see the need for more evidence is about how the evidence-based and results agenda plays out in practice. How it affects the behaviour of development agencies and their staff as well as their ability to support the promotion of the kinds of transformational change which are likely to make a significant difference to the lives of people living in poverty and injustice. It is odd that those that argue for more evidence seem rather reluctant to admit that this is needed!

This is a debate we are keen to pursue further in the upcoming Big Push Forward conference on the Politics of Evidence.

The evidence debate continues: Chris Whitty and Stefan Dercon respond from DFID

Yesterday Chris Roche and Rosalind Eyben set out their concerns over the results agenda. Today Chris Whitty (left), DFID’s Director of Research and Evidence and Chief Scientific Adviser and Stefan Dercon (right), its Chief Economist, respond.

It is common ground that “No-one really believes that it is feasible for external development assistance to consist purely of ‘technical’ interventions.” Neither would anyone argue that power, politics and ideology are not central to policy and indeed day-to-day decisions. Much of the rest of yesterday’s passionate blog by Rosalind Eyben and Chris Roche sets up a series of straw men, presenting a supposed case for evidence-based approaches that is far removed from reality and in places borders on the sinister, with its implication that this is some coming together of scientists in laboratories experimenting on Africans, 1930s colonialism, and money-pinching government truth-junkies. Whilst this may work as polemic, the logical and factual base of the blog is less strong.

Rosalind and Chris start with evidence-based medicine, so let’s start in the same place. One of us (CW) started training as the last senior doctors to oppose evidence-based medicine were nearing retirement. ‘My boy’ they would say, generally with a slightly patronising pat on the arm, ‘this evidence-based medicine fad won’t last. Every patient is different, every family situation is unique; how can you generalise from a mass of data to the complexity of the human situation.” Fortunately they lost that argument. As evidence-informed approaches supplanted expert opinion the likelihood of dying from a heart attack dropped by 40% over 10 years, and the research tools which achieved this (of which randomised trials are only one) are now being used to address the problems of health and poverty in Africa and Asia.

The consequences of moving from expert (ie opinion-based, seniority-based and anecdote-based) to evidence-based healthcare policy, far from being some sinister neocolonial experiment, have been spectacular. To quote a recent Economist headline, ‘Africa is currently experiencing some of the fastest falls in childhood mortality ever seen, anywhere’. It is a great example of the positive side to modern Africa the current excellent Oxfam publicity campaign (right) is all about. This success is based on many small bits of evidence, from many disciplines, leading to multiple incrementally better interventions. Critically, it also involves stopping doing things which the expert consensus agreed should work, but which when tested do not. It is no accident that one of the most evidence-based parts of development is also one where development efforts have had some of their greatest successes.

Proper evidence empowers the decision-maker to be able to make better choices. This is a good thing. In every discipline, in every country, where rigorous testing of the solutions of experts has started, many ways of doing things promoted by serious and intelligent people with years of experience have been shown not to work. International development is no different, except that the communities we seek to assist are more vulnerable, including to our bad choices.

Much of what we all do in international development has very limited evidence that it does any good  (in this it is no different from many other policy areas) – which is not the same as saying it is pointless. Rather we don’t know what is pointless. Some of our actions will work better than we think, much of it will work much less well than we hope, and some of it will be damaging the poorest without us realising it. In the evidence-light areas we just don’t know which are which.

We must have the humility to accept that we are all often wrong, however reflexive the practitioner, however deep their reading and experience and passion to do good. Evidence-based approaches are not about imposing a particular theory or view of the world. It is simply about taking any opportunity to test our own solutions in the best way available, using evidence honestly when it is available to inform (note the word) decisions, and when the facts change, changing our minds.

This honesty includes saying to decision-makers when evidence is methodologically weak, mixed or missing so they know they are on their own, unable to rely on (or make a claim on) the evidence. The worst possible solution, which we know Chris and Ros would also deplore, is using the social power of the ‘expert’ to imply we know the answer when we actually have no solid evidential basis for our opinion or prejudice.

A few false assumptions about evidence-based decision making

Some of those who express unease about evidence-based policy and practice seem to assume that it is always based on randomised trials and quantitative methodologies: not so. Methods from all disciplines, qualitative and quantitative, are needed, with the mix depending on the context. Randomised trials are one tool amongst very many, although a good one in the right setting. The argument that evidence-based approaches can “only apply in cases of individual treatment and not the wider community level” ignores over 30 years of methodology which has done exactly that, with very convincing results.

A sterile argument  between people who are on the one side believe that a  randomised trial can answer any question (they can’t), and people who do not appear to be aware of any  methodological advances since the 1970s except in their own narrow field is a depressingly familiar experience. We know this does not apply to Rosalind and Chris, but listening to people passionately critiquing methodologies they have not taken the trouble to understand does no good to anyone. This applies both to a randomista who seems to believe that all there is to social research is a few focus groups and in-depth interviews, and to people from a more qualitative social science background who would have trouble explaining the difference between cluster randomised and step-wedge design but assume both are irrelevant to social research anyway (both can be used to measure societal rather than individual effects).

It is tempting to take every point the authors make where we have concerns about their factual basis and logical framework but we will take just three.

“Evidence-based approaches are pre-occupied with avoiding bias and increasing the precision of estimates of effect”. On less bias – generally true. Please complete the sentence ‘More biased research is better because…’. On precision – no, incorrect, the range of situations where a more precise answer is a better answer is small.

One statement we would like to address head-on starts “Evidence-based approaches became linked to value for money concerns to deliver ‘results’…”. We agree- and this is a good thing. Doing a pointless thing, professionally delivered and passionately believed in, is always going to be poor value for money. Testing what works and what does not therefore is essential to value for money. More importantly, doing pointless things diverts very limited human and financial resources, in an ocean of need, away from those who could best use them- not what any of us are in international development to do.

Is it “technical approaches” on the one hand, and “power, political economy” analysis on the other?

Rosalind and Chris’ key criticism is that evidence-based approaches “deflect attention from the centrality of power [and] politics […] in shaping society”, and they offer “power analyses” as an apparent alternative to assessing rigorously what works. This creates a false dichotomy, as if a choice has to be made between a “technical, rational and scientific approach to development” and an approach that recognises politics and the role of power. It is easy rhetoric, but troubling and, if taken much further, even dangerous. Understanding power and politics and how to assist in social change also require careful and rigorous evidence, and again, results are not simply what experts would have expected a priori. Recent studies on the positive impacts of female leadership quotas in rural India are for many of us rather surprisingly good news, even if one can fairly worry about its applicability in other settings, while the struggle to find systematically a positive impact of decentralisation and community-driven development programmes is important to internalise in our actions for change, and highlights the importance of understanding contexts and politics. In these cases, it is not a matter of just RCTs, but of rigour, and of combining appropriate methods, including more qualitative and political economy analysis.

Strong analysis of politics and power without offering much in terms of what can be acted upon is similarly unhelpful. They criticise an evidence-focused agenda by stating that “to act ‘technically’ in a politically complex context can make external actors pawns of more powerful vested interests and therefore by default makes them, albeit unintentionally, political actors.” But all actions by external actors will interact with political forces and vested interests. In many of the settings where development actors want to make a difference, power and political institutions are biased against the poor. Being able to act on strong evidence of what works in constrained political settings is crucial.

A reductionist and misinformed view of evidence as purely ‘technical’ or as being only about “what works” is unhelpful – it is also about generating evidence and understanding (and learning) on why interventions and approaches may work, including understanding the social, political, and economic factors that may enable or constrain success of different approaches. Far from the search for evidence pushing us in a ‘technical’, apolitical direction it has reinforced the importance of understanding and trying to tackle the underlying causes of poverty and conflict. There is agreement on the importance of politics and institutions in shaping growth, security and human development. However, the ability of external actors to influence institutions is much less clear and this is where DFID research is now focussed. Ros and Chris have misread the context – the commitment to evidence has opened up the space fundamentally to challenge conventional, technical approaches to aid.

Why it matters for international development

There are large areas of international development where decision-makers are largely flying blind – forced to make decisions purely on gut feeling and ideology not because they wish to because they have no option. Try making difficult decisions in education policy compared to health policy and the difference in usable evidence is dramatic – yet both are complex, social and context-dependent parts of human life. It is always puzzling when people say airily ‘health is easy’- it is not, and is an intensely political and social subject requiring interventions at societal level.

Today we can eradicate rinderpest in cattle and build bridges over the Zambezi based on rock-solid evidence from many disciplines, but do not have anywhere near as clear an idea how to reduce violence against women or tackle police corruption. All are great challenges with social dimensions but in two of them people have set about finding and testing solutions in a systematic way over many decades.

Having robustly tested evidence-based solutions certainly does not eliminate politics: the decision whether to build a bridge, what sort and where, is an intensely political choice – but at least those making the choice now have a fair assumption it will stand up- based on hundreds of years of incremental evidence. The evidence-barren areas in development are a collective, and in our view shameful, failure by us all in the academic and practitioner community. We should never excuse them with the feeble assertion that it is too difficult or complicated. Development is difficult and complicated – but the bases for making decisions will gradually improve if we are serious about improving it.

In conclusion, we collectively have the capacity to be able to give to our successors in every continent a far better basis on which to make their decisions for their lives than our generation have. To imply it is not worth trying to provide the best and most rigorous evidence to those who need to make difficult decisions because they will have other influences as well is like saying to someone going for a walk in dangerous mountains that they do not need a map because there will be many other factors that will determine where they go. That is true – but they are still less likely to fall off the cliff if they have one.

Where evidence is clear-cut we should be making that plain to decision makers – and where it is not we should say that as well, be honest about what is there and try to get better evidence for the future. That, in essence, is what evidence-based decision making is about – and all it is about. If the academic community is serious about trying to assist those working in the field (including in Oxfam), and above all empowering the most vulnerable communities to make the most informed possible decisions available for their own development, we should be putting our greatest efforts into supporting decision-makers to use the best evidence, and finding better methodologies in areas where we currently have very weak evidence. There are many, and this should be tackled as a matter of urgency.

Tomorrow, Chris Roche and Rosalind Eyben respond

The political implications of evidence-based approaches (aka start of this week’s wonkwar on the results agenda)

The debate on evidence and results continues to rage. Rosalind Eyben (left) and Chris Roche (right, dressed for battle), two of the organisers of April’s Big Push Forward conference on the Politics of  Evidence, kick off a discussion. Tomorrow Chris Whitty, DFID’s Director of Research and Evidence and Chief Scientific Adviser, and Stefan Dercon, its Chief Economist, respond

Distinct from its more general usage of what is observed or experienced, ‘evidence’ has acquired a particular meaning relating to proof about ‘what works’, particularly through robust evidence from rigorous experimental trials. But no-one really believes that it is feasible for external development assistance to consist purely of ‘technical’ interventions. Most development workers do not see themselves as scientists in a laboratory, but more as reflective practitioners seeking to learn how to support locally generated transformative processes for greater equity and social justice. Where have these experimental approaches come from and what is at stake?

The origins and critiques of evidence-based approaches

Evidence-based approaches are pre-occupied with avoiding bias and increasing the precision of estimates of effect. In the UK they spread beyond clinical practice when the government elected in 1997 was keen to demonstrate that its decisions would not be driven by political ideology but rather by objective evidence. Evidence-based approaches became linked to value for money concerns to deliver ‘results’ as efficiently and effectively as possible, by a government recently described as ‘truth junkies’.

Yet, even within medicine, the leap from evidence-based clinical practice into evidence-based policy was challenged. A British Medical Journal article by Nick Black in 2001 drew on an extensive body of contemporary literature on policy processes to argue that policy was shaped by institutional arrangements, values and beliefs and a variety of different sources of information.

Opponents of evidence-based education critiqued its positivist assumptions; its linear cause-effect thinking; and the poor understanding of the tensions between scientific and democratic control of educational practice. An OECD report in 2007 on the reasons for the uptake of evidence-based education in Sweden, the UK, Canada, the USA and Australia, noted the increasing pressure for greater accountability of expenditure and effectiveness, an explosion in the search for measurable outcomes, and demands that impacts and effectiveness be given a monetary value.

The same report noted that evidence-based approaches were largely absent in OECD countries ‘less used to empirical and quantitative methodologies in the social sciences.’ The de-politicization of policy making is one of the reasons given by development researchers for its neglect in France.

In UK social policy, evidence-based approaches with their ‘gold standard’ of experimental or quasi-experimental design, have been criticised as being inapplicable to complex issues. In What Works, Tony Harrison argued that evidence-based approaches can only apply in cases of individual treatment and not at the wider community level where multiple perspectives come into play and no agreement exists about the nature of the problem.This of course is the case with most development programmes, and in particular those that seek transformational change.

Evidence based approaches in development: an anti-politics firewall?

Arguably evidence-based approaches build an anti-politics firewall. Development assistance becomes a ‘technical’ best practice intervention based on rigorous objective evidence, delivering best value for money to domestic taxpayers and recipient country citizens mostly without interfering in that country’s politics. They are the latest manifestation of  a certain long-standing approach to development that as Timothy Mitchell wrote in Power of Development, speaks to the sector’s ‘need to overlook its internal involvement in the places and problems it analyses and present itself instead as an external intelligence that stands outside the objects it describes’.

In the 1930s Africa was seen as ‘a living laboratory’ to achieve improvements in the welfare of the populations. Evidence-based approaches are reviving the development-as-laboratory idea. In 2012 the World Bank established a Gender Innovation Lab to design ‘innovative interventions to address gender inequality and to develop rigorous research projects in order to produce evidence on what works and what does not’.  Jeffrey Sachs’ Millennium Villages have been framed as  ‘laboratories to lift people out of poverty’. The most well-known is the J-Pal Poverty Action Lab whose mission is to reduce poverty ‘by ensuring that policy is based on scientific evidence’.

In the absence of political debate, this approach can exacerbate the tendency to see people as subjects requiring treatment, rather than as citizens with political voice. Power silences any challenges to the technical framing of ‘the problem’, foreclosing discussion of the structural causes and consequences of inequity and how these should be tackled. To act ‘technically’ in a politically complex context can make external actors pawns of more powerful vested interests and therefore by default makes them, albeit unintentionally, political actors.

High stakes

Evidence-based technical approaches can therefore deflect attention from the centrality of power, politics and ideology in shaping society. We agree with the view of the Developmental Leadership Program that recent research suggests that  the development sector should be ‘at the frontier of a narrative shift between a technical, rational, and scientific approach to development, and a recognition that politics matters; that poverty reduction is not a technical problem but requires significant social change, and that this social change is, and must be, both political and locally led.’

However this has some significant implications for external actors. We need to be self-aware to avoid disempowering others.  This requires undertaking power analyses with ourselves factored in – as organisations and individuals who can make a positive or negative contribution, often inadvertently.  It means engaging with a wider and more diverse group of policy actors in the state, civil society and the private sector; whenever possible, supporting debate, locally-driven problem solving, and independent research. It means avoiding overly linear project-based aid modalities that demand omniscience before they have even begun.

As Michael Sandel has recently argued in his book about the moral limits to markets, how we put values (and prices) on things can change their meaning, as well as change the relationship between economic actors.  More information, (dare we say ‘evidence’), is needed to draw some firmer conclusions about the consequences of evidence-based approaches to designing projects and assessing results. This is why the Big Push Forward is currently seeking to crowd-source more information from development practitioners about how they actually experience the ‘results’ agenda, and why we believe this issue needs more debate.

And make sure you come back tomorrow for DFID’s counterblast

What does Tolstoy’s War and Peace teach us about Causation, Complexity and Theories of Change?

Just finished reading Tolstoy’s War and Peace, an amazing work, which quite possibly justifies the blurb’s ‘greatest novel in any language’

Leo Tolstoy, development guru

claim (who on earth decides these things and how?). I read it 30 years ago, but to be honest, I’m not sure I understood much of it then.

Tolstoy manages to combine the enthralling human saga of Russia’s experience of invasion by France under Napoleon, and the French’s subsequent retreat, with a profound meditation on the nature of history and change. I started it as holiday reading, supposedly time out from the day-job, but I couldn’t help wondering what Tolstoy would say about some current development debates. At times it feels as if in his frustration with the causal explanations of the day, he is banging on the doors of complexity theory. Some choice quotes, mainly from the concluding meditation on the nature of history at the end of Book Two:

Tolstoy on causation and attribution (are you listening, MEListas?)

‘It is beyond the power of the human intellect to encompass all the causes of any phenomenon. But the impulse to search into causes is inherent in man’s very nature. And so the human intellect, without investigating the multiplicity and complexity of circumstances conditioning an event, any one of which taken separately may seem to be the reason for it, snatches at the first most comprehensible approximation to a cause and says ‘There is the cause’……

There is, and can be, no cause of an historical event save the one cause of all causes [i.e. God}. But there are laws governing events: some we are ignorant of, others we are groping our way to. The discovery of these laws becomes possible only when we finally give up looking for causes.’

Tolstoy on Command and Control and the fallacy of hindsight (cf Ros Eyben’s work on aid)

‘History shows that the expression of the will of historical personages in the majority of cases does not produce any effect – that is, their commands are often not executed and sometimes the very opposite of what they order is done…. Every command executed is always one of an immense number unexecuted. All the impossible commands are inconstant with the course of events and do not get carried out. Only the possible ones link up into a consecutive series of commands corresponding to a series of events, and are carried out.

Our erroneous idea that the command which precedes the event causes the event is due to the fact that when the event has taken place and out of thousands of commands, those few which were consistent with that event have been executed we forget about the others that

were not executed because they could not be.’

Tolstoy channels Amartya Sen on Freedom and Wellbeing

‘All man’s aspirations, all the interest that life holds for him, are so many aspirations and strivings after greater freedom. Wealth and poverty, fame and obscurity, power and subjection, strength and weakness, health and disease, culture and ignorance, work and leisure, repletion and hunger, virtue and vice are only greater or lesser degrees of freedom.’

And finally, I’m definitely with Tolstoy on the meaninglessness of free (read ‘political’) will:

‘In history, what is known to us we call the laws of necessity; what is unknown we call freewill. Freewill is for history only an expression connoting what we do not know about the laws of human life.’

It’s 1500 wonderful pages – get stuck in. As for me, the boxset of the 1972 TV adaptation with Anthony Hopkins has just arrived (see pic). Good times.

Can we demonstrate effectiveness without bankrupting our NGO and/or becoming a randomista?

Back in March there was a fascinating exchange on this blog between Ros Eyben and Claire Melamed on the role of measurement in development work (my commentary on that debate here). Now one of Oxfam’s brightest bean counters (aka ‘Programme Effectiveness Adviser’), Karl Hughes, explains where Oxfam has got to on this:

Eric Roetman, in a recent 3ie working paper, A can of worms? Implications of rigorous impact evaluations for development agencies, tells a provocative tale of the experiences of International Child Support (ICS) in Kenya carrying out randomised control trials (RCTs) in partnership with several  world-renowned quantitative impact evaluation specialists.  ICS saw itself evolve into a “development lab”, where the bulk of its staff became devoted to supporting the organisation’s research, as opposed to development, operations.  Given ICS’s desire to revert back to its roots, it eventually opted to get out of the RCT business.

ICS’ story relates directly to issues further explored in another recent 3ie working paper I recently co-authored with Claire Hutchings, another one of Oxfam GB’s global MEL advisers, entitled Can we obtain the required rigour without randomisation?  Oxfam GB’s non-experimental Global Performance Framework.  The central issue is this: We in the international NGO community are all too aware of our need to up our game in both understanding and demonstrating the impact – or lack thereof – of the various things we do.  But what really baffles us is just how to do so without going down the “development lab” route.  (This is not to imply that “development labs” are bad; in fact, the more their findings inform our programming, the better.)

The bottom line, as outlined in our paper, is that evaluation is research, and, like all credible research, it takes time, resources, and expertise to do well.  This is equally true no matter what our epistemological perspective – positivist, realist, constructionist, etc.  This is perhaps why, rather than using  those offered by mainstream academia, we as a sector are so quick to experiment with seemingly more doable alternatives such as Most Significant Change, social return on investment (SRI), outcome mapping, and participatory M&E.  They’re all very well, but those of us who feel a need to go further find ourselves at a loss.

One popular way of attempting to demonstrate effectiveness, being pursued by several international NGOs, which we comprehensively bash in the paper, is dubbed “global outcome indicator tracking.”  Here, the organisation in question gets all its programmes/partners  to collect common data on particular outcome measures, e.g. household income.  All these data are then aggregated (only the gods know how) to track the welfare of global cohorts of programme “beneficiaries” over time.  If there is positive change in relation to the indicator from time 1 to time 2, the organisation can boast about how much impact it is generating.  Aggregation complexities aside, the underlying foundations of this approach are inherently precarious.  In general, outcome level change is influenced by numerous extraneous factors, e.g. rainfall patterns in rain-fed agricultural communities.  Consequently, even if we are able to capture reliable data on a decent outcome indicator, its status will go up and down and all around not matter what our interventions are and/or how well they are implemented.  Any consideration of attribution is entirely absent.

But what of the fact that donors have been encouraging us to pursue outcome indicator tracking for decades now through instruments such as the logframe, as part of ‘good practice’?  In a paper entitled, The Road to Nowhere, Howard White argues that the United States Agency for International Development (USAID) identified the futility of the outcome indicator tracking strategy some years ago and, consequently, abandoned it.  I worked on a USAID funded orphan and vulnerable children (OVC) programme from 2005 to 2010, and yes we were only required to report on outputs, so perhaps this was the consequence of this realisation.  (Incidentally, USAID also came to the realisation that there was no evidence-base established on what works and what does not in OVC programming after all the billions that it spent and seems to regret not having supported the rigorous evaluation of key OVC care and support interventions.)  To what extent have the other donor agencies recognised the fallibility of outcome indicator tracking?  Sadly, there is plenty of evidence to suggest that many are still operating in this outdated paradigm.

So where does this leave us as NGOs?  While Oxfam GB has not come up with a panacea, it is attempting to pursue a strategy that is reasonably credible.  Each year, we are randomly selecting and then evaluating, using relatively rigorous methods by NGO standards, 40-ish mature interventions in various thematic areas.  The causal inference strategy differs depending on the nature of the intervention.  For community-based interventions, for instance, where we are targeting many people (aka large n interventions), we are attempting to mimic what RCTs do by statistically controlling for measured differences between intervention and comparison populations.  Evaluating our policy influencing and “citizen voice” work (aka small n interventions), on the other hand, requires a different approach.  Here, a qualitative research method known as process-tracing is being used to explore the extent to which there is evidence that can link the intervention in question to any observed outcome-level change.

It is not that the above approaches are free of  limitations.  In the case of large n interventions, for instance, given that programme participants have not been randomly assigned to intervention groups, coupled with the conspicuous absence of proper baseline data, we cannot absolutely guarantee that any observed outcome differences are the result of the workings of the intervention in question.  The  process tracing approach is also retrospective in nature, when ideally the research should take place throughout the life of the advocacy or popular mobilisation initiative.  But, hey, what we are doing is not too shabby, especially considering that we are no “development lab.”  Moreover, every evaluation design –  even the golden RCT – has  inherent limitations.  Nonetheless, if anyone has any suggestions on how NGOs in general and Oxfam in particular can do a better job at both understanding and demonstrating impact, I’d love to hear them.

What does a theory of change look like?

I’ve been working on ‘how change happens’ for a few years now, as regulars to this blog will know, but in the last few months, ‘theories of change’ has gone viral as a new development fuzzword. In meetings and documents, people earnestly enquire ‘what’s your theory of change?’ and you’re in trouble if you don’t have an answer. (Quite a good answer is ‘could you just explain what you mean by theory of change?’ – people often have no idea).

So it’s time to ride the wave and speed up the ‘theories of change’ work programme, and last week I spent some time at IDS and with Oxfam big cheeses thinking through what a joint IDS-Oxfam work programme might look like. Here’s what I’m currently thinking, with a plea to others to comment, send sources and otherwise give me a hand. Hat tips to Thalia Kidder and Jo Rowlands for their suggestions.

Firstly it’s ‘theories’ not ‘theory’. When people talk about a single ‘theory of change’ (ToC from now on), alarm bells should ring – in the worst case it’s just a new jargon for old-school linear change, impact chains, logframes etc. Instead of a deluded search for a single grand theory of everything, we need to learn to recognize and manage a range of theories, throw them at a problem, and see which ones are helpful (see my recent experience of doing this in Tanzania). Yes folks, we’re talking practical post modernism….. Surfacing our deeper, buried assumptions about the motors of change can also help us understand why we keep disagreeing with each other, a crucial skill in coalition-building.

Being able to acknowledge your own ‘preferred ToC’ and yet have the ability to stand outside it and understand those of others is really hard to do – an emotional and intellectual stretch – but it’s an invaluable skill. I think a lot of the practical impact of any ToC workplan is going to lie in helping build such capacities.

To do that, you need some rules of thumb – NGO types are mainly doers and activists, impatient to get on and change the world. They need practical tools to help them apply ToCs in their work. So I’ve been building on some work on ‘archetypes of change’ by Chris Roche and ‘meta-theories’ from Ros Eyben, to come up with this rough typology.

There are three categories, with some overlap between the categories, but broadly they are:

1. ‘Systemic meta-theories’ (apologies – please suggest a less pretentious alternative!), describing the underlying way you see the world and its motors of change. They may lead to particular change strategies, or simply underpin the overall analysis.

2. Archetypes – more specific snapshots of how change happens in a given place and moment. I’ve provisionally grouped them into four clusters: active citizenship, elite-driven change, cross-class coalitions and what I’ve called ‘dynamics’ where the focus is on the rhythm of the change itself, rather than specific drivers. Not sure if the dynamics actually belong in a separate column.

3. Change strategies, adopted by would-be ‘change agents’ to bring about good change/ prevent bad change

Category 1 is free standing; while categories 2 & 3 go together, i.e. the change strategies follow from the archetype of change in the same row. None of these lists are exhaustive, and some overlap with each other – I hope to reduce the level of messiness as the work proceeds, but here it is, warts and all.

Systemic meta-theories

• Rational Choice: change is unintended outcome of individual choice
• Environment/techno determinism
• Long term shifts in deep underlying  norms, values and beliefs
• Purposive individual/ collective action
• Marxist/Structuralist: changes in relations of production and economic power structures key
• Evolution (variation/selection/ amplification)
• Shocks and wars drive change by transforming social, political and economic relations

And here are the more specific archetypes and their associated change strategy:

Clear as mud? I’d welcome all thoughts, especially on clarifying and improving the typology. Next steps are to start identifying case studies, plan some desk reviews, design some training modules and raise some research funding (do get in touch if you want to fund it!) I’ll keep you posted as plans develop.

If not results, then what? The risks of not having a results agenda

The ODI’s Claire Melamed replies to yesterday’s guest post from Ros Eyben:

“Ros Eyben suggests that instead of a results agenda, we should rely on good relationships to deliver good aid.  And indeed, if all relationships were good, and all the people involved in making decisions about aid were thoroughly well-informed, open to new ideas, flexible in their approach, lacking in ego, adept at dealing with cultural and religious differences and aligned with the needs and priorities of poor people, that might just work. 

But just supposing, for a moment, that aid bureaucracies aren’t all like that, let’s think about the risks of not having a results agenda.

If you don’t define in advance what the objectives of an aid programme are, you leave it up to the managers who make the decisions and the politicians who guide them to impose their own values and prejudices onto the aid programme.  Of course if they could all be trusted to make the right decision, there’s no problem.  But evidence suggests that might be over-optimistic.  Exhibit A: attempts to fund the building of a dam in Pergau that had nothing to do with poverty and everything to do with arms sales.  Exhibit B: the ideological pursuit of structural adjustment programmes in the face of substantial evidence of the harm they were causing.

A focus on results can help to rebalance inequalities of power. When the Labour Government created the National Institute for Health and Clinical Excellence (NICE) in the UK in 1999, to ensure that evidence about value for money and effectiveness was used in deciding what drugs to prescribe in the National Health Service, pharmaceutical companies were among the most hostile to the idea. Naturally, from their point of view, they preferred their own marketing ‘evidence’ to help doctors make prescribing decisions. 

Actually, of course, leaving all decisions about prescribing up to doctors – informed by partial evidence – led to inequalities (the dreaded ‘postcode lottery’), and to millions of pounds wasted on ineffective treatments. NICE’s role in bringing together evidence from clinical trials (which included patients’ own assessments and valuations of changes in their health) with the costs of treatment, has started to improve value for money in the NHS and also to take more account of health benefits (or lack of them) from patients’ own point of view. 

A results agenda, as long as the right results are being pursued, can help to rebalance inequalities of power and make the actions and decisions of the powerful more transparent. It helps people to know what the objectives of decision makers are – and so to argue that they should be different, if that’s the case; and also to hold people to account for their success or failure to meet those objectives. Without measurement, there can be no accountability. 

The real question is what results we are looking for, and how to measure them. Of course if donors want to do the wrong things, and measure the wrong things, they won’t get good results. But pointing to examples of the wrong way of using results and saying, ‘so let’s not measure results’, seems to me as big a folly as the, sadly all too popular, pastime of pointing to the latest example of unsuccessful aid and saying ‘so let’s give up on aid altogether’. 

So if the numbers of polio vaccines isn’t the right result to ask for, then let’s look for something that is a better measure of the strength of health systems. And instead of counting the length of roads, let’s measure the strength of solidarity in communities – that’s doable. 

The results agenda is actually a huge opportunity for people who care about relationships, trust, empowerment, rights and complexity to find ways of getting these things firmly integrated into how we measure development.  Then they’d be part of the mainstream.

These things can be counted. There are approaches developed in the UK’s National Health Service, for example, which allow patients to say how much they value different health outcomes, like the absence of pain or the ability to move about normally. Research shows that the values that ordinary people attach to different outcomes are different to those of even the most well-meaning professionals – which should be a warning to us all not to make assumptions about what people want. This information is turned into numbers and used to allocate funding and to measure results.  Imagine if we actually knew what poor people wanted and if they were getting it?  Everyone who works in development should surely admit that we don’t know as much as we should about if we are actually delivering ‘value’ as the recipients of our efforts would define it. 

We should be welcoming the focus on the results, because a world where we don’t know the results of our actions is not one that any of us would want to live in. This agenda should be used too, to  encourage a focus on what results poor people themselves (or, more likely, poor women, poor men, poor people in cities, in rural areas and so on, who would all have different priorities) most want to see, and how they’d define ‘value’ or ‘effectiveness’. 

Information is power. I say, don’t fear it. Use it. ”

Claire Melamed is the Head of the Growth and Equity Programme at the Overseas Development Institute

Update from Duncan: In a desperate attempt to stem the tide of consensus and mutual respect sweeping over the comments, I’ve put up a poll to the right of this post that allows only a yes/no answer to the question of whether the current focus on Value for Money is a Good Thing. ‘Sometimes’ ‘Maybe’ ‘It depends’ type answers all forbidden!