So i think what we need to be discussing is how we balance our results agenda and move away from focusing on one set of results (outputs- the numbers game) at the expense of the others (outcome and impacts – hard to measure ^_^).

I think we have always had a “results” agenda in development and all three levels have their role – for politicians, numbers may be all they want to work with but that does not mean that development managers then neglect the other results (outcomes and impacts) that make our work enjoyable… I always say “outputs just measure how busy one is”, where as “outcome and impacts demonstrate how effective in delivering and sustaining change one is”.

Lets not only welcome the focus on results, lets support a more balanced approach to results with the knowledge that different levels of results will be useful in different ways to different people. As development managers we need results at all three levels and our M&E systems should allow us to have produce and communicate at these levels so as to meet the demands of our diverse stakeholders.

Yes, it is absolutely necessary to show results (although I feel more comfortable with terms such as outcome and impact), otherwise we are at risk of being accused of what we accuse governments of on a daily basis. But my fear is that the pattern the current VFM is taking is to have results/change determined through the lenses of donors. The recent review BAR and MAR processes and outcomes appear to support that

I worry that if we reduce the debate on results to just value-for-money and counting, and thus move aid away from results, we’ll lose this key focus. In my mind, the debate should be less about whether we have a results-based agenda (I’m not quite sure how we can justify going back to the old input-based-planning model) and instead be more about how these results can be defined, how they can be measured and what models we use. Rigid logframes and quantitative indicators clearly have huge and acknowledged limitations, but the fact that they are widely used to this day speaks volumes about the lack of an alternative.

As Serverino and Ray point out in the excellent ‘End of ODA’, “in the absence of credible indicators of aid impact, development agencies are assessed according to an extremely weak proxy: their capacity to generate visible projects and disburse high volumes of funding quickly”.

So results are now seen as something akin to number of children in school and relative cost to the taxpayer in donor country X. But this wasn’t what development results were meant to look like, and it seems the whole idea of results has been hijacked, simplified and re-packaged in the guise of value-for-money.

To me, the debate should be not whether results but whither results.
Can outcome mapping be useful? What do the results we want to contribute towards look like? Can we find a way of setting out a path towards these results (and agreeing how to measure them) which takes into account the in-country reality? How does this take into account broader structural considerations and the interplay of aid, trade, taxation, finance, industry? What role can impact assessments play?

I say this carefully, as I have not quite taken off my DFID hat!Donors pretend to have more control than they do, because that is what they think their political/taxpayer masters want to hear. Maybe it is or maybe we should give taxpayers more credit for their ability to acknowledge that the world is messy?

The reality of course is that in many spheres of development, donors have very little control over how change happens. The challenge for DFID and others is to navigate this tension.

For what it’s worth, I wobbled off the fence to vote yes in the poll

I’m with Claire on this one.

The average size of Oxfam’s country programmes is about a million pounds a year or so (I may be out of touch now, with the consolidation across the different Oxfams that is currently taking place, but indulge me). These are then divided among a number of programmes and within them, projects. Small amounts of money in the grand scheme of things.

Working on evaluation for Oxfam, the biggest challenge was figuring out exactly what was being done in these projects, as they were all described in terms of the impact they were going to have (e.g. “Women’s empowerment”; “Improving the resilience of livelihoods”) rather than the necessarily much more modest means they had at their disposal to actually achieve these (e.g giving money and training to three local NGOs to represent themselves better in national policy circles).

I think that by focusing on results we helped to bring this aspiration gap to light. I got good feedback from staff for the training on logic models that we organised, where everyone was encouraged to draw diagrams describing the change they wished to see, and to think about what is necessary and sufficient to achieve it. This was based on expected cause-effect relationships. IF we do this AND the government does this THEN we think this OR this will happen. Debates about killer assumptions ensued. It was, whisper it, fun.

The things that were then actually measured (as determined by the staff) were just the points of information that gave an insight into whether our diagrammatic story about the future was playing out in reality. The main benefit of the whole results push was this debate about strategy and the need for realism and clarity. Utopianism is for the campaigns teams – achieving development is difficult but worth the effort. Any sensible donor will recognise this.

I believe there is all reason to pursue a focused, accountable, results-based agenda for all the justifications in Claire Melamed’s post.

But for all the reasons laid out in Rosalyn Eyben’s, if that equates to nothing more than that which can be counted in binary numbers, little will those results accomplish.

My own perception of why it has proven so difficult to combine the two assertions -and in practice separate accountability from countability- is that our models of change are flawed. As long as we remain stranded on shoulders of social sciences that brought us far but never beyond linear cause-effect thinking, I don’t see us really getting our heads around it.

More research is advised

/Søren