Comments on: When we (rigorously) measure effectiveness, what do we find? Initial results from an Oxfam experiment. http://www.oxfamblogs.org/fp2p/?p=12012 duncan green poverty to power oxfam development Wed, 19 Jun 2013 10:24:57 -0500 http://wordpress.org/?v=2.8 hourly 1 By: Daniel http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-369369 Daniel Thu, 31 Jan 2013 11:06:24 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-369369 How can you say that 40 x £10,000 was good value for money without knowing what action the country teams actually took....that deifnes the value of the investment - ie, the objective of all this effort? How can you say that 40 x £10,000 was good value for money without knowing what action the country teams actually took….that deifnes the value of the investment – ie, the objective of all this effort?

]]> By: Isabelle http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-275574 Isabelle Tue, 06 Nov 2012 10:16:29 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-275574 Congratulations, this is a truly huge piece of work. At World Vision International we have also been experimenting with different approaches to tackle the effectiveness question and the evaluation challenge. We are taking a different approach – a three pronged one - so I was really interested to read about Oxfam’s approach to this Effectiveness Review, and your successes and challenges. Firstly we are piloting 'Annual Summary Reports' with country offices, based on existing M&E data (we do collect baseline data). These are intended to be simple reports with strategic relevance, focused on internal learning for improving decision making at country office level for programme effectiveness. Secondly, at the global level, to produce a meta-review of these reports, but quality issues are also a challenge. The first pilot report was released internally in September. I hope that by next year we will also be releasing externally as Oxfam has. Thirdly, we are working with an academic institution on an impact study with comparison groups in three countries, focused on health and nutrition. We are hoping this will tell us more about 'impact', but again, results will take time. I am particularly impressed with a few less obvious points in your work: overcoming internal controversy and challenges (no small feat!); posting the management responses online and the overall 2-page summary. Simplicity is the ultimate sophistication. Again congratulations on this mile stone. It has given us more food for thought. I look forward to next year's report! Congratulations, this is a truly huge piece of work. At World Vision International we have also been experimenting with different approaches to tackle the effectiveness question and the evaluation challenge. We are taking a different approach – a three pronged one – so I was really interested to read about Oxfam’s approach to this Effectiveness Review, and your successes and challenges.

Firstly we are piloting ‘Annual Summary Reports’ with country offices, based on existing M&E data (we do collect baseline data). These are intended to be simple reports with strategic relevance, focused on internal learning for improving decision making at country office level for programme effectiveness.

Secondly, at the global level, to produce a meta-review of these reports, but quality issues are also a challenge. The first pilot report was released internally in September. I hope that by next year we will also be releasing externally as Oxfam has.

Thirdly, we are working with an academic institution on an impact study with comparison groups in three countries, focused on health and nutrition. We are hoping this will tell us more about ‘impact’, but again, results will take time.

I am particularly impressed with a few less obvious points in your work: overcoming internal controversy and challenges (no small feat!); posting the management responses online and the overall 2-page summary. Simplicity is the ultimate sophistication. Again congratulations on this mile stone. It has given us more food for thought. I look forward to next year’s report!

]]> By: Michaela Raab http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-259570 Michaela Raab Thu, 18 Oct 2012 08:49:26 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-259570 Hats off indeed! Hats off also to Alex Jacobs, whose post raises the questions that have gone through my mind. I have an extra question: Why does Oxfam select the programmes to be evaluated by random sampling? One could argue that it is the most "neutral" or easiest option - you don't need to think about the criteria you would use to select interventions. But if the key purpose is to generate learning within the organisation and within the wider sector, wouldn't it be more cost-effective to select the projects for evaluation in a more targeted manner? For instance, if you wish to find out about the effectiveness of a specific theory of change in a specific sector, you could organise a set of case studies in a range of countries. A more purposive approach would also enable you to limit RCTs to those kinds of projects where they make sense. That is not just a matter of numbers - many development interventions are just too complex for RCTs to yield meaningful insights. I have posted a couple of interesting presentations on that issue on my blog: http://www.developblog.org/2012/10/evidence-of-what-worked-at-some-point.html and http://www.developblog.org/2012/05/participatory-statistics-and-more.html Meanwhile, many thanks to Oxfam for advancing the debate and research on evaluation in development by making the full reports publicly accessible! Hats off indeed! Hats off also to Alex Jacobs, whose post raises the questions that have gone through my mind. I have an extra question: Why does Oxfam select the programmes to be evaluated by random sampling? One could argue that it is the most “neutral” or easiest option – you don’t need to think about the criteria you would use to select interventions. But if the key purpose is to generate learning within the organisation and within the wider sector, wouldn’t it be more cost-effective to select the projects for evaluation in a more targeted manner? For instance, if you wish to find out about the effectiveness of a specific theory of change in a specific sector, you could organise a set of case studies in a range of countries. A more purposive approach would also enable you to limit RCTs to those kinds of projects where they make sense. That is not just a matter of numbers – many development interventions are just too complex for RCTs to yield meaningful insights. I have posted a couple of interesting presentations on that issue on my blog: http://www.developblog.org/2012/10/evidence-of-what-worked-at-some-point.html and http://www.developblog.org/2012/05/participatory-statistics-and-more.html
Meanwhile, many thanks to Oxfam for advancing the debate and research on evaluation in development by making the full reports publicly accessible!

]]> By: Alex Jacobs http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-258922 Alex Jacobs Wed, 17 Oct 2012 11:28:30 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-258922 Great to see Oxfam trialling new approaches and doing it so transparently. As other NGOs, we've a lot to learn from your method and findings. Thank you for being so open. I've just blogged about the reviews: http://ngoperformance.org/2012/10/17/hats-off-to-oxfam-but-are-they-asking-the-right-question/ I strongly support your experiment and transparency. The blog mentions six main reactions and a couple of questions: (a) which decisions & actions will be most influenced by the reviews, and (b) how do the reviews help field staff do their jobs better? I'm looking forward to continuing to work on these issues together - they are such a high priority for the sector as a whole. Great to see Oxfam trialling new approaches and doing it so transparently. As other NGOs, we’ve a lot to learn from your method and findings. Thank you for being so open.

I’ve just blogged about the reviews: http://ngoperformance.org/2012/10/17/hats-off-to-oxfam-but-are-they-asking-the-right-question/

I strongly support your experiment and transparency. The blog mentions six main reactions and a couple of questions: (a) which decisions & actions will be most influenced by the reviews, and (b) how do the reviews help field staff do their jobs better?

I’m looking forward to continuing to work on these issues together – they are such a high priority for the sector as a whole.

]]> By: Lee http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-257406 Lee Mon, 15 Oct 2012 10:35:44 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-257406 Interesting stuff. In line with B. Osborn Daponte's comments on using a stratified sample rather than a totally random sample - you might want to consider some kind of filter. DFID issued guidance recently which lays out some criteria for which projects should be evaluated: “It is not necessary, or desirable, to evaluate every programme, but the decision needs to be clearly justified. Many DFID country offices have developed evaluation strategies identifying criteria for deciding if an evaluation is needed. Common criteria include: - a weak evidence base, - a contentious intervention, - stakeholder interest, - an innovative or pilot programme, and - a high financial value." http://www.dfid.gov.uk/R4D/PDF/Publications/DFID_guidance_for_evaluating_social_transfer_programmes_June_2012.pdf Interesting stuff. In line with B. Osborn Daponte’s comments on using a stratified sample rather than a totally random sample – you might want to consider some kind of filter.

DFID issued guidance recently which lays out some criteria for which projects should be evaluated:

“It is not necessary, or desirable, to evaluate every programme, but the decision needs to be clearly justified. Many DFID country offices have developed evaluation strategies identifying criteria for deciding if an evaluation is needed. Common criteria include:
- a weak evidence base,
- a contentious intervention,
- stakeholder interest,
- an innovative or pilot programme, and
- a high financial value.”

http://www.dfid.gov.uk/R4D/PDF/Publications/DFID_guidance_for_evaluating_social_transfer_programmes_June_2012.pdf

]]> By: Kate http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-256885 Kate Sat, 13 Oct 2012 20:40:17 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-256885 Thanks - the link to the previous blog post is fine, it's the one to the Oxfam-3ie paper that seems to be broken. I think 3ie have moved things about on their site, it's now at http://www.3ieimpact.org/media/filer/2012/05/07/Working_Paper_13.pdf Thanks – the link to the previous blog post is fine, it’s the one to the Oxfam-3ie paper that seems to be broken. I think 3ie have moved things about on their site, it’s now at http://www.3ieimpact.org/media/filer/2012/05/07/Working_Paper_13.pdf

]]> By: Kate http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-256402 Kate Fri, 12 Oct 2012 10:47:22 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-256402 Could you post the link to the "Can we obtain the required rigour without randomisation? Oxfam GB’s non-experimental Global Performance Framework" paper? The link in the older blog post doesn't seem to work. Thanks. Duncan: Works for me Kate, but anyway, here it is again http://www.oxfamblogs.org/fp2p/?p=6719 Could you post the link to the “Can we obtain the required rigour without randomisation? Oxfam GB’s non-experimental Global Performance Framework” paper? The link in the older blog post doesn’t seem to work. Thanks.

Duncan: Works for me Kate, but anyway, here it is again http://www.oxfamblogs.org/fp2p/?p=6719

]]> By: Luc Lapointe http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-256143 Luc Lapointe Thu, 11 Oct 2012 19:38:15 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-256143 Thank you for taking on this initiative and I look forward to read the report. In an era where we are trying to break silos and where more actors are engaged in aid delivery / development, I hope that we will soon see efforts to measure collective impact. The success of your programs depends on a multitude of factors and organizations that are most likely not associated with you. Your success (impact/outcomes) will greatly increase when we measure the collective impacts in the context of cooperation/collaboration and strengthening partners for greater effectiveness. Thank you for taking on this initiative and I look forward to read the report.

In an era where we are trying to break silos and where more actors are engaged in aid delivery / development, I hope that we will soon see efforts to measure collective impact. The success of your programs depends on a multitude of factors and organizations that are most likely not associated with you. Your success (impact/outcomes) will greatly increase when we measure the collective impacts in the context of cooperation/collaboration and strengthening partners for greater effectiveness.

]]> By: Owen Scott http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-255981 Owen Scott Thu, 11 Oct 2012 15:44:27 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-255981 First, let me say that this is brilliant. I like the focus on reporting both effective and ineffective results. Further, the format of short reports backed by longer and more rigorous reports makes the information both digestible and trustable - a rare combination in this arena. A quick piece of feedback. It would be very useful in the 2-page summaries to have a bit more information about the project - specifically the budget and timeline. For instance, the project review for the Copperbelt Livelihood Project in Zambia (http://ow.ly/eoTKe) reports that: "Women in the intervention village were found more likely to own at least one strategic asset." To properly interpret this it would be very helpful to know, at a glance, what the duration of the project was, and what the elapsed time was between the end of the project and the evaluation. Regardless, this is a very good start, and thanks again for sharing publicly. First, let me say that this is brilliant. I like the focus on reporting both effective and ineffective results. Further, the format of short reports backed by longer and more rigorous reports makes the information both digestible and trustable – a rare combination in this arena.

A quick piece of feedback. It would be very useful in the 2-page summaries to have a bit more information about the project – specifically the budget and timeline. For instance, the project review for the Copperbelt Livelihood Project in Zambia (http://ow.ly/eoTKe) reports that: “Women in the intervention village were found more likely to own at least one strategic asset.” To properly interpret this it would be very helpful to know, at a glance, what the duration of the project was, and what the elapsed time was between the end of the project and the evaluation.

Regardless, this is a very good start, and thanks again for sharing publicly.

]]> By: Mary Healy http://www.oxfamblogs.org/fp2p/?p=12012&cpage=1#comment-255810 Mary Healy Thu, 11 Oct 2012 12:49:34 +0000 http://www.oxfamblogs.org/fp2p/?p=12012#comment-255810 Very interesting, thanks for sharing. I'd love to see the total costs presented, INCLUSIVE of staff time, as this is the real cost to Oxfam and to any possible donors. Very interesting, thanks for sharing. I’d love to see the total costs presented, INCLUSIVE of staff time, as this is the real cost to Oxfam and to any possible donors.

]]>