Lost in translation: the alienation of the development worker

I’m writing this flying over Afghanistan, on my way back from India (blog flurry to follow). The air is extraordinarily clear, so that even from 30,000 feet, I can make out individual fields,Del345114 clusters of mud-coloured houses, nestling among the serrated, snow-topped mountains and winding river courses. At the same time, I’m reading ‘Lost in Translation’, by Eva Hoffman, a stupendously thoughtful, observant study of the fragmented, alienated nature of immigrant identity (in her case, of a Polish Jew in Canada).

The combination brings home the deep fragmentation of what Robert Chambers calls  rural development tourism’. Within the space of ten days, I have been a besuited presenter at an OECD forum, a besandaled interviewer in a Delhi slum, and a laughing Sahib visitor delighting in the dances and songs of girls in a village school run by a bearded sage in an Ashram. And now I’m looking down on what may be (for all I know – the aircraft map is rubbish) the battlegrounds of the Afghan war.

The result is a kind of spiritual/emotional vertigo, a high-speed zooming in and out in terms of scale and emotional landscape. Grappling now with the political economy, then with the life struggles of slum women trying to stop their men from drinking, next with the subtleties and hypocrisies of international networking. You try and stay open to them all, antennae at full receive, assimilating where you can, compartmentalizing where

'Damn, the wifi is down'

'Damn, the wifi is down'

you must. But I do wonder whether overall, it dulls the nerve endings, blunts the edges of the sensibilities, leaving you seeking refuge in the bland abstractions of development-speak.

It resembles Hoffman’s attempt to capture the fragmented consciousness of the immigrant, both in the negatives (the lack of any deep, unconscious sense of belonging) and the positives (the cross-fertilization of different worlds). We may not be immigrants, but we are increasingly all migrants, and there’s some overlap between the two conditions. It makes me wonder if a spell at Walden Pond, amid a single, slowly changing and unpeopled landscape might not be a spiritually healthier alternative. But in this connected age, such a deliberate decision to step outside the information stream feels highly unlikely – have they started marketing ‘only disconnect’ holidays yet, in the world’s remaining ‘no signal’ digital wilderness areas? Tell me when they start.

October 31st, 2012 | 3 Comments

China’s African road; Euro-governance failures; patronage -> accountability; building fragile states; Blattman needs you; grey planet; toxic development; why people cheat: links I liked

The colonialists never built a road around the West African; they only needed extractive roads and railways ‘like the fingers of a hand’, from coastal ports into the interior. But now China is building one. So maybe it isn’t so colonialist after all? http://www.chinaafricarealstory.com/2012/10/china-to-build-west-african-coastal.html
‘From the perspective of Europe or the United States, hesitation and uncertainty [over financial market reform] may look like the price that has to be paid for national level sovereign democracy. From China or some parts of the old South, it looks like a serious governance failure with global consequences – and not so different from the consistent economic policy failures in Africa in the 1970s and 1980s that helped pave the way for the emergence of the governance agenda.’ Mick Moore gets magisterial on the history of the governance agenda and concludes that after 20 years, the jury is still out on its usefulness. http://www.governanceanddevelopment.com/2012/10/is-governance-agenda-going-anywhere.html
How does patronage politics move to accountability? Promising series from Acemoglu and Robinson. Post one explains clientilism http://whynationsfail.com/blog/2012/10/25/patronage-or-programmatic-politics.html
How to do state-building in fragile states: distinguish local state from national; shift from supply to demand – ‘shift from state-building to ‘citizenship-building’; forget RCTs and get serious about qualitative monitoring. Excellent piece + links from ODI’s Rachel Slater http://www.odi.org.uk/opinion/details.asp?id=6884&title=service-delivery-state-building-conflicted-affected-states&utm_source=ODI_Opinion&utm_medium=feed
‘If you, your uncle, or your dog’s previous owner’s second cousin runs a firm in a low-income country, drop me a line.’ Chris  Blattman looking for volunteers for his research programme to study poverty impact of small firms. http://feedproxy.google.com/~r/chrisblattman/~3/9snB4BeQ_RE/
Waste from mining, lead smelters, industrial dumps and other toxic sites affects the health of an estimated 125 million people in 49 low- and middle-income countries. This unrecognised health burden is on the scale of malaria or tuberculosis (TB), a new report has found. http://www.guardian.co.uk/global-development/2012/oct/24/pollution-health-problem-malaria-tb
Population rethink: In 2000, for the first time, there were more people over the age of 60 than children under five. 2/3 of them live in developing countries. Is it time for an international convention on the rights of older people? http://www.guardian.co.uk/global-development/2012/oct/01/un-report-action-need-ageing-population From pyramid to tube – evolving global population distributions 1950-2100 http://populationpyramid.net/WORLD/2100/ via Shanta Deverajan
And a new RSAnimate is always a cause for celebration. This one has Dan Ariely explaining why people cheat, and what can be done about it. Bank reform anyone?
<object width=”560″ height=”315″><param name=”movie” value=”http://www.youtube.com/v/XBmJay_qdNc?version=3&amp;hl=en_US&amp;rel=0″></param><param name=”allowFullScreen” value=”true”></param><param name=”allowscriptaccess” value=”always”></param><embed src=”http://www.youtube.com/v/XBmJay_qdNc?version=3&amp;hl=en_US&amp;rel=0″ type=”application/x-shockwave-flash” width=”560″ height=”315″ allowscriptaccess=”always” allowfullscreen=”true”></embed></object>

The colonialists never built a road around the West African; they only needed extractive roads and railways like the fingers of a hand, West Africa coast roadfrom coastal ports into the interior. But now China is building one. So maybe it isn’t so colonialist after all?

‘From the perspective of Europe or the United States, hesitation and uncertainty [over financial market reform] may look like the price that has to be paid for national level sovereign democracy. From China or some parts of the old South, it looks like a serious governance failure with global consequences – and not so different from the consistent economic policy failures in Africa in the 1970s and 1980s that helped pave the way for the emergence of the governance agenda.’ Mick Moore gets magisterial on the history of the governance agenda and concludes that after 20 years, the jury is still out on its usefulness.

How does patronage politics move to accountability? Promising series from Acemoglu and Robinson. Post one explains clientilism.

How to do state-building in fragile states: distinguish local state from national; shift from supply to demand – ‘shift from state-building to ‘citizenship-building’; forget RCTs and get serious about qualitative monitoring. Excellent piece + links from ODI’s Rachel Slater

‘If you, your uncle, or your dog’s previous owner’s second cousin runs a firm in a low-income country, drop me a line.’ Chris  Blattman is looking for volunteers for his research programme to study poverty impact of small firms.

Population rethink: In 2000, for the first time, there were more people over the age of 60 than children under five. 2/3 of them live in developing countries. Is it time for an international convention on the rights of older people? And from pyramid to tube – evolving global population distributions 1950-2100 Interactive graphic h/t Shanta Deverajan

Waste from mining, lead smelters, industrial dumps and other toxic sites affects the health of an estimated 125 million people in 49 low- and middle-income countries. This unrecognised health burden is on the scale of malaria or tuberculosis (TB), a new report has found.

And a new RSAnimate is always a cause for celebration. This one has Dan Ariely explaining why people cheat, and what can be done about it. Bank reform anyone?

October 30th, 2012 | 1 Comment

How can a post-2015 agreement drive real change? Please read and comment on this draft paper

HOW CAN A POST-2015 AGREEMENT DRIVE REAL CHANGE? DOWNLOAD PAPER

The post-2015 discussion on what should succeed the Millennium Development Goals (MDGs) is picking up steam, with barely a day going by without some new paper, consultation or high level meeting. So I, along with Stephen Hale and Matthew Lockwood, have decided to add to the growing slush-pile with a new discussion paper. We want you to read the draft (see right) and help us improve it. Contributions by 5 November please, either as comments on the blog, or emailed to research[at]oxfam.org.uk.

The paper argues that there’s an urgent need to bring power and politics into the centre of the post-2015 discussion. To have impact, any post-2015 arrangement has to take into account the lessons of over a decade of implementing the existing MDGs, and be shaped by the profound global change since the MDGs were debated over the course of the 1990s and early noughties.  We’re hoping that this will be at the centre of this week’s discussions in London linked to the High Level Panel and in Berlin at the Berlin Civil Society Center on Development post 2015.

The most significant shift is that the new arrangements have to be designed to influence governments, whereas the main impact of the MDGs was on the aid system. Why the shift? Because aid is becoming less important, both because it is likely to decline in volume over the next few years, and because governments’ dependence on aid as a percentage of revenues is falling even faster than aid itself. In any case, aid is a pretty ineffective way of influencing government behaviour, beyond the actual expenditure of donor dollars.

So if influencing governments is the goal, what can we learn from the experience of the MDGs? The first thing to note is a startling lack of research. Many reviews blur the distinction between ‘MDGs’ and ‘MDG policies’/’MDG planning’ (in effect, social welfare). Analysis of the data on improvements in health, education, and other key sectors largely ignores the vital question of how much of that improvement can be plausibly attributed to the MDGs, rather than to other factors such as national politics, economic growth, or technological innovation. Given the substantial political and financial investment in the MDGs, and the need to design an effective post-2015 framework, being unable to attribute – with any certainty – progress due to the MDGs is a truly lamentable gap in our knowledge.

mdg-iconsThere is even less research on (and less anecdotal or circumstantial evidence for) the impact of the MDGs on the policies and behaviours of rich countries, beyond changes in their aid budgets. There is scant evidence that MDG 8’s commitment to a ‘global partnership for development’ has had any impact on rich country behaviour. Understanding this failure is vital, given that many proposals for the post-2015 regime seek to place more obligations on rich countries in areas such as climate change and resource consumption.

What we know is that some governments have adopted the language of the MDGs and have customized them to fit national priorities, while civil society groups have increasingly used them as advocacy tools.

Beyond that, many post-2015 participants seem to think it is not possible to give a more complete answer to the traction question because of the missing counterfactual (how can we know what would have happened without the MDGs?). Not so. It is certainly possible to know much more than we do about attribution through more rigorous qualitative research. For example, in-depth interviews with policymakers could investigate the traction exerted by a range of external and domestic forces on their decisions (avoiding any leading questions on the MDGs). We have yet to locate such research.

So much for the MDGs, what about whatever comes next? International instruments can exert influence in three key ways:

  1. By changing national norms in areas such as women’s rights. However intangible, norms matter, leading to long-term changes in what society considers acceptable or deplorable, which then leads to changes to laws, policies and behaviours.
  2. By directly influencing government decision making, through any of a number of possible carrots (aid, contracts, acceptance, approval) or sticks (sanctions, disapproval).
  3. By giving civil society organisations and other domestic actors more tools with which to lobby, campaign, and secure action by their governments.

In most cases, the main drivers of change will be domestic – the result of national politics and culture. But international initiatives are second-order factors that can nudge things along. We identify six kinds of instrument at global and regional levels.

Big global norms: rallying cries intended to influence the underlying attitudes of decision makers and citizens, such as ‘zero poverty’ or ‘zero hunger’.

Global goals and targets: as encapsulated by the MDGs.

Regional goals and targets: the African Union has been particularly energetic in agreeing regional targets, setting out what its member governments should be aiming for on the Rights of Women (AU Protocol, 2003),or their allocation of spending to agriculture (Maputo Agreement 2003), health (Abuja Declaration 2001) and similar commitments on social protection, and water and sanitation.

Global league tables: the international community and/or civil society can simply collect and publish data allowing a comparison between different countries’ absolute situation and rate of progress, as in the UNDP’s Human Development Index. Anecdotal evidence (and long NGO experience) suggests that league tables can be effective both in attracting public and media interest, and in goading politicians into action – there is nothing a leader likes less than to be seen to lose out to a rival nation.

Data transparency: according to some architects of the MDGs, perhaps their greatest legacy will be the improved quality, collection and dissemination of social data. One option would be to make this the centrepiece of a post-2015 arrangement, and leave it to others (national or regional bodies, international institutions) to ‘mash up’ the data into different indices and use it to advocate for progressive policies.

International law: Most governments are already signatories to dozens, if not hundreds, of international conventions and the role and influence of international law appears to be on an inexorable upward curve, steadily encroaching on previously untouchable areas of state sovereignty.

What are the strengths and weaknesses of these options in influencing norms, decision making or civil society activism? Here we are basically into guesswork/gut feeling, captured in the table below. We’d be interested to hear your views, and very grateful for links to any relevant research.

Possible options for international instruments to drive change post-2015

Instrument Influence on national norms On decision making Civil society take-up
Big global norms Sometimes strong, but often disappear without trace Long-term influence (e.g. shaping future leaders’ world views) Strong, if resonate with national reality
Global goals and targets Partial Transmission via aid system, otherwise likely to be partial Yes, when resonate with national reality

Far stronger if accompanied by national goals, civil society commitment to these, and clear national accountability mechanisms

Regional goals and targets More influence where regional identity is stronger (e.g. African Union) Especially if governments have to ratify and legislate. Rivalry can also be effective Can provide a valuable advocacy tool, especially where regional identity is strong
Global league tables Weak Effective if builds on regional rivalries Can provide a valuable advocacy tool
Data transparency Weak Depends how data are picked up by national actors Depends on civil society capacity to use data for advocacy purposes, alliances with academics, etc.
International law Strong, but slow osmosis into national common sense (e.g. children have rights) Especially if governments have to ratify and legislate, or report publicly on their performance (as with the UNCRC or CEDAW) Depends on civil society capacity to use legal system (and responsiveness of legal system)

Over to you for comments, links etc

October 29th, 2012 | 17 Comments

Why the World Bank is wrong (so far) on large land deals

You’re getting a lot of guest posts this week, not least because I’m in India – expect a spate of India posts next week. Here’s Hannahhannah stoddart Stoddart, Oxfam’s Head of Economic Justice Policy, responding to the World Bank’s response to Oxfam’s call for a freeze on large land deals.

Oxfam’s land grabs campaign, launched on 4th October, highlights the alarming increase in the speed and scale of large land deals in the past decade. It calls on the World Bank – as an investor in land deals, as a global standard setter and as an adviser to developing countries on their land policies – to freeze those of its agricultural investments that involve large land deals for 6 months while it reviews its policies and practices to ensure land grabs are prevented.

The World Bank has responded through official statements, blogs and interventions on panels. Here’s Oxfam’s response to some of the Bank’s main counter-arguments:

  • Extent of World Bank involvement in land-grabbing

Some at the World Bank have suggested that it is not the right target – it is only involved in a ‘few cases’ that could potentially constitute land grabs and at any rate it is not as bad as most other investors. Oxfam stands by its focus on the Bank for a number of reasons. First, given the Bank’s mandate for poverty alleviation, even one land-grab case is a case too many.

Secondly, in reality we know that there are very likely more than a few controversial cases relating to land. 21 cases involving land disputes have been brought by communities since 2008 (Oxfam is involved as a complainant in a number of them). We also know that between 2000 – 2012, 56% of the complaints to the Compliance Adviser Ombudsman (CAO) have been in relation to land. The CAO also confirms that in the past 4 years there has been a growing number of complaints in relation to agri-business.

Lastly, while the World Bank may not the worst culprit when it comes to land-grabbing, it IS the only global bank with a mandate for poverty alleviation and it is a crucial institution for setting the bar high in this area. In other words, we believe that if Oxfam can’t convince the World Bank to raise its standards, we have no hope of getting other financing institutions to do so. If the Bank takes leadership, we hope we can leverage change in other institutions as a result, from regional development banks to private investors.

  • World Bank’s role in agriculture

In reaction to our call for an investment freeze, the World Bank contends that it has increased its agricultural investments precisely in response to calls from organizations such as Oxfam for it to focus on a sector that has been neglected for too long. It argues that to suspend its agricultural investments – which overwhelmingly benefit smallholders – will only end up harming the very people that Oxfam seeks to support.

In response, we have never argued – and never will – that the World Bank should not be investing in agriculture. We welcome increased investment in agriculture by the Bank that genuinely benefits smallholders. This is why we are not arguing that the Bank should get out of agriculture altogether. And this is also why we are not calling for a freeze of all agricultural investments, but for a temporary 6 month freeze on agricultural investments that involve large-scale land acquisition – which the Bank acknowledges is not the majority of its investment portfolio. To put it another way, we’re invoking the precautionary principle – something the Bank has done itself in the past when it froze lending to the palm oil sector as a result of a controversial case in Indonesia.

land grabs logoAs the World Bank’s investment in agriculture has increased from $2.5 billion in 2002 to $6-8 billion in 2012, the risk of some of these investments involving problematic land acquisition is heightened (for the record, this figure was misquoted by some media as being up to $8 billion in land investments, Oxfam has always been clear that the overall figure is for agriculture more broadly, some of which will involve land acquisition).

We welcome models of agricultural investment – both large-scale and small – that benefit communities and genuinely lead to shared benefits based on consultation and consent. We have recently published a paper outlining models of positive agricultural investment, and Oxfam GB CEO Barbara Stocking reiterated this message recently in the Financial Times. What we oppose is a model of agricultural investment that involves the mass transfer of land rights away from poor farmers and communities, a model that  frequently leads to conflict and for which there is very little evidence of pro-poor outcomes.

  • Transparency

The Bank has suggested that it is a leader in the area of transparency. While Oxfam agrees that it has made great advances over the years, we feel that there are still some real areas of concern. First, we can’t even tell the full extent of the Bank’s investment in this area: there is no clarity on the overall size of its land portfolio. For an institution that rightly prides itself on the huge advances it has made in making  its data accessible, this is disappointing.

Second, 17 of the 21 complaints involving land raise issues relating to inadequate transparency. Third, over 50% of lending through the International Finance Corporation (the private sector lending arm of the World Bank) is channeled through financial intermediaries: these investments are far more opaque, and these bodies are also not subject to the same standards as the World Bank. And it makes it almost impossible for Oxfam to judge whether the Bank’s claim is true that ‘only 2% of IFC agribusiness loans in the past financial year involved land acquisition’. Furthermore, the trend towards new lending instruments and technical assistance makes it far more difficult to hold the World Bank accountable for cases where it might not have directly funded a project that results in controversy, but has provided the advice that made it possible.

So if the Bank wants to know #whatwillittake to end poverty, Oxfam thinks taking leadership on stopping land grabs is a great place to start.

Hannah Stoddart is Head of Economic Justice Policy at Oxfam GB

October 26th, 2012 | 1 Comment

Getting evaluation right: a five point plan

Final (for now) evaluationtastic installment on Oxfam’s attempts to do public warts-and-all evaluations of randomly selected projects.jyotsna puri This commentary comes from Dr Jyotsna Puri, Deputy Executive Director and Head of Evaluation of the International Initiative for Impact Evaluation (3ie)

Oxfam’s emphasis on quality evaluations is a step in the right direction. Implementing agencies rarely make an impassioned plea for evidence and rigor in their evidence collection, and worse, they hardly ever publish negative evaluations.  The internal wrangling and pressure to not publish these must have been so high:

  • ‘What will our donors say? How will we justify poor results to our funders and contributors?’
  • ‘It’s suicidal. Our competitors will flaunt these results and donors will flee.’
  • ‘Why must we put these online and why ‘traffic light’ them? Why not just publish the reports, let people wade through them and take away their own messages?’
  • ‘Our field managers will get upset, angry and discouraged when they read these.’
  • ‘These field managers on the ground are our colleagues. We can’t criticize them publicly… where’s the team spirit?’
  • ‘There are so many nuances on the ground. Detractors will mis-use these scores and ignore these ground realities.’

accountability cartoonThe zeitgeist may indeed be transparency, but few organizations are actually doing it.

So while Oxfam’s results are interesting, more importantly the transparent process must be applauded. But as I read these documents, it was deja vu… In the initiatives that used quasi-experimental methods I was struck by Oxfam’s acknowledgement that they didn’t know the ‘why’ of some of the results. For the ones that used qualitative methods (the humanitarian portfolio, citizen’s voice and policy influencing), I kept asking myself, how much did they do better by? It seemed like a zero-sum game: One method meant the absence of the other.

This was one source of familiar dissatisfaction…

As they say, once a ship has sunk, all the mice know how it could have been saved.

So here’s the mouse in me. What can an organization do to answer questions I (and it) have and not wring its (collective) hands regretfully later? Here’s my five point list for what all NGOs should think about before setting up an M&E system (or even after setting it up). It’s operational (I have put one into place), it’s not easy, but it has the potential to quieten most detractors (and people like me):

Point 1: Have a good theory of change/causal pathway/impact pathway or whatever you want to call it. The name doesn’t matter (it’s a rose!)

Theories of change are good for understanding the program, for schematics and great communication tools too. Additionally anHaiti reconstruction cartoonevidence-based theory of change can help you decide where you need most investigation, where a process evaluation is sufficient, where a counterfactual analysis of outcomes is required and where a simple tracking of indicators is useful.

Do: Set one up and ensure everyone who needs to, knows the theory of change along with risks and assumptions.

Point 2: Put in place monitoring and information systems. Track process and process/output and some outcome indicators across program areas. There should be a list of performance monitoring indicators that speak to different sectors (four in the case of Oxfam).

Do: Put together a set of standard operating procedures for collecting information on process indicators. This should contain information on frequency of collection, identify data sources (clinics, households, schools), specify respondents (teachers, nurses, women, children…) and clearly elucidate methods for calculating indicators (even for simple indicators such as enrollment rates).

Do: Write and revise and revise a standard operating procedure manual till you have it pat.

Do: Have a management information system that also includes algorithms for quality checks and have a full time person doing data review.

Do: Train your data collectors and your data base managers;

Measuring babyPoint 3: Think about measuring attributable change. Can you for instance:

-          Assign the intervention randomly from the beginning without losing sight of your final goal?

-          Identify counterfactual sites and start collecting data there? Pros: great reporting to donors; rigorous information; Cons: more expensive than just monitoring data, does require high level of scrutiny in comparison sites especially if you use ex post techniques.

-          Use other methods to establish causality? (Which ones?)

For all methods:

Do: Use protocols and register them (3ie will soon start to register them.)

Do: Use rigorous surveys in implementation sites and in control sites (and get someone who knows how to do them. Don’t do them yourself).

Do: Have standard operating procedures for site level data entry and cleaning;

Do: Use anthropometric measures and bio-physical indicators to the extent possible;

Do: Use and write a field operations manual, write standard operating procedure manuals for data managers that contain range and logic checks for data, and, encourage double data entry.

Point 4: Undertaking cost and cost effectiveness studies. What are the priced and non-priced inputs in the project? Think about whether you want to use these projects in other places? Scale them up? (And no it’s not going to be calculated from your budget statements alone. )

Do: Put together a standardized template with cost categories and measurement methods. (E.g. how will you measure the cost of usingtrapped in rubble cartoongood seeds for the farmer? It’s not just the cost of procurement or transportation but also the cost of additional manure, the cost of storage for seed and post-harvest produce.)

Do: Ensure that everyone in the delivery chain understands and sees this template the same way. (Train, train, train…train!).

Point 5: Focus on implementation research. Systematically documenting implementation factors, and putting together a protocol which contains questions that are relevant to informing all stages of the evaluation. This is where participatory methods, focus groups, observational scrutiny, process research should come in, and also inform your theory of change.

Do: set out a protocol at the beginning that lays out i) the questions you want answered ii) what you’ll ask in your interviews to answer them; iii) a plan for analyzing your qualitative information.

There are many more things one can do. But I believe if you have these covered, you are on your way.

A few more things to bridge that elusive evidence-policy gap:

  • Evidence is required for policy making but most policy makers are looking to affirm (and not inform) their opinions (as a recent article in Time says. See here for an excellent QJPS article also cited there).
  • Be circumspect about what evidence you advocate for. Not everything is worth fighting for (and often leads to evidence-fatigue.) When I have taught policy analysis, I have often used a rule of thumb long known to academic political scientists: if a policy change leads to less than a 10% change in outcome, it’s a flashing red (stop and think before translating that evidence into policy); if it’s a 10-25% change (it’s a lime, go for it but think about transition costs); if it’s more than 25% change, it’s a deep, loud green: Adopt the policy. The costs of transition will be surpassed by the benefits of policy change.
  • Change the institutional incentives: Oxfam is on its way, but will program managers on the ground really adopt this culture change or will it continue to be top down? (See here for an excellent blog by Mead Over and Martin Ravallion.)
October 25th, 2012 | 2 Comments

What do DFID wonks think of Oxfam’s attempt to measure its effectiveness?

More DFIDistas on the blog: this time Nick York, DFID’s top evaluator and Caroline Hoy, who covers NGO evaluation, comment on Oxfam’s publication of a set of 26 warts-and-all programme effectiveness reviews.

Having seen Karl Hughes’s 3ie working paper on process tracing and talked to the team in Oxfam about evaluation approaches, Caroline Hoy (our lead on evaluation for NGOs) and I have been reading with considerable interest the set of papers that Jennie Richmond has shared with us on ‘Tackling the evaluation challenge – how do we know we are effective?’.

From DFID’s perspective, and now 2 years into the challenges of ‘embedding evaluation’ in a serious way into our own work, we know how difficult it often is to find reliable methods to identify what works and measure impact for complex development interventions.  Although it is relatively well understood how to apply standard techniques in some areas – such as health, social protection, water and sanitation and microfinance – there are whole swathes of development where we need to be quite innovative and creative in finding approaches to evaluation that can deal with the complexity of the issues and the nature of the programmes.  Many of these areas are where NGOs such as Oxfam do their best work.

So we would really like to welcome and applaud Oxfam’s new Effectiveness Reviews, which adopt a clear and practical framework for assessing what difference it is making, through its partners, in the development process. It is a big step forward for them – and it would be great if it also inspires other organisations to develop new and interesting approaches to measuring results and undertake rigorous analysis of what works.  Clearly this needs to be done in a way which each organisation can afford and resource – things need to be done in a proportionate way – but the Oxfam initiative shows some of what is possible.

They have chosen quite a practical strategy – picking out a random sample and then probing more deeply and using different techniquescartoon-evaluation_culture to measure impact or use well-tried monitoring of performance indicators.

Of course there is one potential drawback – random sampling may mean there are gaps in what you can say, if key areas don’t happen to have been sampled this time.  Oxfam also notes that the reviews do not necessarily enable full understanding of why a programme is successful (e.g. in Pakistan) and that they now need to go back and undertake some more work.   One way round this is more purposive sampling – we don’t know if this was considered –  or identifying priority themes up front based on what the organisational objectives are, and focusing on them in some depth.  The key challenge is finding a strategy for using the limited resources for evaluation and data collection in a targeted way that gives a nice balance between extensive coverage and intensive analysis.

Another challenge is maintaining the independence and integrity of those carrying out the evaluations.   Finding impartial observers – given that many people and experts have worked for years in these areas and know each other well – can be difficult.

The very interesting study of policy influencing by Oxfam’s partner in Bolivia, Fundacion Jubileo, is worth looking at in some detail.   It made a good case that the grantee was really having an impact on some key aspects of social change in Bolivia The evaluators clearly applied the process tracing technique skilfully and identified the most significant changes – but it must have been difficult to stay objective when doing the interviews, working with the grantee and identifying who was really influencing whom.  Howard White and Daniel Phillips’s paper on ‘small n’ techniques talks a lot about the biases that one needs to avoid in using these techniques.  The appendix to the study provides an excellent and useful set of reflections on the use of the process tracing methodology and what the evaluators learned.

One key assumption is that by doing more work and collecting more data (e.g. from comparison sites in Zambia and the Philippines), they will be able to understand and demonstrate impact.   Actually, based on discussions we have had with Michael Woolcock recently in DFID, we have started to ask a different sort of question.    In some types of programmes, more data and more work may not be the solution Gandhi v logframe cartoon- more innovative methods and approaches to understanding impact can be required and if the programme itself develops as you implement it then the goal posts are continually shifting too.

Looking ahead, and thinking about the next stages of this agenda….first, we would encourage others to share their approaches and experiences in the way that Oxfam has done.   Second, it would be great to see Oxfam and other NGOs sharing resources to develop better methods across the sector, given their common challenge of demonstrating results and the limited resources.   The results agenda is particularly challenging for smaller organisations, whose inputs are increasingly recognised – so can we ask if Oxfam sees itself as in a position to demonstrate leadership in linking with such organisation to jointly share and explore results?

Nick York is DFID’s Chief Professional Officer – Evaluation and Caroline Hoy, its Results and Evaluation Specialist, Civil Society Department.

October 24th, 2012 | 2 Comments

Why are international conferences so bad, and what can be done about it?

Delhi logoLast week I attended the OECD’s 4th World Forum on Measuring Wellbeing. Actually, I sampled it, ducking out to look at Oxfam programmes in Delhi, meet people and give a couple of lectures in local universities. Lots of people do this, so it ought to have a name – conflirting? Condipping? Any better suggestions?

My overall impression was that official interest in well-being and its measurement continues to grow, but has moved to a national level, where numerous governments are seriously trying to put it into practice (here’s where the UK has got to, big report due next month). Although it has set up its 36 country ‘Better Life Index’ (with a funky interactive website where you can construct your own measure of well-being) and has launched the wikiprogress site, the OECD is not driving the debate as it was when I attended the previous Forum in Busan in 2010, (many fewer delegates this time around, and not much new in the debates). That is probably a good thing – national action and experimentation is what really matters.

Back to conflirting, because despite the hard work and dedication of the OECD staff, I suspect one of the reasons people do it is because many international conferences are so mind-numbingly dull, and I’m afraid much ofboring-conference this one followed the standard pattern. A few ‘keynote speakers’, bleary with jetlag, stumble through their papers (Joe Stiglitz and several politicians whose names escape me), or give a speech on their current interest, completely ignoring the subject of the conference (Jeff Sachs). Dry-as-dust panels of disconnected presentations – chairing is feeble in keeping to time and/or panels are over-stuffed with speakers, so there is never enough time for questions or interaction between the speakers.

As the days pass, fewer people turn up (and interestingly, start to abandon smart clothing – everything gets more casual). Even if they do, most people are on their phones doing their emails or tweeting about the meeting (guilty as charged).

Often, the only really useful activity is the networking on the margins (and in the bars, quite memorably so in Delhi, but that’s another story), but conferences take no account of this in their design, except to allow lots of coffee breaks (when those survive encroachment by over-running panels).

In terms of the timesuck of highly qualified people, and the money involved, this seems spectacularly amateurish/cavalier, especially when compared to the huge investment in improving the impact of research and development programming. So come on multilaterals and funders, what about funding/designing a ‘Conference for Impact’ programme. What would you do differently? Some ideas:

Narrow the agenda, broaden the minds: Set a specific question to be answered by all participants. At the same time as narrowing the question, broaden the range of disciplines involved – the Wellbeing conference was largely made up of government and multilateral officials and economists, (with the odd token NGO like me). What about philosophers? Religious leaders like the Buddhist abbot we consulted in Busan? Psychologists? Psychoanalysts?

Avoid academic conference formats, which seem to be the most stultifying. Panel presentations plus Q&A has to be one of the least productive ways to spur creative thinking. Import some of the less cringeworthy methods we use in NGO discussions – groupwork, world cafes, speeddating, sandpits and other innovative formats. I’m sure the private sector has lots of others.

Powerpoint-poisoning

Sort out the presentations: Ban anyone from reading out a paper; find a way to limit Powerpoints to a maximum of 20 words per slide (and urge speakers to use images); install amber and red lights on the mikes, which cut the sound off after the speaker goes into the red. Maximum of 3 speakers per panel, and ask the audience to buzz with their neighbour before going into Q&A, to get some energy back into the room.

Set up a feedback system: A public Ebay-type ratings system to show which speakers/conferences were best. As an extreme method, adopt instant audience feedback, Occupy-style (thumbs up from audience if they like the speaker, thumbs down if they don’t) or a twitter wall behind the speaker to show how they’re going down with the public.

Avoid distractions: One of the reasons people got more involved in Busan may have been the lack of opportunities for conflirting. Delhi on the other hand is stuffed with institutions people want to visit. And (provided the other factors are dealt with to create a useful event), maybe choosing a state-run hotel where the internet keeps going down (as it did in Delhi) is not such a bad idea after all.

Any other conference braindeath survivors want to add suggestions?

And here are some previous, slightly more highbrow, reflections on the purpose of conferences. I probably won’t get invited to any more now. Oh well.

October 23rd, 2012 | 14 Comments

Deconstructing resilience; savage v fragile; China’s flashmob breastfeeders; Africans in the Diaspora; I’m a (Welsh) prof; Why Tories love aid; hierarchies of misery: links I liked

OxfamAmerica wonk Gawain Kripke unpacks the latest devt fuzzword, resilience http://politicsofpoverty.oxfamamerica.org/2012/10/19/thoughts-on-resilience-as-organizing-focus/
For ‘savage’ read ‘fragile states’. Changing development terminology 1821->2012 http://nyudri.org/2012/10/18/new-historical-discovery-the-change-in-development-terminology-from-1821-to-2012/
Flashmob breastfeeding in China (inspired by a similar action in Manchester), take on bottlefeeding + caesarean culture http://www.danwei.com/the-year-of-the-breastfeeding-flash-mob-in-china-a-fathers-account/
Project Umubano – is this the reason the Tories have stuck to the UK’s aid promises? http://www.conservatives.com/Get_involved/Project_Umubano.aspx
Africans in the Diaspora (AiD). Exciting new initiative http://www.aviewfromthecave.com/2012/10/activating-diaspora-for-development.html
OK, don’t laugh, but Cardiff University have appointed (annointed?) me as an Honorary (that means no $) Professor of International Development in their Planning and Geography Department (http://www.cardiff.ac.uk/cplan/about-us/staff/duncan-green). They want me to help beef up their development studies work (one of the few Masters courses that is oversubscribed, apparently), give occasional lectures, and help develop joint research collaborations with Oxfam. And no, I didn’t write the blurb…..
Hmm, what do you think of this ad? Clever and effective, or politically dodgy to ridicule the first world and say ‘the only real problems are those of poor people’?
<object width=”560″ height=”315″><param name=”movie” value=”http://www.youtube.com/v/fxyhfiCO_XQ?version=3&amp;hl=en_US&amp;rel=0″></param><param name=”allowFullScreen” value=”true”></param><param name=”allowscriptaccess” value=”always”></param><embed src=”http://www.youtube.com/v/fxyhfiCO_XQ?version=3&amp;hl=en_US&amp;rel=0″ type=”application/x-shockwave-flash” width=”560″ height=”315″ allowscriptaccess=”always” allowfullscreen=”true”></embed></object>

Top OxfamAmerica wonk Gawain Kripke unpacks the latest ubiquitous development fuzzword – ‘resilience

For ‘savage’ read ‘fragile states’. Changing development terminology 1821->2012

africans_in_diaspora_logo

Flashmob breastfeeding in China (inspired by a similar action in Manchester). Women challenge bottlefeeding + caesarean culture

Africans in the Diaspora (AiD). Exciting new initiative

OK, don’t laugh, but Cardiff University have appointed (annointed?) me as an Honorary (that means no $) Professor of International Development in their Planning and Geography Department. They want me to help beef up their development studies work (one of their most popular Masters courses, apparently), give occasional lectures, and help develop joint research collaborations with Oxfam. And no, I didn’t write the blurb…..

Wherever I go outside the UK, I end up trying to explain why our right-wing, austerity-obsessed government is increasing aid, despite the sustained hostility of the Daily Mail. Is Project Umubano the equivalent of Sandinista coffee brigades for the Left? Here are some prominent Tories explaining for themselves, c/o Save the Children [h/t Alex Evans]

Hmm, what do you think of this ad? Clever and effective, or politically dodgy to ridicule the first world and say ‘the only real problems are those of poor people’?

October 22nd, 2012 | 2 Comments

To close the energy poverty gap, we need ideas, investment…and natural gas. Todd Moss responds to Hannah Ryder

CGD’s Todd Moss responds to Hannah Ryder’s critique of his ‘let them burn fossil fuels’ line on energy povertyTodd-Moss_detail

Thanks to Hannah for raising some good questions about my proposal that the US agency OPIC partially exempt the world’s lowest-income, lowest-emitting countries from the greenhouse gas cap. I think we both agree that 1.3 billion people without access to electricity in the 21st Century is inexcusable. It’s a development problem that can and should be solved. We also agree that the past approach to power has been insufficient, and that to close the energy poverty gap we need new ideas and new technologies. Here’s where we disagree:

  • Greater investment is necessary if we want to close the energy poverty gap. The data on additional generation capacity and access do not, as she suggests, show that increases in the former have no relation to decreases in the latter. Rather those IEA graphs in an apples-to-apples (global-to-global) comparison show the opposite: a clear decline of roughly 25% between 1985-2000 in the total number of people without access to electricity. Yes, this decline is driven by East Asia, but this is also likely to be precisely where the bulk of the investment and capacity additions have occurred (IEA doesn’t provide ungated data on capacity addition disaggregated by region – I’d love to see that). More recent estimates (in World Energy Outlook 2011) show that the number of people without electricity has continued to decline by some 300 million in the past decade. In other words, is seems safe to assume that where massive investment takes place (e.g., China), millions of people are gaining access to power. Thus, the conclusion, including in the paper Hannah cites, is that even more investment is needed (they suggest 5x current levels). If this is the case, it seems odd that we would question whether capacity should really increase or, worse, hamstring our agencies tasked to boost this investment with environmental mandates that have nearly zero effect on global emissions targets.
  • Off grid renewable may be better, but it’s not realistic everywhere. Certain populations may benefit from new technologies and new models, such as off-grid renewable sources. We should absolutely leverage our policy tools to deploy these where we can. But the scale of the problem is such that sizeable populations will still require old-school on-grid power that is (at least based on current economics) probably going to come from fossil fuels. This is especially likely for underserved urban populations and heavy industrial projects. This World Bank paper reports that barely half of poor residents in Dakar and Nairobi have access to electricity. OIL & GAS Production Reaching the rest will come, not from some high-tech solar system, but from hooking up more homes to the grid and boosting generation capacity in big power plants. Ditto for the 97% of large firms in Nigeria that rely on (costly, inefficient, and polluting) diesel generators to provide nearly 2/3 of their power. Similarly, Ghana’s Valco aluminum smelter in the industrial port of Tema is running at 20% capacity for the sole reason of a shortage of low-cost power. Getting Valco to capacity, with all the jobs and spin-off industries that would accompany full production, is going to require new power investments in large-scale power.
  • Natural gas will be part of the solution. My proposal specifically excludes coal, but not natural gas. This distinction is partly political, but it’s mainly pragmatic: many of the same countries that have substantial energy poverty gaps also have natural gas reserves that could be transformed into domestic energy.  Just in Africa in the past few years, Kenya, Tanzania, Mozambique, Ghana, and Cote d’Ivoire have had major new gas finds. And Nigeria still flares much of its gas. Why should we stand in the way of these countries turning these resources into electricity and jobs for their people? We shouldn’t—especially when we have placed no such constraints on ourselves.
October 20th, 2012 | 3 Comments

Why high carbon energy is the wrong solution for low income countries

DFID staff break their duck as guest writers on FP2P with this post from Hannah Ryder (right), a regular blogger on the DFID site and Senior Ehannahryder.thumbnailconomist specialising in climate change and low carbon growth

Economists have a reputation for being sceptical – there is even a book called “the Skeptical Economist”. This has a lot to do with how it is taught. For instance, we are encouraged to be sceptical of the idea that one thing (a “variable”) might directly cause another variable to change. A number of development economists have recently been stressing that “complexity” should make us even more sceptical of these relationships.

Now, I usually avoid wearing the sceptic’s hat. But the other day I came across an article that assumed a linear, causal relationship between two variables. The article was by Todd Moss at the Center for Global Development. He was arguing that the American organisation that provides investment to developing countries “OPIC” should be able to help low-income countries invest in high-carbon energy – such as coal or diesel powered stations, to help stimulate access to energy in those countries. He argued that the limits that OPIC has on this kind of investment are “strategically counterproductive and morally dubious”.

I, like Todd, certainly feel strongly about access to energy. Around the world, 1.3 billion people have no access to electricity. Over 80% of those people live either in sub-Saharan Africa or in South Asia. Access can vary dramatically within regions – over 95% of people lack electricity access in Chad and Liberia versus 25% in South Africa. Although problems are currently worse in rural areas than urban areas, even so about 56% of urban dwellers in Sub-Saharan Africa lack access to electricity.

Han’s Rosling’s latest TED talk cleverly explains why increasing energy access helps reduce poverty. It can expand people’s choices and productivity, particularly for women. It also helps business. A recent survey of manufacturing firms in Nigeria showed that 83% of respondents identified electricity as their top problem. In many cases, even when people or firms get access to electricity they still suffer from blackouts (such as experienced recently in India) and lack of affordability. Related problems exist in developed countries. In the UK, around 19% of households were “fuel poor” in 2010 – meaning they had to spend over than 10% of their income on fuel for adequate heating. Energy poverty matters.

The problem is that, from a quick skim of historic data, there is no good reason to expect that investment in conventional high-carbon energy will solve the energy access problem. These two graphs from the 2003 and 2002 IEA World Economic Outlooks (respectively) illustrate:

hannah ryder graphic

Although the dollars invested in the power sector and installed capacity – most of it based on conventional fuels such as coal, gas and oil – have increased strongly since the 1970s, the number of people with access to electricity has increased somewhat, but not a great deal.

Of course, the problem couldsimply be population growth outpacing investment growth, but the data suggests it isn’t. A 2011 study by a set of global energy experts foundno distinguishable relationship between investment in energy infrastructure and the degree of energy poverty once you control for total population. These expertsinsteadsuggestedthe problem was inequality.  Effectively, in many countries, new energy investment tends to benefit people that already have access. They therefore recommended a five-fold increase in overall energy sector investment in low-income countries, particularly in grid extensions, off-grid solutions and renewable energy – rather than the conventional, high-carbon methods used to date.

Added to this, looking forward, reports such as the European Report on Development and McKinsey’s Resource Revolution provide evidence that commodity prices are likely to rise and become more volatile in future. A number of economists such as Shalizi and Lecocq think some developing countries might regret building infrastructure now that locks them into needing to buycoal or oilor relying excessively on their volatile revenues. While there isn’t much evidence on this yet, it’s probably sensible for most countries to begin to plan for a diversified energy sector, especially if they are also going to try to target poor energy consumers more strongly in future.

These are the reasons why I was sceptical when I read Todd Moss’s article calling for OPIC to invest in high-carbon energy. It’s also why the UK supports the UN Secretary General’s Sustainable Energy For All Initiative, and why DFID specifically helps low-income countries invest in diverse sources of energy, particularly through vehicles such as the Scaling Up Renewable Energy Program, the Results-based Financing Facility and Green Africa Power. Pushing OPIC and others to look in new directions and help forge a new relationship between investment and energy access might actually be a good thing. And with that, I shall remove off my sceptic’s hat.

Todd Moss responds tomorrow

October 19th, 2012 | 2 Comments

Powered by WordPress | Design modified by Eddy Lambert from the Blue Weed theme by Blog Oh! Blog | Entries (RSS) and Comments (RSS).