At last, a sensible suggestion for post2015

After my ‘bah humbug’ paper on post2015, I’ve been largely avoiding the subject as a monumental timesuck. However, a combination of Sabina ‘multidimensional’ Alkire and Andy ‘bottom billion’ Sumner is an unstoppable force, so I’m making an exception for their new paper, Multidimensional Poverty and the Post-2015 MDGs, which is worth a skim.

What Sabina and Andy do is use her previous work for UNDP on multidimensional poverty indicators (MPIs) to square an important circle. They suggest building an ‘MPI 2.0’ based on whatever combination of issues is finally agreed in the post2015 discussion. This would produce a single number that summarizes a country’s overall progress towards the post2015 goals.

That in turn would allow the post2015 process to generate more traction on national governments (the lack of which is the subject of my paper) through league tables. Imagine if every year, all countries (including the rich ones) are ranked on a comprehensive human development table that (unlike the Human Development Index and other similar efforts) has buy in and recognition from across the international community. Each annual report would pick out the countries that have risen/fallen relative to the others. Regional tables could compare India and Bangladesh, or Peru and Bolivia, to generate extra public interest and pressure on decision makers. Within countries, an MPI could highlight regional disparities (see map).

A particular advantage of the approach is speedy feedback for policy makers: The MPI reflects effective social policy interventions immediately. With measures of income poverty, a positive social change – for example in schooling or clean water – may not be reflected for a number of years.

One of the lasting institutional legacies of the MDG process is the investment in better quality data needed to assess progress – this proposal would build on that.

One suggestion though – MPI 2.0 is a dreadful name. Why don’t we just call it ‘poverty’ and argue that it should replace $1.25 a day as the international standard?

And here’s me at a recent IDS seminar explaining why I’m so underwhelmed by the general post2015 debate.

What is the evidence for evidence-based policy making? Pretty thin, actually.

A recent conference in Nigeria considered the evidence that evidence-based policy-making actually, you know, exists. The conference report sets out its theory of change in a handy diagram – the major conference sessions are indicated in boxes.

Conclusion?

‘There is a shortage of evidence on policy makers’ actual capacity to use research evidence and there is even less evidence on effective strategies to build policy makers’ capacity. Furthermore, many presentations highlighted the insidious effect of corruption on use of evidence in policy making processes.’

i.e. you can have all the arguments you like on the nature of evidence – disciplinary and political bias, what constitutes knowledge etc etc (as this blog recently did), but policy makers are often either unable or unwilling to use it anyway – supply doesn’t guarantee demand.

The aid agencies and research councils that fund research are very keen to promote this shift from worrying about supply to wondering how to boost demand (although the researchers are often less keen – they just want to be left alone to churn out papers and develop their careers). What was nice about this conference was the amount of on-the-ground grassroots research on how decision makers actually use (or more often ignore) research in places like Nigeria (‘political manipulation and ambition seem to be among the strongest determinants of factors influencing policy development processes’) and Indonesia (‘Even if technocratic or political – it doesn’t matter – it’s 90% personality’).

One thing I learned is that agonising over per diems is not confined to the aid business:

‘One particularly heated debate concerned the frequent requests from policy makers for ‘sitting fees’ in order to attend training or seminars which could inform them about research issues. Participants agreed that this practice is widespread in most of the African countries represented; however, opinions on how to respond to this differed. Some suggested that those who aim to inform policy makers about research need to just accept that paying these fees is necessary and should therefore include them in their budgets. However others felt that continuing to pay such fees just propagates the problem and that those funding research communication and uptake work should take a ‘zero-tolerance’ approach.’

On the demand side, the report considers both capacity and incentives. On capacity ‘most people don’t know what they don’t know!’ will resonate with researchers in NGOs trying to convince their colleagues to look harder at the evidence. There’s a mountain to climb: a survey of Zambian parliamentary researchers and librarians (and these had positively agreed that they needed to use research) found that ‘only one in three believed there was consensus that the CIA did not invent HIV’.

‘Research-evidence is often used opportunistically to back up pre-existing political decisions/opinions (confirmation bias)’. That preference for policy-based evidence-making is alive and well in the big aid donors and NGOs too, of course……..

And unfortunately, research from Ghana, Sierra Leone, Uganda, and Zambia concluded that ‘a lack of capacity to understand research was perceived as beneficial to policy makers since it ‘allowed’ them to ignore evidence and instead follow their own agenda. Thus, there is not only a lack of capacity but also a disincentive to build capacity.’ Oh dear.

How to build the incentives to use research, assuming these political obstacles are not insuperable? On HIV policy in Pakistan, DFID ‘built the capacity of civil society organisations representing marginalised groups to demand policy change’.

Other useful tips:

• including policy makers in the design phase of research projects (get them on the advisory board, guys, don’t just see them as seminar fodder once you’ve finished the research)
• networks and linkages between researchers and policy makers are necessary but definitely not sufficient
• researchers need to change the (often dire) way they communicate their work – in one case study from Ghana ‘photographs of real people suffering from mental illness is far more powerful in influencing opinions than any policy brief could be.’ (Well duh)
• target the ‘policy entrepreneurs’ with influence over decision-makers (the Minister’s old university professor etc)
• ‘There is a tendency for researchers and research intermediaries to focus their communication efforts on elected representatives and appointed officials but to ignore the crucial role that technocratic staff play.’

All good stuff, but the report reminded me of the governance debates of a few years ago, in that even though it recognized the problem is incentives and politics, kept drifting back to the comfort zone of supply issues (if they don’t want research, we just have to get better at communicating or building their capacity), rather than thinking harder about the demand side. For example:

• Anyone involved in advocacy knows that the openness of policy makers to new ideas is episodic, and linked to things like changes of administration, scandals, crises and failures. So how does research need to be redesigned to capitalise on such brief windows of opportunity?
• Opposition parties are often much less well resourced, and much more malleable in their thinking as they cast around for clever ideas that will help them win power – to what extent should researchers concentrate on those without power, rather than those currently in office?
• Young minds are (generally) more open to new ideas than old ones: should researchers target future leaders (who are pretty easy to identify by faculty and university) rather than waste their time on the current generation?

The evidence debate, you won’t be surprised to hear, continues……

Beyond Horsegate: comparing the supply chains of the big 10 food companies

Erinch Sahan (right), a private sector policy advisor at Oxfam GB, introduces Behind the Brands, a big new report and company scorecard, launched today.

So we didn’t know we were eating horses. What else don’t we know about the supply chains delivering our food? 18 months ago, Oxfam posed this question to the Big 10: the world’s 10 largest food and beverage companies. In alphabetical order, they are Associated British Foods, Coca Cola, Danone, General Mills, Kelloggs, Mars, Mondelez, Nestle, Pepsico and Unilever. Today, we launch the results of our research and make it the centre piece of a brand-spanking new campaign: Behind the Brands.

The results aren’t pretty. The Big 10 seem unengaged with their supply chains (they don’t seem to know what’s going on and how to address it). Nor do they tell us much about where their commodities come from and next to zilch about how they use their power to shape the behaviour of suppliers. We came up with numbers to show this and ranked them across some important issues. This is how we did it.

7 critical themes
We chose seven themes that directly or indirectly change conditions for the poorest people in the food system: women, small-scale farmers, farm workers, land, water, climate, and transparency; asking whether the Big 10 are:

• improving conditions for women, small-scale farmers and farm workers;
• promoting equitable and sustainable access to and use of land and water;
• reducing emissions and helping farmers adapt to climate change; and
• being transparent about their supply chains and broader corporate activities

How the Big 10 stack up
After 18 months of analysis by Oxfam – and a long process of consultation with academics, industry experts, Oxfam staff on the ground, organisations working on international supply chains and the companies themselves – we came up with the following scores:

So what lies beneath these scores? The following questions transcend the themes and form the back-bone of the scorecard.

Are they telling their suppliers to do the right thing?
Most of what goes into the products does not come from farms operated by the Big 10. So the most important issue is how they shape the behaviour of their suppliers. On this point, the most relevant documents are their supplier codes (and supplier guidelines), which contain the standards they ask their suppliers to meet. An example is Unilever’s Sustainable Agriculture Code, which tells its suppliers to work with farmers’ groups to provide training to farmers and to provide safe working conditions for workers.

We had two problems in assessing supplier codes. Firstly, we cannot know if the codes are enforced, partly due to a lack of transparency around the auditing of suppliers. Secondly, most of the codes contain little meaningful detail. So we could not rely on supplier codes alone and had to broaden the scope of our scorecard.

Are they aware of the broader issues?
Across the themes, we assess if the Big 10 recognize the challenges faced by agricultural communities. For instance, under the women theme, we rewarded Coca Cola, Nestle, Pepsico and Unilever for publicly acknowledging that women lacked access to training related to food markets and for recognizing that small-scale farmers need assistance in adapting to climate change. While this can seem disconnected with their actual practices, for a company to address a problem, it must at least be aware of it.

Do they know details of their own supply chain?
Do companies know the relevant details of their supply chains? A company that knows where there is poorer land governance, will know it needs to focus on that part of its supply chain; a company that identifies water-stressed regions that they operate in is better equipped to channel its efforts.

Are they committed to improving conditions?
Once they identify an issue, we want to see a company commit to tackling it. This could be a target or a general commitment to tackling a problem. For instance, under the water theme, we rewarded nine of the Big 10 for setting a target to reduce water use in their own operations (since this impacts availability of water for local farmers). In farmers, we looked for a commitment to ensuring that small-scale producers receive a price that allows them to earn a decent income (none of the Big 10 does this).

Do their projects address core issues in their supply chains?
The Big 10 are active in philanthropic projects but we focused on whether they are doing anything to address the core issues in their supply chains. For instance, we looked for projects that improve farmer productivity, women’s empowerment, wages, land rights, resilience to climate change and access to water (we have a long list). Under most themes, we also required that they work with a relevant organisation, such as a local union or a farmers’ organisation. Few of the projects conducted by the Big 10 met our criteria.

What remains unanswered
Whether the Big 10 use their power to make their suppliers do the right thing is not very clear. It is hard to get information about who they do business with and exactly where the commodities in their products come from. We know even less about how they engage with their suppliers and, after assessing information in the public-realm, are left with unanswered questions such as:

• How much emphasis do they put on social and environmental issues when negotiating contracts?
• Do they know how much it would cost for their suppliers to do business responsibly and do they pay enough to allow this to happen?
• How much information do they provide to their suppliers in terms of advance notice of upcoming orders and quality requirements?
• Who bears risks relating to transport and weather-related disruption and fluctuating demand?

Greater transparency about how they manage these issues with suppliers is an essential first step – starting with some facts on whether and how they incentivise their buyers to take account of these issues. Until the Big 10 stop hiding behind the excuse of “commercial sensitivity”, they are not serious about being held to account for their power to improve the lives of the marginalised, many of whom are growing the food we eat.

What do the Big 10 need to address?
The Big 10 have several gaping holes in their policies but Oxfam suggests that they prioritise taking action as follows:

1. Make explicit commitments to recognize and fix the injustices in their supply chains.

2. Identify areas of high risk and analyze and disclose their impact on supply chain issues.

3. Make clear their expectations of their suppliers and support them to do their part to fix the injustices.

Erinch Sahan led the team across Oxfam International that put together the scorecard.

Community-based tourism in Ethiopia – aka where I’ve been for the last two weeks

Just got back from something of a busman’s holiday – two weeks in Ethiopia with my wife Cathy. The highlight was some community-based tourism, a magical four-day trek across the highlands near Lalibela.

First the community bit. The trek consisted of daily walks, with the next village providing a donkey for the bags (v welcome at 3000 metres), a donkey minder + local guide (we also had an English speaking guide from the central CBT organization – see below). Pic right, as promised. Every night we stayed at comfortable purpose-built huts. The community got about 40% of the $600 we paid for the four days, (lots of elaborate writing of receipts etc as our guide handed over the cash) plus about another $100 in the tips we were encouraged to hand over directly (I was less comfortable about this, but we were assured it was acceptable and transparent).

Each day would end with a conversation by the flickering light of a eucalyptus wood fire, with community leaders, cooks, or whoever else was around. The exchange was more relaxed than the structured interviews of a work field trip, with a bit more two way conversation (we were particularly stumped by trying to answer ‘so what is the motor of the economy in London, where you live?’).

And the tourism bit was wonderful, as we worked our way across the Meket Escarpment an hour south of Lailbela – a plateau of flat fertile farmland, edged by plunging cliffs and heart-lifting views (particularly from the toilets at each campsite – nice touch. See pic left). No electricity for 30 miles, so the stars were unbeatable (saw a bright red shooting star for the first time in my life). Great birds for the twitchers (Abyssinian Ground Hornbill? No problem) troupes of lion-maned Gelada baboons, a family of rock hyraxes sunbathing next to one camp site.

But the highlight was the encounter with two priests of the ubiquitous Ethiopian Orthodox Church, pickaxes in hand, digging a series of underground churches out of solid rock on the flanks of a remote and huge canyon. The centuries-old ‘Rock-hewn churches’ attract thousands of tourists to Lalibela, but Gebre Meskel and his younger companion had decided to build some new ones, which they’ve christened Debre Sion. They’ve been at it for a year now, and reckon they will open for services within another year. Meskel has nothing on paper, just a design in his head, with lots of plans to start more churches once this series of underground chapels and connecting tunnels is complete. Initially, their colleagues were sceptical, but now just a couple of hours’ walk away, the priests are hailing a miracle – for how else could two men with handtools build an underground church this fast? Wild.

But these kinds of anecdotes don’t capture the cumulative impact of the walks, the sights and the

conversations. At times it felt like a mobile immersion, and by the end I was quite overwhelmed by the human drama, the combination of undoubted progress and daily grind and above all the beauty, both human and natural.

If you want to go, contact the Lasta Lailbela Community Tourism Guiding Enterprise (LLCTGE). Conversations with other trekkers suggest it might also be worth talking to (or specifically asking for) our guide, the endlessly patient and helpful Temesgen Damtew Tsegaye –  temesgendamtew[at]yahoo.com (right – yep, the huts even had beer). More on Community-based Tourism here, but what I can’t find is any directory to help you identify good CBT projects, hopefully including next year’s holiday. Any suggestions?

Off on holiday. It involves a donkey. Back in a couple of weeks.

Exhausted by the ferocity of the evidence debates, I am off for an African holiday. I believe a donkey is involved at some point. Pics may follow.

Aid and the private sector: a love story

Oxfam private sector adviser Erinch Sahan (right) summarizes a critical new review of the growing interlinkages between aid and the private sector

Donors have a new love: business. And it will end poverty. Aid chiefs across the world have concluded that if we need growth to end poverty and the private sector drives growth, isn’t aid most effective where it focuses on the private sector?

Well, no. At least not usually. And a new report from Canada sheds light on why.

The Canadian Council for International Co-operation and the think-tank the North South Institute have assessed the policies of donors towards promoting growth and the private sector. They break down (very adeptly) what the OECD donors are doing, how they’re thinking about the private sector and where they are falling short. It’s not only a sorely needed assessment of the rise and rise of private sector-focused aid, it also reminds us that aid is increasingly promoting a very specific economic model (thy name is neo-liberal capitalism) with some questionable assumptions that should attract serious scrutiny.

You can read the report here but here are some highlights, along with a few thoughts from me.

Just feed the growth machine

Donors are focused on plain old growth. They don’t have a meaningful approach to improving the distribution or pro-poor impacts and too often merely pay lip-service to issues to do with the quality of growth. They’re not linking possible interventions; such as support to government capacity to protect labour rights, effectively collect taxes and redistribute the benefits of growth; with their private sector programmes. In addressing gender, too often they simply focus on getting women into markets, ignoring the social and political drivers of gender inequality. And direct support to the poorest and most marginalised is falling out of favour.

OJ or kool-aid?

While the donors may differ on their rhetoric (some are good at sprinkling in buzz words like “inclusive” and “sustainable” when discussing growth), their private sector work ends up looking eerily similar. French and Belgian aid goes furthest in prioritising equality and accompanying growth with social services. However, for most donors, private sector programmes are effectively stand-alone growth feeders. The report reminds us that sustained growth in many developing countries (particularly in Africa) has failed to put a dent in unemployment. Growth driven by extractives is one such example that’s cited. So just feeding the growth machine doesn’t automatically mean the poor will benefit.

Can aid really drive growth?

The report finds that donors focus on interventions at the macro (e.g. business enabling environment and international CSR support) and meso (e.g. PPPs and financing investments) levels. However, it’s the micro-level programmes (e.g. training women farmers) that “have a much larger redistributive impact for poor and marginalized populations”. But can aid money really be a (meaningful) driver of growth? Why not focus on helping the most marginalised, rather than obsess over catalysing broader economic growth?

Win-win-win-win… always

Public-private partnerships (PPPs) are the new black. For OECD DAC members, PPP spending has risen from US$234m in 2007 to US$903m in 2010. Donor strategies suggest they represent wins for everyone: recipient governments, business, donors and NGOs. But donors seem to ignore the complexity of interests and agendas among those involved in PPPs. The pitfalls and inefficiencies of PPPs are scarcely mentioned and the politics of the PPP negotiating process is swept under the rug. This, I suspect, comes from an ideological underpinning to private sector strategies, which holds that business interests are aligned with development interests. Donors need to be reminded that this is only sometimes the case.

Crony capitalism?

Donors are also looking to include their national firms in their private sector strategies. For example, Australia’s Mining for Developmentprogramme (AU$127m) partners with Australia’s mining giants on a range of projects in developing countries. Denmark’s Business Partnerships programme and the UK’s Food Industry Retail Challenge Fund are only open to national firms. This all seems dangerously like a veiled attempt to subsidise the foreign investment and CSR strategies of national companies – as long as some development story can come about. And isn’t this neo-tied-aid?  In the report’s own words “Donors sometimes favour their own commercial interests to the detriment of developing countries’ domestic policies for development.” Without more transparency around decision-making and a clear results framework for private sector partnerships (the report finds this is sorely lacking), we are left to interpret decisions and priorities as cynically as we’d like (I bags the cynical view).

The report makes a plethora of sensible recommendations, including the need to support democratic ownership of the growth agenda and ensure additionality in private sector development programmes (ie; that an investment, CSR approach or any job-creation wouldn’t have happened even without the programme). But what comes flying off the page is the need to question fundamental assumptions about growth and aid. Some questions that I’m left with are:

• What type of growth should donors be promoting?
• Can aid really drive growth?
• Are aid bureaucrats equipped to do business with business (e.g. through partnerships and PPPs)?
• Why doesn’t aid focus on building the rights and power of the most marginalised?

What do you think?

Bad Governance leads to bad land deals – the link between politics and land grabs

Ricardo Fuentes-Nieva (right) and Marloes Nicholls (left) crunch the numbers to find that big land investments sniff out countries with ‘weak governance’ – aka no accountability, no regulation, no rule of law, and a green light for corruption.

If you had bags full of money and wanted to buy land, where would you go for a good deal? If you’re only looking for ways to make a good profit and control your risk exposure, surely you would look for a place where you can influence the terms of the deal. This is the intuition behind the analysis published yesterday by Oxfam

The results of this analysis show that the global rush for land is mostly taking place in countries with weak governance.  We analysed the link between national governance and large scale agricultural land deals by combining information from two important databases – the Land Matrix[i] and the World Governance Indicator (WGI) Project. To do this, we cross referenced Information on over 200 countries and territories from the two databases. Using the Land Matrix we aggregated the total number of deals reported in each country and their average size; from the WGI, we used estimates of Voice and Accountability, Regulatory Quality, Rule of Law, and Control of Corruption. Once the two databases were merged, we analysed the link between the countries where large-scale land deals were – or were not – taking place and the four governance indicators for the period 2000 – 2011.

The results reveal a strong and significant link between land deals and weak governance. The majority (78%) of the 56 countries where land deals are taking place have below average WGI, and on average the pool of countries where land deals take place have 30% lower indicators than those without. These results are consistent over time.

A quick comparison between two countries shows that the land availability does not appear to be a significant factor in investment decisions. Guatemala, which scores below average on all four World Bank Governance indicators, has seen an estimated 87,000 hectares of land under deals between 2000 and 2011 despite high levels of hunger and malnutrition in rural areas. This is in stark contrast with Botswana which has a similar area of arable land per person (.11 and .13 hectares per person in Guatemala and Botswana, respectively) but which scored well above the average on World Bank governance indicators and did not record a single large-scale land deal in this period.

These results are hardly surprising. Other studies have found similar results. Researchers at the IMF ( here and here), using a different database and methodology, have previously found that “countries with weak land sector governance are the ones most attractive to investors – at least as gauged by the number of land-related investments.” They suggest that investors might pick countries with weak governance because “it is easier to obtain land quickly and at low cost where the existing protection of land rights is weak, given that public protection may not matter to investors who can muster their own resources to defend their property rights.” Research by the World Bank found that deals were often formulated for the benefit of investors rather than the countries involved. They report that “in many cases the nature and location of lands transferred and the ways such transfers are implemented are rather ad hoc – based more on investor demands than on strategic considerations.”

Why Might Weak Governance be Good for Business?

The story behind land deals and weak governance is one of power imbalance and destitution. It’s a story where the interests of local communities are set aside to promote the interest of large investors.

There are usually three actors in any land deal – the investor, the local community who owns or uses the land and the government.  The national government often acts as the intermediary (in wonk parlance, the government is the agent for the local community or the citizens, who are the principals). Weak governance – which basically reflects a gap between the interest of citizens and governments – enables investors to sidestep costly and time consuming rules and regulations, which for example, might require them to consult with affected communities. In countries where people are denied voice, where business regulations are weak or non-existent, or where corruption is out of control it might be easier for investors to design the rules of the game to suit themselves.

This analysis is only the first step towards a more in depth research project. Next steps include a more in depth analysis on the determinants of the number and location of deals (a double-hurdle estimation? suggestions appreciated from econometricians out there). We will also look at the geographical distribution of deals within countries to see if there is a link between the location of land deals in countries and socioeconomic indicators in those areas.

Land is such an important element of millions of people around the world that any issue related to use, access and ownership of it should be carefully analyzed. Agricultural investment is sorely needed but it should not be at the expense of people’s rights and access to land. There are potentially catastrophic implications of bad land deals. Poor accountability and regulation only means that people affected by land deals have fewer tools to defend their livelihoods and rights.

And if this all sounds a bit abstract, here’s what we’re talking about

So What do I take Away from The Great Evidence Debate? Final thoughts (for now)

The trouble with hosting a massive argument, as this blog recently did on the results agenda (the most-read debate ever on this blog) is that I then have to make sense of it all, if only for my own peace of mind. So I’ve spent a happy few hours digesting 10 pages of original posts and 20 pages of top quality comments (I couldn’t face adding the twitter traffic).

(For those of you that missed the wonk-war, we had an initial critique of the results agenda from Chris Roche and Rosalind Eyben, a take-no-prisoners response from Chris Whitty and Stefan Dercon, then a final salvo from Roche and Eyben + lots of comments and an online poll. Epic.)

On the debate itself, I had a strong sense that it was unhelpfully entrenched throughout – the two sides were largely talking past each other,  accusing each other of ‘straw manism’ (with some justification) and lobbing in the odd cheap shot (my favourite, from Chris and Stefan ‘Please complete the sentence ‘More biased research is better because…’ – debaters take note). Commenter Marcus Jenal summed it up perfectly:

‘The points of critique focus on the partly absurd effects of the current way the results agenda is implemented, while the proponents run a basic argument to whether we want to see if our interventions are effective or not. I really think the discussion should be much less around whether we want to see results (of course we do) and much more around how we can obtain these results without the adverse effects.’

There were some interesting convergences though, particularly Whitty and Dercon’s  striking acknowledgement of the importance of power and politics, which are often assumed to be excluded from the results agenda. But what they actually said was

‘Understanding power and politics and how to assist in social change also require careful and rigorous evidence.’

True, but what about reversing the equation? Does understanding the role of evidence in development also require a careful and rigorous understanding of power and politics? They never fully address that crucial point, which is at the heart of Roche and Eyben’s critique.

Both sides (rather oddly, as acknowledged experts in their fields) decried the role of experts. Whitty and Dercon called for ‘moving from expert (i.e. opinion-based, seniority-based and anecdote-based) to evidence-based policy’. Ah, turns out that what is actually being suggested is a move from one kind of expert (practitioners) to another (evidence/evaluation).

As a non number-cruncher I also took exception to their apparent belief that only those who understand the methodological intricacies of different evaluation techniques are eligible to pass judgement. On that basis politicians would be out of a job, and only rocket scientists would get to pronounce on Trident.

There was also a really confusing exchange on the hierarchy of evidence. Whitty and Dercon show a surprising (to me at least) commitment to multi-disciplinarity: ‘Methods from all disciplines, qualitative and quantitative, are needed, with the mix depending on the context….. it is not a matter of just RCTs, but of rigour, and of combining appropriate methods, including more qualitative and political economy analysis.’

Music to the ears of the critics, but is it actually, you know, true? Everything I hear from evaluation bods is that DFID does actually see RCTs as the gold standard, and other forms of evidence as inferior. Roche and Eyben returned to the attack on this in their response, arguing that what Whitty and Dercon call the ‘evidence-barren areas in development’ are only barren if you discount sociology and anthropology, among others, as credible sources of evidence. By the way, Ed Carr has a brilliant new post on the (closely linked) clash between quants and quals, arguing that while quants can establish causation, only quals can explain how that causation occurs.

But the exchange did provide me with one important (I think) lightbulb moment. It was about failure. Whitty and Dercon were particularly convincing on this: the evidence agenda ‘involves stopping doing things which the expert consensus agreed should work, but which when tested do not’. This is a nice Popperian twist – the role of evidence is not to prove that things work, but to prove they don’t, forcing us to challenge received wisdom and standard approaches. This is indeed what I noticed about Oxfam’s recent ‘effectiveness reviews’ – if you find no or negative impact, then you (rightly) start to re-examine all your assumptions. But if this is the proper role for the evidence agenda, is it politically possible? By coincidence I have just read Ed Carr’s forceful critique of Bill Gates’ approach to evaluation, arguing that failure is often airbrushed out in order to safeguard funding and credibility. That seems a pretty fundamental contradiction.

The comments were just as thought-provoking. One of the key messages that emerged is the gulf between these debates and what those in charge of gathering results in aid agencies actually face – highly constrained resources, crazy time pressure, and the need to deliver some (any!) results to feed the MEL machine. Oxfam’s Jennie Richmond reflected on the gap between theory and practice yesterday.

Commenter Enrique Mendizabal asked whether we are demanding a different role for evidence in poor countries than in our own.

‘In the UK, health policy is decided by a great many number of factors or appeals (evidence, sure, but also values, tradition, biases, political calculations, etc). We may complain about it but we accept that it is a system that works. But health policy for Malawi (or other heavily Aid dependent countries) is decided mainly by evidence (or what often passes as evidence at the time) and usually by foreign experts…. would we be happy with USAID funding a large evidence-based campaign to reform the NHS or our education policy?’

But he took his argument a step further – if the final decision should be left to the interplay of evidence (of different sorts), politics and negotiation, then DFID and other donors would be better advised to boost the ‘enabling environment’ for such debates and decisions by investing in tertiary education in developing countries:

‘strengthening economic policy debate is a more adequate objective than achieving policy change (even if it is evidence based).’

Commenter David highlighted a fundamental point that rather went missing in the initial exchange – how the results agenda does or doesn’t work in complex systems:

‘The results agenda approach tends, by presenting development as objectively knowable if broken down into discrete and small bits, todrive attention toward small, more easily measurable interventions to test, particular those that are suited to situations that are simple or complicated rather than complex. Current processes around evidence-based results fail to grapple with complex systems, interaction effects, and emergent properties that dominate most aid project landscapes.

A fundamental critique of the evidence-based revolution is that it actually diminishes efforts to get rigorous evidence about addressing complex challenges. We all want evidence, it’s a question of whether the current framing of “evidence-based” is distorting what types of evidence we gather and value. For those who think that the current emphases on methods to test what works are distorting how we value the evidence coming in (RCT=gold, qualitative methods=junk), this offers little other than platitudes about lots of other methods existing.

Personally, I would be a bigger proponent of the evidence-based revolution if it was coming to folks interested in power, politics, and development, and asking them what their questions are and what evidence might contribute to their work. Absent a learning agenda set to fit complex space and concern itself with power, it will continue to seem to me to be an instance of methods leading research – or searching for keys under the light rather than inventing a flashlight.’

To be fair, Roche and Eyben explicitly chose to focus on the politics of evidence, rather than the implications of complex systems (for example, the question of external validity in complex systems – or lack of it – raised by Lant Pritchett in our recent conversation.)

Final thoughts? After about 500 votes, the poll went narrowly to Whitty and Dercon (34% v 31% for Roche and Eyben, with a pleasing late rally for the ‘totally confused’ camp – my natural habitat). I think Chris Roche and Rosalind Eyben need to work on their communication style (more punchy, less abstract, more propositional). Chris Whitty and Stefan Dercon should give some examples of gold standard anthropological or sociological evidence to allay the doubts over their true commitment to multi-disciplinarity, and take the complex systems question more seriously.

A massive thankyou to all who took part, and please can you come back for another go in a year or so? This one isn’t going away.

Theory’s fine, but what about practice? Oxfam’s MEL chief on the evidence agenda

Two Oxfam responses to the evidence debate. First Jennie Richmond, (right) our results czarina (aka Head of Programme Performance and Accountability) wonders what it all means in for the daily grind of NGO MEL (monitoring, evaluation and learning). Tomorrow I attempt to wrap up.

The results wonkwar of last week was compelling intellectual ping-pong. The bloggers were heavy-hitters and the quality of the comments provided lots of food for thought. However, I was left wondering what it all meant for those of us who work in NGOs, trying to generate and learn from ‘evidence’ on a daily basis. I found myself unable to simply vote, so instead I blog….

The results and evidence agendas have brought some real benefits to NGOs in my view. First and foremost, it is important and right that those of us who claim to work in the interests of the poorest people in the world and are stewards of other people’s money, should set ourselves high standards for our own impact. In its simplest form the results agenda asks us to justify the trust others have placed in us, by demonstrating whether we are actually bringing about positive change. In Oxfam GB, accountability has long been held as a core organisational value. It is not the results and agenda that has got us thinking about how to capture and communicate our effectiveness, but it has provided a helpful additional push.

A further positive is that space has been created both within our own organisations and in the wider sector, to stop, listen and learn. MEL-istas (as Duncan calls us) 5 years ago struggled to get the ear of senior managers (let alone Ministers). But the results agenda has increased the stakes around MEL – encouraging organisations not only to increase investment, but also to listen to the findings coming from our own data gathering and analysis.

However, it has also increased the demand and the expectation, which are not easily met by all NGOs. In Oxfam GB the investment in MEL has increased over the last couple of years, undoubtedly, but still it is a real stretch to deliver the ever-more ambitious demands from donors, to develop tools to tell the story of our broader organisational impact, and to ensure that we are developing innovative ways of measuring cutting-edge programming areas, such as resilience, enterprise development and influencing.

And we are one of the largest international development NGOs in the UK. How much more difficult for the smaller and niche NGOs, or those who lack the flexible financing that permits investment in MEL and innovation? We are conscious in Oxfam that we and other large NGOs need to guard against distorting the NGO market place by pushing the boundaries on MEL and impact too far, and thereby creating expectations that cannot be met by everyone. Somehow we all need to keep our sights on a proportionate approach.

It is not just important to generate evidence, but also to use it properly. There is increased demand for serious, evidence-based conversations about what works.  None of us can get away with decisions made purely on gut instinct, force of habit or ideological leaning. We are challenged by the ‘evidence’ question to collate and distil from the broad knowledge base we have at our disposal. And this has in some cases led to surprises. Rigorous studies, whether based on qualitative or quantitative methods, can challenge our preconceptions – showing us impact where we were not optimistic, or the opposite. The test, of course, comes when new programmes are designed. Will the body of evidence be applied – will we be able to find it for starters (in our often not-so-state-of-the-art knowledge management systems), and will it be politically acceptable in our own organisations to apply it to practice?

So, how can we use the results and evidence agendas and make them useful to us as NGOs?  We need to do this in a way that a) is true to the actual work we do (which in the case of Oxfam includes a great deal of work that drives for political change and influencing) and b) does not distort decision-making away from the right decisions (i.e. what most suits the specific needs and opportunities of each context) in our efforts to be able to measure and communicate what we are doing.

One of the concerns raised in last week’s blog was that in some institutions, evidence becomes synonymous with impact evaluations, and even specifically with Randomised Control Trials. As all the bloggers agreed, the default use of one research method for interventions of all types is simply nonsensical. You only have to look at the enormous variety of the things we do in international development (from campaigning for policy change to delivery of bed-nets, from building of bridges to raising awareness of the rights of citizens) to realise that one approach is just not going to cut it.

Another challenge is that so much of what we do in international development is extremely hard to measure. How can we trace the input through to impact chain and clearly demonstrate the ‘on the ground’ changes we have brought about in people’s lives when the investment is in budget support or core funding?  How can we reduce the process of a community standing up against acts of violence against women to a Value for Money calculation? The ethical dilemmas and practical difficulties wrapped up in measuring and ‘evidencing’ many of the processes we are involved in are huge. And, as Eyben and Roche point out, much of what we engage with in international development is messy and political. We need to make sure that the tools we have at our disposal for evidence generation are sophisticated and nuanced enough to acknowledge this messy political reality, and that we are sharing ideas on how to do this in a practical and affordable way.

The push for evidence should go hand in hand with a more entrepreneurial approach to development, opening up space for honest

MEL that - US military mindmap of Afghanistan

reflection on both success and failure. That is the theory. But, of course, there are obstacles to this becoming a reality. Our systems in large institutions, including NGOs, are designed to demonstrate success. We all have our logframes and our KPIs, and we want to be able to put a tick in the box. No-one wants their project to be the one famous for not achieving what it set out to do, even if the real story is that it helped enormously to generate learning for future projects. Complexity thinking is having some influence right now, which helps to raise the right questions about process and incentives. However, we have a long way to go before even in the most reflexive learners in NGOs and other development institutions want their project to be hailed as the great failure.

So, we proceed with caution – welcoming the increased space the Results Agenda provides to consider ‘what seems to work’, and the profile it gives to the need to take a thorough and transparent look at the information coming out of our programmes. But, wary of the dangers of distorting what we do in order to make it measurable; of placing the MEL ‘bar’ for NGOs too high to reach; of the over-emphasis of certain methodologies; and of the danger of ignoring political realities in the work that we do. It is certainly helpful to keep reflecting and questioning, however, from all sides of the debate – so the wonkwar of last week was welcome.

Why do some (better) alternatives to GDP get picked up, while others sink without trace? Useful new study on political economy of indicators.

Took me a while to overcome my reluctance to read a document subtitled ‘Deliverable 1.1’ on the front page (yuk), but I’m glad I did so. The paper ‘Review report on Beyond GDP indicators: categorisation, intentions and impacts’ (cracking title too…..) is published by BRAINPOoL – ‘Bringing Alternative Indicators into Policy’ (is that the sound of teeth grinding?).  The authors come from Eurothinktanks CUEC (Czech Republic) and the new economics foundation (UK).

But enough of the snark, because the paper is really good. Rather than add yet another indicator to the smorgasbord of measures that go ‘beyond GDP’, it tries to understand why some of these indicators get picked up, while others sink without trace. Asking the right question, as Einstein said, is a lot more than half the battle.

The researchers sought to answer that question with a mix of desk research, interviews with indicator nerds, and media analysis. First they tried to categorize the ever-growing list of alternative indicators. They came up with:

• Level of impact – international, national, local
• Indicator domains – environmental, social, economic
• Indicator approaches – subjective, objective
• Indicator types – single indicator, set/dashboard, aggregated, composite
• Envisaged users – politicians/policy makers, public, experts
• Link to GDP – adjusting GDP, “replacing” GDP, supplementing GDP

Media analysis showed that single figure indicators get far more pick up than complex dashboards, as do those that enshrine ‘simple and meaningful concepts’ like the Human Development Index or Ecological Footprint, which lead the pack by some way (see chart).

But a media splash, while useful, is not the main purpose of the indicators studied, which was overwhelmingly to influence policy. When have they achieved it and why?

• The most basic influences noted were transmission to or reference by a decision maker.
• The next step up from this is use of Beyond GDP indicators in assessment. Examples of this include GPI, ISEW and QUARS being integrated into regional or local official assessment frameworks in the US, UK and Italy. Also noteworthy is the Ecological Footprint being used to set environmental impact targets in several national governments.

• Actual policy changes connected to indicators were also reported. These include (i) the implementation of schemes to improve habitat for farmland birds after changes to the UK Defra Sustainable Development Indicator set (ii) the investment of $15 billion by the UAE government to develop alternative energy sources spurred on by a poor ranking in the Ecological Footprint.
• The most prolific successes, however, were achieved by local level indicators, perhaps because the distance between producer and user of the indicators is much smaller, making it easier to achieve a better ‘fit’ while also achieving legitimacy and relevance. For example, the Jacksonville Community Council Indicators are believed to have been instrumental in reducing infant mortality rates, recidivism and water pollution in the area. The Council of Europe SPIRAL project, meanwhile, achieved change to the delivery of food relief in France and the transformation of the approach to homelessness in Belgium.

And what lessons did the authors draw from all this?

“Indicators were successful when they had real relevance for policy makers. Crucially they need to measure something that policy makers believe they can influence. Subjective well-being indicators when used incorrectly can appear to fail this test, which explains efforts to build the evidence base for how policy can influence wellbeing. Another factor here is cost. In the current climate, indicators that provide clues for low cost policies, or indeed those that can help save money, are of particular interest.

Salience for a broader audience is also crucial and entails the elements of simplicity, understandability and good communication. Initiatives are effective when they allow one to produce a simple and attractive message that relates a meaningful concept. Using communications experts and avoiding taboo words were also identified as being important.

Indicators need credibility and legitimacy. Aside from the requirement of quality data, the appearance of neutrality was seen as the best route to achieve this with some interviewees comparing advocacy organisations’ data unfavourably with that of National Statistical Offices.

Developing the indicators with the audiences at whom they are targeted and/or encouraging participation (in the way that the Jacksonville Community Council Indicators initiative has done) is also seen as a key success factor. The importance of relationship-building also applies to policy makers with most initiatives that had achieved policy success citing direct face-to-face channels as vital.

Several barriers to the success of Beyond GDP indicators have also been identified. The economic crisis was viewed by many interviewees as a challenge for this agenda as it has moved the policy focus. Ideology and vested interests are also noted barriers with subjective well-being and composite indicators receiving strong resistance from those with libertarian or right of centre political views.”

Not sure I buy that last point – isn’t a crisis just as likely to create opportunities for adoption of new indicators, especially if the failings of existing indicators are seen as partly responsible for the crisis? I remember being struck by a presentation from the South Korean statistics office (I really do have all the fun), arguing that Korea was adopting a new quality of life index, based on over 100 objective and subjective indicators of wellbeing, because GDP was incapable of explaining why an ‘economic miracle’ like Korea had the highest suicide rate and the lowest birth rate in the OECD. Failure is a great source of innovation.

The Brainpool project continues and is worth following (especially if they ditch the naff titles). Oxfam is getting involved, working with nef in Brazil and India to explore likely reactions by politicians and voters to possible indicators. It will build on Oxfam Scotland’s Humankind Index, which was an earlier foray into this field. I hope the research includes applying a more serious power analysis to the findings in this report: what fractions of the state, private sector, political class or popular movements adopt new indicators, either individually, or in coalition with others? When and why? That could get very interesting.