Lant Pritchett v the Randomistas on the nature of evidence – is a wonkwar brewing?

Last week I had a lot conversations about evidence. First, one of the periodic retreats of Oxfam senior managers reviewed our work on livelihoods, humanitarian partnership and gender rights. The talk combined some quantitative work (for example the findings of our new ‘effectiveness reviews’), case studies, and the accumulated wisdom of our big cheeses. But the tacit hierarchy of these different kinds of knowledge worried me – anything with a number attached had a privileged position, however partial the number or questionable the process for arriving at it. In contrast, decades of experience were not even credited as ‘evidence’, but often written off as ‘opinion’. It felt like we were in danger of discounting our richest source of insight – gut feeling.

In this state of discomfort, I went off for lunch with Lant Pritchett (right – he seems to have forgiven me for my screw-up of a couple of years ago). He’s a brilliant and original thinker and speaker on any number of development issues, but I was most struck by the vehemence of his critique of the RCT randomistas and the quest for experimental certainty. Don’t get me (or him) wrong, he thinks the results agenda is crucial in ‘moving from an input orientation to a performance orientation’ and set out his views as long ago as 2002 in a paper called ‘It pays to be ignorant’, but he sees the current emphasis on RCTs as an example of the failings of ‘thin accountability’ compared to the thick version.

In a forthcoming paper (which I will definitely link to when it’s published), Lant defines thick accountability as ‘an “account” in the sense of a justificatory narrative of my actions, the story of my actions I tell to those whose opinion of me is important (including myself, but including family and kinsmen, friends, co-workers, co-religionists, people I respect and desire admiration from) that explains why my actions are in accord with, and deserving of, a positive view of myself.    In contrast, thin accountability is “accounting”, which is that small part of the account about which objective facts can be established.’  He sketched out the inevitable 2×2 matrix for me

The challenge in most development work is to move from top left to bottom right. There are occasions when thin accountability/high performance works – typically routine functions like delivering mail or building roads. But anything involving the messiness of people and institutions requires thick accountability, involving deep bonds of trust and reciprocal relationships that are likely to be defined by a setting’s unique history and geography – what he calls ‘folk practices, from which formal organizations can (re)emerge’.

He argues that the randomistas just don’t get this. His critique of RCT culture ranged pretty wide:

• The politics of RCTs: ‘RCTs are a tool to cut funding, not to increase learning.’  ‘Randomization is a weapon of the weak’ – a sign of how politically vulnerable the argument for aid has become since the end of the Cold War. ‘Henry Kissinger wouldn’t have demanded an RCT before approving aid to some country.’ And I can’t see the military running RCTs to assess the value for money of new weaponry before asking for more cash (mind you, if they did, that might at least save some money on Trident….).
• The lack of interest in theory: ‘the randomistas are going back to alchemy – atheoretic experimentation’.
• RCTs test at most a few project variants using ‘project vs non-project’, whereas interventions are typically multiple, overlapping and synergistic (i.e. the whole cannot be reduced to a sum of parts).
• No-one evaluates the evaluators. At the very least, given how much RCTs cost, you need to know that the findings are useful elsewhere (so-called ‘external validity’). But once you have multiple RCTs on the same issue (and their spread is starting to produce such comparable studies), you find very little external validity – the results of an RCT in one country and time are not replicated elsewhere (with the possible exception of deworming in schools, but even that iconic RCT story is contested). This is the big contrast with real science, where replicability is a key condition of validity.

Patronising? Overpromising? Nah....

In another recent paper, he argues instead for ‘structured experiential learning’, which involves rigorous and intelligent conversation, rather than the illusory certainty of numbers. Get people in a room, agree what the problem is, agree to try out some experiments to solve the problem, and set up rapid feedback to identify failure and/or build on success. In another recent paper, he calls this ‘Problem Driven Iterative Adaptation (PDIA)’. It sounds very similar to the conclusions of the Africa Power and Politics Programme, which I reviewed recently. In yet another paper (he’s horribly prolific), he also draws a neat distinction between experiments and experimentation:

‘Perhaps surprisingly, the experimentation and experiments approaches are not at all the same. I argue that experiments, while a terrific method for generating PhD dissertations and published papers, will have impact on development and development practice only insofar as they are embedded in an experimentation approach (which they are often not).’

The feeling I got from these conversations was of two tribes encamped and preparing for battle. That line from Henry V comes to mind: ‘from camp to camp, through the foul womb of Boston night, the hum of either army stilly sounds.’ On one side are the ‘best fit’ institutionalists and complexity people, with their focus on path dependence, evolution and trial and error. On the other are the ‘universal law’ experimentalists, offering the illusory certainty of numbers, and (crucially) comfort to the political paymasters seeking to prove to sceptical publics that aid works. It’s hard to see how they can both be right, or happily coexist for long. Time for a wonkwar on this blog, I think…..

How can a post-2015 agreement drive real change? Please read and comment on this draft paper

The post-2015 discussion on what should succeed the Millennium Development Goals (MDGs) is picking up steam, with barely a day going by without some new paper, consultation or high level meeting. So I, along with Stephen Hale and Matthew Lockwood, have decided to add to the growing slush-pile with a new discussion paper. We want you to read the draft (see right) and help us improve it. Contributions by 5 November please, either as comments on the blog, or emailed to research[at]oxfam.org.uk.

The paper argues that there’s an urgent need to bring power and politics into the centre of the post-2015 discussion. To have impact, any post-2015 arrangement has to take into account the lessons of over a decade of implementing the existing MDGs, and be shaped by the profound global change since the MDGs were debated over the course of the 1990s and early noughties.  We’re hoping that this will be at the centre of this week’s discussions in London linked to the High Level Panel and in Berlin at the Berlin Civil Society Center on Development post 2015.

The most significant shift is that the new arrangements have to be designed to influence governments, whereas the main impact of the MDGs was on the aid system. Why the shift? Because aid is becoming less important, both because it is likely to decline in volume over the next few years, and because governments’ dependence on aid as a percentage of revenues is falling even faster than aid itself. In any case, aid is a pretty ineffective way of influencing government behaviour, beyond the actual expenditure of donor dollars.

So if influencing governments is the goal, what can we learn from the experience of the MDGs? The first thing to note is a startling lack of research. Many reviews blur the distinction between ‘MDGs’ and ‘MDG policies’/’MDG planning’ (in effect, social welfare). Analysis of the data on improvements in health, education, and other key sectors largely ignores the vital question of how much of that improvement can be plausibly attributed to the MDGs, rather than to other factors such as national politics, economic growth, or technological innovation. Given the substantial political and financial investment in the MDGs, and the need to design an effective post-2015 framework, being unable to attribute – with any certainty – progress due to the MDGs is a truly lamentable gap in our knowledge.

There is even less research on (and less anecdotal or circumstantial evidence for) the impact of the MDGs on the policies and behaviours of rich countries, beyond changes in their aid budgets. There is scant evidence that MDG 8’s commitment to a ‘global partnership for development’ has had any impact on rich country behaviour. Understanding this failure is vital, given that many proposals for the post-2015 regime seek to place more obligations on rich countries in areas such as climate change and resource consumption.

What we know is that some governments have adopted the language of the MDGs and have customized them to fit national priorities, while civil society groups have increasingly used them as advocacy tools.

Beyond that, many post-2015 participants seem to think it is not possible to give a more complete answer to the traction question because of the missing counterfactual (how can we know what would have happened without the MDGs?). Not so. It is certainly possible to know much more than we do about attribution through more rigorous qualitative research. For example, in-depth interviews with policymakers could investigate the traction exerted by a range of external and domestic forces on their decisions (avoiding any leading questions on the MDGs). We have yet to locate such research.

So much for the MDGs, what about whatever comes next? International instruments can exert influence in three key ways:

1. By changing national norms in areas such as women’s rights. However intangible, norms matter, leading to long-term changes in what society considers acceptable or deplorable, which then leads to changes to laws, policies and behaviours.
2. By directly influencing government decision making, through any of a number of possible carrots (aid, contracts, acceptance, approval) or sticks (sanctions, disapproval).
3. By giving civil society organisations and other domestic actors more tools with which to lobby, campaign, and secure action by their governments.

In most cases, the main drivers of change will be domestic – the result of national politics and culture. But international initiatives are second-order factors that can nudge things along. We identify six kinds of instrument at global and regional levels.

Big global norms: rallying cries intended to influence the underlying attitudes of decision makers and citizens, such as ‘zero poverty’ or ‘zero hunger’.

Global goals and targets: as encapsulated by the MDGs.

Regional goals and targets: the African Union has been particularly energetic in agreeing regional targets, setting out what its member governments should be aiming for on the Rights of Women (AU Protocol, 2003),or their allocation of spending to agriculture (Maputo Agreement 2003), health (Abuja Declaration 2001) and similar commitments on social protection, and water and sanitation.

Global league tables: the international community and/or civil society can simply collect and publish data allowing a comparison between different countries’ absolute situation and rate of progress, as in the UNDP’s Human Development Index. Anecdotal evidence (and long NGO experience) suggests that league tables can be effective both in attracting public and media interest, and in goading politicians into action – there is nothing a leader likes less than to be seen to lose out to a rival nation.

Data transparency: according to some architects of the MDGs, perhaps their greatest legacy will be the improved quality, collection and dissemination of social data. One option would be to make this the centrepiece of a post-2015 arrangement, and leave it to others (national or regional bodies, international institutions) to ‘mash up’ the data into different indices and use it to advocate for progressive policies.

International law: Most governments are already signatories to dozens, if not hundreds, of international conventions and the role and influence of international law appears to be on an inexorable upward curve, steadily encroaching on previously untouchable areas of state sovereignty.

What are the strengths and weaknesses of these options in influencing norms, decision making or civil society activism? Here we are basically into guesswork/gut feeling, captured in the table below. We’d be interested to hear your views, and very grateful for links to any relevant research.

Possible options for international instruments to drive change post-2015

Over to you for comments, links etc

Why don’t Africa’s politicians invest more in small farmers? The political economy of ag policy.

Interesting if rather impenetrable new(ish) paper from the Future Agricultures consortium on the political economy of Africanagricultural policy. It seeks to answer an important question – why hasn’t the spread of democracy produced more investment in the smallholder farmers that form the majority of the electorate in many countries? Here’s the summary:

“Theories of policy neglect of, or discrimination against, agriculture in Africa include urban bias and the narrow self-interest ofautonomous elites. Whilst structural adjustment removed much of the previous tax burden on African agriculture , the sector also saw declining investment from international development partners and through national budgets. Whilst there has been some recovery in public investment in agriculture over the past decade, signalled by the 2003 Maputo Declaration, investment in the infrastructural and institutional public goods needed to support smallholder-led agricultural growth remains disappointing. As a result, the contribution of the agricultural sector to growth and poverty reduction objectives in Africa is widely believed to have been below potential.

In theory, democratisation, which has proceeded unevenly across Africa during the past two decades, should encourage pro-poor agricultural policy, as the majority of voters in many countries remain rural and poor. This paper draws on case studies of recent policy change (attempted and actual) in eight African countries, plus an analysis of the political systems in these countries, to explore the evolving role of competitive electoral politics in agricultural policy making.

An important observation is that politicians are as likely to rely on ethnic allegiances and forms of social or political control to secure votes as they are to engage in policy competition. Moreover, the political incentives facing senior policy makers in the agricultural and rural development sphere may be inimical to the development of strong institutions to promote smallholder agricultural growth. Instead the paper finds that it is exogenous factors – macroeconomic dependence on agriculture and, most strikingly, sustained threats to regime survival – that create positive incentives for agricultural investment, even where social or political control is relied on to secure votes.”

So what I think Colin Poulton, the author, is saying is that since many African leaders don’t rely on having good policies to win elections, it doesn’t matter much whether those policies are popular with voters. As he later explains:

‘Critically, the argument that democratisation may strengthen political incentives for policy support to smallholder agriculture in Africa assumes that politicians primarily exchange policies for votes. Drawing on existing literature and the country case studies, the paper  argues that this assumption does not hold in most of Africa.’

In fact, they only start worrying about investing in agriculture where, as in Burkina, Malawi or Ethiopia, it is particularly vital to generate the foreign exchange they need to run the government (or buy a new Mercedes), or the whole government is under actual or potential military threat (Rwanda, Ethiopia). When those moments arrive, then political leaders are more likely to remember small farmers and listen to well-meaning technocrats, but at other times they will largely ignore them.

There are a few exceptions like Ghana, where policies do seem to matter more (some great policy debates going on in the current election campaign on issues such as healthcare).

Can civil society help fill the democratic deficit and generate the pressure for change? According to Poulton:

‘In Latin America mobilization of poor groups by diverse social movements (Vanden 2007) arguably began to exert an influence on electoral outcomes around 15 years after the return to democracy. However, there are various reasons why such mobilization might take much longer to achieve in much of Sub-Saharan Africa than in Latin America, including deeper levels of absolute poverty, lower education levels (although recent increases in school enrolment are a positive sign in this respect) and little prior history of awareness raising amongst the rural poor in Africa (some early cooperative development activity aside).’

Your thoughts?

What can political economists tell us about Africa, aid and development?

This post also appeared on the World Bank’s ‘People, Spaces, Deliberation’ governance blog

There’s a clutch of different research initiatives trying to understand Africa’s political economy and its impact on development and aid. Often, the tone of the political economists can be quite discouraging – Alex Duncan gives a tongue-in-cheek definition of a political economist as ‘someone coming to explain why your aid programme doesn’t work’. There are few practical ‘take aways’ either for large bilateral aid agencies, or NGOs other than ‘give up and become a researcher’.

And that’s pretty much the tone of a logotastic ‘joint statement’ from 5 research programmes based (loosely) in the UK, Denmark, and the Netherlands (The Africa Power and Politics Programme, Developmental Leadership Programme, Elites, Production and Poverty: A Comparative Analysis, Political Economy of Agricultural Policy in Africa, Tracking Development). Here’s some highlights:

From the summary:

“African countries badly need to embark on processes of economic  transformation, not just growth, and they are not helped to do so by insistence on  prior achievement of Good Governance, meaning adoption of the institutional ‘best  practices’ that have emerged in much richer countries.”

From the full statement:

“Our single most important message is that development outcomes in poor countries depend fundamentally on the political incentives facing political elites and leaders….. Because of the way democratisation affects politicians’ incentives in poor developing countries, the introduction of competitive elections is a mixed blessing for achieving the economic transformation that Africa needs.”

“The reasons [a number of South-east Asian countries] achieved sustained, pro-poor growth [and Africa has not] over the 50 years since 1960 are mostly about policy differences. During the early decades of the period, Indonesia, Malaysia and Vietnam invested heavily in rural development, driven by urgency, outreach and expediency. They did so under a variety of political regime types, none of which were free of major corruption. They made some progress towards democratisation only after achieving  substantial economic  transformation.”

The paper’s ‘big idea’ is that “What shapes the ability of policy to drive economic transformation is the extent to which mutual interests, cooperative relations and synergies emerge between three  large groups of actors. [see diagram] [Usually] the relationships are not mutual, cooperative and synergistic, but antagonistic, exploitative and perverse. [But the key to improving aid practice is] understanding exactly how and why exceptions occur.”

“Because politicians are typically constrained to generate and use rents to cement their alliances,  ‘good politics’  can  result in ‘bad economics’” Elites need cash to funnel to their supporters and so have to milk the state for short-term rents, rather than investing in the future, as the Southeast Asian elites did, (supporting pioneer firms, building roads, providing health and education etc). The trouble is that in such a system “the introduction of formal multi-party competition into such an environment does not alter the basic logic. Clientelism in Africa is to a greater or lesser extent competitive under both authoritarian and more democratic regimes…. Typically, multi-party elections formalise and sharpen this competition with often mixed results for development.”

The paper identifies two broad kinds of exceptions to the clientilist rule:

Big-picture exceptions: In a somewhat desperate search for developmental states in Africa, the paper comes up with “the early-independence regimes of Houphouët-Boigny (Côte  d’Ivoire), Kenyatta (Kenya) and  Banda (Malawi) [and today,] Ethiopia and Rwanda.”  These have all “achieved centralised rent-management [thus freeing them from the distractions of competitive clientilism] and this led to significant economic transformation and social advance for a period.” Ah, so the best way to improve on competitive clientilism is to eliminate the competition, not the clientilism.  Oh dear.

Small-picture exceptions: successes in Asia were in many instances the result of breakthroughs in particular sectors or commodity chains which only later became generalised… [African examples include] sugar in Mozambique and dairy in Uganda.”

What does all this mean for aid donors (the authors are basically talking to the big money donors, not relative minnows like Oxfam)?

“The central message that needs to be got across is that the conditions which keep the African masses in poverty are the result of decisions by politicians who are responding to incentives  that change slowly and are not in the  short  term very  favourable to development. More immediately, they stem from the inability of sector actors to overcome their collective action problems in the face of unsupportive if not predatory state behaviour. They cannot, therefore, be addressed by merely transferring economic resources from the global rich to the global poor. Indeed, such attempts can make matters worse, by further weakening those political incentives that work in favour of domestically driven economic transformation.

If this is true, aid needs to become far less supply-driven and more focused on supporting processes that show real promise, based on an informed assessment of the local situation and the lessons of history. In particular, there should be no implication that donors know best what institutions poor countries need.”

The political reality check is excellent, but the lack of genuine ‘so what’s’ (despite the paper’s protestations to the contrary) is a real problem. As is the casual abandonment of human rights, democracy, citizenship and social struggle. Citizens, social movements, the Arab Spring (see left) barely get a condescending mention in all this high level grown-up talk of governments, elites, donors and political settlements.

Still, for NGOs who usually prefer Gramsci’s ‘optimism of the will’ to his accompanying ‘pessimism of the intellect’ (this paper suffers from the opposite problem),  it should be at least food for thought that a cold-blooded look at Africa’s recent history persuades a group of serious academics that the best Africa can hope for is benign dictatorship, epitomised by Rwanda’s Paul Kagame, which delivers “social provision under a centralised rent management regime [and] illustrates how a  powerful upward accountability of public service-providers and local administrations  to the national political leadership can remarkably improve service quality, more  than adequately substituting for the downward accountability to users that tends to  be stressed in donor rhetoric.”

What’s more, I think that a more progressive reading of this work is possible – outsiders must give up trying to impose blueprints, and concentrate on spotting and supporting positive developments as they emerge (with many of them coming from the very social movements that this group ignores or dismisses). But that will be of little comfort to the big official agencies, which are too big and too politically constrained to pursue that more entrepreneurial approach. They are left with what I’ve previously termed the ‘decent chap’approach – picking political winners by “focusing  particularly  on sectors  or areas  where  government  attitudes are positive for  political reasons and  sector  actors have  revealed a collective ambition to move ahead.” Shame Paul Kagame says he doesn’t really want more aid.