What is the evidence for evidence-based policy making? Pretty thin, actually.

A recent conference in Nigeria considered the evidence that evidence-based policy-making actually, you know, exists. The conference report sets out its theory of change in a handy diagram – the major conference sessions are indicated in boxes.

Conclusion?

‘There is a shortage of evidence on policy makers’ actual capacity to use research evidence and there is even less evidence on effective strategies to build policy makers’ capacity. Furthermore, many presentations highlighted the insidious effect of corruption on use of evidence in policy making processes.’

i.e. you can have all the arguments you like on the nature of evidence – disciplinary and political bias, what constitutes knowledge etc etc (as this blog recently did), but policy makers are often either unable or unwilling to use it anyway – supply doesn’t guarantee demand.

The aid agencies and research councils that fund research are very keen to promote this shift from worrying about supply to wondering how to boost demand (although the researchers are often less keen – they just want to be left alone to churn out papers and develop their careers). What was nice about this conference was the amount of on-the-ground grassroots research on how decision makers actually use (or more often ignore) research in places like Nigeria (‘political manipulation and ambition seem to be among the strongest determinants of factors influencing policy development processes’) and Indonesia (‘Even if technocratic or political – it doesn’t matter – it’s 90% personality’).

One thing I learned is that agonising over per diems is not confined to the aid business:

‘One particularly heated debate concerned the frequent requests from policy makers for ‘sitting fees’ in order to attend training or seminars which could inform them about research issues. Participants agreed that this practice is widespread in most of the African countries represented; however, opinions on how to respond to this differed. Some suggested that those who aim to inform policy makers about research need to just accept that paying these fees is necessary and should therefore include them in their budgets. However others felt that continuing to pay such fees just propagates the problem and that those funding research communication and uptake work should take a ‘zero-tolerance’ approach.’

On the demand side, the report considers both capacity and incentives. On capacity ‘most people don’t know what they don’t know!’ will resonate with researchers in NGOs trying to convince their colleagues to look harder at the evidence. There’s a mountain to climb: a survey of Zambian parliamentary researchers and librarians (and these had positively agreed that they needed to use research) found that ‘only one in three believed there was consensus that the CIA did not invent HIV’.

‘Research-evidence is often used opportunistically to back up pre-existing political decisions/opinions (confirmation bias)’. That preference for policy-based evidence-making is alive and well in the big aid donors and NGOs too, of course……..

And unfortunately, research from Ghana, Sierra Leone, Uganda, and Zambia concluded that ‘a lack of capacity to understand research was perceived as beneficial to policy makers since it ‘allowed’ them to ignore evidence and instead follow their own agenda. Thus, there is not only a lack of capacity but also a disincentive to build capacity.’ Oh dear.

How to build the incentives to use research, assuming these political obstacles are not insuperable? On HIV policy in Pakistan, DFID ‘built the capacity of civil society organisations representing marginalised groups to demand policy change’.

Other useful tips:

• including policy makers in the design phase of research projects (get them on the advisory board, guys, don’t just see them as seminar fodder once you’ve finished the research)
• networks and linkages between researchers and policy makers are necessary but definitely not sufficient
• researchers need to change the (often dire) way they communicate their work – in one case study from Ghana ‘photographs of real people suffering from mental illness is far more powerful in influencing opinions than any policy brief could be.’ (Well duh)
• target the ‘policy entrepreneurs’ with influence over decision-makers (the Minister’s old university professor etc)
• ‘There is a tendency for researchers and research intermediaries to focus their communication efforts on elected representatives and appointed officials but to ignore the crucial role that technocratic staff play.’

All good stuff, but the report reminded me of the governance debates of a few years ago, in that even though it recognized the problem is incentives and politics, kept drifting back to the comfort zone of supply issues (if they don’t want research, we just have to get better at communicating or building their capacity), rather than thinking harder about the demand side. For example:

• Anyone involved in advocacy knows that the openness of policy makers to new ideas is episodic, and linked to things like changes of administration, scandals, crises and failures. So how does research need to be redesigned to capitalise on such brief windows of opportunity?
• Opposition parties are often much less well resourced, and much more malleable in their thinking as they cast around for clever ideas that will help them win power – to what extent should researchers concentrate on those without power, rather than those currently in office?
• Young minds are (generally) more open to new ideas than old ones: should researchers target future leaders (who are pretty easy to identify by faculty and university) rather than waste their time on the current generation?

The evidence debate, you won’t be surprised to hear, continues……

Book Review: Knowledge, Policy and Power in International Development: A Practical Guide

This review appears in the Evidence and Policy journal, where it is now available free online (after I protested about the scandalous, rip-off $30 they were charging). Or you can just read it here. Note to self: in future, I will not write anything for journals that are not open access (thanks to Owen Barder for that suggestion).

In recent years, the public and policy debate over climate change, ‘climategate’, and the debacle of the Copenhagen Summit (and seemingly the wider UN negotiations) has brought home the tenuousness of the links between knowledge and public policy-making. ‘Do the research and they will come’ is clearly not a credible doctrine. Knowledge, Policy and Power, written by a group of researchers from the Overseas Development Institute, tackles some important aspects of these links, building on ODI’s strong track record on the interface between research and policy-making.

The book has good instincts – sceptical of all things linear, of researchers claiming to know more than they do, stressing the importance of values, beliefs, assumptions, taboos and other group pressures, hidden power  and in/exclusion in what are often portrayed as neutral processes of research and debate. There is ample discussion of the relative strengths and weakneses of different kinds of knowledge, whether derived from practice, ‘pure’ research or the people themselves.

Knowledge, Policy and Power argues that four key dimensions need attention in understanding how research translates (or doesn’t) into policy:  the political economy of the knowledge-policy interface, the actors who engage at it, the types of knowledge used and the role of knowledge intermediaries. It devotes a chapter to each of these, and concludes by summarizing its ‘core messages’ as:

1. Systematic mapping of the political context is necessary to improve the success of knowledge-policy interactions. Adopting the position that ‘it’s all down to political will’ is not only inaccurate but also counterproductive.
2. Understanding the role and behaviour of actors is not a simple matter of imputing self interest, but of considering the interplay of actor interests, values/beliefs and credibility and the power relations that underpin these.
3. Research needs to be complemented by other forms of knowledge, based on local conditions and practical experience.
4. Anyone working in this field as a ‘knowledge intermediary’ needs to think through a range of possible approaches to ensure their role is effective.

Along the way, it scans a vast literature to cull numerous useful typologies – of states, schools of thought, influencing factors, forms of knowledge etc, which can provide useful tools for those seeking guidance. The chapter on ‘facilitating knowledge interaction’ is the most practical and useful, setting out and discussing a spectrum of roles for ‘knowledge intermediaries’ (which I guess includes people like me), from low level ‘informing’ to ‘engaging’ to ‘building adaptive capacity’. With sensible guidelines on how do decide which approach to use in a given situation, it came closest to fulfilling the ‘how to’ promises of the book.

For the book claims to be a practical guide, which brings me to my first of three main criticisms. It isn’t very practical. The style doesn’t help: Firstly the language is variable, (chapters feel like they have been written by very different authors), but veers overall towards the opaque, with the verbiage of post-modernism (plural contexts mediated by contested discourses etc etc) scattered liberally over the text, seriously blunting its ability to communicate a clear message. Even the (very welcome) case studies seem too abstract! Example: ‘The difference between an informal designation process (Viet Nam) and a dual system where birthright and elected leaders share control (Morocco) is manifest in the degree of regulation and openness.’ Yeah, right.

That may be why, although I had regular glimmers of recognition and the odd wry smile, I had no ‘aha moments’ when reading this book. That is unfortunate– I think revelatory ideas are probably in there somewhere, but are so buried beneath the dense language, that several readings would be required to uncover the gems, and few people will have sufficient time or patience.

My second problem with Knowledge, Policy and Power is the alarming extent to which it blurs (or more accurately, ignores) the boundaries between research and advocacy. The book recommends that researchers consider ‘shaming techniques directed at veto players’ and ‘building wider movements’ as part of their day job.  It contrasts the Brookings Institution (high credibility, but limited advocacy role) with the much more overtly partisan and activist Heritage Foundation, and the authors seem to prefer the Heritage model, but don’t discuss the costs of doing so. The section on credibility is rather weak and ignores the issue of reputational damage.

I work for an organization which specialises in ‘research for advocacy’, but even I was alarmed by this – if research organizations veer too blatantly towards activism or ‘policy-based evidence making’ they risk reputational damage that can be close to permanent. Instead, I would have liked to see much more discussion on the kinds of alliances researchers can make to improve impact, while preserving their reputation, and the challenges they face in forming those alliances (for example NGOs typically work on much shorter timescales than researchers, resulting in much mutual frustration). The book seems to assume that researchers can do it all – they can’t, and nor should they.

My final point is that Knowledge, Policy and Power includes only passing reference to shocks, or ‘critical junctures’ as they are categorised in Why Nations Fail. The discussion portrays a largely steady state world of research, engagement with policy makers, and civilised debate, but in advocacy terms, readiness for such junctures is all. Very often, it is scandal, failure, crisis and disaster that drive change in policy, and that carries important implications for researchers and advocates. The most obvious is that when a shock hits, researchers should be repackaging existing research to show its relevance to the current crisis and making every effort to get it into the hands of policy makers, even if that means temporarily abandoning the cherished five year research programme. A discussion on the use of research before and after elections would have provided another excellent example of influencing in practice.

Overall, I think there is enough in here to warrant close study by researchers seeking to improve the policy impact of their work, but be warned – you will have to work at making this book produce practical guidance.

Do’s and don’ts on research -> policy and the state of Development Studies in Ireland

Spent a couple of enjoyable if tiring days in Ireland last week (they always look after visitors well….). I was there to speak to Ireland’s Development Studies Association on “What’s changed in development in the last five years, and how do we respond?” (here’s the powerpoint). It coincided with the first copies of the new edition of From Poverty to Power coming back from the printer (V exciting), so we did a book signing and sold out, which gets any author hyperventilating. But enough about me – here’s some random impressions of how the development and development studies scene in Ireland compares to the UK.

First the DSAI is new (this was only its 3^rd annual conference) and buzzing with energy: 100+ people packed into the venue, and the Minister for Trade and Development Joe Costello TD showed up and more or less promised government support for the future. The event had much more NGO and government people attending than the UK equivalent, which feels more distant and academic – in Dublin even the academics were activist/practitioner oriented.

That may be down to history (isn’t everything?). According to Minister Costello (who was launching a new book on teacher training in Africa) ‘there are two things embedded in the Irish psyche: famine and education’. To which I would add missionaries – although the Church is in deep disgrace with a lot of people in Ireland (some even talk about Ireland being a ‘secular society’ – blimey), the missionary tradition is deeply rooted. It has even spawned secular equivalents – the government has a long history of encouraging volunteering, and is now looking at how to enable retiring civil servants to work overseas. A lot of immigration during the boom years has also created a more cosmopolitan society and student body.

And in case you’re interested, here’s the notes for my 10 minute pitch on that old favourite, how to link research and policy.

Q: When does research influence policy?

A: It usually doesn’t. Policy making is seldom rational. But the political process tends to open up to research input at certain times:

-          After serious failures, scandals and shocks (eg change of leader, disasters, conflicts)

-          In accordance with political timetables (eg drawing up manifestos, leadership elections)

-          When something new occurs and the government is looking for guidance (eg a new issue to which it must respond, a new outbreak of disease, or a new technology or institution)

-          When significant implementation gaps exist between government policy and reality

Researchers need to get better at identifying such windows of opportunity and responding (e.g. by dropping current work and repackaging previous research relevant to the opportunity) before those windows slam shut again.

How can you maximise your chances of research influence?

Do:

Include human stories, killer facts, powerful images (see left) and spend lots of time on the executive summary (no-one will read anything else)

Pull out the ‘so-what’s’ – the things governments, companies or whoever should do differently. If there aren’t any, you have a problem.

Use multiple platforms: one pagers, blogs, op-eds, infographics, twitter etc

Involve the research targets (eg governments) in the design and governance of the research (‘publish it and they will come’ is a rubbish philosophy)

Find out who the research target listens to (parastatal thinktanks, former leaders, captains of industry, their former prof at university) and try and involve them in the research

Pursue alliances with NGOs based on relative strengths – NGOs have field presence and good comms (sometimes); researchers (on a good day) have rigour and credibility and access to academic funding

Don’t:

Strew your paper with awful academic/post-modern jargon. Avoid deconstructing, discoursing, contestation etc

End with no real conclusions except ‘needs more research’, ‘more data’, ‘needs more civil society participation’ and especially ‘everything is complex and/or context specific’.

More on this topic here.

Can theories of change help researchers (or their funders) have more impact?

Got dragged into DFID this week for yet another session on theories of change. This one was organized by the DFID-funded Research for Development (R4D) project (sorry, ‘portal’). A lot of my previous comments on such sessions apply – in DFID the theories of change agenda seems rather dominated by evaluation and planning (‘logframes on steroids’), whereas in Oxfam, it is mainly used to sharpen our work in programmes and campaigns. But the conversation that jumped out at me was around ‘how do we influence the researchers that we fund to use theories of change (ToCs) to improve the impact of their research?’

It’s risky to generalize about ‘academics’, but I’m going to do it anyway. Let’s apply some ToCs thinking to academia as a target. Applying ToCs to try and understand why academics don’t use ToCs may feel a bit weird (like the bit in Being John Malkovich where Malkovich enters his own brain), but bear with me.

Let’s start with the 3i model – processes and decisions are influenced by institutions, interests and ideas. Because academia is largely non-profit making, institutions and interests are pretty much the same thing, and come down to incentive and career structures. Here I think DFID has a problem in getting researchers to be more concerned with impact - whatever favourable ideas are around in terms of academics wanting to change the world are likely to be neutralised by the institutional culture:

Career progression takes place largely through peer approval rather than through any ability to influence the world outside (in fact, being dubbed a ‘media don’ can damage your promotion prospects).

One of the big risks for an academic is being rubbished in public for being wrong, naive or insufficiently nuanced – academics love snark and gossip (not like NGOs then…) and that kind of kicking can damage your reputation for years. So there are strong disincentives to set out clearly your assumptions about how change happens (especially if they’re really naff, like ‘all you need is robust research to convince grateful-but-dim policy makers to change their misguided ways’, which I suspect is actually the theory of change behind a lot of research).

That fear of clarity may explain why when I worked as a publisher, I watched how perfectly good, clear writers started a PhD and were lost to me, entering into several decades of inaccessible post-modern gibberish before emerging blinking into the light as self confident, respected professors once again able to communicate in normal English (e.g. talking to a potential young author on Mexico. Me: ‘so who has the guns then?’ Author – light dawns after baffled look – ‘Oh, you mean the repressive apparatus of the state!’)

What other ideas might ToCs suggest? That you need to reward and build alliances among the drivers of change (eg encouraging young Blattmanesque bloggers who ‘get’ communications and influencing, while doing your best to neutralise ‘blockers’ – custodians of the peer-reviewed flame, perhaps?).

Or that you need to spot and capitalise on windows of opportunity, since change is seldom smooth and continuous. In the UK, one such window of opportunity is the new version of the Research Assessment Exercise (RAE), the enormously influential scheme by which UK universities are assessed for state funding. The next round of the RAE, now renamed the ‘Research Excellence Framework’ concludes in 2014. Importantly, it will allocate 20% to impact, defined as ‘reach and significance’. Could DFID and other funders pick that up and use it in their own assessments?

How else could DFID help turn this around? It has a lot of clout, largely coming from its sizeable research budget (about £200m a year last time I looked). Here’s a few ideas, in no particular order and mixing up sticks and carrots.

How to get researchers to understand the minds and lives of the non-researchers they hope to influence? How about insisting that any recipient of a DFID research grant not only identifies the non-academic targets of their research, but gets credit if they manage to arrange to shadow these targets for a few days to find out how they absorb and use information (I learned more about advocacy from shadowing a UK Development Minister for a day than from dozens of workshops).

Publish (and require recipients to publish) stats on blogging, citations in the media (not just journals) and any other indicators of communications and/or impact, by named academics, in order to generate some positive competition. Let the league tables commence….

Start ‘a window of opportunity fund’ that specifically excludes new research in favour of funding previous or actual research recipients to rapidly repackage existing research in response to major new opportunities in terms of demands for new thinking – e.g. change of leadership in target institution, scandal, external shock etc.

In funding applications, insist on a proper power analysis/theory of change, including which target institutions are to be influenced, what the opportunity timetable looks like (eg new legislation or drawing up manifestos). If anyone limits their ToC to ‘changing the discourse’, they should probably be taken out and shot (unless they can plausibly suggest how they aim to achieve that).

Ask researchers to explain how they will involve both influencing targets and communications people in the governance of their research from the outset (rather than completing the research and then saying, ‘oh blimey, how do we communicate this to keep DFID happy, we’d better organize a seminar and send a copy to the Minister’).

There are also risks here – people are sometimes scarily ready to blur/erase the boundaries between advocacy and impartial academic research – more on that to follow.

I’m sure there are lots of other ideas – please send them in

Previous thoughts on getting research into policy here and here.

Other thoughts from the workshop here.

How can we get better results from working with consultants?

OK this is a bit internal, but I thought it was interesting. We had a great 3 day session in Oxford last week with our rapidly expanding Global Research Team (see right – the prominent fella in the front is Martin Walsh, who’s our Global Research Adviser and your best point of contact if you want to talk to them). They’re a bunch of bright sparks from every corner of the globe with a big challenge: to make sure Oxfam’s thinking, programming and advocacy is informed by the best possible evidence. That means training and supporting other staff (lots of people in Oxfam commission or conduct research), working with partners who commission research, sometimes with Oxfam money, plus (increasingly) seeking research funding [note to all you academics out there – need a partner for your next research proposal with people and contacts on the ground and a strong global communications and advocacy capacity? Please form an orderly queue.....]

So what (other than the general excitement at seeing the team all together) stood out from the conversation (or at least the bit of the meeting which I managed to join)? Above all, just how hard it is to get good results from consultants:  the standard model for development research is to outsource it to specialist researchers. Some of them are local, some are expats, and all of them present a tricky management task. Often they are really expensive, with fees inflated by big aid donors all desperate for quality analysis. Many are unreliable, reporting late and often not sticking to the terms of reference. And then they disappear with all that acquired knowledge and all you have to show for your money is a bit of paper and a Word document.

It doesn’t have to be like that – I know and have worked with some great consultants. We’ve though a lot about this and even published a book on it, but it doesn’t seem to get any easier. So what makes sure you get value for money?First, be clear why you are putting everyone through this. Commission research to answer clear questions, not to avoid decisions – it should never be a substitute for direction, or a means of prevarication.

Second, take time to find the right consultant; ask around; have preliminary chats with possible candidates; don’t just call the person you always use. Insist on competitive bids (although sometimes you have to skip that if a local guru is clearly the right person for the job and will tell you to get lost if you demand lots of CVs and form-filling). As with any appointment, hiring the right person is 90% of the battle.

Third, take time on the briefing: write very specific Terms of Reference, avoiding broad questions like ‘what is the role of agriculture in country X’ that allow them to regurgitate the literature and don’t help you take decisions. Don’t ask them to do your job for you (‘design an advocacy campaign on Y’).

Finally, don’t give them a contract and say ‘come back when you have a draft report’ – you need regular contact to avoid nasty surprises. Here are some top tips from my colleague Kate Raworth:

 As research manager, your attitude should be “eyes on, but hands off” – that is, engaged in the process but not trying to influence the actual findings.

 Is there a possibility for any staff or interns to work alongside the researcher for part of the work, to learn from them, and to get involved in the issues?

 Ask the consultant to present an early draft of the findings to a group of colleagues. Discuss the results and help the researcher keep focused on the key questions (but don’t be tempted to add new questions now – it is not fair to add to the TOR unless you extend the contract).

 When commenting on the first draft of the report, make sure you don’t simply ask a list of new questions. Look back to the TOR: have these original questions been answered? What other specific information is needed to complete the answer?

Do all that, and you’ll often get good results, but not always. We should also explore the alternatives, like hiring staff on short term contracts as a (perhaps) more manageable and cheaper alternative. We need to get better at working with PhDs and other forms of slave labour students – their language and timescale is often just so different from our own that we don’t manage to sort anything out. Has anyone worked out how to collaborate to both sides’ benefit? What about commissioning high-end local journalists instead, given that sometimes everything we commission seems to need a rewrite into more accessible language?

And often we are not even the ones commissioning the research. Instead, we are funding partner NGOs who in turn hire consultants, and there the challenge is even greater, especially with partners who are understandably prickly about funders trying to backseat drive and micromanage.  How to ensure quality control without Oxfam control? Lessons there include being proactive – offer to go and talk to partners, take part in meetings etc. Building relationships with both partners and their consultants is more likely to work than trying to order people about. Accompanying partners or consultants on a field trip can be a great way to build relationships and see what is going on.

And just in case you think I’m laying all the blame on them, a shout out to all those consultants out there: what advice would you give NGOs and agencies  about how to be better research contractors?

And here’s a Dilbert strip about management consultants – not the same thing at all, but who cares?

Do research funders have to chose between rigour, innovation and impact?

Another day, another committee discussing funding for development research (again, no details, sorry). This one produced a really interesting conundrum, perhaps even a trilemma, that suggests it may be impossible to simultaneously achieve three ‘Good Things’ sought by research funders: innovation, rigour and policy impact.

First, rigour: Western research funders do their level best to judge the rival merits of research proposals based on the importance to poverty reduction of the research, and its methodological rigour. They require lots of detail on the methodologies to be employed, rely on anonymous peer review and deliberately ignore or minimise the importance of reputation – whether of the institution or the lead researcher. Only fair, surely? The last thing we want is a nepotistic old boys (and girls) network of decent chaps who all went to college together bunging each other research grants with no questions asked, right?

Up to a point, Lord Copper (sorry, Scoop reference – feel free to ignore). Suppose you want to fund innovation: as Tim Harford argues in Adapt, it may be better to find and back innovative people, even if they don’t have a nicely worked out project. Harford points to the outstanding record of the Howard Hughes Medical Institute, which explicitly urges (and finances) ‘researchers to take risks, explore unproven avenues, to embrace the unknown’. And the best innovators may well be at the start of their careers (Einstein and Newton did their best work in their 20s). In the UK, the Wellcome Trust uses this approach with its ‘New Investigator Awards’ that “support outstanding researchers who are no more than five years from appointment to their first academic position, but who can already show that they have the ability to innovate and drive advances in their field of study.”

who ya gonna fund?

But what about policy impact, increasingly sought by funders of research? Evaluations of why some research has more impact on policy and practice add another personalistic element. The distinguishing factor between research that sinks without trace, and research that has impact is often the relationships of the researcher with decision-makers, and that favours well established researchers, who know everyone (and taught half of them as students). There’s a link to the idea that research has maximum influence after a shock (scandal, collapse of a previous policy, financial crisis, natural disaster, conflict etc). In such circumstances, decision makers are winging it, desperately looking for new ideas, and they are much more likely to turn to people they already know and trust (like their old professor) for those ideas.

That goes for institutions too: in many countries, if you want to influence the state, who does the research matters at least as much as how good the research actually is. Will officials trust the source? Working with parastatal thinktanks in countries like China and Vietnam is often the best way to ensure research finds a fast-track to policy influence.

If a funder wants to pursue three such contradictory goals, what can they do? They could decide to fund larger projects with separate components, but I’m not sure such different approaches and personalities could work together. Or they could create different funding pots. The difficulties include being more willing to accept higher failure rates or being less insistent on methodological rigour (in the case of backing innovators).

As for the risks, what can be done to prevent this just becoming a pretext for some lazy nepotism? All thoughts welcome.

How to write winning research funding applications

Recently I’ve been involved in some fascinating exercises in allocating large dollops of institutional funds for research (can’t give any more details, sorry). This has involved reviewing and discussing dozens of applications from different academics. Here’s a quick download of what I learned about the art of writing winning applications:

Mixed methods rock: Quants and quals seem to have a really hard time talking to each other, but proposals that manage to integrate the two (genuinely, rather than have entirely separate processes) go down well.

You need serious impact plans: a surprising number of proposals still seem to think that once the research is done, you publish in a couple of journals, organize a seminar or two, and it’s job done. If you want research to actually have an impact, you need to try a bit harder – think about who you are trying to influence, when they might be most interested in listening to you, perhaps involve them in the governance of the project (see ODI’s RAPID programme for more on this).

A bit of innovation really helps: Yes, using mobiles either in data gathering or dissemination still gets brownie points, but not all innovation has to be tech-based – I have seen a roomful of profs and luminaries gurgling with delight because a proposal has included street theatre among its dissemination plans.

Make sure the Principal Investigator is not just a figurehead: The PI leads the project, but an alarming number of proposals reckon he/she can do that on a couple of hours a week. Not usually very convincing.

Have a clear research question or hypothesis to test: seems obvious, but it’s surprisingly rare to get a really clear crisp hypothesis with a good explanation of how the research proposes to test it.

Don’t be greedy: If the application range is £100,000 to £300,000, don’t automatically stick in for £300,000 – it’s more credible if the budget gets to £275,000. But if you do, think about value for money (as funders increasingly do). If the day rate for your researchers is at the top of the range, then reviewers start to get irritated.

Don’t be tokenistic about involving southern researchers: reviewers notice if the ‘capacity building’ element is not much more than employing a lot of African PhDs to do data collection at bargain basement rates. You need clear evidence of ownership by local universities in both research design, and dissemination.

The topic matters as well as the research: panels like research on new/sexy issues, as well as well-designed research processes.

How you respond to referees matters: most proposals are sent out for review and the anonymised comments sent to the applicants. Take the comments seriously – if you just dismiss them, or question the reviewer’s credibility, it makes you look brittle and unconvincing. But yes, they can be very annoying, so if you suffer from ‘reviewer rage’, read the referee’s comments, then walk away for a day or so before replying……..

Finally, remember you are writing for a mixed audience: you may have specialist referees who like nothing better than a pointy-headed exchange, but there will also be generalists like me in the room, so make sure you tell us important stuff like what’s new in your proposal and why it matters, preferably in a language approaching English.

Any other do’s and don’ts from reviewers our there?

Anyone want my job? Oxfam’s looking for a new Head of Research

So here’s the deal. After seven happy years in charge of Oxfam’s policy research team, I’m moving sideways to take up a new post as senior strategic adviser (though I personally prefer my colleague John Magrath’s suggestion of ‘chief opinionator’) and we’re looking for a new head of research. Here’s the blurb from the ad in the last week’s Guardian (the cartoons are added):

The role
This is one of the most influential positions within Oxfam. You will be managing a team that will deliver high profile research in support of Oxfam’s global campaigns. You will also play a key role in building the capacity of the whole organisation to deliver high quality, high impact research in countries where we work, including developing a vibrant community of practice, creating high profile research collaborations and securing significant research funding. Moreover, you will play a large role in shaping the intellectual leadership of Oxfam on the challenges facing development, positioning Oxfam as a cutting edge development thinker. You’ll represent us externally, both as a media spokesperson and as a writer of key articles.

What we’re looking for
A strong conceptual thinker, and recognised researcher of development, humanitarian and poverty issues, you’ll have proven your ability to link grassroots programmes to policy development and be equally comfortable with the use of statistics and modelling. You’ll also be an adept strategist, bringing an appreciation of the impact research has on making change happen, and with the written and verbal communications skills to influence through lobbying and media. You’ll have the leadership and management skills to drive complex organisational change that delivers high-impact, high quality research.

I can’t recommend the job highly enough – the team is great, there’s a major new campaign to resource and interest in research across the organization is at an all-time high. We could be at the threshold of something new and exciting – want to lead it?

Full details and online application here. Closing date is 4 September, so get your skates on.

Please tell any suitable friends and colleagues, and if you want to pick my brains, feel free – just use the comments box and say if you don’t want it published for other blog readers to see.

How can research funders work better with international NGOs like Oxfam?

I spoke recently to a meeting of the UK Collaborative on Development Sciences. It’s a great initiative, bringing together 13 UK funders and stakeholders with an interest in international development research, but is ‘collaborative’ really a noun? Anyway, the topic was how research funders (mainly state funded) can link up more effectively with large INGOs like Oxfam. Let me talk you through the powerpoint……

First, why do INGOs do research? Above all, to improve impact of programmes and advocacy in three broad areas, according to a nice distinction made by my colleague Kimberly Pfeifer at Oxfam America: ‘tactical research’ (reactive to broader events and policy agendas); formative research (setting new agendas and directions) and evaluative research (MEL, learning lessons). INGO people are doers and activists, with little time for theorising – they think in terms of guidelines and toolkits. That is probably why UKCDS wanted to talk to me, because government is increasingly demanding that researchers demonstrate the impact of their research, rather than the beauty of pursuing knowledge for its own sake.

But what do we mean by the word ‘research’? For INGOs it is often much more about a clear narrative than about data. The risk is doing violence to a complex reality, but the upside is that we tell stories that stick in the heads of policy makers and others. There is also a priority on case studies and bearing witness – exploring how large scale phenomena (climate change, food prices etc) affect the lives of people living in poverty. What some academics dismiss as anecdotes are for INGOs (and most normal people) closer to reality than some massive number-crunching exercise (though we still need to be careful about correlation v causation and attribution). For some examples, check out the Oxfam publications website.

What does good policy research look like, from an INGO standpoint? A clear story, bringing together a decent review of the academic literature with those real life stories; preferably relevant to what is on the agendas of decision-makers over the coming months; drilling down into the issues of power, inequality and social relations that often go missing in conventional research. For impact it also needs a sprinkling of killer facts, an answer to the inevitable ‘what’s new in this research?’ question, and clear and convincing recommendations and solutions.

Are INGOs any good at research, thus defined? Generalizations are perilous, but here goes:

Strengths: at its best INGO research is rooted in real life, the experiences of partners and communities (e.g. our work on the impact of the global financial crisis, or forthcoming stuff on food prices); INGOs have been pioneers on participatory methods; the research packs a punch both in content and in the ability of INGO media teams to make a media splash that gets it noticed. And they have a global constituency and reach that many academic researchers can only dream of.

Weaknesses: often stronger on qualitative than quantitative; sometimes a bit cavalier on methodology (although we outsource a lot of research to academics which, if they’re any good, should fill that gap); weak systems of peer review (and some confusion over what constitutes a ‘peer’); suffers from short INGO attention spans, so few examples where research builds up over time; patchy links to developing country research institutions and always short of cash and capacity compared to the formal research institutions.

How can funders improve the relevance and use of research by INGOs? Well, they could fund it directly of course, but that is often going to be difficult given the way they are set up, so here are some other ideas. Insist that research institutions work with INGOs to co-design research programmes (the norm is alas, for an institution to decide on a largely irrelevant agenda  and then approach the INGO as an afterthought to help with the communications, or ‘do the voices of the poor bit’.) Sure, we could (and do) take the initiative and approach research institutions with our own ideas, but the timescales, interests and approaches are often just too different to find common ground. Funders could provide incentives to help bridge the gap.

That means understanding what research INGOs are going to need over the next few years. Luckily the level of intellectual herding is pretty high, so if you get a bunch of them in a room, they will probably all come up with a similar set of priorities (current ones would probably include climate change, scarcity, food security, theories of change, measuring impact, multipolar world and the absence of gender and disaggregated data from most research questions).

And a few more specific suggestions for the higher education researches themselves, (and where prodding by funders can probably help):

If you want access to communities, the research had better be relevant to the people and partners (e.g. testing new approaches through action research). It needs to be properly discussed at draft stage and disseminated and discussed locally on publication. INGO staff time and direct costs (and those of their partners on the ground) should be properly funded. Finally, you need buy-in at country level, where harassed staff may have very different priorities from INGO HQ.

What is at stake is, I think, pretty important – building a regular and productive interchange between funders, higher education institutions, thinktanks and INGOs. Funders could help by creating incentives for better links between these groups, requiring researchers to demonstrate impact and relevance. They could also help create a space for collective reflection on research priorities among INGOs (that only happens in a very ad hoc way at the moment, for example by everyone commissioning papers from Alex Evans……) and build INGOs’ capacity to understand, commission and use existing research (as well as do a bit themselves).

Any other suggestions?

What development issues do we need more research on?

Every development research paper I ever absent-mindedly skim pore over with fascination seems to end with NMR (needs more research) – a blatant piece of self-justification, but usually justified (anyone got any candidates for areas where we need less research? Anything

Top placard from Jon Stewart's recent Rally to Restore Sanity

involving cross country regressions or ‘discourse analysis’ perhaps?) But research on what? Research funders are ever-hungry for the next big idea, preferably a few years in advance in order to allow the research machine to lumber into action in time to produce some useful results when they are needed.

So when asked recently for my suggestions to add to the customary list (governance; food security; civil society; technology; beliefs and values; risk and resilience; urban etc) of possible focus areas for research, I tried a change of tack. Rather than thematic areas, what about some cross-cutting ones, for example:

1. The role of shocks in triggering change: research papers are usually written as if policy changes are decided in some University Senior Common Room, by enlightened leaders who debate the evidence and then calmly decide on the necessary changes. In reality, the political process is far more chaotic than that, and big shifts are often linked to big shocks. Examples from the UK include women’s suffrage (World War One) and the creation of the Britain’s National Health Service (World War Two). In the developing world, natural disasters often lead to political change (Ethiopia, Nicaragua), as do wars and civil conflict (Rwanda). So how well do we understand the situations in which different kinds of shocks do/don’t trigger change?

2. The limits of measurement: the metrics fundamentalists are in the ascendant at the moment, arguing that Einstein was wrong and everything that counts can be counted. That may be true in theory (discuss) but in the real world of harassed civil servants, spending cuts and intense pressure for Value For Money, the easily measured (vaccinations, schoolrooms, roads) is highly likely to squeeze out the tricky-but-vital stuff (rights, empowerment, well-being, insecurity). The current debate is unhelpfully polarised between true believers and angry rejectionists, so how about a more distanced look at what measurement can/can’t achieve in a world of limited resources. How far have we got in developing cheap and practical ways to include the tricky stuff in our metrics? What alternatives are there to crude metrics that can help us judge the success/otherwise of a particular intervention, and provide similar levels of accountability?

But I’d be interested in hearing from you guys – what development topics have you been reading/thinking about that might warrant that awful cliché ‘out of the box’. Candidates from this blog include obesity, ageing and disability, interestingly all issues that transcend old ‘North-South’ distinctions. What others have you got up your collective sleeves?

And yes, I realize that this obsessive search for funky new ideas is a bit of a problem if the real obstacles to development are the age-old themes of poverty, skewed land rights, lack of health and education, gender injustice, discrimination etc. All very boring, unless you happen to be experiencing them directly. It’s harder to get research funding (or academic credit) for ‘more of the same’, even if that’s what needs doing. Anyone seen any good advice on how to dress up old issues as new ones in order to secure funding?

Honourable mention (and who knows, maybe massive injections of funding) for the best suggestions. I may even take it to a vote……