Busara 183d ago

Both non-profits and their funders rely on impact evaluations to determine which social programs work -- and therefore should be scaled -- and which do not. But do impact evaluation methods as currently practiced deserve this level of trust? In a new article in New Directions for Evaluation, Patrick Forscher, Director of the CREME Meta-Research Team at Busara, along with Benjamin McNamee, Director of Strategic Partnerships at DARO, and Michael Lenczner, CEO at DARO, argue that the answer may be "no", at least under current practices and incentive structures. To make their argument, the authors draw on the recent history of another field dedicated to creating credible knowledge about how people behave: social psychology. Social psychology has recently gone through a crisis of credibility -- sometimes called the "replication crisis". This crisis was caused by the need to obtain positive results, which are deemed more "publishable" than negative ones, combined with: ✅ a competitive, publish or perish environment ✅ incentives for buzzy, newsworthy results, and ✅ the use of research methods as rhetorical devices rather than ways to enhance result credibility The authors note that similar incentives and exacerbating factors face non-profit impact evaluators, who must show that interventions provide value-for-money or risk losing the funding upon which their non-profit depends. This incentive could undermine the usefulness of impact evaluations for learning, especially when combined with exacerbating factors like: ✅ the "starvation cycle", where non-profits cut overheads to show value-for-money and therefore rob themselves of the money and operational support necessary to do high-quality impact evaluations ✅ pressure to use research evidence to drive sales rather than for scientific learning ✅ thoughtless valorization of particular methods -- especially RCTs -- as the best and only ways to do impact evaluations, regardless of circumstances Impact evaluators should therefore consider the possibility that their field is in crisis and, similar to what has happened in social psychology, consider the tools and strategies that could help safeguard its credibility. Intrigued? Check out the conversation between Patrick and Ben below -- or read an open access version of the full paper at the following link. https://lnkd.in/da-A3bSS