{"type":"rich","version":"1.0","provider_name":"Transistor","provider_url":"https://transistor.fm","author_name":"80,000 Hours Podcast","title":"#30 - Eva Vivalt on how little social science findings generalize from one study to another","html":"","width":"100%","height":180,"duration":7289,"description":"If we have a study on the impact of a social program in a particular place and time, how confident can we be that we’ll get a similar result if we study the same program again somewhere else?\r\n\r\nDr Eva Vivalt is a lecturer in the Research School of Economics at the Australian National University. She compiled a huge database of impact evaluations in global development - including 15,024 estimates from 635 papers across 20 types of intervention - to help answer this question.\r\n\r\nHer finding: not confident at all.\r\n\r\nThe typical study result differs from the average effect found in similar studies so far by almost 100%. That is to say, if all existing studies of a particular education program find that it improves test scores by 10 points - the next result is as likely to be negative or greater than 20 points, as it is to be between 0-20 points.\r\n\r\nShe also observed that results from smaller studies done with an NGO - often pilot studies - were more likely to look promising. But when governments tried to implement scaled-up versions of those programs, their performance would drop considerably. \r\n\r\nFor researchers hoping to figure out what works and then take those programs global, these failures of generalizability and ‘external validity’ should be disconcerting.\r\n\r\nIs ‘evidence-based development’ writing a cheque its methodology can’t cash? Should this make us invest less in empirical research, or more to get actually reliable results?\r\n\r\nOr as some critics say, is interest in impact evaluation distracting us from more important issues, like national or macroeconomic reforms that can’t be easily trialled?\r\n\r\nWe discuss this as well as Eva’s other research, including Y Combinator’s basic income study where she is a principal investigator.\r\n\r\nFull transcript, links to related papers, and highlights from the conversation.\r\n\r\nLinks mentioned at the start of the show:\r\n* 80,000 Hours Job Board\r\n* 2018 Effective Altruism Survey\r\n\r\n**Get this episode by subscribing to...","thumbnail_url":"https://img.transistorcdn.com/VO1STE7hN95RRg9QdLo4soV2VhhbR9PF5ZZlRhDYcwE/rs:fill:0:0:1/w:400/h:400/q:60/mb:500000/aHR0cHM6Ly9pbWct/dXBsb2FkLXByb2R1/Y3Rpb24udHJhbnNp/c3Rvci5mbS9zaG93/LzQxNDAyLzE2ODM1/NDQ1NDAtYXJ0d29y/ay5qcGc.webp","thumbnail_width":300,"thumbnail_height":300}