Subscribe
Share
Share
Embed
Welcome to GiveWell’s podcast sharing the latest updates on our work. Tune in for conversations with GiveWell staff members discussing current priorities of our Research team and recent developments in the global health landscape.
Elie Hassenfeld: [00:00:00] Hi, everyone. This is Elie Hassenfeld, GiveWell's co-founder and CEO. Today we're going to do something a little bit different. Instead of diving deep into one area of our research, we're going to talk to three researchers about the work that they do at GiveWell. We're going to hear about what they do, but primarily we're going to focus on how they work and what it's like to work at GiveWell.
And so as we get into their work, we're going to understand the way they approach the questions that they have in their work, times that they've learned things that have changed their mind, how they make decisions about when to go deeper versus when to call a project done and move on to something else.
All of these questions are ones that our research team has to ask every single day, because GiveWell is a group that aims to do very deep, almost academic-style research, but then we exist not because we're doing research and producing research, but because we're trying [00:01:00] to put money to work in the world to help people in need.
And so the question of how we work and how we balance what we focus on, how deeply we focus on it, and when we move on to other questions is always very salient for us and on our minds, and so each of them talks through that in their work.
I think these stories also illustrate, you know, a major focus for GiveWell right now in our expansion, which is we're doing two things at once. We're both trying to go broader and cover more ground, because covering more ground gives us the opportunity to find more of the programs that are among the best in the world at helping people. And at the same time, we're also going deeper. We're trying to get better information about questions that matter to our grantmaking. We're doing work to improve the quality of our decision-making and ensure that our work is accurate and reflects the most important considerations, so that our decision-making leads to the best funding decisions that it can.
You know, right now GiveWell is growing rapidly, and we have a lot of positions open on our research team. And so if you're [00:02:00] interested in the kind of work that we're doing and think you can help, please do look at our jobs page and apply. We need to grow the team to enable us to do more to help people in need as much as we can.
So today I'm joined by Dilhan, James, and Meghna. Thanks for joining me. Can you each introduce yourselves?
Dilhan Perera: Great. I can go first. Thanks, Elie, for having me back on the podcast. My name's Dilhan Perera. I'm a senior program officer on GiveWell's New Areas team. I've been at GiveWell for about three years now, and prior to that, I worked in behavioral science, impact evaluations, and public policy.
James Watson: Hi. James Watson. So I'm a senior researcher on the malaria cross-cutting team. Been at GiveWell for a year now, and before that I was working for about 10 years, mostly in Southeast Asia, as a malaria researcher in academia.
Meghna Ray: Hey, Elie. Thanks so much for having me. I'm Meghna Ray. I'm a research analyst on GiveWell's New [00:03:00] Areas team. My background is in infectious disease epidemiology and global health. I've been at GiveWell for about eight months now. I started on the Commons team, which is where all the new research analysts start, and have been on the New Areas team for about three months now, and have primarily been supporting Dilhan on our oxygen work and TB.
Elie Hassenfeld: Great. Thanks everyone, excited to have this conversation. So I asked each of you to bring an example of something you've worked on so we can give people a feel for what doing research at GiveWell actually involves, behind the curtain, I guess.
So Dilhan, I know you've been looking at ways to prevent children dying of pneumonia. You looked into that a bit. Can you just tell us the story of what you looked into, and you know, where that process started?
Dilhan Perera: Sure. So maybe just to step back why we were focusing on pneumonia. So as I mentioned, I work on GiveWell's New Areas team, and part of our role within GiveWell is to explore areas of global health that are less familiar to GiveWell and where there might be substantial room [00:04:00] for cost-effective impact.
So pneumonia is one of the biggest killers of children under five globally. I think it accounts for something like 10 to 15% of under-five child deaths globally, which is similar to malaria, which is one of our big areas of focus. But despite the massive number of children that die each year from pneumonia, we hadn't funded programs at a particularly large scale focused on pneumonia deaths before.
So when I turned to this, one of the ideas that was suggested to us was this program called Integrated Community Case Management, which basically involves community health workers, who are people who are living in communities and have some basic health training but are not nurses or doctors, going around their communities trying to find sick children, and in particular, children who might have, you know, a small set of very common diseases in this context. So it's typically malaria, pneumonia, or diarrhea. Finding those [00:05:00] children, identifying which of the conditions they might have, and then either providing some treatment on the spot or referring them to a health facility where they can get more intensive treatment.
On the pneumonia side, what this typically looks like is trying to identify the kids with pneumonia who might benefit from antibiotics and giving those children amoxicillin on the spot. And also sometimes it involves identifying more severe cases of pneumonia and making sure that those children get referred to a health facility or a hospital.
So when I first looked at this program, you know, what we typically do is we try to put together initial cost-effectiveness analysis to help us understand, you know, how impactful is this program relative to its cost, and how does that compare to everything else that we could fund.
And when I initially did that I thought this type of program could be very cost-effective. That was due to a couple of reasons. One is that amoxicillin is just very cheap. Another is that there are a lot of children out there who suffer from pneumonia [00:06:00] each year, even if only a small proportion of them end up dying. Just finding those kids who have pneumonia relatively early, making sure they have antibiotics, seemed like a pretty effective way of making sure pneumonia cases don't progress to the stage where they need more severe treatment.
But as I started talking to researchers in this space, particularly those who had been studying why the children who die of pneumonia and other diseases, why they had actually died, I think a few things made me a bit more pessimistic about the extent to which amoxicillin in particular, or amoxicillin alone, could actually make much of a dent on, pneumonia deaths. Amoxicillin only treats a certain type of bacteria, and it seemed like from these very intensive studies of why children were dying, that it was actually a different type of bacteria called gram-negative bacteria, which were becoming more important in explaining why children die from pneumonia. And in addition to that, there's actually been a vaccine focused on the same [00:07:00] type of bacteria that amoxicillin treats called the pneumococcal vaccination, PCV. Uptake of that vaccine has been increasing over time. And so actually, there was a sense that a lot of the deaths that were still happening were due to viruses and other bacteria that are not treated by the vaccine, which would not be responsive to amoxicillin.
So that updated me negatively on the cost-effectiveness of this type of program. But through those conversations, a different pathway to averting pneumonia that became more salient to me, which was ensuring that of the kids who are referred to hospitals with more severe pneumonia, making sure that those kids have the treatment they need once they get to the hospitals, could be much more impactful, and that specific treatment is medical oxygen. It can be more expensive to provide than amoxicillin, which is why people hadn't initially suggested it to us. But it is much more targeted on the children who are most likely to die from pneumonia, and also can help prevent deaths that are due to a wide range [00:08:00] of causes, so like viral pneumonia, pneumonia due to other types of bacteria that are not responsive to amoxicillin.
So that's the current focus of my work, and I'm pretty excited about that. But yeah, it's a bit of a change from what I initially expected to focus on when I first started looking at this.
Elie Hassenfeld: You know, it sounds like this program fits almost like a paradigmatic GiveWell focus program. It's high level of need. You know, many children were dying from pneumonia, a treatment that has strong evidence behind it and is cheap to deliver. And so you came into this with the expectation that using amoxicillin to treat children with pneumonia could be extremely cost-effective, could be a way that GiveWell could have a lot of impact. I think it's often challenging to change one's mind and to go from thinking that something is very promising to something is not promising and not worth pursuing. What does it take to go from "this is promising" to the opposite belief: "we should pursue something else"?
Maybe paint a picture a little bit more of how you went through the [00:09:00] process of changing your mind and what that took? Because I think, from my perspective, the kind of process you went through here is some of what is so critical to keep happening at GiveWell.
It's like we're not coming from the vantage point of, you know, we want to fund amoxicillin for pneumonia. We're coming from the vantage point of we want to help people as much as we can, and that means doing this open-minded research where going to be driven where the evidence and the opportunities appear.
Dilhan Perera: Yeah. So it's a good question. And then I want to say upfront that, you know, while I think community-based treatment of pneumonia with amoxicillin is less promising at this stage than increasing access to medical oxygen, I think it still could be a very impactful program, and it's just, you know, it's all relative, and we just have to make decisions about what we prioritize in the near term.
There were a lot of assumptions in my initial cost-effectiveness analysis but there wasn't a single source of data or evidence that would give me the answer. I was triangulating across different papers, all of which didn't quite answer the question [00:10:00] I was interested in or had some limitations, and also taking into account what various implementers were telling us.
And in this case, the specific parameter I think was what percentage of deaths due to pneumonia could be averted by making sure someone was given amoxicillin relatively early on. And I knew it was less than fifty percent because there's all of these other causes of pneumonia deaths that are not responsive to amoxicillin, but, you know, I thought it was somewhere between twenty-five and fifty percent.
I reached out to the authors of one of the papers that had informed my initial estimate, and when I was speaking to them, they didn't give me their best guess of the number I cared about, but they did start expressing some skepticism about how much impact increasing access to amoxicillin alone would have. And they emphasized the other causes of pneumonia deaths—of viruses and other types of bacteria—more than I was expecting them to.
So that was sort of a qualitative signal that maybe this number I'd had in the [00:11:00] CEA was too high. Then once I had that conversation, I went back to some of the implementers who were suggesting this type of amoxicillin-focused program to us and presented my logic for why I thought maybe this wouldn't be as impactful as focusing on the oxygen.
And they mostly agreed with me. They didn't have that much pushback. Their reasoning for why they'd initially proposed the amoxicillin-focused programs is just that oxygen programs typically need much larger upfront investments. They can be more complicated because it involves strengthening systems across facilities nationally.
So they had just sort of assumed that it didn't look like the type of program that GiveWell would have funded in the past. But yeah, we obviously just care about finding the programs that are most impactful and cost-effective, so we were trying to push them past that. And then I started speaking to other stakeholders in the space to just stress test my comparison or understanding about the relationships between these two types of programs, and I think there was general agreement that there's a lot of impact to be had with medical oxygen and [00:12:00] that it could be cost-effective if targeted well and done well.
And at that point I went and did the cost-effectiveness analysis for the oxygen programs and they appeared very cost-effective despite the higher per person costs. So I think it was sort of a gradual process like that, rather than an obvious point where I made this pivot.
And I think it just reflects that there's a lot of judgment calls we have to make in our work, and there's a lot of ambiguity, and that's one of the challenges and interesting things about being a researcher at GiveWell.
Elie Hassenfeld: Yeah. I mean, I think it's such an interesting story, so I want to draw out a couple things that you said to emphasize them.
So, you know, first we were talking about amoxicillin treatment and I said something earlier that, you know, it wasn't that promising, and I think one thing that's just important for us to keep in mind that you noted is that GiveWell's trying to spend a large amount of money for any individual organization, but a small amount of money in the scheme of things, and so we're really trying to get those funds to the most impactful opportunities at the present margin. And so there are a lot of programs that would help people a lot, but, even if [00:13:00] they are extremely impactful for a dollar, we're trying to find that small slice that enables us to help people as much as possible. And so we're always asking the question, you know, not just is this program good, but is there something that could be more impactful, a way that we could help people to a greater extent with the money that we have? So yeah, I appreciate your drawing that out.
And then I think another point that you said that was interesting in this story is that people, you know, often groups that we work with have an impression that there are certain kinds of programs that GiveWell tends to fund. It tends to be programs that are more like the treating children with amoxicillin, a fairly straightforward delivery of a health commodity, and that is much of what we have done in the past. But certainly now we're expanding our scope a significant amount, and there are many other kinds of programs that we're considering, and, you know, with a larger, more experienced team, we're in a position where we can, you know, look at a broader range of programs, including ones that are as complicated and challenging as medical oxygen.
But then in your story specifically, what I thought was really interesting, it sounded to me like you, when you started, [00:14:00] you knew from the outset that there was a key input into our overall assessment of cost-effectiveness, which was the proportion of cases of pneumonia that would be treatable successfully treatable with amoxicillin. And then you knew that you had to go about, like, getting a better estimate of that number. You started somewhere, you took a, you know, I don't know, a reasonable guess based on what you knew, and then that involved reading research, reaching out and speaking with academic experts on the research, speaking to implementing organizations directly, and both about, like, what you believe to see what they thought about your beliefs because, you know, they work on this all the time, they have feedback.
And then also, you know, not only checking the parameter, but then when you reach this conclusion that medical oxygen was worth pursuing, asking them for input on the trade-offs between medical oxygen and amoxicillin treatment. You know, all of that together was this process you had to go through, but took you from at the outset, I'd say guessing that you'd move ahead with the amoxicillin path towards pivoting in a very different [00:15:00] direction towards medical oxygen.
Dilhan Perera: Yeah. I think that's right, And one thing I've been mindful of in this entire process is how different types of programs might build on each other or interact with each other, and I think one of the reasons for the pivot is that, you know, if we are successful in increasing access to medical oxygen, that could make the community-based programs look more cost-effective in the future because to the extent that they are referring kids, very sick kids to facilities, it's more likely that they'll get the treatment that they need.
Elie Hassenfeld: Yeah, and I think this is just something that is just so important to understand about GiveWell and how we work. Like, you can imagine a version of a global health funder that says, "You know, we just want to do X. We are just an amoxicillin funder." And with that scope, you'd end up missing out on the programs that could potentially be more impactful. And so instead, what we're trying to do is ask the question: How can we help people most?
Often that means identifying some of the most decision-relevant data that could change our mind and cause us to go in a different direction. And then, you know, here's a good example where it took us [00:16:00] towards medical oxygen. Just very briefly before we move on and talk to others, just tell us where medical oxygen stands today so we're not left hanging with that. You know, just an update on where that work stands right now.
Dilhan Perera: Yeah, so I think we are trying to figure out both how cost-effective this type of program is in different geographies. When you're targeting different types of populations, it could look very different when it's mostly adults being treated versus mostly children being treated.
But at the same time, trying to think about what's the best approach to funding this area. I think it's a lot less straightforward in some other areas because there isn't a single type of intervention that will increase access to medical oxygen. I think depending on the place, the type of interventions required could look very different.
So yeah, we're trying to figure out what our approach should be, and we're talking to other more experienced funders in this space to help inform that. But yeah, so far it looks very promising, and it would also be, you know, a different way of funding programs relative to [00:17:00] what GiveWell has done previously. We're excited to see if this could push us to evolve a bit more.
Elie Hassenfeld: That's great. Well, thanks Dilhan. And for anyone who wants to know more about medical oxygen, episode 32 is a conversation I had with Dan Brown, who leads the new areas team, where we covered medical oxygen among other new areas and new programs that we're going into.
James, something you look at is in some ways related to the topic we were just talking about with Dilhan, which is, you know, what is the total size of the need that we're facing? And so maybe you could just tell us a little bit about where you've focused on this and the project you worked on, you know, in the space of malaria where you work.
James Watson: Yeah. So it is, kind of similar in a way, because it's the sort of problem that maybe from, you know, a naive point of view you think this is actually probably quite simple. You've got some RCTs [randomized controlled trials], you put them together, it looks great. But then actually kind of have to dig into the complexity of the biology and things that are changing quite drastically over time.
One of my focuses—I'm in this group called the Malaria Cross-Cutting team, which [00:18:00] tries to help with problems that run across all the different malaria programs, whether it's to do with drugs or vaccines or bed nets. And a sort of fundamental ingredient that goes into every cost-effectiveness analysis is in a particular location, how many children—I mean, so these are places where predominantly malaria is a childhood disease—how many children die because of malaria? I guess you would think, "Well, we should know this." And in fact, well, you know, when you dig into this sort of fundamental epidemiological question, actually we don't really know. I would say we know very little how many children die of malaria, and different organizations, individuals will have very different points of view on this.
Elie Hassenfeld: So why is that? One of the things that I often say about GiveWell is that we focus where we do in global health programs and often in averting child deaths because it's, you know, among the most measurable outcomes that one could focus on in philanthropy.
I mean, in addition to [00:19:00] being a gigantic tragedy that the number of children die who do, and small amounts of money can make a huge difference. What do you mean when you say we know so little, and why is it so hard for us to know the number of children who are dying from this disease?
James Watson: Well, so first of all, I guess, the context of this is countries where there's no good vital registration system. So A, just knowing the total number of births and deaths is actually not so straightforward. But let's, you know, let's assume you have a reasonable estimate for the number of births and the number of deaths. The issue with identifying the cause of death is that most children in tropical areas, most children who come to hospital who are sick have the same symptoms.
They've got fever. I mean, if you're two you can't really say where it hurts or, you know, what exactly is wrong with you. And actually even, you know, in hospital where you have very experienced, say, research teams who are looking at children who are very sick, they will have a hard time actually saying, "Is this child very sick because of [00:20:00] malaria or are they very sick, say, because of bacterial sepsis?"
And now if you think of what's the main input for some of our estimates of the number of deaths from malaria, it's actually based on what's called a verbal autopsy study where if there's been a death of a child, you will then go and interview generally the mother, and ask them around, you know, the history of what happened until that death.
And, you know, most of the time this will be, "Well, it was a fever and then they were floppy," et cetera. And, you know, it's almost impossible to attribute causality to that being malaria versus something else. The other issue is that verbal autopsy studies also assume that individuals die of a single cause, whereas actually, you know, you could think more of malaria as just a general risk factor.
So in these places where there's lots of malaria, you're getting infected possibly every single day. It makes you very anemic. It makes you vulnerable to all sorts of other things. So in a sense, you know, a malaria program can have a much bigger effect than the sort of total number of [00:21:00] so-called malaria deaths.
Elie Hassenfeld: Okay, so why this is hard. So, now, I'd love to just understand what approach you're now taking to estimating this important part of our overall view, and how did you arrive at the approach that you're taking today?
James Watson: So I mean, it's trying to be as simple as possible because the current methods rely on quite complicated models which are often very difficult to really understand what's going into those models and how they're coming to their, you know, final numbers.
And I wanted something where it was based on quantities that you can reliably measure. And for malaria, there's a sort of agreement in the malaria research community that you can measure the intensity of the transmission, so how much people are getting infected, by looking at prevalence in school-aged kids because they're kind of like the Petri dish of infecting the community.
Elie Hassenfeld: James, are those kids getting tested regularly? Is that why we're able to measure prevalence in school-aged kids?
James Watson: Well, [00:22:00] you can do a representative survey. So you can go to a place and select, you know, a few kids at random and you do a very simple, it's a bit like the COVID rapid test. You take a little drop of blood, you put it on a strip, and it says infected, yes, no. So it's very easy, you know, in, I mean, obviously in some areas they're difficult to do because they're difficult to access, but at least in theory you can do representative surveys.
And that prevalence—you know, lots of people have looked into this over the last 50 years—that kind of predicts how much, say, how many mosquitoes are actually infecting individuals every day. So you have this prevalence which, you know, in, say, parts of northern Nigeria where there's a lot of malaria, that could go to, say, 60%.
So, you know, 60% of your kids, six, seven-year-olds, if you gave them a malaria test, they'd test positive. And then in lower burden areas, maybe that's like 10%. And the basic idea is that, the more the prevalence goes up, the more the share of all-cause mortality that's attributable to malaria increases.
Again, that's sort of [00:23:00] also justified by randomized trials that were done in, say, the '90s where, you know, you could see large reductions in transmission and then large reductions in all-cause mortality. So I sort of discussed this idea with some malaria epidemiologists who didn't think it was completely mad.
And then the way that I tried to actually work out what this relationship between prevalence and share of all-cause mortality was to download all the data from—what are they called—the DHS surveys. So these are surveys that have been historically funded by USAID and a kind of way of gauging the health of communities.
And they've been run periodically in countries in Sub-Saharan Africa for the last few decades. All the data are—they're anonymous, but you can download them and analyze them. And then looked at the relationship between prevalence as estimated in these surveys and all-cause mortality as a function of age, and adjusting that for the things that we expect might confound that relationship. So like, you know, access to [00:24:00] vaccination, clean water, GDP per capita, et cetera. And, you know, you can throw everything into that model, but you get quite a robust relationship between prevalence and this sort of increased all-cause mortality.
And what's, I think, the most convincing to me is that that relationship, you don't see it for neonatal deaths, but you see it for post-neonatal deaths. So the neonatal period is the first month of life. So it's your highest risk period as a child. So in some places, you know, a third of all deaths will be in that first month of life. And most of those deaths have nothing to do with malaria. They have to do with, say, birth complications or infections that you get shortly after birth. And actually you're protected against malaria in your first months of life because you've got your maternal antibodies. And so that suggests, well, there's not, you know, loads of confounding going on because we don't see any relationship with neonatal deaths. We see this very strong relationship with post-neonatal deaths.
Elie Hassenfeld: Okay. So let's just try to track this back and make sure that I'm able to hold all this. [00:25:00] We provide a lot of funding to malaria programs. And deciding how much funding to provide, what programs to support and where, an incredibly important input into that, the quantitative part of that decision-making, is how high is the need. Meaning how many people, mostly how many children, are dying from malaria in this area. And there are existing estimates of the number of children who are dying from malaria in different locations, countries, subnational estimates. But because the underlying data about this is scant, there's good reason to question the reliability of those existing estimates.
And so you wanted to take a different approach that relies on all-cause mortality. Basically start with the total number of children who are dying in a given area and then try to estimate the portion of that that's coming from malaria, and that seems to be a more robust approach to arriving at this important input into our decision-making.
James Watson: Yes. I guess the two things to say is that the prevalence of [00:26:00] infection is something you can reliably measure, and also that all-cause mortality you can reliably measure because what DHS surveys do, so they randomly select households in communities, and then they'll ask all women of childbearing age their complete birth history.
So the woman will say, "Well, you know, I've had this many births." And for each birth say whether that child is still alive or died. And that gives you a sort of unbiased sample of the sort of recent past of the mortality rates. So those two quantities you can reliably estimate.
Whereas I suppose the argument is verbal autopsy, it is just not a reliable source of data. And actually something that's very interesting about this is, if you look at verbal autopsy data, this suggests that actually about a quarter of malaria deaths in Africa are in adults.
Now, if you then go and ask people who work in hospitals in malaria-endemic countries in Africa, they will tell you, "Well, we never see an adult in hospital with malaria. [00:27:00] Never." So, you know, where are all these adults dying of malaria? And I think, I mean, that's really, for me, that's the one piece that says, yeah, this source of data is not trustworthy as a kind of primary source for calculating the burden of disease.
Elie Hassenfeld: It's all very interesting. So let's interrogate this a little bit. I'm curious, you know, how the other groups that use different modeling approaches would respond to your critiques. What would they say about the approach you're taking?
James Watson: I'm not completely sure yet. So I have sort of slightly unusual setup at GiveWell, which is I'm part-time GiveWell and then part-time still a researcher. So my part-time researcher, I'm putting some time on writing this up as a research paper, and the nice thing is, you know, you put that out and you see what people say, and people might hate it or they might like it.
Elie Hassenfeld: What do you think are the biggest weaknesses, I would say, in your approach vis-a-vis the current approach? So, you know, we have the current approach, and it has the problem of having [00:28:00] relatively few clearly measurable inputs that go into it. And so you wanted to rely on something that has these two inputs that we know we can measure as your starting point. That has an intuitive appeal that we can clearly rely on prevalence and all-cause mortality. And then I'm curious, what are the downsides of the approach you're taking relative to an alternative modeling approach?
James Watson: I guess the main weakness is what I'm saying is, you know, if you have two areas, say two states in Nigeria that have, let's say similar all-cause mortality, but one has half the prevalence of the other.
I'm interpreting this as causally there's, say, half as much malaria deaths in this state versus that state. Maybe the first objection would be, well, this is probably confounded by all sorts of other things that might be going on. And then I said, well, it's a sort of unanswerable question, is this confounded or not? The other thing I'd say is that these numbers probably shouldn't be taken really at face value, because in a sense it's sort of an unmeasurable [00:29:00] quantity. But it's more sort of semi-quantitative of approximately how bad do we think this is.
But I guess maybe the issue is that the numbers, say, that come from the World Health Organization or for, you know, modeling organizations, a lot of people don't realize just how much uncertainty should be attached to those numbers.
Elie Hassenfeld: Right. All right. Yep, that makes sense. How different do these numbers, where do you land? Where does GiveWell land relative to the existing numbers? You know, how close are they, and then what kind of difference does that make to our grant making?
James Watson: So this is something I think's really interesting, is that if you look at today, let's say 2025, actually, oddly enough, and I really didn't engineer this, the WHO's sort of total numbers and the, you know, the various modeling agencies, kind of land about the same, so it's somewhere between 500,000, 600,000 deaths per year. The World Health Organization reckons in the year, say, 2000, there were maybe let's say 800,000 deaths, and this has gone down to, say, [00:30:00] 600,000. Now, the population under five in sub-Saharan Africa has increased by 80% in that same period. But it's still quite a disappointing reduction over 25 years. Because if you think, you know, the year 2000 as being this really bad year where there were no insecticide-treated bed nets, you know, the drugs weren't working, there was no chemoprevention, there were none of these interventions that we're currently funding.
Actually, you know, the progress that we've made over sort of 25 years seems quite disappointing—and a lot of money has been put into malaria control. Whereas actually if you look at this all-cause mortality versus prevalence, you get almost double the decline in sort of relative terms. Which I guess fits more my priors on just the year 2000 was really, really bad and things are—I mean, they still are quite bad today, but they're a lot better.
Elie Hassenfeld: And then I think even in what you said, I mean, there's wide uncertainty around these numbers. I think you said something like the different estimates come out roughly in the same place, you know, 500 to 600,000 deaths per year. And [00:31:00] even that, I mean, that's not like a huge range, but, you know, that represents a 15% difference in the total need for the program and so even that represents a fairly wide range of uncertainty. How much does the estimation affect more localized decisions? Because, you know, something that we do in our malaria grantmaking is not only ask the question, you know, should we support more of this program globally, but should we support more of this program in this country? And even should we support more of this program in this subnational location in this country, which you know, in many ways is challenging because it's relying on even less data, but also there can be big differences in the need in different locations within a country like Democratic Republic of Congo that has very high need. And so, I don't know, does this play into those kinds of decisions?
James Watson: Yes, well, I think it's essential. So I've started looking at this specifically for Nigeria and, I guess interestingly, if you take, say, World Health Organization numbers for Nigeria, [00:32:00] there's not that much variation in terms of malaria mortality. And again, you know, approximately these two methods, once you take into account the sort of indirect deaths adjustment that GiveWell makes, we land at about five deaths per thousand per year in under fives.
You know, very similar, but this kind of simpler method gives you a much wider range in the different states of Nigeria than the existing estimates. And again, this is part of this issue about the existing estimates being quite difficult to parse and not being very transparent.
So, you know, these are complex what we call geostatistical models, where there's a lot of smoothing and all sorts of adjustments going on. And so, you know, sometimes things look very homogeneous, but actually that's an artifact of, say, how the model is being put together. But, you know, I mean, that's sort of conjecture. Whereas here, I think, you know, you can more clearly see there are differences in all-cause mortality rates, you know, from these DHS surveys. And there definitely are differences in patterns of death across the [00:33:00] ages. And so, you know, that drives more heterogeneity than otherwise we'd assume.
Elie Hassenfeld: Yeah. Well, it's really interesting, so thanks, James. I mean, I think it's such a good illustration of some of the work that GiveWell does, you know, with this whole conversation. It's a number that GiveWell has used going back I guess now nearly 20 years. I bet you could go find whatever number we plugged in in 2007. And so it's like you have a number, but then, you know, even after so many years, there's a lot of work that we're doing, I guess you're doing, to try to come up with better inputs into our decision-making, and it's really going, you know, exceptionally deep, trying to create the state-of-the-art for estimating malaria burden in different locations.
And so just, you know, in some ways it's on the other end of the spectrum of the broadening is the deepening and I think equally important, because both of them are necessary for making good funding decisions. And I think this is a good opportunity, Meghna, to bring you in and talk about your work, especially like this work that we've done to do a better job checking the estimates we make and trying to find errors and ensure [00:34:00] that what we're calculating is calculated well and accurately.
Meghna Ray: Yeah, absolutely. So just for a little bit of context, in 2025 GiveWell made 128 grants, we conduct research like what James and Dilhan describes, and build a cost-effectiveness analysis spreadsheet which lays out our inputs, our assumptions, our calculations, and some of those are quite large.
Just as an example, our spreadsheet for seasonal malaria chemoprevention, or SMC, has 215 rows on the main tab alone, plus eight additional tabs. And these analyses are a very important input into our funding decisions. A wrong formula or an input can meaningfully change how cost-effective a program looks, so it's really important that we get those right as they do impact our grantmaking decisions.
And so previously a research analyst would go through the spreadsheets pretty much cell by cell, line by line to make sure everything was accurate. [00:35:00] And I had the idea of this seems like a thing that AI could be useful for; it could help speed up some of our reviews while maintaining the accuracy of our research.
I thought it would be pretty straightforward and quickly realized that that was not the case. I found that AI was pretty good at checking if a formula was correct, but didn't catch a lot of the things that a human reviewer might catch, such as like if the assumption behind a formula actually makes sense.
And so it ended up being a very iterative process to figure out how we could make this work for us. To try and test and improve the skill, I ended up working with all of the other research analysts to try and compile a list of all of the errors that they had found through this vetting process of checking past cost-effectiveness analyses, to create a comprehensive set of instructions for the AI skill. That was pretty critical as I, at that point in time, was, like, two months into my time at GiveWell. So I was learning how to build these [00:36:00] analyses while also trying to build a tool to help review them. It was also my first time building an AI skill, so I worked very closely with Brendan, who's our resident AI expert, to figure out how to structure a skill and also rigorously test it.
That testing process was pretty comprehensive. I would run the tool, compare it to the human findings, and then see what the skill missed. So I did that for, I think, over 40 past spreadsheets, and in the first couple of tests it was scoring, like, 30%. And then as we continued to improve the skill, it's now catching over 90% of errors.
So the tool is now pretty complex. It has over 30 what we call sub-agents that do different types of checks, and do multiple waves of checks. So now all research analysts are using it as a first pass for our vetting of spreadsheets before we make a grant decision or publish a grant page. So that's saved us a lot of time, to have more research analyst [00:37:00] time solving some of the important questions that James and Dilhan are working on.
Elie Hassenfeld: Yeah. This is a process that's close to my heart. So this is something we call at GiveWell vetting, and I think the origin of vetting is—GiveWell started in 2007, and at some point in, I think, 2009 or 2010, GiveWell was three people at that point, and Holden, Natalie, and I decided to go back and look at some of the research we had published and, check how accurate it was. And we were, um, dismayed, shocked by the number of errors we found when we went back to it a couple years later and looked for errors.
And so ever since then, we've had this process of, at one level of depth or another, of what we call, quote, "vetting our research." And I think there are many, many people who have come through GiveWell over the years who have spent many, many hours poring through spreadsheets trying to find errors and fix problems.
And so this seems like a great way to, hopefully, like, increase the accuracy of our work while also reducing the amount of time that [00:38:00] was needed. Yeah, I'm really curious, how this went at the beginning, and what were the kinds of things that you found where, you know, it wasn't going as smoothly as you would've hoped?
Because sort of by default you might imagine, ask AI to check the calculations is around the, I don't know, sort of like intuitively the thing it should be able to do. And so, can you help us understand more, like, what it couldn't do and then what it took to get to the point where we are today?
Meghna Ray: Yeah. That's a great question. I think just as a very concrete example, say James is working on a malaria spreadsheet and he's using DHS data from 10 years ago and didn't realize that there was newer DHS data available. James, of course, would never make that mistake, but say he did. Initially the AI tool wouldn't have been able to catch that versus maybe a research analyst could catch that.
So now the tool is checking if there is more recently available data. As another example, maybe Dilhan used 45% in one of his medical oxygen [00:39:00] spreadsheets and 30% in a different one. A research analyst who'd looked at both might be able to catch that previously, and the AI tool initially missed that, and now it's looking across various different versions of our spreadsheets to make sure that we're being consistent. So that was definitely part of the process of iterating and seeing where the gaps were and trying to figure out ways to solve those.
Elie Hassenfeld: And so how did we do that? Like, how did we—it sounds like we're building up this list of gaps that need to be filled in. How did we create that list, and where does that work stand today?
Meghna Ray: Yeah, it was definitely building up that list across a lot of testing, countless weeks of testing at this point. But it was truly running a spreadsheet in the skill, and then I would go through and see what the AI skill caught versus what the human vetter, the research analyst, caught, and then seeing where those differences were.
That's how I caught things like not catching more recent data or a parameter being inconsistent. And so, based on what it missed in that test of the [00:40:00] vet, I was able to add that as part of the instructions for the overall skill. There's probably hundreds of different things that the skill is checking for now that we've built out over time.
So now, the skill is doing really well. It's used by all vetters as the first pass. There is still human interpretation that's needed: The skill will produce an output spreadsheet of all of the different errors, or questions it has, and it still takes a research analyst to sift through that and see what they agree with, and then, adding comments to the original spreadsheet for a researcher to review so we can decide how to actually incorporate the changes.
Elie Hassenfeld: And so we still have that research analyst review. Does that mean we're continually building up this, like, longer list of additions to the skill? Is that ongoing work?
Meghna Ray: Yes, exactly. So now after a research analyst runs the vetting skill, at the end it'll ask if there were things that it missed, if there were false positives, things that it flagged as an error that weren't actually errors and don't need to be flagged in the future. [00:41:00] And so the goal is to continue to improve the skill so that it helps save more and more research analyst time. I could see a world in which one day we're just running the skill, but we're not there yet.
Elie Hassenfeld: I think you told me what the average time savings was. You know, so you said we made, like, 130 grants or so last year, and, you know, each one of those, on average, took some amount of time to vet before the skill and takes that amount of time now.
Meghna Ray: So I think it depends a lot on the type of spreadsheet and intervention. But in some cases, it could take a week to vet a spreadsheet thoroughly, if it's a really big, important grant, versus if it's a smaller spreadsheet and it's pre-decision, then maybe it's a day or two. But now for all of the spreadsheets, it takes a couple of hours to run the skill and then a couple hours to interpret the findings. So, right, a huge, huge number of hours saved.
Elie Hassenfeld: Yeah, and it's pretty amazing because we've also hired so many research analysts this year, and in the past, so much of that time, your time I guess, would've been going to just vetting spreadsheets, [00:42:00] and now we're able to do a lot more beyond that.
Meghna Ray: Yeah, exactly. Now I'm able to support Dilhan with some of the medical oxygen research rather than vetting a spreadsheet. Both important, but able to reallocate my time.
Elie Hassenfeld: What do you think is next for this vetting skill? You think it's just slow iteration and slow reduction over time of person effort, or is it going to be something else?
Meghna Ray: I think it's probably slow reduction of person time as we continue to improve the skill. As I mentioned right now, it's catching 90% of errors or more. And so I think we're starting to hit the margin where making tweaks to the skill doesn't catch more errors or improve it significantly. So I'm not exactly sure, but I'm sure there's room for improvement.
Dilhan Perera: Meghna, can I just, I'm very curious, as you added more checks to the skill, did you find that the number of false errors—errors that weren't actually errors—that it was flagging was also going up? And has that become more of a problem over time, or has that fallen? [00:43:00] And if it is a problem, how do you think we could deal with that issue?
Meghna Ray: Yeah, that's a good question. It definitely did increase initially where it would flag that, for example, an assumption that we had made didn't make sense even though in, for example, a cell note the researcher had explained why they made that specific assumption so it wasn't an error.
And so it took more time for a research analyst to sift through. And so now we've integrated into the instructions not only what types of errors it should look for, but what types of things are not errors so that we're avoiding some of those false positives.
Dilhan Perera: That's cool. Thanks.
Elie Hassenfeld: Well, thanks everyone. So we talked about each of your projects. I guess I want to ask a couple questions for you all as a group. And so, you know, one thing that I think is somewhat unique about GiveWell is that the kind of research we do is academic in style.
You know, James, you're talking about your modeling and potentially being able to publish this as an academic paper. And then on the other hand, it's not academic [00:44:00] in the sense that, you know, on an ongoing basis, we're putting money out into the world to make a difference.
There's real, I don't know, there's real urgency to making decisions with imperfect information. And so I guess my question is, for each of you, how do you think about this question of when have I done enough that it's time to act versus, you know, when should I keep going and do more research?
Because I think that there are environments one could be in where not that much research is done, and you're mostly just acting, and there are environments where you just want to keep analyzing because the job is to analyze. And we're at a very, I don't know if I should say unique, but at least rare point on the spectrum that is really pushed very hard, both in terms of the—you know, one of our core values of truth-seeking. We really want to get into it and understand it. On the other hand, we only exist to direct money to programs that help people, so action is critical. Yeah, and so I would just love to hear how each of you think about that.
Dilhan Perera: Yeah. It's such an important question and I don't think there's any easy [00:45:00] answers. In fact, Meghna raised this exact question in one of our New Areas team meetings a couple of weeks ago, and I think all of us are still figuring out how to do this better on an ongoing basis. I think for me, I think about things like how important would a change to this number be in terms of the total amount of funding that it could direct, and trying to take a somewhat longer-term view of that. So not just for the decision I'm trying to make now, how much of a difference would it make? But how much of a difference could it make for future decisions that are related but in different contexts, for example, where the bottom line cost-effectiveness estimate might be much closer to our threshold than in the current case.
And then I also think about, is this a number that is easy to get better information about or is it something that's just fundamentally hard to measure and observe? And I think in the latter case, there becomes a point at which additional work is just not going to give you more certainty, so you have to make a call about, is this amount of uncertainty acceptable given the amount of [00:46:00] funding we're talking about? Do we just accept the risk and move forward, or is there such substantial uncertainty that, like, really undermines the case for the thing that we're considering?
Elie Hassenfeld: What's an example of the latter? So something where it matters a lot, but it's just hard, and you just don't think it's particularly tractable to get a better answer.
Dilhan Perera: Yeah, I think maybe one—I mean, obviously, these things exist on a spectrum, but I think one thing that is hard to observe in a lot of our work is, say a child has pneumonia. What percentage of those children would die if they didn't get any medical oxygen or didn't get amoxicillin?
And I think, you know, obviously for ethical reasons, you can't run an RCT on that to deny children who would otherwise have gotten oxygen or amoxicillin just to understand what this parameter is. So I think those sort of case fatality ratios, as you call it, in the absence of treatment sometimes is a key input to our model and is just very difficult to observe, and there we just have to triangulate lots of imperfect information and make our best [00:47:00] guess.
Elie Hassenfeld: Yeah.
James Watson: So my experience of this is slightly different because the malaria cross-cutting team, sort of small team and we're not really a big grant-making team. So we're not really investigating, you know, large grants that are going to delivery. We're more investigating where there are these problems, uncertainties, and what research could be funded to give you actionable evidence.
I suppose, I mean, there you're thinking about the value of information of a specific piece of research. And that is I think kind of much harder to do. There's a lot more uncertainties, how things can pan out in the future. You know, all sorts of things could happen to make a trial work or not work.
I guess what I find difficult is when you know that you have clearly a trial that would give useful, interesting information, but you're not going to fund it because it's not going to move the needle maybe—that can be quite difficult to explain to researchers. And it is a very different way of thinking about research than any other funding [00:48:00] body. But it's—well, I'm biased, so I think it's the right way of thinking about it. But it's hard.
Elie Hassenfeld: Right. Thinking about research from the perspective of how much would this research change decisions and how much does it matter rather than like are the questions interesting and are there open questions that we'd like to answer? Yeah, that makes sense.
Meghna Ray: I feel like I've fallen down so many rabbit holes just following interesting questions, and a really helpful piece of advice that Dilhan gave me is to just write down all of the questions as I'm doing research instead of jumping into them immediately, and then thinking about which of these is actually going to change our decision-making and really help us in making our decision. So that's been helpful. But, I feel like you could easily do a PhD in a lot of these topics, and we can't do that.
Elie Hassenfeld: Are there any questions that come to mind where you remember going down the rabbit hole and you're like, "Oh, shoot, I wish I hadn't. That wasn't worth it"? And then on the other hand, questions where now you recognize that possibility beforehand and are able to avoid them because of this [00:49:00] approach of being explicit about what questions you're going to pursue?
Meghna Ray: As one example, with some of the TB work, I've been trying to figure out which programs for active case finding could be most impactful and trying to answer this question of what would a world without active case finding look like and how many deaths are currently caused by TB.
A lot of the data is historical data, so trying to think about how data from 100 years ago in a different context applies now, and could easily spend countless hours reading every single paper, and trying to make some judgment calls, but taking a step back and seeing what's the main question and how I can make progress on that.
Elie Hassenfeld: All right. Let me close with a final question, which is what's a part of your work working at GiveWell doing research that someone considering applying for a job at GiveWell might not expect but, you know, really is important to you, you know, [00:50:00] you really enjoy about the work, but someone might not know from the outside?
Dilhan Perera: I don't know how surprising this is, but I think some people have a perception that the work we do at GiveWell is very focused on spreadsheets and reading evidence and, you know, doing a lot of thinking in our heads. But I think one of the most valuable bits of my work at least is just talking to both experts, like academic researchers, and people who are actually designing and implementing programs and seeing how they go and learning from them.
And I think there's just really useful information for our decision-making that I get from each of those types of expert stakeholders. And so that's a really important part of my work, and probably a bigger part of my work than I expected before I started.
James Watson: I think the thing I've enjoyed the most is the diversity of backgrounds in the wider research team. So coming from an academic background where, you know, often you're working with a set of people who often have quite similar backgrounds. And you know, for instance, I think, one unfortunate [00:51:00] aspect of academia is, say, that health economics is often quite divorced from a lot of medical research, and those two worlds don't talk to each other. Whereas they definitely do at GiveWell. So that's, yeah, it's really opened my sort of horizons a lot from having exposure to these various people.
Meghna Ray: Jumping off what James said, my favorite part of GiveWell definitely has been just the people and how kind and smart everyone is, and how collaborative.
I think I was surprised at how much GiveWell is continuously changing and improving, and how even being fairly new to GiveWell and just a research analyst, I was able to change a process—so that's been really cool to see.
--
Elie Hassenfeld: Hi, everyone. This is Elie again. I hope you found that conversation interesting and that it gives a better flavor not only of the what of GiveWell and the types of programs we recommend, but the process that we go through in making [00:52:00] our decisions, and that process includes building cost-effectiveness models, doing deep research to better understand some key inputs into those models.
Also engaging meaningfully with the outside world, with academic experts and implementers, people who see how the programs work in reality, when they're actually serving people. Also people who have spent their lives thinking about the questions that we're trying to focus on. And all of that's supported by work that aims to check our research to ensure that it is accurate, we're identifying errors and ultimately making good decisions, the best decisions we can about very high-stakes outcomes. So thanks again for your interest in our work.
We're hiring now. If you're interested in working at GiveWell, please do see our jobs page. We're looking for more people to help us in our mission of helping people in need as much as we can. Thank you so much.