Subscribe
Share
Share
Embed
Welcome to GiveWell’s podcast sharing the latest updates on our work. Tune in for conversations with GiveWell staff members discussing current priorities of our Research team and recent developments in the global health landscape.
Elie Hassenfeld: [00:00:00] Hey everyone, this is Elie Hassenfeld, GiveWell's co-founder and CEO, and today I'm gonna be talking with Stephan Guyenet, a senior researcher at GiveWell, about vitamin A supplementation.
Vitamin A supplementation is one of the programs we funded for the longest. We support programs through Helen Keller Intl and Nutrition International across more than 10 countries in Africa, and these programs reach millions of children every year. The cost of this program is remarkably low, about $1 to $2 per child per year. And with these programs, we estimate it costs somewhere between $1,000 dollars and $8,500 per life saved, and that depends on the location that we're working in.
When you look at the evidence of this program, which is 11 large randomized controlled trials, we estimate they show about a 20% reduction in child mortality. It's extremely low cost, it can reach massive scale—you know, it really looks like what I think a lot of people would expect from a GiveWell Top Charities program.
It's [00:01:00] evidence backed, it's cost effective, it's scalable, but the case for vitamin A supplementation is much less straightforward than it initially seems. The evidence base for vitamin A supplementation is unusual. And I think in some ways this is unique to vitamin A supplementation, the level of complexity, but I also think it's really a great illustration of how all programs are a lot more complicated than they initially seem.
Vitamin A supplementation has one of the strongest evidence bases of any program that we've looked at. And the vast majority of programs you could look at in the world have much less compelling evidence overall. And still, there are these big questions about it that we're going to talk through. And so I hope that this conversation about vitamin A supplementation helps illustrate the point that even the interventions with some of the strongest evidence bases have really hard, genuinely unresolved questions that really matter for funding decisions.
I think a lot of what we do at GiveWell looks simple from the outside, but it gets a lot more complicated underneath. And this conversation is a chance to show what some of that complexity actually looks like. Hopefully, it also shows how GiveWell's approach has evolved over time, how we've improved the research we do so that we're handling this uncertainty and we're addressing these unanswered questions in what I think are more sophisticated, effective ways. And that is enabling us to do a better job supporting programs that do the most good.
One more note: we are focusing just on the evidence here [00:02:00], and that's because there's a lot to talk about in the evidence for vitamin A supplementation. So it's just about how do we apply and interpret the evidence of this program's impact on mortality to the programs we might fund today. There are a lot of other big questions about this program that we're not going to even get to.
So, with that in mind, Stephan, before we dive in, can you just introduce yourself and your role at GiveWell and your background?
Stephan Guyenet: Sure. Hi, my name is Stephan Guyenet. I am a senior researcher at GiveWell on the nutrition team, and I've been with GiveWell for about eight years now.
I have a BS in biochemistry from the University of Virginia and a PhD in neuroscience from the University of Washington. As a postdoc at the University of Washington, I researched the neuroscience of obesity. I have a long-standing interest in nutrition and have been studying it and writing about it for many years.
But at GiveWell, I've mainly worked on problems of nutritional deficiency, like the health consequences of vitamin A and iron [00:03:00] deficiency, I've been at GiveWell for eight years, and over that time I've worked on a number of projects, mostly related to the evidence and cost-effectiveness modeling that guides our nutrition grantmaking.
Elie Hassenfeld: Yeah, it's amazing. I mean, let's just start off. Vitamin A supplementation has really strong evidence. Walk us through that, what is the evidence base for vitamin A?
Stephan Guyenet: Yeah, so first let me just explain what vitamin A is, where do we normally get it, what are its biological effects.
Vitamin A is a fat-soluble vitamin that we would normally get from foods like eggs, dairy, leafy greens, and certain other vegetables. The vegetable form of it has to be converted by the body. In the US, few people are deficient, but in many low-income parts of the world, deficiency of vitamin A is common.
Vitamin A plays several important roles in the body, particularly in development, in vision, and in immune function. And it's really that immune function effect that we think is the most important to the cost-effectiveness of [00:04:00] vitamin A supplementation.
The role in immune function is to maintain the integrity of mucus membranes, so like in the mouth, sinuses, digestive tract. These are the first line of defense against many pathogens. And without enough vitamin A, mucus membranes weaken and they let pathogens into the body. Vitamin A also supports the function of several types of immune cells that fight pathogens. So without enough vitamin A, those immune cells don't work as well.
And those immune effects are what we think are critical. And the places where we find vitamin A supplementation, infectious diseases are the leading cause of death in children under five. So anything you can do to strengthen immune function is important. So that's the biological rationale for it.
Elie Hassenfeld: I guess like an important takeaway from that is a combination of the fact that it makes sense that vitamin A supplementation would improve outcomes for children in a context with high levels of infectious disease. And, there are places around the [00:05:00] world where kids have very high rates of vitamin A deficiency that are much higher than what we experience in high-income countries.
Stephan Guyenet: Yeah, exactly. But having a biological rationale, you know, having an effect that seems plausible, is not enough. You need direct evidence that it's effective. And in this case, we have an abundance of direct evidence. We have 11 large randomized controlled trials that were designed to measure mortality effects.
So, as you know, this is a rigorous type of study design that compares a group receiving supplements to a group not receiving those supplements and measures the difference in deaths of those two groups. According to our custom meta-analysis, those trials overall report an average 19% reduction in the risk of death from any cause in children six months to five years old.
That adds up to a very strong evidence base and a large effect size. In fact, I'm not aware of any other intervention in global health that has [00:06:00] that strong of an evidence base in terms of the number of large RCTs and the effect size. The only intervention I know of that really comes close is insecticide-treated bednets for preventing malaria, which we also fund.
Vitamin A supplementation, as you mentioned, is also extremely cheap—$1 to $2 per child per year, and most of that is the delivery cost. So if you can go to places where one out of a hundred children under five die each year and avert one-fifth of those deaths by giving them an extremely cheap vitamin twice per year, that would be a huge win, right? But unfortunately, as you said, it's not quite that simple.
Elie Hassenfeld: Right. And so I think that setup is important. It's extremely unusual to have a program where there are a large number of randomized controlled trials that cover a very large number of children that directly measure an effect on mortality, meaning death, and is then very cheap to deliver that program.
Like it sounds, [00:07:00] what you're saying, and I think this is right as far as I know too—it's like vitamin A supplementation and malaria nets are the two programs where we're sort of in that range of evidence, and everything else like really doesn't come close.
Stephan Guyenet: Yeah. And you know, there are, despite that, a lot of people who would say that it doesn't work very well because of the limitations in taking that evidence and applying it to modern-day, current situations where vitamin A supplementation is provided.
Elie Hassenfeld: Right. So walk through that. So there's this sort of like extremely unusual and very strong evidence base that, at least at a first glance, we'd say very few other things, I mean, basically nothing else compares, but yet there are some questions. And so where do those questions come from?
Stephan Guyenet: Yeah. So we can break them down into two categories.
One is called external validity. So, as you know, that's how we take the evidence from the trials and apply it to situations outside where that [00:08:00] evidence was collected. And then there's internal validity, which relates to the integrity of the evidence itself.
And our biggest, most impactful questions really revolve mostly around external validity: how we take that evidence and apply it to other contexts. We have these 11 large randomized controlled trials showing a 19% reduction in under-five deaths. But most of those trials are from the 1980s and 1990s.
And over the course of the last 30 or 40 years, there have been big changes in the infectious disease landscape in low-income countries. Kids are dying less from infectious diseases due to improvements in vaccination, water quality and sanitation, availability of medical care, and other things. And on top of that, we think vitamin A deficiency is less common today, so there's less of a problem that vitamin A is able to fix.
And highlighting that concern, there was a massive vitamin A supplementation trial, [00:09:00] the DEVTA trial, which was conducted in 1999 to 2004 in Uttar Pradesh State, India, that reported a small and non-significant 4% reduction in the risk of death in children under five. So some people see this whole thing and they say, okay, it used to work in the eighties and nineties, but we don't think it works anymore.
Elie Hassenfeld: I think that it can seem straightforward to say, well, there's a lot of rigorous evidence demonstrating that this program works, we're done. But when you look closely, the question is really: What does that evidence say about the places and the programs we might support today? And how, if at all, are those places fundamentally different from the context in which the original evidence was generated and the studies were conducted? And that's where a lot of the questions come from. It's also a lot of what, you know, we have to focus our work on on an ongoing basis.
Stephan Guyenet: Yeah, exactly. Even when you have really strong evidence, it is still hard to [00:10:00] predict what a program is going to do in the real world.
And part of that is that we have a lot of uncertainty about how well the program works in each specific location we model. Vitamin A deficiency surveys—where they go out and they measure the prevalence of vitamin A deficiency in children under five—they're not available for every country, and the quality and recency of the surveys that are available varies.
And even under the best of circumstances where we have good recent surveys, vitamin A deficiency surveys turn out to be surprisingly hard to interpret for technical reasons. So an extreme example of that is a nutrition survey in 2021 in Nigeria that reported a 45-fold discrepancy in vitamin A deficiency rates, depending on which biomarker you look at. So 54% versus 1.2% in children under five.
Adding to that, there's also the issue that the trials on average gave vitamin A every five [00:11:00] months while our programs give it every six months, and that itself could matter. So those are the external validity questions.
Then when we turn to internal validity—again that's about the integrity of the evidence itself—the biggest threat to internal validity that we have dealt with is publication bias. That's when researchers are more likely to publish certain types of studies and not other types of studies. So, for example, researchers might run a study and find a disappointing result and say, eh, it's not worth it to publish this. And so at the level of the entire scientific literature, you in that case would get a bias where effect sizes would look larger and more significant then they really are.
And so these researchers, Rafe Meager, formerly Rachael Meager, and Witold Wiecek found statistical evidence of strong publication bias in the vitamin A [00:12:00] supplementation trials. And using statistical methods to correct for that actually cut the effect size in half. So if that's right, the evidence base is a lot weaker than it looks on the surface.
Elie Hassenfeld: And so, our challenge then is, you know, we start with this very strong evidence base and then we see that there are all these challenges. And then our job is to try to do the best we can to interpret what we make of these challenges, how we try to deal with them, which is what we're gonna talk about next. So, yeah, Stephan, tell us a little bit more about how you’ve thought about these challenges and what you've done and what GiveWell has done to try to get better answers.
Stephan Guyenet: Yeah, absolutely. So, with our higher research capacity, we've been able to dig into some of these uncertainties more deeply, and that has resulted in significant adjustments to our cost-effectiveness model and higher confidence in our cost-effectiveness estimates of the programs we fund. I'll again break this down in terms of external validity and internal validity. [00:13:00]
Again, external validity is taking the evidence and applying it to real-world scenarios. And the bottom line here on external validity is this is really the most impactful thing. This is really the thing that has impacted our estimates the most due to differences between trial and program context.
We do think that vitamin A supplementation probably has a much smaller effect on mortality than what the trials reported—in current program contexts. So the trials on average reported a 19% reduction in mortality. In our cost-effectiveness analysis we're estimating, across the locations we're modeling, a 1% to 11% reduction in mortality with a median effect size of 4%. So that's quite a bit smaller than what we're estimating from just taking those trials and pooling them together.
So one of the things that we've done is rather than just adjusting for differences in the overall risk of dying of infectious [00:14:00] diseases between trial and program context, we did a deep dive to figure out which specific diseases are prevented by vitamin A supplementation and adjust for differences in those. The evidence is far from ironclad, but it suggests that vitamin A supplementation mainly reduces deaths from diarrhea and measles. There may be others, but those seem to be the big two. So one of the things we know about how the infectious disease landscape has changed over time is there's a lot less measles than there used to be in many low-income settings, because of the measles vaccine. So being able to adjust for that more specifically is, we think, a more accurate way of adjusting for changes in the infectious disease landscape.
We've also spent more time thinking about DEVTA, that very large trial in India that reported a small and non-significant 4% reduction in risk of death from vitamin A supplementation. And I'm not going to get into a lot of the technical [00:15:00] details, but the bottom line is we think DEVTA is somewhat informative, but it does not really greatly impact our view of vitamin A supplementation. And that's for a few reasons.
The first one is that on the internal validity side, we think that the vitamin A supplementation coverage rates that were reported in that trial are probably higher than the coverage rate that was actually achieved, accounting for some of the smaller effect size versus previous trials. So basically, their coverage of vitamin A supplementation in the intervention groups were not as high as some of the other trials and were overestimated, we think, in the publication.
Elie Hassenfeld: So we have this study and it shows a 4% non-statistically significant effect on mortality. It also reports coverage of the children in the study. And if it's overreporting coverage, if it actually got lower coverage, it's less surprising that it had a lower effect on mortality. And so we've taken that into account in our [00:16:00] analysis. How do we know that? Like where would that idea come from, that it is overreporting coverage rates?
Stephan Guyenet: Yeah. So essentially the methods of achieving and measuring coverage are, in the paper, not very well-documented. So part of it is like it's hard to verify the numbers that they claim, which is 86% coverage, which would be a good coverage rate if it were true.
However, the limitation of that is it was not drawn from a random sample of each group. It was really drawn from a convenience sample—in other words, children that were able to be reached more easily than the average child.
And I think that's really one of the overarching limitations of the paper is it seems like the vitamin A supplementation and the estimation of coverage rates were really biased towards children that were more easily reached. And so, both of those are [00:17:00] probably overestimated.
Elie Hassenfeld: Got it. Okay. So what we're trying to do here is just go through these big problems and then how we deal with them. And the bottom line is we started with our estimate of a 19% reduction in mortality from the trials. And you know, there's some variation in what we think that mortality is today based on where we're funding, but a median reduction of 4%. And so it's these problems that we're talking through lead us to estimate much lower effects in mortality that then came through in the trials alone.
And notwithstanding that huge reduction, we see the cost of delivering this program and that effect size as, you know, making this a really, cost-effective program overall, taking it all into account.
Stephan Guyenet: Yeah, yeah, absolutely. There are a couple more things I want to say about DEVTA before we move on. The coverage issue, that's not going to explain the discrepancy between 4% and 19%. But there are some other things to keep in mind when interpreting [00:18:00] this, and an important one is that one trial in one particular context does not erase the result of 10 other trials in a variety of other contexts, no matter how large it is, and DEVTA was very large.
In the trials, vitamin A supplementation worked much better in some contexts than others—even if we exclude DEVTA—for reasons that we mostly don't understand. And that suggests that context matters a lot. And it's possible that vitamin A supplementation just isn't that effective in Uttar Pradesh, India, for reasons that we don't really understand. BUt that doesn't make a compelling case that it's not effective in other contexts when we have trials from other contexts reporting that it is effective.
But the other point I want to make is that, after adjustments, the effect sizes we estimate in our cost-effectiveness analysis aren't actually that different from DEVTA. DEVTA reported a non-significant 4% reduction in under-five mortality as its central estimate, with an uncertainty range going from a 3% increase in [00:19:00] mortality to 11% decrease in mortality. That uncertainty range is actually very similar to the range across different locations in our CEA. Our median value is 4%, and it ranges from 1% to 11% across locations. So I don't think the finding is actually really in conflict with the estimates that we're getting anyway.
So, putting DEVTA aside, we've also done a deep dive into methods for assessing vitamin A status so we can interpret survey results better. So I told you about that crazy 45-fold discrepancy in vitamin A deficiency rates reported in that 2021 nutrition survey in Nigeria. We still have a lot of uncertainty about how to weight those two figures, how to incorporate those into our model. And we've spoken with a bunch of experts. No one really has a clear answer. But we did put those two measures together, so we are weighting them together, and that resulted in a large downward adjustment to our estimate [00:20:00] of vitamin A deficiency rate in Nigeria, reflecting evidence from both biomarker methods.
But there are fundamental uncertainties here about how vitamin A deficiency is measured that are very obvious in what I'm saying that we can't address using desk research. The amount of evidence that is currently available through published papers and experts is not sufficient to answer these questions. We actually need new science, and I'm excited to say that we are funding a study that will partially address this issue, particularly the discrepancy in Nigeria. I'm not able to share details publicly yet, but we'll be sharing that in the coming months. And I think that is a perfect example of how our expanded capacity is allowing us to not just comb through existing research, but actually fund new research where there are critical knowledge gaps that impact our grantmaking.
The last piece on external validity that I'll mention is one that I'm particularly proud of. [00:21:00] We did an analysis of the relationship between how often vitamin A supplementation was delivered in the trials and what the impact on the risk of death was. And what we found is that the more frequently vitamin A is given, the larger the reduction in the risk of death, independently of cumulative dose. So even if the same amount of vitamin A total is given per year, the more often you give that—so more frequent, smaller doses—the larger the reduction in mortality that you tend to see. And for that analysis we got the help of a professional statistician at UC Davis named Charles Arnold.
I think this is really important because as far as I know, this is the first time anyone has identified a significant predictor of why some vitamin A supplementation trials worked better than others. And it matters because the trials in our meta-analysis delivered vitamin A every five months on average, while our programs deliver it every six months.
And [00:22:00] it turns out that when you give vitamin A supplementation, it only lasts about four months in the body before it's depleted. So any interval longer than that is basically leaving children unprotected for part of the year. Accounting for that resulted in a 17% decline in our estimate of how effective modern vitamin A supplementation programs are at reducing mortality.
Another thing that came out of that is that vitamin A fortification or consistent dietary intake probably reduces mortality more effectively than vitamin A supplementation does. And this is basically what almost any expert would say is probably true. Vitamin A supplementation was always intended as a kind of stopgap measure that you use until you can get people to eat a more nutritionally complete diet. But there was never any direct evidence for that until now. And that also affects how [00:23:00] we look at the future funding opportunities in this space, for example, vitamin A interventions beyond supplementation like fortification and dietary diversity.
Elie Hassenfeld: Something that's like worth seeing in all of this is how our larger team today is able to do a lot more than we could in the past. So just in talking about dealing with the external validity issues related to vitamin A supplementation, we have a deeper dive on the infectious disease landscape that we understand its implications for vitamin A supplementation. Well, that kind of analysis takes a lot of time. That's not always time that we had.
You know, we're funding external research on vitamin A deficiency to get better answers to scientific questions that the answers just don't exist.
And then when there's really deep technical analysis that needs to be done, that's outside the scope of our normal work, we can get someone's help, you know, a professional statistician who can really help us figure out how to synthesize the data that we have [00:24:00] to arrive at better conclusions and then, you know, pour that all into the analysis that we're doing.
Stephan Guyenet: Yeah, absolutely.
We've also done work on internal validity. We talked about the publication bias issue that was raised by Rafe and Witold, which initially looked like it might cut our estimate of vitamin A supplementation effectiveness in half. And this one was particularly satisfying to me because many of the questions we grapple with about the effectiveness of vitamin A supplementation do not have high certainty answers. The best we can do is reduce our uncertainty somewhat. But in this case, we think we came up with a fairly clear answer using an approach that we developed in-house.
The issue with publication bias, as we discussed, is that some studies go unpublished, especially smaller studies that have fewer resources invested in them. Researchers get a disappointing result and they feel like it's not worth it to publish. What we did to address this is to restrict our analysis to the [00:25:00] 11 large trials that were designed and conducted to measure mortality effects. These were big resource-intensive trials that would've been very hard to conceal from other researchers.
We then interviewed vitamin A supplementation trialists, Keith West and Reynaldo Martorell, to ask them whether they know of any mortality trials that went unpublished. Keith West in particular has been around for all the trials, he conducted some of them, and he has an encyclopedic knowledge of them. Both Keith and Reynaldo were confident that there are no unpublished mortality trials. And that gives us fairly high confidence that there is no publication bias within the subset of 11 vitamin A supplementation trials that are large and designed to measure mortality effects.
So we did a custom meta-analysis of just those 11 mortality trials, and that is what yielded that 19% [00:26:00] reduction in mortality figure that we've been talking about. And our cost-effectiveness analysis is now based around that 19% figure, which we don't have to adjust for publication bias because we don't think there is any.
Elie Hassenfeld: You know, one thing I can imagine someone asking in listening to this entire conversation is in some ways it seems like we ended up where we began. You know, vitamin A supplementation, this, you know, really evidence-backed program. Then we articulate all these issues in the evidence, but in some ways it almost seems like we've just dealt with all of them and safeguarded vitamin A supplementation and it sort of remains. GiveWell has had Helen Keller Intl as a Top Charity for many years now, and those years have spanned all of this work that we've been doing on vitamin A supplementation.
So I guess the question I have for you is, at the end of the day, GiveWell is not an academic research institution. We're a group that is trying to get money to organizations that are delivering programs in the world. And so the real question that matters on some level, or maybe the only question that matters is, you know, how big a deal is this for our grantmaking? Like, how [00:27:00] differently have we funded, are we funding, will we fund as a result of all this work?
Stephan Guyenet: Yeah, so our current cost-effectiveness analysis is very different from our 2017 cost-effectiveness analysis, back before I joined GiveWell and before we did all the work that I described today. It's so different that it's pretty hard to compare. But I can tell you that back then we estimated that vitamin A supplementation through Helen Keller Intl was about nine times as cost-effective as our benchmark. And our latest published grant to Helen Keller for Vitamin A supplementation in eight countries in 2025 has an estimated cost-effectiveness of 25 times our benchmark. So it looks like the net effect may have made vitamin A supplementation look better, though it's hard to say because our old CEA doesn't break the estimate down by location, and so I can't say the locations being compared are the [00:28:00] same.
But to me what the most important thing is, is that we can make grants more confidently today. We have a better ability to identify the best locations for vitamin A supplementation. We have a better idea to help implementers refine how they run their programs because we've identified and gone down rabbit holes that we didn't have the capacity to go down before.
And really, as you said, it's honestly pretty impressive to me that with the number of rabbit holes we've gone down and the amount of critical scrutiny that we've applied, vitamin A supplementation has held up pretty well so far, but we're always looking for that next update.
Elie Hassenfeld: Yeah, and I think we've talked about something and you said something here that is really material and I think maybe people don't fully understand, which is, in our earliest days, GiveWell supported programs and organizations. So we would support vitamin A supplementation at Helen Keller Intl. And that's changed now where we're looking very [00:29:00] individually at program locations at a national level or even a subnational level for a program that depend on the cost and the impact we expect in that location.
And so these more refined estimates were, you described a range of, you know, a 1% reduction in mortality to an 11% reduction in mortality—those are very large differences in the estimated effect of this program in different locations. And, the work that we're doing is enabling us, like you said, to make those location-specific decisions much more effectively than we did in the past.
And while there are some programs that are either so obviously worth funding or so perhaps, not going to be the, you know, the most cost-effective use of our funding that we don't fund. Well, those probably don't change very much because of this research, but the ones that are closer to the line and, you know, they could go either way, the work that we're doing here ultimately makes a difference to, you know, whether we allocate those funds to that [00:30:00] program or somewhere else. And of course that's really important because we want to allocate funds where they'll do the most good.
Stephan Guyenet: Yeah, and not only that, a lot of the things that we've looked into, they didn't tank vitamin A supplementation, but they could have. And so I feel like we ended up in a place where we're still funding vitamin A supplementation, but that was not clear when we started working on some of these projects that it was going to end up that way.
Elie Hassenfeld: Right, right. That makes sense. What are you thinking about going forward, with respect to the evidence for vitamin A supplementation? You know, what else is on your mind that could really make a difference in the future to our thinking about this important program?
Stephan Guyenet: Yeah, so I'm an evidence guy. You know, I'm working mostly on intervention research and modeling, and so from that perspective, we think that we have picked most of the low-hanging fruit we can pick by sifting through published evidence, speaking with experts, and incorporating that information into our cost-effectiveness model.
Elie Hassenfeld: And I think it's funny that you call [00:31:00] it low-hanging fruit. It's years of work that you've and others have put in, but we'll call it low-hanging fruit, relatively speaking.
Stephan Guyenet: Yeah. And there, you know, there could always be things that come up that are things we haven't thought of yet that maybe are important. Maybe someone will come to us with an idea that we need to look into more deeply. But for right now, our big research questions that we can address by desk research and speaking with experts, we have mostly addressed those.
So the main thing that remains right now on the intervention research side is managing new research we're funding or exploring that we think could substantially improve our grantmaking. So, for example, the research I mentioned to clarify the usefulness of various methods for assessing vitamin A deficiency. Research looking at alternative ways of estimating the impact of vitamin A on mortality, like measuring the impact of the rollout of historical vitamin A fortification programs. Possible new vitamin [00:32:00] A deficiency surveys in key countries. And then possibly even doing a new vitamin A supplementation trial.
Elie Hassenfeld: And say more about that last one because that's really interesting. In some ways the new trial could answer a lot of questions about the effect of the program today in the contexts we're looking at.
And so those are the reasons we would do it. You know, what are the reasons we wouldn't, how likely do you think it is that we do?
Stephan Guyenet: Yeah, yeah. Can't leave people on a cliffhanger on that one, right? Yeah. So I mean, ultimately, the very best way to determine how well vitamin A supplementation works in the contexts we fund would be to run a new randomized controlled trial.
And we've been looking into that possibility with the UCSF team that ran the MORDOR trial, which was a trial of azithromycin mass drug administration in Africa. I won't get into the details of it, but it was a large, high-quality trial that is very analogous to the type of trial we would want to run in this context.
And we would love to [00:33:00] run a new trial—it's really not clear whether it will be possible. There are a couple of major barriers to this, and those fall into the categories of practical barriers and ethical barriers. The practical barriers are, given declining death rates and vitamin A deficiency rates, which is good, the trial would have to be very large and expensive to be able to detect a mortality effect. So, as you know, the more rare an outcome is, the bigger a study you need to detect it, and that makes studies more logistically complex, more expensive. We did an analysis of how big the trial would have to be in different locations to detect the effect size we anticipate. And there are only a few locations where we think we could do the study with fewer than a million children. So potentially very large trial, not clear that it would be feasible on a practical level.
There are also ethical concerns. So we're already funding [00:34:00] vitamin A supplementation in the places that we think need it the most. And so practically speaking, those are also the places where it would be best to conduct a trial because there's high mortality, there's high vitamin A deficiency rates. But since we think vitamin A supplementation works, we can't ethically withdraw it from people who are currently receiving it in order to have a control group that doesn't receive vitamin A supplementation. So we have to get creative with location and study design to look for opportunities that don't cross any ethical boundaries.
And yeah, those are two formidable challenges, and right now it's not looking great for being able to work that out. But we're still investigating it, and it is possible that it could happen.
Elie Hassenfeld: Great. Well, that's super interesting. Thanks so much, Stephan. This was great.
Stephan Guyenet: Yeah. Thank you, Elie.
--
Elie Hassenfeld: It's Elie again. Well, I hope that episode was interesting and helped explain a lot about GiveWell and [00:35:00] also a lot about how GiveWell has changed and what it means for the work that we do.
GiveWell is a place where we know that we have to go deep on the programs that we look at to understand them well and make good decisions. And Vitamin A is certainly an illustration of that. We went through, as Stephan described, a lot of twists and turns, went really deep on a lot of different questions. Some of those questions, as Stephan said, could have tanked the program overall. And others, you know, didn't but allowed us to have better estimates of the overall cost-effectiveness of this program in different locations, which means that, you know, because of all this work, we're able to direct donor dollars to higher-impact places per dollar than we otherwise would have.
And then I think it also says a lot about how GiveWell has evolved. The depth of our research and what we've been able to do has changed dramatically over the last 10 years. Where in the past we did our best with a small staff to make reasonable decisions about the evidence we saw, but today with our larger team, [00:36:00] we can do deeper desk research, we can fund research to generate new and improved data, and we can retain outsiders who are technical experts to help us do the kind of work that we're not able to do ourselves.
To me, all of that means that as we've grown and are responsible for directing larger amounts of money, it's not only the case that our research is maintaining the level of quality that it had in the past, but, in my opinion, it's reaching a whole new level where we're making much better decisions than we ever have about how to allocate funds.
So as always, thank you so much for your interest in our work, for your support, and for taking the time to listen to what we're doing. If you'd like to support our work, you can do that on our website, by hitting the donate button.