Subscribe
Share
Share
Embed
Welcome to GiveWell’s podcast sharing the latest updates on our work. Tune in for conversations with GiveWell staff members discussing current priorities of our Research team and recent developments in the global health landscape.
Elie Hassenfeld: [00:00:00] Hey everyone, this is Elie Hassenfeld, GiveWell's co-founder and CEO.
We're going to do something a little bit different in this episode. At GiveWell, we have an internal podcast, which is often a conversation between two staff members— often a researcher—to talk in detail about their work, and it helps staff learn. And today we're going to share one of those internal conversations with you.
This conversation will be more casual, and it will assume some knowledge. But when I listened to this conversation in GiveWell's internal podcast, I thought it was really interesting and informative, and so we're excited to share it. It focuses on an assessment we did of a past grant that we made to the Against Malaria Foundation for distributing malaria nets in Democratic Republic of Congo. This conversation was between three people, our Chief Research and Program Officer, Teryn Mattox, Program Director for our [00:01:00] cross-cutting team, Alex Cohen, and a Researcher on the cross-cutting team, Steven Brownstone.
So it covers this work that we did to assess the impact of a grant that we'd previously made, and as far as we can tell based on all this work, the impact of the grant was consistent with our expectations. Basically, what we expected to happen did in fact happen. As you'll see in the conversation, we primarily looked at two things in this post-grant assessment.
First, we looked at child mortality in Democratic Republic of Congo following the net campaign, and in the conversation, Steven is going to describe the approach we took to this quantitative analysis. And then we also supported additional qualitative research, and this was trying to find other kinds of issues that we might have missed or might not have picked up in our quantification that could help us understand how the grant went. The conversation is just an overview; we'll be sharing a more detailed writeup on our website in the coming weeks.
[00:02:00] This conversation was recorded for an internal audience, so there are many terms that are not defined. We're going to post a glossary of key terms in this episode at blog.givewell.org. A few things just to know going in, so we'll use the term CEA, that stands for cost-effectiveness analysis. DHS, which is the Demographic and Health Survey that collects a lot of data for low-income countries. We'll talk about AMF, which is the Against Malaria Foundation, and then DRC, which is the country, Democratic Republic of Congo.
You know, at GiveWell, we do a lot of research before we make a grant. It's obviously important to do that kind of intense research, but I think it's also really critical to follow up afterwards and see whether the real world impacts matched our assumptions. So I'm just really excited that we're able to share this work and this conversation with you.
–
Teryn Mattox: Alright, this is Teryn [00:03:00] again with another episode of the pod, here with Alex and Steven Brownstone to talk about our recent lookback on what has happened in DRC with AMF's nets programming. Thank you guys for joining.
Alex Cohen: Glad to be here.
Steven Brownstone: Happy to be here.
Teryn Mattox: Glad to have you. So for context for folks listening, because DRC is very big and has lots of people and lots of malaria, the malaria need there goes broadly unfunded beyond a certain level. So this is a place that we've funded nets before, but one thing that we really wanted to do was look back on how previous net campaigns have gone in DRC so that we could just understand, did they go well? Are there any surprises? Did they go better or worse than we thought?
Alex and Steven are part of a bigger team that was doing this. Didn't want to pull everybody from that team, but I think that lots of folks on cross-cutting were working on this lookback, so just wanted to acknowledge that up front.
And then I guess I wanted to start out [00:04:00] with, can you guys just walk me through, you did so many different things on this lookback. Part of the reason I wanted to talk to you is because I think it was very cool what you did, and I wanted to dig in on some of those cool things. Can you walk me through like the different chunks of what was involved in this lookback, and maybe we can go through those in more detail in a second?
Steven Brownstone: Sure. Do you want me to take that out?
Alex Cohen: Yeah, do it Steven. And I should say too that I was technically manager on this project, but Steven and Brian and Katie did all the work. So, any credit goes to them. But yeah, Steven, go ahead.
Steven Brownstone: So I guess, I mean there were basically three broad sets of analysis that we did. So I think the first was maybe the more traditional lookback approach of just really diving into the CEA and figuring out how that's changed since we made the 2023 grant, and what that can tell us about what parameters and factors might matter. That was really led by Brian.
The second big chunk of work we did, which I think was relatively novel, [00:05:00] was really diving into what we could glean from Demographic and Health Survey micro data. So there was a recent Demographic and Health Survey in the DRC that allowed us to really get a lot of insight into various aspects of bed net distributions there. And so we did kind of a deep dive on that, in addition to the standard thing of looking at data shared by AMF. And so trying to triangulate that to some of the Demographic Health Survey data. And then we also had some quantitative data that Rosie had been working on with a survey firm in the DRC that we were also triangulating against. So we had kind of like three sources of relatively rich quantitative data that we were looking at.
And then third, which is somewhat unique, is we did also sort of a big push on qualitative information about net campaigns in the DRC. So in addition to scheduling more OnFrontiers calls than usual, we also commissioned our survey firm for the quantitative survey, Marakuja, to do in depth [00:06:00] qualitative interviews with every level of people involved in bed net campaigns across five different provinces in the DRC. And so this was from health zone level administrators, all the way down to focus groups of mothers in villages, to understand qualitatively, how these net campaigns actually happen and where the gaps might exist.
Those are the broad buckets of things that we did.
Alex Cohen: Yeah, and just to give some background on this, relative to previous lookbacks. So lookbacks are relatively new. We did our first official lookbacks where we looked retrospectively at how grants went, published those last year. And it was typically, we’d do the CEA retrospective, talked about how our research changed, maybe include some of the monitoring and evaluation and costing data that we had, and maybe in some cases, have a conversation with the charity to learn about implementation. But definitely didn't do all these other pieces.
And that was [00:07:00] partially by choice. We were, you know, these were first versions of them, and were limiting the capacity that we spent on them. But, yeah, just to give a sense of what's a little bit deeper here. We've got more of this qualitative data, more of this original empirical work, not using the M&E from AMF but using these other sources too. We also got this Justin Sandefur initial report at the same time, which was in sync with some of the findings here, which was a nice, I don't know, coincidence that that came in around the same time here. But lots of extra stuff that we didn't have in our original lookbacks.
Teryn Mattox: Yeah, I had forgotten about the Justin…so this Justin Sandefur grant was something that we commissioned, that Alex, I think you worked with him to commission, to understand, are we really filling actual gaps in these places where we make these mega Top Charity grants? Or if we didn't do them, would somebody else come in? And that report, I think I have a draft in my Asana inbox that we just like hot off the presses have a [00:08:00] report from him on that.
Okay, cool. So I don't want to spend too much time. I think I'm really excited to talk about those latter two categories on using these two novel approaches, because it's just like exactly the type of thing that we couldn't have done a year ago. Or definitely not two years ago, probably not a year ago. Like the capacity we've got, the team we have on crosscutting is really set up to do—and on the net team, frankly, is set up to do this kind of work now—that we couldn't have done historically. And I find it very exciting and very compelling.
But I do want to start quickly with the traditional approach. What were the top one or two findings from this work of diving back into the CEA, figuring out what has changed, what did we learn from that?
Steven Brownstone: Yeah, I think surprisingly there…there was basically a lot of adjustments that changed. There was sort of how we thought about funging really kind of changed between the [00:09:00] CEAs. And even I think this move from thinking about there would be no campaigns in our absence, to us just shortening the interval of the campaigns. So I think there was like structural changes about how the nets team thinks about the broader kind of landscape of bed net funding, that sort of changed the cost effectiveness, between the retrospective CEA and doing the CEA now.
I think going through the CEA, it struck us how much, I mean I guess the nets team is aware of this, but how much the net durability estimates matter, right? Like if you look in the CEA, the coverage in the CEA we assume declines really rapidly from year one to year two to year three. And so I think that kind of guided some of the empirical investigation.
Then relatedly population is this very important parameter in the CEA and that, I don't think we made as much progress there, but it sort of highlighted the challenges that we have in understanding population estimation.
Teryn Mattox: So on that [00:10:00] first one, on the adjustments that changed, I know there's all of these like little things, and they kind of add up over time. But I'm curious to learn a little bit more about how we think about what we're doing here. Which is like, are we making sure that campaigns happen at all, or are we shortening the interval between campaigns? Can you talk me through what we thought and now what we think, and why?
Steven Brownstone: So, I think historically, we thought that net campaigns would happen regardless of GiveWell funding, and what GiveWell funding was doing was really just reducing the time from one campaign to the next. And so that means that basically people would get some protection in our absence of the campaigns, from the nets that were left over, but because nets do degrade, there was this advantage of people getting nets earlier.
And I think a combination [00:11:00] of things happened. Like one, I think particularly the DRC, we got updated data that the nets decay faster than we thought they did. So that by year three, our best guess is that there's not…I mean, there's some meaningful coverage left over, but it declines pretty quickly after year three.
Teryn Mattox: Gotcha. So we're like, okay, there's nobody coming to do this in our absence. Maybe DRC makes a decision to roll out nets more slowly. So maybe there literally are campaigns that are going to happen more slowly, but the nets are degrading so fast that we think there's effectively no coverage past a certain point.
And can you tell me a little bit about this net durability declining rapidly? So this was an update for us, is that right? And what data did we use, and how confident are we? How much is this DRC-specific, or is it like all-net-specific?
Steven Brownstone: So I think this was an update before the lookback, so I think this is a thing that nets team had been aware of, and this is something I think AMF had flagged. So this [00:12:00] is from the PDM data that AMF was doing, they look at net durability, they track it over time. PMI also does very official net durability studies where they track the chemical efficacy of these nets. So that's actually why the DRC used to be on 36 month cycles, and it was reduced to 30 month cycles, partially because of this data that AMF had put together on low net durability in the DRC.
Teryn Mattox: And do we know if it's just in DRC, or is this data that should update us everywhere?
Steven Brownstone: This is DRC specific because it actually came from like…they're running these PDM data, like their data system exists everywhere right? But it was the DRC where there was this sort of concern that was flagged.
Teryn Mattox: And do we have any sense, maybe this came from some of our other work, of why it would be different in the DRC?
Steven Brownstone: Yeah, I think the best understanding, is it just…[00:13:00]no, I'm so sorry. The short answer is no. I think there's, there are guesses, like people talk about just the houses being less well-constructed, or rainstorms being worse, or bugs being worse. But I don't think…
I tried to test for some of this in the DHS survey data and didn't find much of anything. It doesn't mean these factors aren't real. But yeah, I don't think we have a strong, conclusive story as to why DRC net durability is lower than other countries, aside from maybe it just being worse weather.
Teryn Mattox: And how does weather play into it?
Steven Brownstone: I think just like rainstorms and moisture are bad for nets.
Teryn Mattox: Mm-hmm. Right. Kind of, yeah, bad for most materials. Yeah, that makes sense.
It's interesting, I feel like this is just such a common [00:14:00] thread in vector control is like, we have these places where it seems like durability is just way off from what we would expect, and I feel like I'm never satisfied. We never know the answer to why. It seems like it's something that we need to understand better.
Okay, cool. Okay, great. So let's move on. That all makes a ton of sense. And then, yeah, let's move now to this DHS micro data work, kind of like these novel analyses that you did. Can you kind of…well, why don't you tell me what you did, and then we can dig into what I think I'm most interested in.
Steven Brownstone: That sounds good. Yeah, so what I did started with just the observation that despite the DHS happening at one point in time, the time since the last campaign varied a lot across the different provinces, because of this staggered nature of campaign timing in the DRC.
So I think it actually might have started with just me saying, you know, these provinces that the campaign [00:15:00] happened three months, or less than six months ago, what can the DHS tell us as like a fake post-distribution monitoring survey. But then, I started running some regressions and realized that there's this really sharp linear relationship between time since campaign and net use. And then that was sort of…I just kept running regressions.
And then also in the back of my head, I think in a separate track, I had found this paper when we were doing planning for the lookback that used basically the timing of children's births relative to bed net campaigns to try to measure an all-cause mortality effect of the bed net campaigns. And I thought, oh, now we have all these cool AI tools, maybe we can replicate this, but with the more recent DHS data.
And initially I gave up. I got the replication package from that paper, and then it seemed like they were using some control variables that would be hard to reconstruct in the current data. But then as I pushed forward on the basic [00:16:00] analysis and got to understand what the richness of the DHS data was better, I was like, why not? Let's try the all-cause mortality stuff.
And actually it worked partially because I had forgotten that the old paper had only nine provinces of variation, and now the DRC has 26 provinces. So there was a lot more variation in the modern DHS data across provinces than there was in the old paper that I was working off of.
Teryn Mattox: So what's going on here is we have kind of this rolling set of campaigns. So in a certain province, a campaign might start January 1st, and then another province, a campaign might start June 1st, and then another province, January 1st the next year. You know, these things happen over several years.
And so what we can look at is children that were born right before a campaign and compare them to children that were born right after a campaign. Like crudely, this is what you're talking about, right? And see what happens. And because so much of the mortality for malaria is concentrated at younger ages, this is a really kind of like, [00:17:00] like, oh yeah, it's, it's, I'm getting sad…because it's, deaths happen at such young ages. This is where a lot of that mortality is concentrated, so we would expect, especially if this net decay information is right, that those kids that are born right before a campaign effectively have no coverage, right? No net coverage. And so those kids that are born right after a campaign would have much more.
So what you're saying is this is almost like a little experiment. Of course, the province assignment and timing is not random, so we can talk about how you dealt with that—or I think it's not random? But basically, what you did is said like the very stylized version is, let's look at kids that had effectively no net protection when they were born and kids that had effectively full net protection when they were born and see the difference. Is that right?
Steven Brownstone: Yeah, that's roughly right for the mortality. I think the slight hinge is we're actually looking at death risk. So it actually, like even kids that were born a little before the campaign, we count them as protected as soon as [00:18:00] they're protected. Not just the status of the net when you were born, but it's like the full time horizon of when you had a net and didn't have a net. And the key identifying variation for that is actually just when you were born, relative to the time of the campaign.
So you can use province fixed effects because the key comparison, you think, two kids in the same village—one got lucky and was born a little after the campaign happened to happen, and one was unlucky and was born a bit before. And the idea is that the parents are not planning the timing of their births around these campaigns, which end up being like…we can't even predict well when these campaigns are happening, and we're the people that fund them. So that's kind of where the identifying variation for the mortality analysis came from.
Teryn Mattox: Gotcha. That makes a ton of sense. And then, so can you just walk me through your key finding here?
Steven Brownstone: Yeah. So the key finding is that it looks like all the risk of [00:19:00] dying, so this is the Cox hazard rate, decreases by about…it's a 27% reduction in the mortality hazard. So that's like, I think the easiest way to think about that is sort of like the risk of dying in the “bed net world” is 27% lower than the risk of dying in the “no bed net world.”
Teryn Mattox: And that's all-cause?
Steven Brownstone: And that's all-cause mortality. Or, I guess the idea is that the fraction of other causes of mortality should be kind of fixed, right? So the fact that it's all causes are attenuating. But because we have the baseline and the end line risk, the kids are dying of other things, but those other things shouldn't be changing, just around the timing of the bed net campaigns.
Alex Cohen: That's pretty close, that 25, 30% magnitude, that's pretty close to what we're assuming in the CEA?
Steven Brownstone: It's pretty close to…so this is where it gets a little more complicated. [00:20:00] So, this analysis has to pool across age year. So for power reasons, we can't really do the decay thing that we do in the CEA. And so it's sort of a mix of the year one and year two effects that are in the CEA. And so I think it is close to the CEA, but it's not exactly apples-to-apples because the year decay thing is kind of hard to exactly reconcile across the two approaches.
Alex Cohen: Yeah, and probably wide confidence intervals too, it's not like a tight 27.3%.
Steven Brownstone: Yeah, yeah, the 95% CI is between 8% and 32%. Yeah, I'm not advocating, we just plug this 27% number into the CEA tomorrow, but I think it's compelling evidence that there is, sort of like these mortality effects that we would expect to see are real.
Teryn Mattox: Right. It's astonishing. I think this is so great. And something that I was thinking about last [00:21:00] night as I was just thinking about how great this is that you did this. I mean, 27%...even if it ends up being 15 or whatever, like wherever it lands, reduction in all-cause mortality is just so enormous.
And it is surprising to me that we haven't in other places, using other data sets, been able to tease out the mortality effect of our programming in this way. I know that we've looked historically, I think we've tried to tease this kind of effect out and have just found that maybe mortality measurement was too noisy or something. I feel like we haven't really been able to get at this before with triangulated data.
Steven Brownstone: One of the issues with using triangulated data is that you're maybe comparing mortality that's being collected in different ways. But the advantage here is we're just taking the same DHS birth histories for essentially the same women in the same villages and just really leveraging this fine [00:22:00] brain data on campaign timing. But that kind of allows controlling for province fixed effects, we have kind of rich controls from malaria ecology and household demographics.
So I think part of maybe why this analysis works, whereas just kind of comparing all-cause mortality in different regions of the world doesn't work as well, is there's like so many things that can drive differences in all-cause mortality. And so what you really need is these almost quasi experimental bits of variations so that you can hold all those other things that are driving differences in all-cause mortality constant to really see these effects.
And so I haven't looked at our other analysis, but my guess is that was part of the issue with when we've had to do this in the past is we haven't been looking as tightly within the same samples.
Teryn Mattox: And also maybe there's something that's just, the timing seems very fortuitous here as well. Is that right, or is the mini DHS going all the time and we [00:23:00] can do this again?
Steven Brownstone: I think, yeah, I think anywhere we have detailed…I think the other thing that makes it sharper in the DRC than maybe some other contexts in sub-Saharan Africa is that the vast majority of nets in the DRC are campaign nets, like even more so than other countries. So I think they are really reliant on nets that just come from these bed net campaigns, which are on this very fixed timing.
Whereas, I think in other countries, the routine net distribution system through antenatal care visits and immunization may be working a bit better, or people may be buying more nets on the private market, which we see in Nigeria. And so that attenuates, to some degree, the steepness of these cycles, or like the degree to which campaign timing matters, which is really stark in the DRC. But I think a next step could be trying to replicate this analysis in other countries, like the basic structure could be replicated.
Teryn Mattox: Yep.
Alex Cohen: And we do have [00:24:00] some…we've got an older page on more macro studies on this, looking at across country variation in net coverage and other malaria programs, and some more micro studies. But I think, like Steven's saying, we've got maybe a little bit more variation in this case, even though it's within DRC. But yeah, there is some other evidence there. And I think we probably, you know, it's not like we put 90% weight on this one study. This is one piece of evidence. But yeah, I do think there are some unique benefits here.
And also, I should add, this is something we've talked about doing before, we're like, we should do this, there's like timing of net coverage, and DHS, and regression discontinuity. Why don't we do this? And just like never got it off the ground. And when Steven proposed this, I was like, “I don't know, Steven, we've just thought about this before and it never worked out.” And then he just did it, which is really cool. And I think for GiveWell people listening, especially newer people, if we say, “yeah, we've thought about that before and it didn't [00:25:00] work out,” don't necessarily believe that. We've got a lot more capacity now to do things that we may not have been able to do before.
Steven Brownstone: I think in the world, even a year ago, I wouldn't have been able to do all of this in the time that I had allotted. Shout out here to Claude code, but it did serve as a really, really good research assistant. I think especially when you have these relatively well-defined econometric ideas like regression discontinuity, or implement this Cox Hazard thing, it's great, you can tell it like look really closely at this data help file for Cox Hazard, think hard on that and then write it out, you know?
So I think the ability to do that, but like just all the sort of data merging, data munging, pulling from different parts of the DHS, I think would've taken just orders of magnitude more time than in the world without AI. And I think the DHS was particularly well suited for it because they actually have [00:26:00] the core STATcompiler indicators, how they're calculated is available on a public code repository. And so you can just tell Claude code, look at this, this is your model, build on that. Which I think really helped making sure that the data pipeline is as clean as possible in a relatively costless way.
Teryn Mattox: Incredible. I'm curious, so I heard noise about this from the vector control team. It just seems like, if we are seeing this sharp of a decline in all-cause mortality, it seems like the natural next question is, should we shorten campaigns further, right? Beyond the 30 to 36 months. At what point does that stop being cost effective?
I think the team is thinking about that as last time I spoke with them, they essentially said that this is on their list. And I think that's because of this analysis, my understanding is that this analysis pretty much confirms this concern we had about net decay. But yeah, do you have anything to say about that? [00:27:00]
Steven Brownstone: I mean, I will say before I did any of this analysis, our models spit out the risks of basically this ratio of all-cause mortality difference between year one and year three after the campaign. So I think this is a case where this information was already what we were assuming in our cost effectiveness model essentially. This triangulation made it maybe more salient, or increased kind of confidence that this was the correct information.
But I think there's multiple approaches, right? So I do think shortening that timing is one approach. I think there are risks that, as I mentioned earlier, we tried to move from 36 months to 30 months based on information on that decay. And the DRC hasn't been able to achieve that due to logistical delays and other delays. And so running these campaigns, and that's the whole other part of this project, there's just so much logistical challenge in getting these nets out to these places.
Another approach is giving more nets in the [00:28:00] campaign and hoping that households store them, which there's some evidence that they do store them. But there's also a possibility that some are diverted. And so I think that's another approach and something that can be done experimentally, like we wouldn't have to do an entire province with more nets, but we could experiment with more nets in certain places.
And I think a third approach, which we looked at a bit in the lookback but could dive into more, is just trying to improve routine coverage. So the other approach is just trying to get more nets out through like the antenatal care channel, which in some ways is the ideal way to give bed nets, because you're necessarily covering pregnant mothers and newborns if you give them nets through antenatal care. And I think in the past, the vector control has looked into this a bit, but obviously health system strengthening, as we know, is a complex subject. But I think it's possible that there's more targeted interventions we could do around supply chain management there, just making sure that the hospitals have nets in stock that [00:29:00] might move the needle.
So I think it suggests a variety of grant making opportunities. And I guess I'm sort of passing the buck to the vector control team to decide which one makes the most sense, or maybe all of them make sense. But those are the three that we discussed in the report.
Alex Cohen: Yeah, it seems like, I mean, this is a maybe an opportune time to think about ways to beef up these campaigns, in these ways or these other ways. Seems like these are worth exploring, especially when DRC just looks so cost effective right now, like what are things that you could layer on top to increase the impact even further. I don't know, it seems really exciting to me.
Teryn Mattox: This whole line of thinking is really exciting, especially as we move into a world where we might just have more funding and more ability to do things like this.
I'm really curious about this Marakuja group, how we found them; big picture, what they found. This is the group that we commissioned to do qualitative interviews with like literally every level of group involved in this campaign.
Steven Brownstone: Yeah, [00:30:00] so I'd heard about Marakuja for a long time and was really excited that we got the chance to work with them. But more proximately, Rosie had already commissioned them to do the quantitative DRC survey. And so we thought it was a natural extension to say, hey, can you also do some qualitative interviews, in addition to this quantitative work that they'd already done. So they'd already had some understanding of net campaigns and had trained enumerators on how to ask these kind of basic survey questions on nets, and had made connections with the PLNP staff in at least two of the districts, or provinces that we were interested in.
And so what we'd asked them to do was, basically we took kind of an expanded version of the OnFrontiers interview guides that we had put together, because we'd already made these plans to, through On Frontiers, talk to people involved in campaign distribution, people at the health zone level. And we modified them and sent them across to Marakuja and said, hey, can you interview [00:31:00] people, these are the provinces we're interested in. And they got back to us and said, actually, we can do all the provinces. It was actually more of us kind of raining down, they were very excited to make a really big scope of work.
And so our key focus was basically the lookback questions, did the nets get to the households? If not, why not? Are the households using the nets? If not, why not? And I think that focus really helped. And you can kind of see in the interviews, their enumerators sort of probing on some of these issues.
And the other really useful nudge I think they had made, which I appreciated, was to not just interview random households, but interview religious leaders and community leaders. And so I think some of the best quotes came from community leaders and religious leaders complaining about things that had gone wrong, you know, net diversion, or people that had gotten more nets than they were supposed to for various reasons, because I think these community leaders really try to pay attention to issues of equity in the community.
Teryn Mattox: What were the most surprising findings from this? Or was it all generally [00:32:00] confirming what we had expected?
Steven Brownstone: No, I think it actually it's one of these things, it's hard to parse. I think if I had just had the Marakuja data, it would be a very different lookback report, right? It would be much more negative. Because I think they, I mean, taking our lead, really looked for these cases of misuse, cases of corruption, cases of disruption.
And so I think there were these really creative uses like lashing them together into giant ropes to tie together log barges, or discussions of households because their daughter was the romantic partner of the community distributor, they had gotten a bunch of extra nets. So those are the sort of rich qualitative data of all the ways things can go wrong.
But on the flip side, you know, this one case that really stuck out to me is there's the Kitawala, which are these very fascinating Jehovah's Witness related group of people that reject all Western medicine, anything Western, they live a 40 kilometer trek into the forest. But apparently, they really wanted bed nets. And so some poor distributor had to make a [00:33:00] 40 kilometer trek into the rainforest to give this group that otherwise rejects every concept of modernity in Western medicine, their bed nets. And so I think there's also stories like that where it's really clear that people see the impact, like the impact of the bed net is very salient to people, and so they really want them.
Teryn Mattox: One of the things that we felt really uncertain about the last go round when we made a bunch of grants to the DRC was just how complex of an operating environment it is there and this idea of fraud, this idea of net loss. And it's really interesting that we found a lot of these cases and still came out broadly very positive, I think, about this grant. And so I just think that's like an interesting thing, you know, like we can still have lots of loss, we can still have lots of cases where people are misusing, lots of cases of fraud and yet the overall impact is still strongly positive. Is there anything that you want to say about that kind of tension?
Steven Brownstone: Because the nets are so valuable and perceived as so valuable, and this comes through [00:34:00] across Marakuja, when they're misused, it really stands out as salient to members of the community. And it's often "I heard," or like "there's one person in the village that did X," "there's one village over there that did Y," but I think these cases of misuse are just really salient to people because they know how powerful the nets are as a lifesaving tool.
And so I think there are these misuse cases, but these stories spread because people feel that these are such valuable commodities. And it really upsets them when they hear about someone misusing them in these kind of dramatic ways or engaging in corruption in these dramatic ways. So I think part of it is you can have both happening because the misuse is just really salient if you value something, or if you realize how important it is as a lifesaving tool.
Teryn Mattox: Exactly. If we believe that all-cause mortality is dropping by a quarter after net campaigns, then there's just no way moms aren't noticing that, you know what I mean? And I would get so mad if somebody over there got extra nets and like I was…you know, anyway. Yeah, I can totally see that makes sense.
Steven Brownstone: Yeah. You read that, [00:35:00] you read like, you know, in the qualitative interviews, moms are saying this, right? Yeah.
Teryn Mattox: Right. Amazing. Well, great job. This is really awesome, really exciting, and just so helpful. I think as we're thinking about what to do next, and this is a case I think where all of this kind of confirmatory triangulation is really strengthening and bolstering the case. This kind of work is really important, and I'm really glad you did it, and I'm very excited to see more of this over time. Thank you guys.
–
Elie Hassenfeld: Hey everyone, it's Elie again. So arguably the most important takeaway from this conversation is that we made a large grant to AMF because we expected it to save children's lives. And looking back, both with Steven's quantitative approach and the qualitative research that we supported, it looks like our expectations for the grant held up well. Delivering nets in DRC reduced child mortality by about one quarter, 27%, and that's a really massive impact that we're very proud to have had with your [00:36:00] support, in partnership with AMF and its implementation partners.
You know, this kind of analysis is something we could not have done a few years ago. Now we've grown our team, and we have more capacity and capability to do the kind of quantitative analysis that we described here, and the ability to go out and identify and support partners to do the kind of qualitative follow-up that we discussed. Also, new tools, like Claude, the AI tool, enables us to do research that we don't think we would have done without it. We will be publishing a lot more detail about this on our website, so if you really wanna dive into the details, you will be able to in the near future.
And finally, if you wanna learn more about this kind of work, this kind of post-grant assessment, we're hosting a webinar on June 9th, and that will feature me and also program directors Alex Cohen and Julie Fowler, where we'll be talking through other examples of this kind of work, and we'll be taking questions from attendees.
You can register at the link in the episode description in [00:37:00] your podcast app. We'll also have that in the episode summary that we're going to publish at blog.givewell.org. I hope you'll join us. Thank you so much.