Subscribe
Share
Share
Embed
Welcome to GiveWell’s podcast sharing the latest updates on our work. Tune in for conversations with GiveWell staff members discussing current priorities of our Research team and recent developments in the global health landscape.
Elie Hassenfeld: [00:00:00] Hey everyone, this is Elie Hassenfeld, GiveWell's co-founder and CEO. Today we're going to get pretty detailed in our conversation, so I wanna start with the big picture. GiveWell's aim is to direct funds as cost effectively as possible to help people in low-income countries, and that means we're trying to get funds to programs that will do the most good per dollar spent.
To do this, we spend a lot of effort evaluating potential grants before we make them. That means assessing academic evidence, building cost-effectiveness models, talking to people in the sector who know the program well. And then second, it means evaluating how grants went after we made them to see what we got right and what we got wrong so we can get better over time. That means that we can make better decisions about the kinds of grants we should make in the future.
One of the places that we do this evaluation most commonly is when we're considering a grant for renewal. What that means is we've already directed funding to a particular program and now we're deciding whether or not we should continue to support it. We do some additional research on the program that we haven't done before, but a [00:01:00] major input into that decision about whether to renew the grant is evaluating the results of the program that we funded.
Normally, when we consider a grant renewal, the results are positive, and they're consistent with what we expected. So by and large, we renew the large grants that we've made because we see them having major positive effects that we expect. Sometimes they don't have the positive effects we expect and we don't renew. And so today we're going to talk about one of those examples of non-renewal.
I'm going to talk to Erin Crossett, our Senior Program Officer who leads our water team, about a renewal decision that we recently made, where based on the results that we got, we decided to not renew the grant. This grant was to support a program called Dispensers for Safe Water, which provides chlorination dispensers that people use to add chlorine to water, so that the water is clean and that improves health. This grant was in three countries, Uganda, Malawi, and Kenya.
And I think this is a really interesting conversation, because it illustrates a couple of important things about GiveWell. [00:02:00] First, how we are different from other funders. I think that this focus on collecting and using monitoring data in the way that we do is relatively unique.
And then second, how we learn from our own track record, not only to make better decisions within the specific programmatic areas that we're supporting—meaning, deciding whether or not to renew a particular grant, like water in this case. But also taking lessons from this grant and applying it more broadly across all the work we do across many causes in our grantmaking portfolio. And ultimately, I think this conversation will be a great illustration of how GiveWell works and the kind of work that we do to ensure that the funds we direct help people in low-income countries as much as possible.
Erin, before we dive in and get started, can you just introduce yourself and your background?
Erin Crossett: Sure. Hi everyone. So I'm a senior program officer at GiveWell and I lead our water team, as Elie mentioned. And previously, I've been in a variety of research-related roles, including I was on the evidence-based policy team at Arnold [00:03:00] Ventures, funding randomized control trials of social programs in the US. And I was also at Development Innovation Ventures, or DIV, at USAID, which is a tiered evidence fund that funds across all sectors within international development.
Elie Hassenfeld: Thanks, Erin, for doing this. Let's just start off, so we're talking about this case study of Dispensers for Safe Water. What happened?
Erin Crossett: So the headline here is, as you mentioned, we made, GiveWell made a $65 million grant to an organization called Evidence Action, back in 2022, to scale up dispensers across three countries—Kenya, Uganda, Malawi.
And so over the course of the last 18 months, we'll dig into this later, but we learned that something was amiss with some of the data. We funded independent surveys and learned that only about a third as many people were using dispensers and actually chlorinating the water as Evidence Action's monitoring suggested.
So, [00:04:00] instead of the 5 million people that we estimated were using dispensers in Malawi and Uganda, the independent data showed that this was actually less than 2 million people. So why? What happened?
We'll discuss today, there are multiple small biases in Evidence Action's monitoring process that we think compounded. So, a couple of these are subjective measurement tools. So using a color wheel there's a lot of subjectivity involved in reading that. Household sampling methods that skewed towards finding and interviewing more engaged users, people who are more likely to chlorinate their water. And then also Evidence Action's reliance on internal staff who were collecting this data.
So, I think of it as like there wasn't one single smoking gun, but five or six things that each added a few percentage points, and together roughly tripled apparent usage.
And so what did we [00:05:00] do as a result of this? We, GiveWell decided not to renew our funding in Malawi and Uganda. Our best estimate is that a renewal grant would've been about two times as cost-effective as our benchmark, which is below our current threshold of eight times. And Evidence Action has decided to discontinue the program in Malawi and reduce its program to half its size in Uganda and Kenya.
And actually before we dive into the details, I just want to really quickly say, I just think it's worth highlighting Evidence Action's role in this. And the fact that we have this data, and Elie, you and I are having this conversation about what went wrong and why, this is all downstream of Evidence Action's willingness to open themselves up to scrutiny and support an external survey to test the robustness of the data. And I think, you know, I'm hugely grateful for that as a program officer here, and this is the kind of attitude I think that we want [00:06:00] all of our grantees and partners to emulate.
And we'll talk about this more later, but Evidence Action is taking concrete steps here. They're scaling down the program in Kenya and Uganda, and they're planning a 24-month transition period in Malawi and Uganda, which, you know, they say is a lot longer than the sector norm. And that matters because still nearly 2 million people across Uganda and Malawi were relying on dispensers to access safe drinking water. And so winding down responsibly, I think, is really important here.
Elie Hassenfeld: Yeah, totally. And I think something that is important also to just note here is that we ended up with this finding that about a third of the people as we expected were accessing chlorinated water via dispensers. That's due to challenges in monitoring, you know. It is not fraud or, you know, malfeasance. It's just the challenges of trying to collect high-quality data in a very difficult context. And so, just appreciate Evidence Action's work and the work of everyone who helped us better understand the impact of this program. So, you know, definitely agree with that.
Okay. So [00:07:00] that's the big picture. You know, we had this grant, $65 million, a few countries. We were in a position to collect more data. It showed us that it wasn't reaching as many people as we thought, so therefore it is less cost-effective.
But let's kind of move more slowly now through the details. And let's just start with, so this is Dispensers for Safe Water. You know, what is a dispenser? How does it work? What does it accomplish?
Erin Crossett: Chlorine dispensers are tanks of liquid chlorine that are installed next to communal water points, most often in rural areas. So the idea is you twist a valve, chlorine dispenses into your container, typically a jerrycan. You then fill the container with water at the water point and chlorine disinfects the water. So the idea is that this reduces waterborne diseases, which are a major cause of child mortality in low-income countries.
And each dispenser is managed by a community volunteer, what Evidence Action calls promoters, we'll talk about [00:08:00] today, and they keep it filled with chlorine and encourage its use. And so Evidence Action has run the Dispensers program for many years in Malawi, Kenya, and Uganda. And just to highlight the scale, by the end of 2023, Evidence Action was maintaining over 50,000 dispensers across these three countries. So it's a very large program.
Elie Hassenfeld: I think like one illustration of how large this program is, is that when I was in Malawi last summer, I did not go with the intention of seeing dispensers. And, basically everywhere we went in the area of Malawi that we were in from specific villages to just, there were times when our colleague Teryn was just like driving down the road and like got out of the car that she was in to go like see the dispenser on the side of the road, like they're really ubiquitous in Malawi now.
So, what did we think about these dispensers when we made the grant, and what do we think now?
Erin Crossett: So again, we made a $65 million grant in 2022. At that time, we estimated that that grant was seven times as cost-effective as our benchmark. And Evidence Action's monitoring suggested that 5 million people were using dispensers in Malawi and Uganda.
So now, what do we think now? The independent surveys, so University of Chicago, the Development Innovation Lab, partnered with Innovations for Poverty Action. GiveWell commissioned the survey; they're the ones who actually executed this. They identified that the real number was less than 2 million. So about a third of what we initially believed, with some rounding.
We estimated that the renewal grant for Uganda and Malawi would've been about 2x our benchmark cash, which was well below our 8x cash funding threshold, so we decided not to renew. And again, just I think something that's worth calling out now is that I think it's easy to say like, oh, you know, 2x, that's well below your threshold, [00:10:00] it's not a cost-effective program, but again, you know, this is a program where we're still helping nearly 2 million people access safe water. I think it's still a good program, I think it's a program that's worth continuing. It just doesn't meet our very high bar anymore, and we think that there are other programs that can do more good per dollar.
Elie Hassenfeld: Like in a world of unlimited resources, we would love to be supporting this program because it's helping people. And then in the world that we're in, with limited resources and very great needs, we think the large but still limited set of funds that we have, we think we should direct them elsewhere.
How did we come to learn this? Like, what's the story of our updating about the number of people who were using chlorinated water via dispensers.
Erin Crossett: Yeah, I think it's helpful to think of this as like three key stages on the journey to getting this data.
So the first stage was, I call like the Kenya red flag. So this was in late 2023. So Development Innovation [00:11:00] Lab, DIL, was running actually a separate GiveWell-funded evaluation, called Kenya Study on Water Treatment Child Survival, KSWTCS. And importantly, that study overlapped in geography with Evidence Action's Kenya program. And Dispensers was kind of like one of the programs that that study was following up on to measure child mortality.
Their data implied significantly lower chlorination rates than Evidence Action's monitoring. Now, the methodologies and sampling frames and areas of overlap in terms of geography, they were too different for direct comparison, so it wasn't an apples-to-apples comparison. But the gap was concerning enough for DIL, GiveWell, and Evidence Action to all say like, okay, this is weird, something is amiss here, I think we should push on this.
Elie Hassenfeld: So this was a totally separate evaluation that we had supported, and why were we supporting this study? Like what was it that we were trying to learn?
Erin Crossett: [00:12:00] So this study was a follow-up onto what's considered to be a very canonical study in the sector. I won't get into details, it was a huge multi-arm trial, testing a bunch of different interventions, one of which was dispensers. The idea, we wanted to fund a long-term follow-up on the initial study to measure child mortality after a certain period of time.
And the reason being that that's an important input into our cost-effectiveness analysis across chlorination. We have a lot of uncertainty around what is the effect of chlorination on all-cause mortality for children under five, and so to the extent that we can conduct more long-term follow ups, even if they're individually underpowered, if we can pull them all together into a meta-analysis, then this could be a very useful piece of information.
Elie Hassenfeld: And maybe like one thing just to notice here that's interesting is how many different questions need to be answered in order to be, [00:13:00] to know about the effect of different programs.
So, you know, one question that we're mostly talking about in this conversation is, you install a dispenser, how consistently do people use that dispenser to chlorinate their water to keep the water safe? And then there's a totally separate question, which is, assuming the water is chlorinated and safe, what do we believe about the cleaner water's effect on health outcomes, as measured by, in one example, by child mortality.
And so this first study that sort of was the, you called the Kenya red flag was supported by us via a group at the University of Chicago to try to answer that second question, you know, what's the health effect of chlorinated water?
Erin Crossett: Okay. So then we go to the second stage. So stage two was, we all think something's amiss. Evidence Action conducts its own Kenya survey, so this is in 2024. And to their credit, Evidence Action took this very seriously. So they designed what they referred to as a simultaneous census and monitoring [00:14:00] exercise.
So the idea was that independent data collectors did a census, alongside Evidence Action's internal staff, their field officers who were doing their routine monitoring in the same set of villages, around 70 villages.
So the headline result here was that the census found that chlorine usage, dispensers usage, was about 60% lower than Evidence Action's monitoring. And as a result of that, Evidence Action decided to wind down approximately half of its Kenya program, starting in 2025.
Elie Hassenfeld: Got it, so the second part of this was just Evidence Action itself taking this issue seriously. And because of that, using two different methods to triangulate what was happening, their standard method and then a more expensive, more intense method, essentially. And the more intense method, that they did on their own, convinced them that their internal monitoring was overreporting [00:15:00] chlorination rates.
Erin Crossett: That's right. So we get that, we say, okay, this is a big negative update. As a reminder, both the KSWTCS, the first stage, and the second stage, Evidence Action's Kenya survey, both were in Kenya, so we say, to what extent do these issues generalize to the rest of the Dispensers program in Uganda and Malawi?
And so at this point GiveWell funds an independent survey in both countries, and this was in 2025.
Elie Hassenfeld: And then can I ask a question? What led us to want to fund an independent survey? Like why not say, you know, we have this negative update from the University of Chicago independent study, we now have this Evidence Action data. Why was that not sufficient?
Erin Crossett: I think there's a couple of things here. The first thing is that, you know, this is a really big program, $65 million, again, 50,000 dispensers, we now know 2 million people were using it. The stakes feel very high, it felt important to get this right. And so I think that there were enough limitations in the stage one data we got, the KSWTCS data, again, compelling, notice the red flag. Not [00:16:00] enough on its own.
The second stage I think was important because, or it was useful, but still had limitations. You know, Evidence Action was still sort of driving that effort. And so to the extent that we wanted to move towards independence, or some more external validation, it felt important to bring in a fully external partner and, again, understand whether the result of this pattern was consistent across geographies. Maybe there's something about Kenya that you know was problematic but didn't apply to the other programs. There are somewhat, on the margins, programmatic differences between countries.
Elie Hassenfeld: It's just such a big decision, and I think this is something that I think we feel regularly in our work. We always want to make the best decisions we can with funding and, you know, one argument you can make, you can already update your expectation to believe that this program is less cost-effective than you previously thought, therefore you should just move on.
But there are real effects to deciding to move on from a program as you said, 50,000 dispensers, 2 million people using them. And it sounds like what you're saying is it's really worth getting the data to be very confident before we went ahead with not renewing. So that's a, you know, a big and weighty decision that we were making.
Erin Crossett: Right. And let me talk about stage three, because I think this illustrates also like a new thing we were getting from this stage three data collection that we didn't get in the stage two.
And so, okay, stage [00:18:00] three, GiveWell funds this independent survey. We fund DIL and Innovations for Poverty Action to conduct both an independent census, so kind of considered the gold standard. Plus, and this is what's different, replicating Evidence Action's monitoring protocol, but with independent surveyors.
So this is important because this allowed us to determine where the bias was coming from. So, is the monitoring design itself flawed, or is it about how Evidence Action was implementing it?
And so this ultimately confirmed this pattern. I think we put more weight on the census because again, it's considered sort of the gold standard, and it found that dispenser usage was, again, about a third of what Evidence Action's monitoring suggested.
Elie Hassenfeld: Got it, it's really that last point where we felt like we really had the answer, quote unquote, to the question of how well this program was working. And that's the one third of the reach, relative to our initial expectation.
I'm just curious, how much did this stage three cost? Because I think like a challenge with monitoring [00:19:00] is always balancing additional costs spent on knowledge versus using those same funds to directly help people in need. So what kind of magnitude are we talking about?
Erin Crossett: In total, the grant was a little bit over $900,000 to DIL and IPA to do the census and adoption monitoring replication exercise in two countries, for both dispensers and then also for in-line chlorination, which was a separate program in Malawi.
Elie Hassenfeld: So really in the scheme of things, in terms of like proportional to the grant size, like a fairly small cost, which is, is important to know.
Let's now just talk a little bit more in detail, you've mentioned it like how Evidence Action monitoring works, but talk in more detail about how it worked, and what went wrong.
It might just be helpful to talk for a second, remind people about how the actual program works. And so maybe I'll do that very quickly, which is in rural areas in these countries, often people access water via a water point, which might [00:20:00] have a pump or a well. And right beside that pump or the well is a dispenser. I've visited these dispensers in Kenya and Malawi myself. Erin, you've visited them in other countries too.
And the challenge is that people come, and they bring their jerrycan and fill it with water, and then they need to turn this wheel and have chlorine come out. Some people just don't do that at all, either they forget to turn the wheel or they don't want to use chlorine. That could be one reason that people are not using these dispensers even when they exist.
There could be times when people want to use chlorine, but for one reason or another, chlorine is empty. Evidence Action in part of its program was consistently monitoring and refilling the dispensers so that they were usable. And so there's just a lot of ways in which this program could fail to deliver chlorine to the people who needed it. That's why this monitoring is so important.
But I think it's just like worth having in mind as we talk about the monitoring approach, [00:21:00] what kinds of questions the monitoring was trying to answer or why it was so critical to have an answer to the question of how many people are being reached, meaning using the dispenser to consistently chlorinate the water that they're collecting from the water point.
Erin Crossett: So how does Evidence Action's monitoring typically work? So here's the process, and this is done roughly quarterly. Evidence Action's monitoring and evaluation team randomly selects water points where dispensers are installed. Evidence Action sends their own surveyor, again, internal staff, who visits the water point unannounced, so no one in the community should know.
And they work with the community promoter to draw up a list of households who use the water point. And then the surveyor randomly selects four households from the list and visits those households. They ask for a sample of stored drinking water, and they test it with a color wheel.
So what this looks like is they take a little sample cup of water, they add a [00:22:00] reagent, and the water turns some shade of pink if chlorine is present, depending on the concentration of chlorine. And the surveyor then matches the shade, by eye, to a reference wheel, a color wheel.
Two issues here that I think we'll probably focus on. So the first is the color wheel problem. The color matching process, it's just a lot of room for interpretation here, and we think that, again, surveyors who are Evidence Action staff were unsurprisingly, you know, maybe motivated or just tended to report ambiguous results positively.
So I think one really illustrative data point here is that when Evidence Action last year, they introduced a requirement that surveyors have to take photos of the water quality test and submit for review, chlorination rates fell 20 percentage points. So I think that just shows that, you know, it's tricky, you're trying to understand, like is it clear, is it like a [00:23:00] very faint shade of pink. You might have some sort of subconscious bias that's like, well, I think it's working, I think there's some chlorine here, so we'll say it's there.
And like another interesting point that I think illustrates the scale of this problem is that in the DIL survey they noted that about 40% of chlorination ratings that were positive were at exactly this 0.2 milligrams per liter threshold. And so that's the faintest shade of pink, that's like the lowest amount of chlorine that you can detect with a color wheel. So, there's zero chlorine, it's just clear, and then there's 0.2.
And so, you know, you can kind of feel for these surveyors who are trying to detect very subtle color changes that you know look different in different lighting. When I was in India, I was following around enumerators, I was wearing a red shirt. It reflected in the water, it made it really difficult to tell whether the water was more pink or if it was just the reflection.
Elie Hassenfeld: So like the measurement of the water itself, and we have some pictures of the color wheels and the dispensers that we've [00:24:00] seen in our trips, we can share as part of our blog post that summarizes this conversation.
I remember when I was using the color wheel in Kenya, just like if you held it up to the sky, it looked different than if you held it up to the trees, right, because the background color is different. And so small things like that come up.
And then of course, the inherent challenge always is there are certainly tools that measure chlorine more precisely, but those tools are expensive and in an environment of resource constraint, it's difficult to make the trade-off between reaching more people with more water more directly, versus more expensive tools that allow for better measurement. But certainly the measurement tool itself, and then the, I don't know, the subconscious or not, incentives or biases of the people doing the reading can have a big effect on the way the data comes out.
And to me it's like such an important point because we spend all our time thinking about the question of, where should we direct funding, and what do we know about how well a program works? And so we're comfortable and used to the [00:25:00] fact that getting highly confident or certain answers to questions is not really possible. But, you know, we're talking about certain issues with respect to data collection here: using a color wheel, or the survey firm, or an independent evaluator, but the questions go all the way down even to the measurement tool itself.
And I think there are just real limits to what we are able to know. I think to some extent, our job is to keep digging into thinking about the trade-offs between spending money on more knowledge versus spending money on programs. But the bottom line is that there are very resource-constrained places in the world where it is quite difficult to get good information and we're doing the best that we can, and it still comes with major uncertainties.
Erin Crossett: Right. Okay, so the second problem I want to highlight in Evidence Action's monitoring, which again, it just wasn't self-evident that it would be a problem, but it's what I call the promoter list problem. So, you know, the logic I think made sense here in theory and, if implemented correctly, I think should yield unbiased [00:26:00] results.
So, you know, not everyone uses a dispenser's water point, approximately 50% of villages in Malawi, according to DIL's data. And so we found the promoter listing approach was a really pragmatic way to isolate the target population who we think might use a dispenser.
Elie Hassenfeld: And the issue is like you might go to a village, you end up at a dispenser, and then the question is, which households are even coming to this particular water point to get their water?
And this can just be difficult because, you know, in my experience, sometimes there are houses that are very nearby and it's like fairly dense in the scheme of things, you know, dense for a rural area, and you can see many households nearby, and there are multiple water points which have dispensers.
But there can be other places where distances are much larger, approaching, I don't know, half a kilometer or something to even find the next household. And it can just be difficult to even know who's coming to this spot to collect their water. And if you're then trying to interpret what proportion of [00:27:00] people are using a dispenser, but they're not even coming to the water point where the dispenser is present because they get their water elsewhere, that would introduce faulty data into your collection.
Erin Crossett: Right. And so in practice, this was all done on paper rather than using like a mobile app. And there was limited quality control checks. Evidence Action certainly had checks, but they were a little bit more limited, and so it was easy for surveyors to end up surveying the most engaged households or households who were closer.
Again, surveyors have, you know, quotas, number of households they have to survey per day, they're under a lot of pressure. It makes sense. It's hard to definitively prove that this was a factor, but one reason to think that this was an issue and this was happening was that the independent survey found a far lower share of people self-reporting that they use the dispenser, around 30%, versus 66% in Evidence Action's routine monitoring. So this implies that Evidence Action's surveyors were surveying people that were more likely to be [00:28:00] dispensers users.
Elie Hassenfeld: Right. And so that, yeah, you can just see how like the approach to monitoring starts to introduce issues into the quality of the data that we're getting.
Erin Crossett: And again, because it was paper based, you don't really have an audit trail. You can't tell like, okay, so let me look and see, were the households that were visited, were those actually the ones that were randomly selected? And it's hard, it's challenging, you know, you go to a household, maybe you did go to a randomly selected household, but no one was home. And so rather than following like the replacement protocol, randomly select another household, maybe they just went next door.
Elie Hassenfeld: Got it. So we've kind of talked about what happened, right? We funded this program, we had some data. We had supported some other research that let us see that the data that we were relying on to judge the reach was too optimistic. Evidence Action got some more data. We funded additional information, and now we learned, you know, really it's reaching about a third of the number of people that we previously thought.
Now I want to kind of switch gears and just reflect on what this means about GiveWell, like, the questions I want to ask are what did we get [00:29:00] wrong here? You know, what mistake did we make? What should we have done differently?
Erin Crossett: Yeah. So on the, like, did we make a mistake question: so I think there's two subquestions here. So there's like, was the grant a mistake? And based on what we knew at the time, I think it was a reasonable grant at the time of decision in 2022. I think we had, you know, a strong RCT evidence of impact, clear theory of change, credible implementing partner with the track record performance across multiple countries. You know, this was in the range of our cost-effectiveness threshold. And it was very unlikely that someone else would step in at this funding level. If I remember correctly, like dispensers was really on a lifeline at that point. So it was possible that without our support, it would've shut down. I wouldn't call making the grant a mistake.
Was the monitoring oversight a mistake? Could we have caught this earlier? Yes. I think the answer is yes. We fell short here, right. I think we had the information to be more skeptical of the protocols. Evidence Action's protocols were shared with us in 2022. And in [00:30:00] retrospect, I think the risks were there. I just don't think we had an appreciation for all of the ways that things could go wrong or all of the subtle ways that bias could creep in, but the risks were there, right. You had internal staff who were doing the M&E, the measurement. You had these subjective measurement, you know, you have these color wheels, which rely on subjective judgment. You have paper-based processes, so there's more limitations in your quality control.
And so I think like any of those individually might have been manageable, but they all just kind of compounded. And yeah, I think the main reason we missed it is, I don't know if we were as attuned to how hard water quality monitoring was.
And I'm not trying, you know, I'm definitely not trying to say that other interventions' monitoring is easy, but to give examples like for some other grants, a lot of times you have more objective measures. So for bed nets, can you physically observe a bed net hung over a bed? Yes, no, it's a binary answer. And color wheel testing is just like [00:31:00] fundamentally more subjective. And again, that's so much worse when the person who's actually in charge of doing the reading has an incentive to report that there's chlorine. And I don't know if we fully appreciated that enough.
Elie Hassenfeld: Like one thing that I think about here is, I think GiveWell is relatively unique as a funder in that we are supporting and utilizing data in an ongoing and consistent way to determine how well programs are working. And to some extent, we are using our own work and our own track record to learn and improve over time.
And, I think that the mistake that I feel like we made here was not recognizing that there was significant value in relatively low-cost triangulation of data. Like I think on some level, when you think about $900,000 to triangulate data on a $65 million grant, in purely quantitative terms, it is like [00:35:00] obvious that you should do that because you're talking about a very small cost in order to either verify or falsify or raise questions about a much larger amount of money.
That is a mistake. And then you could say, well, why would we make that mistake, or what caused us to reach the decision that we did in the past? And why didn't we see this earlier? And I think there's two main reasons. I mean, one of them is, I do think it's just challenging that we are doing something that is potentially unique, if not extremely uncommon. And that just means that we're trying to figure out how to do this well, and we don't have other models to go on. I think in some ways that illustrates some of the value that we're adding, [00:36:00] but it certainly means we're going to make mistakes.
And then, you know, the second one is, our big challenge historically has just been capacity. And so, an effective, high-quality grant for independent data collection, that's not a small add-on. It's a small add-on in terms of dollars to a large grant, but it's effectively one more big piece of analysis that we need to do. We need to figure out how to do this monitoring well.
I think it is only very recently, like in the last couple of years that—you're describing the efforts that we made to engage with the University of Chicago and Evidence Action, and then fund this extra survey—that we really had the capacity to do this, these extra evaluation and learning grants alongside our normal grantmaking.
I'm very glad that GiveWell is in the position today where, you know, we could do that in the last couple of years, obviously it helped get to the right answer in this case. But also we're in a position to do even more of it in the future with our increased capacity.
Okay, so we've talked about dispensers, like what's happening with this program now?
Erin Crossett: So as I mentioned, we're not renewing our funding in Malawi and Uganda. Evidence Action is ramping down in Malawi entirely and reducing their footprint to about half in Uganda and Kenya with other donor funding. And they're planning a 24-month transition and pursuing, I think, prioritizing government handover when possible.
And we're doing a bunch of other things in response to these findings, I think across the water portfolio. So I don't think that this is the end of the road for dispensers with Evidence Action or other implementers. So with Evidence Action, we're planning to consider funding Evidence Action to pilot dispensers in northern Nigeria. I think this is interesting because it sort of shows how nuanced [00:38:00] our grantmaking can be. So you know, even taking, with this updated data and taking a hit to chlorination rates, it's possible that implementing this program in northern Nigeria could be above our cost-effectiveness threshold, just because disease burden is dramatically higher there than in Malawi and Uganda.
And Evidence Action is also thinking about changes to the program. So one example is using financially incentivized promoters rather than relying on community volunteers. We are also funding four other smaller chlorine pilots with organizations that haven't previously implemented the program, three in Nigeria, one in Ethiopia. The goal here is to test new approaches. So what are tweaks that we can make to Evidence Action's existing program in higher-burden locations to try to drive up chlorination rates.
And then I think, bigger picture, in the water quality space, we're also thinking about ways to treat water that sort of remove some of the behavioral complications we've talked about with [00:39:00] dispensers. So in-line chlorination is one option. Also thinking about water quality interventions that don't rely on chlorination, so filtration, inline UV, couple of other potential technologies.
Elie Hassenfeld: Well that's great. Well, thank you so much, Erin, for having this conversation about this important topic.
Erin Crossett: Thanks, Elie.
–
Elie Hassenfeld: Hey everyone, this is Elie again. I hope this conversation helped you understand a bit more about how GiveWell works. I wanted to pull out a few things that I think are worth highlighting as we close out here.
First, I think [00:35:00] this whole project, and this conversation, illustrates something a lot of people may not realize about our grant making, which is that grants that we make, even grants for specific Top Charities, are generally targeted to specific locations rather than just to a program as a whole. The dispensers program has different cost-effectiveness in different places, and so that local context really matters. And that's true of many of the programs that we fund. And so, our understanding of a program in one place and its promisingness can differ from how it might work somewhere else. And so even though we've chosen not to renew this program in east Africa, it's possible that we'll be supporting dispensers in other regions around the world another time.
Second, and I think this is something that is really worth emphasizing, is that the kind of work we do requires a relatively rare type of organization for us to be working with. Evidence Action didn't just let us poke around in their data, they didn't just submit to GiveWell's evaluation. When they saw concerning results coming out of [00:36:00] Kenya, they flagged it, they came to us. We worked together to design an independent set of surveys to figure out what was really going on.
That kind of collaboration, that willingness and desire to be transparent, even when the news might not be good, that is not something that you should take for granted. That's extremely rare in the nonprofit world. And we are extremely grateful to Evidence Action, and to other organizations that we work with, that they engage with us in that way. That is what makes this whole approach possible.
And then finally, I think a broader lesson here is about learning and getting better. GiveWell is trying to do something that is extremely rare in global health and development, which is to support many programs at scale over time, and then use that data to make good decisions about how to allocate funds cost effectively in the future. The data we collected here is exactly the kind of information that many other decision makers just don't have access to, they're not using. And so we're very glad that we're in a position where we do get that data, and [00:37:00] then we can support additional data collection and learn from it, and then reallocate resources based on what we find.
That data's going to enable us to be a more cost-effective funder in the water space. It also is going to inform and influence all of the giving that we do across all of the causes we work in. Increasingly, with more team capacity, with additional funding, we're able to incorporate more sophisticated and rigorous monitoring and evaluation work across all of our grant making. The lessons from cases like these are helping us to get better at all of the challenging work that we do.
So we're grateful for your support and the support of donors who make all of this possible. You're enabling us to deliver funding that reaches people, to invest in the learning and data collection that makes our work better over time, and you're with us on this journey as we figure things out, getting things right, but also learning from mistakes and ultimately improving over time. Thank you so much.