Subscribe
Share
Share
Embed
“Research Ethics Reimagined” is a podcast created by Public Responsibility in Medicine and Research (PRIM&R), hosted by Ivy R. Tillman, PRIM&R's executive director. Here, we talk with scientists, researchers, bioethicists and some of the leading minds exploring new frontiers of science. This season, we are going examine research ethics in the 21st century -- and learn why it matters to you.
Welcome to Research Ethics Reimagined. I'm your host, Catherine Batsford.
Dan McLean:And I'm Dan McLean.
Catherine Batsford:Today, we're discussing who gets counted in public health, data trust, and the ethics of representation with Doctor. David Dutwin, the executive director and senior vice president of AmeriSpeak, NORC's premier multi client panel based research platform. He is a senior fellow of the program for opinion research and election studies at the University of Pennsylvania, a nationally recognized survey methodologist. He also teaches at the University of Arizona and the Westchester University. We are delighted to have you on our podcast to discuss how representation, participation, and trust influence the data that ultimately public health policy.
Catherine Batsford:First, we would like to start off asking you how you found yourself here, where you are today, a little bit about your career path and how you became interested in the
David Dutwin, PhD:Yeah. Sure. I think probably a lot of people have unique stories to tell. Mine is similarly unique. I went back to grad school after working on Capitol Hill and thought I was gonna be a political speech writer and was taking communication courses and and rhetoric courses.
David Dutwin, PhD:And finally, the in the last semester, I had waited to the very last second to take that dreaded statistics course. And I also happened to take a course on public opinion. And it just kind of ended up as like, wow. Like, I actually like this statistics stuff, and I'm good at it. You know?
David Dutwin, PhD:Like, who knew? And so I went on at a different school to get my my PhD and was solely focused on public opinion research and statistics. I really just caught the bug. And and then the rest is history, I guess.
Dan McLean:Before we get too far into the details, and there's a lot of really interesting studies and work that you all have done that we are going to explore a bit today. But can you explain the relationship between NORC and University of Chicago? And I know the acronym for NORC is N O R C. And could you elaborate on the origins of that a bit as well?
David Dutwin, PhD:Yeah. Sure. So a little over eighty years ago, the National Opinion Research Center was founded at the University of Chicago. Like a lot of academic institutions, it was a small survey shop to to do research for professors and whatnot. And about fifty years ago, NORC had gotten so big that the university basically said, it doesn't make sense for you to be solely and universally under our umbrella.
David Dutwin, PhD:And so NORC split off, but basically the agreement was that we would still be affiliated with the university. At least half of our board has to be university administrators, professors, etcetera. But we really are quite autonomous. We have a major office in Downtown Chicago, as well as one in in DC. The NORC name is funny depending on, the age.
David Dutwin, PhD:Sometimes we were the National Opinion Research Center and other times it's just NORC. Right? And it sort of depends on whatever marketing firm we have at the time recommended would be the best foot forward, essentially. So, yes, we are we are officially NORC as of the, about a decade ago.
Dan McLean:AmeriSpeak, which is really that's your domain inside of NORC. Right? How does that fit into to NORC overall? Yeah.
David Dutwin, PhD:So NORC generally is a research firm that does a lot of large scale federal government, state and local government, and international research. And that research includes surveys and focus groups and things like that, but also a lot of direct policy research and program evaluation going into Pakistan and evaluating their education system and making suggestions for improvement. And similarly in The US, going in and and evaluating different sorts of policies and making recommendations. So AmeriSpeak was, in some ways, a move to to try and diversify and also be more inclusive of doing research for foundations and academics and even for profit companies. And so we established a probability panel.
David Dutwin, PhD:So the survey research has gone through many ages. In the fifties and sixties and in the into the seventies, door to door was sort of the only thing out there. But then all of a sudden, technologies developed to be able to randomly sample telephone numbers. And so telephone sample surveys became the predominant resource. And in the seventies, it was very common.
David Dutwin, PhD:You would get 30 to 40% response rates. 30 to 40% of people would pick up and participate in your survey when called. Then cell phones came into play, and so we had to switch to cell phone frames. And then we entered sort of this age of spam and and telemarketing Mhmm. Where back in the in the seventies, if you got a strange call, 50% chance it was probably a survey, and then 50% chance it was just I don't know, some some other reason, doctor of a or something.
David Dutwin, PhD:But, of course, nowadays, we know that probably 29 out of 30, if not 99 out of a 100 strange calls you get tend to be telemarketers or or some such. And that's really, I think, jaded the perspective of Americans. There's a there's a lot of other reasons too as to why now response rates are are typically more like three or 4% when someone tries to do a telephone survey. Mhmm. But the the impact of that is that, first of all, it becomes more unreliable when such a small percent respond.
David Dutwin, PhD:And secondarily, it's very expensive to administer that kind of survey when you have to call and literally go through hundreds and hundreds of numbers just to get one person to participate. So a probability panel is a new sort of survey technology just really in the past ten years where we'll recruit people ahead of time to agree to take surveys, still using random methods. AmeriSpeak specifically uses, basically, the US Postal Service database of all addresses in The United States, randomly samples from that. And then we'll send out mailers. We'll text.
David Dutwin, PhD:We'll phone. And importantly, AmeriSpeak will go door to door and recruit people on their doorstep because at the end of the day, door to door is expensive, but you can get 30% of people to still join with that method versus again, four or 5% with other methods. Very scientifically rigorous because, again, it's a random sample that we draw from to get our recruits.
Dan McLean:One of the studies that caught my eye as I was reading over things again was the study that's titled an annual nationally representative study of young people to inform solutions to help them thrive. And this was one of for one of your clients that was started in 2024 and is ongoing today. It really explores the state of young people, and it looks like there's a sample of 1,500 young people, and it explores four topics, sociopolitical divisiveness and healing, civic education, artificial intelligence, and mental health. And I was wondering if you could share a little bit about this and how the, there's a mirror speak. There's a team panel that's part of it, and it sounds like that helped inform the results.
Dan McLean:This is something that I think is an issue that's been top of mind for a lot of people in the last few years. So I'd be curious to hear what insights you were able to gain.
David Dutwin, PhD:Yeah. Sure. So we do AmeriSpeak is is really a suite of a number of panels. We have the the main AmeriSpeak panel is adults 18 and older, but we also have a teen panel. We have a veterans panel.
David Dutwin, PhD:We have a couple of state and regional panels to be able to really drill down. Chicago speaks. We have a panel in in Texas and in Ohio. We've a Asian American panel that interviews in four different languages besides English. So the team panel, basically, we first of all, we try to build these things the best we possibly can.
David Dutwin, PhD:You know, when we started AmeriSpeak, we're like, oh, let you know, when someone joins, we we ask them a whole bunch of profile questions to understand who this person is on politics, finances, health, education, travel, technology, social media use. And we roster the whole household so we know, oh, this this household has a teenager. And we would sort of initially say, hey. You know, we'd love for your teenager to also join as well. And and the take rate wasn't so good.
David Dutwin, PhD:And then we realized, oh, we really need to establish trust with these folks first. And so what we do now is we let the adult kinda go through. They do four surveys, five surveys. They they kinda get what who we are. They trust what we're trying to do.
David Dutwin, PhD:And then we'll ask if their teenager would would be willing to join, our separate teen panel. And so we we do have a couple thousand teens on the panel now. We have about 50,000 adults on the panel. And, you know, it's it's great because, again, teen research, like a lot of research today, there are other ways to interview teens specifically with what are called nonprobability panels. These are panels of convenience.
David Dutwin, PhD:A teenager sees an ad on their on their LinkedIn pay or, sorry, their their Instagram page. And do you wanna do a survey? Sure. Maybe they're on LinkedIn. I don't know.
David Dutwin, PhD:But they're definitely on Instagram.
Catherine Batsford:Oh, maybe.
David Dutwin, PhD:Or TikTok and and yeah. Right. Hey. Do you wanna do a survey? Yeah.
David Dutwin, PhD:Sure. Okay. Well, the problem with those sorts of methods, of course, is they're not scientific. They're they're self selected. And the teens who participate are, you know, people teens on social media likely to take a a survey.
David Dutwin, PhD:And I know this isn't answering your question, but just why is why is random sample important? I always say the the best synonym to the word science is randomization. Right? I think all your listeners probably understand. Mhmm.
David Dutwin, PhD:If you randomly divide 500 people into two groups, one gets a sugar pill and one gets a new pill to Mhmm. For weight loss or whatever. And lo and behold, the condition that takes the weight loss pill loses tons of weight, and the people who take the sugar pill don't. Nobody questions. That is a scientifically valid finding.
David Dutwin, PhD:Right? Well, surveys, it's kind of the same way. We take every address in The United States. We randomly choose a subset to reach out to. And so it's the same principle.
David Dutwin, PhD:We get really scientific, highly valid results. We're basing it on scientific approaches of random sampling. And also, again, we're making sure that we we get trust from those parents to really agree to be comfortable. We have an IRB, an institutional review board. This is something that most universities have.
David Dutwin, PhD:And that review board does review every project we do to make sure that we are protecting the people who are participating. And that's important because a lot of researchers when they when they come to us, oh, I wanna interview your teen panel. I wanna interview them on bullying or or drug use or sexual violence. Or usually, our IRB comes back and says, well, you know, if you wanna interview your teens on sexual violence, like, you need to let their parent know first and let make sure their parent's okay with you talking to them about this. So we really make sure that that those folks are protected.
David Dutwin, PhD:Doing a survey right now on inventorying social media harms. What are the ways in which social media may or may not be, hurting teens, in terms of social media bullying and and all of these other elements? Certainly, AI is a big topic right now. How are how are teens utilizing AI? And now we're starting to see researchers really getting interested in wanting to learn the degree to which teens are kinda treating AI like another person and confiding in that AI.
David Dutwin, PhD:Unfortunately, there are stories out there, right, about, AI and and possibly teens approaching AI to talk about suicide ideation. You know? And, obviously, like, as a society, we need to understand these sorts of phenomenon so that we can really develop approaches and policies and ways to communicate with our teens that ensure that they're growing up as healthy as they can in such a challenging culture.
Catherine Batsford:So when you're thinking about public health data, what does that mean in collection terms? Is that the same kind of process that you're talking about that researchers would come to you with ideas and then they would address your panels? Or is there a bigger public health data collection initiative that you work with?
David Dutwin, PhD:Yeah. So NORC as a whole does a great deal of public health research, particularly federal government research. We run the national immunization survey. We have a number of other surveys on Medicaid recipients and other sorts of folks. And and that's important because the federal government needs to get official statistics on a lot of phenomenon out there, including health pro proclivities, what percent of Americans have this malady or that malady, and or how are they exactly using their health insurance?
David Dutwin, PhD:I was actually part of a group that started for many years doing a very large health interview survey in Massachusetts, which they use those data to really understand the needs and status of Massachusetts residents in terms of how they're using their health care, what kind of insurance they have. And that survey directly fed into policymakers that developed what ended up being called Romney Care at the time, which eventually evolved into the principles of Obamacare. That all I mean, not solely. There was a lot of other smart thinking and research being done, but that survey was really quite instrumental in informing policymakers to help build that those policies. So that's just one really great example of the way I think a lot of people don't realize the ubiquity of of survey research in The United States and in the world.
David Dutwin, PhD:You know, we we're swimming in data in general right now, right, compared to society, you know, ten, fifteen, forty years ago. But surveys are are being used a lot of people I think the only surveys they know about are are sort of like horse race polling. Right? Like, is who's gonna win the election? And it's unfortunate because that is, first of all, a very strange outlier type of survey that is done.
David Dutwin, PhD:But secondarily, it's probably less than 1% of all surveys done in The United States are about politics. Most of them are out there measuring public health, measuring health insurance, education, crime statistics, etcetera, etcetera. And these are all done for for the good of trying to develop better policies and and understand what's going on out there in the world. We can use that information, hopefully, to improve the world. And that's really been NORX mission is to provide that nonpartisan insight on the host of ways that we as human beings have to sort of, you know, deal with the world.
David Dutwin, PhD:Certainly, when COVID happened, AmeriSpeak was doing multiple studies every week throughout COVID. But but anytime there's any sort of malady, probability panels are used to get that rapid ability to get in the field rapidly and figure out what's going on, be it whatever, avian flu, SARS, mad cow, like all of them. Health issues happen, they often happen very fast. And a full blown, fully throated federal survey often takes months to get in the field to get designed and and executed, but we can be in the field the same week if need be to track when when things happen in the world.
Dan McLean:I wanted to go back to something you mentioned a couple times about building trust with people who are participating and with their families. And that's something that Premier has spent a lot of time on in recent years and trying to build trust in the scientific research process at broad scale. And I was curious about some of your efforts there. And I was reading about how your IRB works a little bit and that NORC also provides certificates of confidentiality to participants in the panels as well. So I'm just curious of when you approach families and then their kids in some cases, how do you lay that out to explain the process, to build the trust, and explain what a certificate of confidentiality is?
Dan McLean:Because that's important if you're gonna be talking to people in a panel and you want them to to speak freely and honestly.
David Dutwin, PhD:Yeah. So we always have introductory language at the beginning of every survey that lets respond reminds respond you know, because, again, this is a panel. So in most cases, you've heard this before. But but even in NORC standalone surveys, we'll we'll always start with a bit of language to really let respondents know their rights. They have the right to refuse to answer any question.
David Dutwin, PhD:They you know, this is not a deposition. This is just a survey. But, also, that language will be very clear to them that their data is going to be anonymous and confidential. The second the survey ends, the dataset that's created, it doesn't have their name in it. It anonymizes them to a case ID, just a number that denotes them.
David Dutwin, PhD:The researchers that have paid to NORC to do research on their behalf never get that personalized information unless there's a specific need to do so. And if there is, the respondent is informed that it's gonna happen and asks if they're okay to continue under those conditions. So we really work hard, and the the certificate of confidentiality really sort of memorializes that assertion that the respondents are going to be absolutely confidential. It's particularly important in health, public health research, because we often asking very sensitive questions about their health. I've there are surveys out there that the federal government, you know, and others partner violence surveys.
David Dutwin, PhD:Of course, an extremely sensitive topic. We've asked surveys asking about past crime or instances where Mhmm. Crime has been committed against them. Certainly, in terms of health information, people willing to reveal health conditions they've had in the past conditions. And so we want all that as protected as possible.
David Dutwin, PhD:And in the event, even if the federal government comes to us and says, we're gonna subpoena you for this information, our answer is no. Very sorry, but this has been, you know, formally protected by the HHS certificate of confidentiality. We will not give you personal information that you're able to then link to the actual data that we gathered on that person.
Dan McLean:Has that ever happened?
David Dutwin, PhD:Yes. I I have known a couple to happen. And in in the two cases that I know in my career, it turned out personal information was not shared because of the assortment of of confidentiality. Yeah.
Catherine Batsford:So you said before that the populations that you're surveying is fairly randomized, but do you do you feel like there's certain populations that are perhaps asked questions more frequently? Is there a methodology for not having study fatigue by certain populations that researchers could be thinking about?
David Dutwin, PhD:Yeah. There I would answer that two ways. The with regard to survey fatigue, luckily, numbers are on our side. There's there's I think I just looked it up, the most current number. I think there's two hundred and sixty seven million eighteen plus people in The United States right now.
David Dutwin, PhD:So the odds that that you get hit up survey after survey is very low. Of course, the caveat with that is if you've joined Amerispeak, have joined with the expectation you're gonna get a lot of surveys over the course of your tenure, which, of course, people can quit the panel at any time. We really work hard to make sure that we have our panelists kind of in a Goldilocks zone. What I mean by that is if you don't engage them enough, they tend to sort of disappear on you. On the flip side, if you're hitting them up daily, they're gonna get survey fatigue to your point.
David Dutwin, PhD:They're gonna get burnt out, and they're gonna leave because of that. So we we try to sort of make sure the panel is large enough so that panelists don't get invited to more than about a survey a week. We've also done a lot of research on what's called panel conditioning, which is this notion that be you're you're kinda becoming a professionalized respondent by being part of a panel, and that may result in you behaving differently than someone who you catch fresh off the street per se. Notably, that change could be good or bad. Right?
David Dutwin, PhD:People tend to think, oh, that must be a bad thing. People who do a lot of surveys may start behaving badly. We cut talk about the three s's in our field, speeding Mhmm. Skipping, or straight lining, or you just sort of answer every right down the line. But there's also the possibility that panelists learn to pay attention more and be more thoughtful.
David Dutwin, PhD:And in fact, the research out there shows that panel conditioning is very rare. It's very modest. And if there is any effect whatsoever, it is that panelists tend to actually pay a little bit more attention to the what they're being asked. So but it is something we we definitely worry about. We we have a a bit of an algorithm that measures their triple s's.
David Dutwin, PhD:And if they've if they've shown if they break the law once, right, they've sort of done a survey that's supposed to be a fifteen minute survey, and they did it in four minutes. Well, probably, they weren't really paying attention. So we'll shoot them a note, and we'll let them know, hey. We noticed you really, you know, sped through that survey. We want you to really give us the best data possible.
David Dutwin, PhD:Can you please not do that? And if you continue to do it, we may need to stop asking you to participate. So it is something we really focus on, making sure the data is as high quality as possible and that our panelists really try to be as engaged as possible. We send them newsletters. We send them results of their research.
David Dutwin, PhD:We ask them, why did you join? The most frequent response is that people wanna be heard, and they wanna see their results Mhmm. Out there in the public, helping the public. So we make sure we circle back and show them, hey. Yeah.
David Dutwin, PhD:Look. Here's the result from this study that you participated in. And that I think that really helps them feel good about what they're doing here. And just occasional reminders of the good they're doing that I don't know if we wanna segue into this topic, but, I worry about these low response rates. Right?
David Dutwin, PhD:I always try to encourage people when I do podcasts or whatnot. If you're listening and you get a a call to do a survey, you know, participate. It's not it's not some horrible thing. Most people, when we call them after they do a survey, say, yeah. It was a it was a positive experience.
David Dutwin, PhD:I always say trying to be an American. What does it mean to be an American? It means being an individual. It means having a voice. We have such limited ways to participate in our democracy.
David Dutwin, PhD:Basically, we can vote. We could show up at a a protest or a volunteer, but it's a pretty small list. Right? Well, taking a survey is yet another way you can kinda be participative. Let people know what you think about things.
David Dutwin, PhD:Have your voice be heard. It's there's in some ways there's no better way to be an American, right? Than the to have your voice be heard. So, you know, I always I always try to get that plug in there whenever I I speak, because I I'm passionate about it, and I feel badly that a lot of people, I think, because of the age of telemarketing have become so jaded when they're reached out to from a stranger. It's a good opportunity again to to have your voice be heard.
David Dutwin, PhD:Better that than to have nobody care about what Two you think,
Dan McLean:things real quick. You mentioned a priority of sharing the results with participants, and it's something we've just been talking about recently. In fact, the NIH director has just prioritized that as goal across the research enterprise to get entities to attempt to share those results back with participants in all sorts of human subjects research trials. I think it's interesting to hear you say that as well. There was also we were talking about the the groups that are participating in your panels, and I was hoping you could speak a little bit to how it's how you structure them to be representative.
Dan McLean:But it says AmeriSpeak is a probability based panel with a household selected from documented sample lists, technically called a sample frame. So can you kinda break that down for us? How are these panels structured so they're not just one niche that you've tapped into?
David Dutwin, PhD:Yeah. Yeah. It's a great question. So always start with the fact there's two types of panels out there. As I mentioned a little bit earlier, there's probability based panels and nonprobability.
David Dutwin, PhD:Nonprobability are these self selected. You may get an advertisement. Oh, you wanna take a survey? I got invited to one of them once because, I was living on the East Coast. I was using Amtrak a lot, and I was in their miles program.
David Dutwin, PhD:And I guess this panel company had a partnership with Amtrak to get their list. And it turns out that that probably panel also had partnerships with the American Airlines program. And so you have to ask yourself, okay. Well, if I wanna do a survey on travel, how representative is it gonna be if I recruited to my panel, you know, from Miles programs. Right?
David Dutwin, PhD:So nonprobability surveys have have some problems. That's a whole another topic. But in probability, like AmeriSpeak, again, we start with a national frame that the NORC national frame is the US Postal Services database of all addresses in The United States. But that database, we estimate likely covers about only well, not only, but about 92% of all US households. What are the other 8%?
David Dutwin, PhD:People who only get mail through PO boxes, people who live in extreme rural situations, like in Appalachia and Alaska that have what are called simplified addresses that are not easily deliverable to. So what the NORC National Frame does is it takes that US postal database, and then we actually go out there into the field and physically enumerate and may oh, there is a house here. Yes. Okay. Make sure it's on the list.
David Dutwin, PhD:And by doing that, we're able to get our coverage of all households in The United States from 92 to about 97, even 98%. So the goal is to start with a frame where every American is in that frame. Right? And so that's what the NORC national frame does. Then, like I mentioned, the word probability is synonymous to random sampling.
David Dutwin, PhD:So we take that list of all households and we randomly select. And I think, you know, Pew Research did a great video on this to sort of give visualize this to your viewers. You have a bowl of Campbell's tomato soup. Right? It's just this red mass of soup.
David Dutwin, PhD:Right? There's nothing distinct about it. Then the question is, how many of the people you that you've randomly selected actually participate? And is are those people the same as the people who don't participate? And that's an important thing.
David Dutwin, PhD:When I talked about earlier response rates having plummeted from the mid thirties to single digits, right? If only one out of 20 people actually participate in your survey, well, that doesn't matter if the one person the people who do participate look exactly like the people who don't. But the reality is we know there are some differences. Historically, those differences were largely explained by some key demographics. People with higher educational attainment tend to participate more.
David Dutwin, PhD:Whites tend to participate more. Older people tend to participate more. Well, luckily, the census has great data on those those metrics, and we're able to do statistical waiting on the back end to fix for any disparities between who we got in our survey and who the census is out there. Just to give you the simplest example of that, this would never happen, but let's pretend that only 25% of a survey is female and 50% of The United States is female. Well, a sub a survey weight would would basically count every female twice to get that population up to 50%.
David Dutwin, PhD:And then, of course, men would be weighted at a number under one to shrink them down. Every pretty much every survey on the planet since the nineteen sixties has been weighted to do this because, yeah, it is true that, gee, I don't know the exact numbers, but 15% of the population is age 18 to 25. Our survey only got 12%. So let's give them a little bump so that they look like 15%. But and that has been incredibly effective at making sure that samples that we get in surveys are indeed representative of The United States.
David Dutwin, PhD:But now we're entering a phase where there's concern and good evidence out there that it's not just demographics. There are other factors that are leading to the decision as to whether to participate or not. People who have very low trust in institutions and other people tend to not participate, people who mistrust science. The problem with that in in the the eyes of election surveys, for example, is that those are the kinds of people that also have been much more likely to vote for Donald Trump than not. And so now, in the past couple elections, surveys have resulted in being a little bit Democrat depending on who did it slightly to very skeptic democratic because those low trust individuals are not participating in surveys.
David Dutwin, PhD:So what do we do about that? Well, we're we're really making sure in it, like, our speak, for example, we have really good language in our recruitment that, first of all, is accessible, low trust individuals tend to also have very low socioeconomic status, low educational attainment. These folks are not National Center for Education Statistics shows that these people have lower levels of literacy and literate proficiency. So we try to design our materials to have sort of a low word count to use really straightforward language. When we talk about who AmeriSpeak has done surveys for before, we really make sure we we give a really wide net.
David Dutwin, PhD:Very early in the day, we say, well, you know, we we do we've done surveys for x, y, and z. And x, y, and z were Northeastern, perhaps liberal Harvard University, whatever.
Dan McLean:Mhmm.
David Dutwin, PhD:No. Let's make sure people understand. No. We also do research for the American Bible Society and for the American Enterprise Institute and the George W. Bush Library.
David Dutwin, PhD:Really make sure that we hit up, right, every so it's it's trying to recruit in a way that is inviting of everyone, not just educated older people. And and then the field is is exploring a lot more advanced ways to do statistical waiting to make sure that those low trust people get weighted up and and are adequately represented in in our samples. So, I mean, the short of it is that surveys that are done really well work really hard to get high response rates to all the best practices out there to to try and make sure that the people who do participate look as much like the people who don't as possible are still, research shows, highly representative and doing really well. But surveys that are more, say, quick and dirty are definitely having representation problems because these low trust individuals are not participating as much as as higher trust individuals. And it's definitely a challenge that the survey research industry has been tackling for the last decade or so.
Catherine Batsford:So let's say I am a researcher and I'm designing a study today. What's one change they can make immediately to maybe improve the representation and trust in that study?
David Dutwin, PhD:Great question. Using a probability based approach is certainly important. Vetting and and selecting a vendor that you feel will execute a very high quality study has a track record of doing the best practices. Because particularly in the case of panels, like, we've already we already have the panelists. Whether whether we have a representative sample or not has happened prior to you coming to us to do a survey.
David Dutwin, PhD:But that said, it's important to have it open inviting language at the beginning to not sort of give away. You know, back in the day, it was very common, like, oh, we're calling on behalf of CBS News. Well, nowadays, that might be problematic. Right? Because people have been sorted into their camps, and people who live in a camp that don't like CBS News are not gonna participate.
David Dutwin, PhD:Right? So in fact, nowadays, with mail surveys, it used to be the case that the best practice was maybe you put something on the envelope. Hey. Survey enclosed. You know?
David Dutwin, PhD:But now the best practice is to make that envelope have nothing on it but the address because that way, the the recipient has to open it to know what it's about. And that's what it we don't want them to self sort themselves out before they even open the envelope and see what we're asking them to do. Right? So those envelopes now are very plain so that they have to open them. But, again, then when they open it, having whatever you can do, you know, Don Tillman, who was a real sort of father figure in the field, was at Washington State University.
David Dutwin, PhD:And when he did studies of Washington State, like, when you open that letter, there'd be, like, pictures, like the space needle, an apple, because there's lots of apple forms. Try and connect to the recipient a little bit in a meaningful way so that they they do get a little sense of like, oh, okay. What is this about? I'm a Washingtonian too. Like, let me see what this is about.
David Dutwin, PhD:And so there's there's a there's a lot there's literally thousands of pages of research out there where people try to look and try and figure out these things. So I guess the short answer again is, hire a firm that has methodologists that know all this literature, you know, and really know how to make sure that your survey is gonna be signed, the best it possibly can in order to to be all inclusive to everybody that you're trying to survey.
Dan McLean:I have a question on on some of the, case studies of work that you've done. There's a portion of people in the survey who are part of the Amerispeak part, and then there's more of a more traditional survey as well. Seems like that there's a blended pool in some way. So I was I was curious of how you decide how many come how much information comes from the panel versus a different type of survey. Two of those that are looking at was that one was really interesting, I thought, about the political spectrum in The United States.
Dan McLean:And you guys identified five different categories of people where they're segmenting themselves. So I thought that was an interesting one. And then another one, which was a completely different topic about food allergies, but it's the same kind of methodology where you have part of the people coming from the panel and then part from another source of information. That I'm curious of how you break those data sources down.
David Dutwin, PhD:Yeah, sure. In a perfect world, would do everything 100% from the panel because, again, we know we've recruited at a very high level of quality. We've even gone door to door to recruit people. But there are use cases where we just can't get everybody a researcher wants from the panel. In the second example, the food allergy example, you're talking about somewhat of a low incidence population.
David Dutwin, PhD:Right? So we may have upwards to 50,000 people on the panel, but the number of people that have had severe food allergies, for example, might only be a thousand of those. And we don't force our panelists to take every survey. We invite them to each and every survey we want them to do, and they choose, and typically about half do and half don't. So you might only get in this example 500 from the panel, but maybe the researcher wants a thousand.
David Dutwin, PhD:So then we have to use a partner, something else. So we'll use a partner panel or maybe we'll do fresh cross sectional interviewing, just calling or mailing households, or we we may blend with that, a nonprobability panel I talked about before. That's gonna be a little bit lower quality. But but we have statistical techniques to blend those samples in a way that tries to sort of reduce any bias that may be inherent in the lower quality panel than the higher quality panel. So that's the first situation.
David Dutwin, PhD:I think that or or your second example, the food allergy. I think in the first one, if if, for example, that's the study Vocast, well, Vocast surveys 80,000 people nationwide so that they can get really great data on the state level and and even even metropolitan level. And again, Amerispeak probably if we invited everybody on the panel to do a survey, we'd we'd only be able to do only in quotes, I guess, about 25,000 interviews. Vocast wanted 80,000, so they have to so those are kind of the two use cases, a very small population or a survey that wants to just get tons and tons of respondents for, the ability to precision into smaller areas. But, you know, in general, we we try to keep it as much on Amerispeak as possible because, again, we know the quality.
David Dutwin, PhD:We trust our methods, but it's just not always possible.
Catherine Batsford:So we always like to wrap up thinking about the future. And what what does survey technology look like in five, ten, twenty years if we were to revisit and discuss?
David Dutwin, PhD:Yeah. It's this is a good opportunity to mention the hot topic of the month and probably for the next year or two, that is AI. How is AI impacting the survey research community? And the answer is there are now synthetic data products coming out. What is a synthetic data product?
David Dutwin, PhD:Where it's a survey of a thousand people, except they're not people. It's an LLM, a large language model like ChatGPT or or Claude or one of the others, taking the data they have and making up the answers, basically. Making a survey of a thousand quote unquote respondents, all of them AI generated, answering every question in the survey. Where is this headed? I will say it's I think, it's similar to non probability.
David Dutwin, PhD:So when not the non probability polls started happening twenty fifteen, twenty years ago, it's like any other adoption of a new technology. The first phase is, oh my god. This is gonna totally replace everything we do. And then cooler heads prevail, and we realize, no. Not really.
David Dutwin, PhD:There's use cases for which this might be appropriate, and there are use cases for which it's not. I I will bet as any amount of money that the federal government is not gonna, in any time soon or probably ever, decide to do a major official statistic survey using synthetic data. They need their goal. These studies literally, in some cases, billions of dollars are spent based on the policy that's developed. Again, thinking of the Romney Care study I mentioned to you before.
David Dutwin, PhD:Mhmm. It is important the the the government rightly is saying, we will spend top dollar to do the survey because that top dollar is still pennies on the dollar to the billions of dollars the federal government are is gonna spend on this health care or whatever. So those surveys are still they're gonna be human based, I think, forever. But we'll see where this goes. And notably, what's really important is, like anything else in AI, the quality of what you get back is based on the quality of the inputs that the model has.
David Dutwin, PhD:We and I think a lot of other people are are are engaged with people who want to are realizing that a really good synthetic data product is probably one that is built on really high quality data in the first place, data like AmeriSpeak or the general social survey or some of these really great public datasets that are out there. A synthetic data product that's built only on respondent or whatever is gonna is kinda gonna be garbage in and garbage out. So it's a brave new world out there. It's a really fascinating world. It's just gonna mean that more data is generated.
David Dutwin, PhD:More people are gonna be using data to make decisions, which has been the trend line in this society and in the world for twenty, thirty years. But it is just a really interesting new brave frontier for survey research, like a lot of other fields. Right? Real quickly to the one other way AI I mean, there's a lot of ways AI is impacting survey research, but another way is people are now testing synthetic interviewers. Right?
David Dutwin, PhD:Why do a telephone survey with live interviewers, when maybe you can have, an AI bot administer that interview?
Catherine Batsford:So interesting.
Dan McLean:It's funny to hear you talk about the synthetic, data and AI. I've got to admit, I'm guilty of conducting my own AI polls. I asked chatbot. If you asked a 100 experts in this field this question, what would they say? And it it gives me the caveat that, well, I can't actually ask them, but this is what they'd say.
Dan McLean:And then it tells me 80 would say this and 20 would say that. And it's kind of an interesting little pulse.
David Dutwin, PhD:Well, you know, your your use case, it gets back to the main mantra of the field of survey research is fit for purpose. If you're Mhmm. Asking AI something and the consequences of it being wrong are like, yeah. Well, okay. I was wrong.
David Dutwin, PhD:Right? As opposed to, again, if if the federal government were to ask how many people were suffering from asthma, and I I actually give you a better example. I did a survey years ago, and we did it both probability and nonprobability. Do you intend to buy a new car or light truck in the next year? And the nonprobability sample got double the number that the probability sample got.
David Dutwin, PhD:Now, if you if the fit for purpose of that survey was to decide whether to build a whole new plant for $7,000,000,000 and you look at this data, oh my god, like 30% of people are going to buy a car next year. We got to ramp up. We got to build a new plant. And then it doesn't happen. Right?
David Dutwin, PhD:The fit for purpose there is such that you're not going to use synthetic data to answer that question. You're still going to use real people to get the right answer. Right? So it it really is all about fit for purpose. So so we won't we won't shame you, Daniel, for asking AI that question.
David Dutwin, PhD:It's your fit for purpose was probably pretty low. You were not billions of dollars were probably not contingent on your your research.
Dan McLean:I asked it if I should change my brake pads and and when. If I asked a 100 mechanics and Yeah. Weighs in. It gives me some guidance. Yeah.
David Dutwin, PhD:Absolutely.
Catherine Batsford:Well, thank you so much for joining us today. I learn a ton every time I hear you speak, and it's just it's fascinating everything behind the scenes that's going on. And we'll put that plug in. If a survey comes by, you should take it. That's your Yes.
Catherine Batsford:That's your choice as an American. Make your voice heard.
David Dutwin, PhD:Yeah. Thanks for for having
Dan McLean:This was great. Thank you very much. We really appreciate your time and your your insights here.