Subscribe
Share
Share
Embed
Who thinks that they can subdue Leviathan? Strength resides in its neck; dismay goes before it. It is without fear. It looks down on all who are haughty; it is king over all who are proud. These words inspired PJ Wehry to create Chasing Leviathan. Chasing Leviathan was born out of two ideals: that truth is worth pursuing but will never be subjugated, and the discipline of listening is one of the most important habits anyone can develop. Every episode is a dialogue, a journey into the depths of a meaningful question explored through the lens of personal experience or professional expertise.
PJ (00:01.666)
Hello and welcome to Chasing Leviathan. I'm your host, PJ Weary, and I'm here today with Dr. Jeremy Weber, professor at the University of Pittsburgh, and we're talking about his book today, Statistics for Public Policy, and I love the subtitle, A Practical Guide to Being Mostly Right, or at least Respectably Wrong. Dr. Weber, wonderful to have you on today.
Jeremy Weber (00:24.888)
Thanks so much for this opportunity.
PJ (00:28.059)
So Dr. Weber, why this book? What led you to write this?
Jeremy Weber (00:34.584)
This book was not something I decided to do on a whim, rather it was the culmination of many experiences over more than a decade that just led me to the ideas and the urge and the felt need to write a book like this to say these things. And that experience was
I would say going back to graduate school when I was fortunate to be in a program where the professors typically collected a lot of their own data, going out into the field and talking to people, developing their own surveys. That gave me an appreciation for, you know, it matters how you ask the question. You can get really wildly different answers and statistics depending how.
PJ (01:27.171)
Hmm.
Jeremy Weber (01:33.108)
you ask people a question. Then my first job out of graduate school was in the Economic Research Service, which is a federal statistical agency within the Department of Agriculture. And unlike most people who would go into a strictly academic environment where your primary activity is teaching and then researching, presenting to fellow academics, I was in an environment where
I did some presenting to other academics, but a lot of engaging with people who were not academics and communicating statistics to them, thinking about what statistics would be most helpful. Then from there, I transitioned to graduate school, I mean, to the Graduate School of Public International Affairs as a professor and began teaching quantitative methods classes to policy-oriented students.
And I inherited a course and it was much to like about that course in the book. But as I went on, I realized I don't think I'm using the time as well as I could. And I think my students are leaving these classes and very few of them have the confidence to use what I'm teaching them. So I, there's some gaps. There's some statistical mentorship gaps.
PJ (02:54.966)
Hmm.
Jeremy Weber (03:01.336)
that I'm not providing, and that's making this whole exercise a waste of time. Because if they're not using what I'm teaching them, I mean, I don't know what I'm doing, what we're accomplishing. And then after that, I had the opportunity to spend a year and a half working in the White House in a statistical analysis capacity, and that further clarified some things. What are these key things that I want students to be able to do well?
And that's a long answer, but it's somewhat of a long journey to this book. It was a culmination of those epiphanies, those experiences, those gaps I saw, those insights that I came to mind as I was speaking to people about statistics and realizing they didn't know what I was talking about. So I'll stop there and you can prod.
PJ (03:33.837)
Hehehe
PJ (03:59.242)
Yeah, so in a very layman's sort of way, I've always been interested in statistics. So I've read things like How to Lie with Statistics, Naked Statistics, Storytelling with Data. And so a lot of like what you have in here is familiar to me, but the for public policy part is really, I think, what distinguishes your work.
Is that a fair way? How would you place what you're writing? Because there is that popular level of what you're doing, but yours has a much more specific purpose. Can you speak to how it fits in the constellation of statistics books?
Jeremy Weber (04:36.117)
Yes.
Jeremy Weber (04:39.892)
Yeah, definitely. There are books that are oriented more to the statistical consumer, how to understand figures and how not to be duped by
certain statistics. There are obviously statistics textbooks that walk you through the formulas, the theorems. Here is how you actually calculate this thing.
Jeremy Weber (05:17.88)
I saw and hopefully have filled a gap in what I call statistical mentorship. This is a little bit deeper than what somebody who just wants to be a equipped consumer of statistics would want. But it's not regurgitating, rehashing what a statistics textbook will do. It is what I call statistical mentorship for someone.
who is trying to help a decision maker.
understand an issue better and they're going to use their statistical savviness to further the understanding of somebody who's in a position to say or some group that's in a position to make a decision about some policy topic and so it's that policy aid
that I have in mind. So this is very practical. This is addressing things like what are, what are things that you've got to get right? And if you don't get these things right, all the details will be wrong. Uh, these sort of matters of first importance, um, that you want to do well so that the person you're serving or group or audience that you're serving, uh, is not, they're confused.
You don't bring confusion, you bring clarity to the conversation. And so I think that's a little bit different than a lot of what's out there.
PJ (06:53.79)
Almost like a rhetoric for academic statistics, is that?
Jeremy Weber (07:00.152)
only covers that because
Jeremy Weber (07:04.996)
to provide statistical information that's helpful, you have to be thinking about what argument is this statistic feeding into? What is its role? And something that I'm big on in the book is thinking hard about what's been calculated. I talk about some,
people who do statistical work, we often think that we're done when we've calculated the number. And I say, we're just halfway there. Most, a good portion of the work is still to be done. And that is thinking about, what is, is this a large number? Is this a small number? How should my audience understand this? And that can require further digging.
to provide the appropriate context so that then the audience knows how to take your number. The one degree increase in temperature.
If you're just given that it's like, well, I don't know that it's one, that sounds fairly close to zero. Um, so that seems small, but does a one degree increase mean ecological devastation and loss of harvest? You know, you've got to do more works to understand the meaning of one degree.
PJ (08:20.386)
Hehehehe
PJ (08:40.022)
Yeah, like, what, 88 to 89, I'm like, that doesn't change my day, right? Like, that's...
Jeremy Weber (08:45.26)
Right, right. And so I get into this when I talk about, in a chapter called, show that you've been to table school, that's basically show that you know how to communicate statistics well. And I say, a general principle for doing this well is present only what's needed and no more. Well, that principle applies to the digits in a statistic. Are you saying nine?
PJ (08:54.603)
Right at the start of that.
Jeremy Weber (09:14.952)
9.2, 9.27.
Jeremy Weber (09:20.488)
That matters, like nine sticks, you get that really quickly, that sticks. 9.273, to read that on the page or to hear it is a lot more burdensome. And whether or not that, what comes after the decimal matters, depends on context. If we're talking about, you know, being an Olympic competitor in the 100 meter dash, you know, then the hundreds place is
distinguishes, you know, gold medal from no medal. But in other contexts, that's just noise. So you have to really know your context well and you have to know is a hundredth of a second a big deal here or is that total noise and just distracting my audience? And nine is plenty of precision given the context.
PJ (10:18.826)
What, I think to help set this up for our audience, and I'm not asking you to like name names or anything unless it's like, you know, historically relevant or, you know, ancient enough that you're not stepping on anyone's toes. What's the worst failure you've seen or how would you describe someone the worst failure you've seen due to poor stats use in public policy?
PJ (10:49.087)
Or one of the worst, yeah.
Jeremy Weber (10:50.464)
Yeah, one...
Jeremy Weber (10:56.292)
Two that come to mind that are by no means the worst and they're not the... maybe even...
Jeremy Weber (11:06.116)
seem that serious, but they stand out in my mind. One is...
Jeremy Weber (11:14.18)
From my own experience, I wrote a paper, this was when I was working at the Department of Agriculture, I wrote an academic paper about the effects of agricultural subsidies on production, which is a sensitive topic, it's an important topic, because the US has agreements that constrain the type of policies we can do with regards to agriculture. We can't be...
doing policies that flood the market with cheap corn and then affect foreign producers. Anyway, wrote this paper and then I had the opportunity to present it to the USDA's chief economist and I gave that presentation and it was a great presentation for an academic seminar. But the chief economist and others in the room...
quickly started asking questions, more about implication, more about the policy context, the meaning of the numbers, that I just wasn't, I didn't have anything to add. I didn't have the rich context that they had. And so they started having a conversation among themselves in the room. And I was...
kind of left out or left on the sidelines because I had spent so much time thinking about how I calculated the numbers and the basic conclusion, but they had so much more knowledge of policy context and significance or lack thereof of the numbers and follow-up questions that I felt like, wow, I was like a little leaguer who just entered.
um, a big league conversation where, where the number was helpful, but it was just kind of a starting point for the conversation. And so I thought, you know, the number, you know, that's the show stopper. This is it. This is the conclusion. And it was just the beginning. And so it was somewhat embarrassing because I felt I was so ill equipped to engage with
Jeremy Weber (13:38.144)
what this number does mean or doesn't mean or what sort of policies might it imply are inappropriate. So that was one where it was fairly benign. I mean, it just meant, you know, I kind of became irrelevant very quickly. My number helped to stimulate a broader conversation among people who were far more steeped in the topic.
Jeremy Weber (14:01.448)
Another one that comes to mind is I was listening to a national media outlet talking with a public health expert about
Jeremy Weber (14:15.936)
a new variant of COVID. And it was a variant that was hitting youth harder, supposedly, and the journalists asked the public health expert, should we be concerned about this new variant?
And the public health expert responded by saying, the number of you young folks hospitalized because of COVID is hitting records.
That was the main response. And this was while I was working on this book. And I thought, he didn't answer the question. He did not answer the question. I mean, my chance of a heart attack is hitting records every day just because I'm getting older. If you hit a record from a really small number, it's not a big deal.
PJ (15:16.387)
Sorry.
Jeremy Weber (15:22.612)
And I'm not saying that the record number was or wasn't a big deal, but what he, he gave was he reported a measurement. You know, we hit a record. He didn't answer the question. Should we be concerned? Which would require a lot more information. Okay. Yeah. How many people are being hospitalized? Not just that it's a record. How many, what are the effects of those?
experienced by those who are being hospitalized? What percentage of them are in more serious condition or dying? How does this threat compare to other things that are harming youth? Give me that context. I mean, there are a lot of youth dying from all sorts of things. Is this the number one cause now? Is it the number 10th cause? Right?
That is a very common practice for people who love the numbers, love the crunch data. We like to report statistics and then just call the day done. And I use an example of this in my book. An example of somebody who you drop your car off. I dropped my car off at a fictional example, but it illustrates the point. I dropped my car off
at a auto place and I'm wondering if I should get new tires. I had the tires for a while and they look like they're now getting old and I asked the mechanic who measures the tire, I said do I need to get new tires and the mechanic reports well you have six millimeters of tread.
and just looks at me as if he answered my question. I didn't ask him how many millimeters of tread were left. I asked him if I should change my tires. Whether six millimeters, what I do with that depends on, I don't know, did the tire start with 12 millimeters? Did it start with seven millimeters? At what point does the risk of the tire blowing out or me sliding off the road jump?
PJ (17:17.132)
What is-
Jeremy Weber (17:44.88)
That's the contextual information I need to make a decision. But the mechanic at that point is just, they just wanna be a measurement reporter. And what I really need is somebody who's going to do more. And I think that's maybe the number one error or limitation of current practice of statistics is we like to report numbers.
And we don't do a good job of explaining their meaning for the context or purpose at hand.
PJ (18:22.682)
To use, to kind of combine your example with the story that you gave of the public policy director, that'd be like the mechanic saying, well, it's more dangerous than it was last week. And it's like, well, what does that mean? Like, I mean, of course it's worn down more than last week. Does it mean it's like, is it way more dangerous? It's like, you don't even have, you didn't give a measurement. You're just like, it's records. Like, well, there's, you know, there's, what kind of records are we setting? So
Jeremy Weber (18:43.297)
Yes.
PJ (18:51.754)
That may, like, this is that connecting piece. Yeah. And that's what, I mean, really, what you've been, we've been talking about here is that first chapter, the big picture, right? Like, just the context. And I think that leads into, nicely, like the next chapter you talk about, know your sample and your data. And that's where, that's that richness of that policy that you're talking about is the, you find the big picture through the context.
Jeremy Weber (18:52.051)
Exactly.
Jeremy Weber (18:56.982)
Exactly.
PJ (19:23.11)
I'm sorry, you find the big picture through actually digging into the surrounding context, like the details that surround that. What are some common mistakes that people make when asking questions or not filtering the questions and paying attention to what's really being asked? And how can you avoid that?
Jeremy Weber (19:44.932)
Are you talking about when somebody's collecting data or when somebody's reporting statistics from the data?
PJ (19:50.99)
That's fair. Yeah. I mean, and I did, I literally just asked two questions because I said, when you're asking the question, and I said, knowing what questions were asked, those are two different things. That's very fair. The, well, for your target audience, you're talking about making sure you understand the questions that were asked, right? And that would be, which would be people who are dealing with the data after it's been recorded.
Jeremy Weber (20:12.78)
Yeah, yes.
Jeremy Weber (20:19.556)
Collect it. Yeah. You know, for whether you're going to be calculating the statistics yourself or just consuming them.
Statistics, they're just observations. They are summaries of observations of people, of places, of businesses.
Jeremy Weber (20:44.936)
It always matters.
Jeremy Weber (20:49.144)
who would consider what's the scope of your observations? Just like, you know, if you wanna know what it's like to live in the city of Pittsburgh, where I live, and you ask two people who, you just talk with a person who lives in a particular neighborhood, in a particular type of house, you ask about their experience and what it's like to live in Pittsburgh, they can tell you true things.
If you only talk to a certain group of people who live in a certain place, your takeaway conclusion from this conversation is clearly going to be driven by you only talk with people who are older than 75 and living in the Bloomfield neighborhood.
That matters who you talk to. What are these folks that then feed into our statistic? What are they like? Were they selected at random? Were they, um, did they volunteer to, uh, come in? What sort of rules, eligible, eligibility rules did the person collecting the data have? Oh, we, we only allowed, you had to live in the city of Pittsburgh for at least x years.
All right, those rules that then constrain who you're learning about or who's reflected in your statistic are hugely important for understanding what it will and what it won't capture and the meaning of the number. The example I use in the book is it's a little disorienting to go to the Department of Agriculture Economic Research Service website
and find that for the last 20 or so years, the median farm household income in the United States has been negative.
Jeremy Weber (22:46.708)
Every year farm households are making are losing money from their farming activities according to this statistic. Not a recent thing. It's essentially since they've been collecting household level data in the beginning in the mid 80s always negative. Wow really a lot of suffering going on in the heartland a lot of suffering economic distress.
If you dig a little deeper and ask, wait, what is a farm household? What is a farm? How do you get this survey from which the USDA is collecting this information and calculating their statistic of household income? If you ask those questions, you'll realize you're probably a farmer. You just might be a farmer. The definition is so broad.
PJ (23:40.828)
Hahaha
Jeremy Weber (23:45.56)
that it includes far more people and places than most of us would expect. And that most of us would probably consider, wait, that's not what I had in mind when I hear farm household. All you need is the potential to have...
a thousand dollars in agricultural goods or services in a year. So it's just the potential. And by the way, I think a horse counts for like a thousand points. You need like a horse or two to qualify. So suddenly now, Oh, so everybody who has like two horses,
would be in this population that would then be sampled. Oh, well this, you can very quickly begin to imagine that these people are not farming the way I think of farming. They have a hobby and most of our hobbies don't make us money. So this is just an example of where knowing how the population is defined, in this case, how a farm is defined,
is key to understanding why the median farm household is making negative dollars from their farming enterprise. If you start filtering the data to those that have considerable investment and look a little more like what we would think of as a farm, then the numbers start to make a lot more sense.
PJ (25:26.378)
Yeah, even as you say that, all of a sudden I have a, I live in a kind of suburban, but on the other side of me is rural. I have several friends who are kind of out there. And I now get to inform them they are, I think you're actually farmers. That's like, I was not even there, even close to, I think they would find that to be a surprise. I have a...
Jeremy Weber (25:35.989)
No.
Jeremy Weber (25:44.431)
Thank you, Farmer.
Jeremy Weber (25:52.928)
Yeah. They might only need a grass field. You know, they have some grass, some large grassy area that is the potential of growing some hay, you know, they might be, or a couple fruit trees, you know, if they're, if they're really, um, fruitful fruit trees, they could easily be producing, have the potential to produce thousand dollars. Yeah.
PJ (26:03.287)
Yeah.
PJ (26:14.35)
Right, potential, which is also a big, yeah. No, the husband and wife, the husband's a lawyer, but the wife works with the local animal shelter. So she just continually takes animals on, and they have ducks and chickens, and for her, they're mostly all just pets, right? And then, but it's like, yeah, that's really interesting. Even what you said about the Pittsburgh type of census, when you say things like...
Jeremy Weber (26:28.722)
Oh.
PJ (26:40.586)
Let's say, you know, we only will ask people who have been living in Pittsburgh for three years. It's a different question than we will talk to people who have been living at the same address for three years. Right?
Jeremy Weber (26:53.912)
That is exactly those sort of distinctions will actually work and can matter a lot. Are we picking up the transient population? Are we picking up the long timers, the more rooted population? They look really different. So depending on what sort of requirements you imposed when collecting your data, it's going to affect your statistics and what we might learn from them.
PJ (27:24.69)
And I think this is kind of, you know, kind of the next... I almost hate asking this question to you because I think this is probably like one of the things that statisticians get asked out about the most, but you have this chapter on what is the difference between correlation and causality and how do people confuse them and what is that? What problems does that cause?
Jeremy Weber (27:48.228)
Mm-hmm. No, it's a great question. And I think most people, many people, are intuitively familiar with the difference. They know that just because a lot of people, a lot of umbrellas emerge at the same time it's raining doesn't mean
that the umbrellas are causing the rain sort of thing. There's a famous, one of the common examples is there's a correlation between ice cream sales and murder rates. And so as you have more ice cream sales, the times of high ice cream sales are also times of high murder rates.
That's the correlation part. They're moving together. If you had one, you could predict with some degree of accuracy an increase in the other. And but we, and people intuitively get, oh yeah, that's nonsense. It's not, you know, ice cream cartels fighting it out amongst themselves for the ice cream market profits. That's just
People tend to be more active in the warmer weather. Murders are more common in warmer weather. And that's also when we tend to sell more ice cream. I don't think that is the example that's given in any sort of empirical classes or in, there are examples like that. And there's a doubt, you know, they make the point. Correlation is not necessarily indication of a causal effect.
But I think something's lost by giving such a trivial example. And here's what's lost. Most of the time, the real challenge is not, we were going to clamp down on ice cream sales because we really did think they were driving the murder rates. No, it comes in instances where there's a intuitive, plausible causal connection. All right, you know.
Jeremy Weber (30:02.452)
let's say more resources, more school funding, and test scores and graduation rates. All right, two things that are correlated.
There's a plausible mechanism to relate those two. You got more money, you can probably hire better teachers. Better teachers then result in students being more engaged, learning more, doing better on the test and graduating. So there's a causal story in place. And we observe just in the raw data, yeah, schools that get more money, they have higher graduation rates.
much higher graduation rates, much higher test scores.
The real and more, what I say, the more pervasive problem is where the correlation overstates or maybe understates, but misstates the real causal relationship. So in the raw data, we just look, oh, here are high income schools and or high resource schools and they have high test scores. And the difference is striking.
So then we think, okay, it's about resources. And then we say, okay, we need to increase the property tax rates or get more federal funding. Let's give them a boatload. Let's drop cash down on that district or on those lagging districts. And then we're surprised when, wait, we're not getting that jump in test scores that we thought. Well, that's because the correlation overstated.
Jeremy Weber (31:48.856)
the actual relationship. There is a true causal relationship between money and student performance, but it's maybe not as strong as we thought. And if you more carefully separated out some other factors that tend to go with resources, so the family environment, the home environment, which tends to also be more stable.
and reinforcing of what's happening in the school when it's higher income. All right, we were thinking that it was just all about the money. When in fact, the money part was masquerading or masking another driver that we weren't paying attention to. And so the cost of this is that the cost of
seeing correlation and thinking that indicates the true magnitude of the causal relationship is that we then run with that, do things that we think are going to have big impacts based on their correlation and they don't have nearly the big impacts that we thought or hoped they would. That's the more common problem and the common downside of confusing correlation.
as being all about causation.
PJ (33:16.934)
And is that links with kind of Why we need regression analysis to discover those links. Is that is that the right way to think about that?
Jeremy Weber (33:27.624)
Yeah, exactly. That's why we need, I would just say, more work. And that can be done through regression analysis where we're trying to incorporate additional variables like family background, parents' education level, when we're trying to isolate the effect of school resources on student performance.
It just, yeah, it raises the need for thinking about.
Jeremy Weber (34:03.98)
what we call experimental design or research design. Your research design is meant to uncover what you want to uncover. So in statistical research design or looking for the cause or effects of school resources, I need to think about a research design that's likely going to isolate the effect of the thing I want to study. When I just take the data and I run the correlations, I'm not thinking at all about
PJ (34:12.554)
Hmm.
Jeremy Weber (34:32.524)
research design and whether or not that correlation is isolating the effect of one variable and the other. I'm just calculating a number. So yes, regression and there are other related techniques. Sometimes we don't even need much of a regression. We just need to compare two averages. It can be as simple as that. But that depends on the research design that's gone into the setting.
PJ (35:01.594)
Can you talk a little bit more about the pothole example and how research design can matter in these, especially in these class distinctions, and racial distinctions?
Jeremy Weber (35:20.264)
Yes, so a common technique taught in many statistics classes, especially for people wanting to do policy related research is regression. This is a method that seeks to hold constant some variable, estimate the relationship between two variables, and it allows you to
in a way hold constant other variables. And that's very powerful. It's hard sometimes to understand how it holds constant, the regression calculation is holding constant other variables. This chapter is meant to explain that a little more in detail and just show the usefulness of being able to account for some other variable.
while I'm making a comparison of some main or primary variable. And the example I give, it's a fictitional example, but, and craft to illustrate the point is, somebody starts complaining that in this city, in this place, high income neighborhoods are prioritized for road spending. So the city's...
public infrastructure spending, road spending budget is disproportionately going to give wealthy communities just the best roads and neglecting lower income communities. And imagine that somebody looks into the data and finds actually the dollar spent per mile of road last year was the same in high and low income communities.
Well, that would seem like, okay, it's just not the case that one type of community is receiving more than it's share.
Jeremy Weber (37:25.06)
but ask an additional question that is, wait, maybe the roads in the low income communities are so much worse that they actually should be getting more road spending dollars. So if we thought about holding constant road quality, are they getting the same amount of money?
What we want to do, the thought experiment we really want to see if unfairness is happening, is take two streets, one in a low-income neighborhood, another in a high-income neighborhood, and ask, and two streets that are of the same quality, they have the same number of potholes.
And then we want to compare those streets and say, are they getting the same funding? So in the example, when you hold constant road quality, it turns out that the high income communities are getting way more money given their road quality than they should. And if you weren't able to hold constant,
road quality through this regression and you just compared road spending per mile in these two types of neighborhoods, it's going to look like there's no problem. But once you incorporate the road quality, which is a measure of need of spending, it becomes clear that the higher income communities, they're getting the same amount of money, even though their roads really don't even need it. Whereas the low income communities really should be giving
be getting a lot more given the lower quality roof.
PJ (39:17.046)
Um, yeah, and I love that in the last chapter you give us like a real life example and talk us through that. Can you, for our audience, talk through what this looks like in a real life scenario and the, uh, how it can look good and how it can be messed up?
Jeremy Weber (39:37.004)
Well, I imagine you're referring to the very last chapter, which is a case study in using statistics for policy decision. Real life case, the University of California, a number of years ago, was getting pressure to remove standardized test requirements from its application. So in other words, people who were applying,
PJ (39:40.754)
Yes. Yeah.
Jeremy Weber (40:05.08)
to the University of California system, they had been required to take an ACT or SAT test.
And the university wanted to reconsider that requirement.
And so the president of the university system put together asked for a task force to study. How are these standardized tests requirements working? Are they disproportionately excluding certain groups of students? Is our different testing requirements?
no testing requirements going to better advance the interest of the university and its purpose. That was the task. So then this task force did an excellent job of combing through the numbers thoughtfully, prepared this massive report. And I just draw out some highlights from the report, especially ones that illustrate principles.
that I talked about early in the book. And you know, this issue comes down to a few key empirical questions that they seek to answer using sensible statistic tools, well suited to the purpose. For example, do SAT or ACT scores actually predict college outcomes?
Jeremy Weber (41:49.94)
If we know your SAT score, for example, are we better able to predict what your freshman GPA will be at our university or whether or not you will graduate? All right, that's a big question because if there's no information content in SAT scores, like people who score well and people who score poorly are equally likely to do well at the university, then why are we...
requiring this test, especially when it's costly to take. And we know that certain groups, underrepresented minorities in particular, tend to have lower test scores, all right? So if it's not predictive, we could just get rid of it and lose nothing. Well, this task force does a very thorough job of answering that question. If we have these other factors like high school GPA,
and we're accounting for them. Does SAT or ACT scores add any predictive power? Is it correlated with first year GPA, graduation rates and so on? And it turns out it is. It's got a fair amount of predictive power. And so the task force in the report goes through a series of empirical questions like this. Is it predict, does it help us select applicants?
who are more likely to succeed at the university. They walk through basic questions like that and they come to the conclusion that we don't think the university should drop its testing requirement. It's not the primary or even the secondary reason why the University of California has a lower percentage of minorities than what graduate high school in the...
State of California, it's not the main cause of that. And it's helping among underrepresented students, it's helping us select the ones most likely to succeed here. All else constant. So the task force made a very clear recommendation. They recognized, look, there's a lot in this decision that statistics doesn't cleanly resolve. But based on what we have learned, based on...
Jeremy Weber (44:16.972)
values that are explicit by the university in the university zone documents, we think for now these tests make sense. That was a recommendation, very clear recommendation, flowed from thorough, clear, understandable analysis.
university decides to abandon the test anyway.
They were already being sued to drop the test. And I write that, you know, the task force shouldn't be now cynical or upset necessarily. They did what they were supposed to do. They furthered the understanding of the issues through clear, credible, transparent use of statistics.
They gave the university regents what they needed to understand a part of the issue. And then the regents, you know, who have a much bigger...
Jeremy Weber (45:22.284)
perspective, you know, they're thinking about a lot of things. They're the ones that are in the proper position to make the call. They're going to have to bear the consequences of whether that works out or not. Uh, and they made the call and it was different than what the recommendation was. But a point of my book is that statistics are rarely decisive for policy. They're often informative, sometimes irrelevant, but they're rarely, ah, just show me the number and there's no debate needed.
the number resolved all the issues. No, because policy debates are multifaceted. There is rarely a number or two that tells you so obvious what you should do. And so this is one of those cases. You can argue with their conclusion. That is, you could argue that the region sort of done something else. That was their call.
And one point of the book is that I'm writing to the policy aide who's trying to equip the decision maker. They are not the decision maker. They shouldn't confuse themselves with the decision maker. They're not going to get the heat if the decision turns out to be wrong or they're not going to get the fame if it turns out to be great. Their job is just to serve and to help the understanding of that decision maker. And if they did that well, they should be pleased.
what they're worth.
PJ (46:52.466)
And it seems like by recognizing, you know, that takes us to you talking about table school, right? Like there's that last piece where you're like, OK, how do we make this very clear to remove the cognitive work? Like you're doing the cognitive work for the policy decision maker. And how does representing magnitude?
affect that? And what are the dangers of, I see you mentioned that it's kind of controversial sometimes to like to represent magnitude, but even as we talk about things like, you know, one degree, like, oh, the worth, you know, what does that mean? Can you talk about the importance of magnitude and also of, yeah, you alluded to this earlier, but properly telling the story with data?
Jeremy Weber (47:32.164)
Right? Right.
Jeremy Weber (47:43.181)
Yeah.
Jeremy Weber (47:46.968)
One thing that if we've taken a statistics class, it's certainly a visual depiction of data class, we've probably absorbed the recommendation that you always want to start your vertical axes at zero. That is, if you're showing, let's say, some values over time, you want the vertical axes, the y-axis, to start at zero, as opposed to starting at, say,
80 and then any change from 83 to 84 suddenly looks like a big deal. That's one of the key ways that people can be manipulated by statistics is that you can make a small change look large. And I understand where that shaming of that sort of practice comes from, but I think it misses the mark.
Jeremy Weber (48:46.418)
Any number.
Jeremy Weber (48:50.096)
Even if it's really close to zero, it can be big, can be small, depending on the context. One degree might be nothing, it might be huge. You have to first think about the magnitude of, let's say we're observing something over time, and the basic question is, has it gone up? Has it gone up by a lot? You have to ask what a lot means.
Jeremy Weber (49:18.132)
A lot of people just don't even attempt to put an adjective next to their number or numbers. And if they do, they don't have good reasons for that adjective. It just seems small. Oh, it only went up by one degree. One seems small. And so you just put only next to the one degree. And my argument is, and my recommendation is, you just have to think about
whether or not that number is big or small and why. You've got to have reasons for your adjectives. And I give different principles for developing ways to develop reasons for considering a number to be small or large. And then let's say you determine, no, this is a big number. The one degree increase is a big deal.
then your figure should show it as a big deal. If you show a one degree, if you show, and this example is in the book, the 10 year moving average temperature at the Honolulu airport over decades, if you have the vertical axis at zero, the temperature looks like it hasn't changed since 1960 or whenever the beginning of that data series was.
It just looks like a flat line. If you start your axis at 75 or so, now it shows a clear rise in temperature, about a 1.5 degree rise in average temperature at the Honolulu airport. If that 1.5 corresponds to serious heat waves that kill lots of people and
massive wildfires and devastation of the islands, coffee plantations, then you should show 1.5 as a big jump. You should start your axes at 75 or 76. If you don't and you show a flat line, the audience, whoever sees that figure is going to conclude, Oh, there's been no change. Nothing of significance.
Jeremy Weber (51:45.4)
has happened with the temperature over the last 60 years. And that would be the wrong conclusion. Assuming you had determined that the 1.5 degrees was a big deal because it had a lot of real impacts on the island. So that's why you can't make a figure without thinking about, you can't make a good figure without thinking about magnitude.
And my recommendation is that figures are to be faithful to whatever your assessment of magnitude was. They show something that's small as small, and they show what's large as large.
PJ (52:32.882)
and making those reasons transparent. Right?
Jeremy Weber (52:35.729)
making those reasons and making those and giving. Yeah, exactly. It's not just, I declared it to be small. It's, it here's why you should think that this is a small number. And then you go through and give your reasons and they could, your audience might disagree. And if it's a contentious issue, somebody will vehemently disagree with your assessment of magnitude. But now we're having a discussion. Now it's in the,
PJ (52:56.447)
Right.
Jeremy Weber (53:03.36)
as opposed to just slipping it in there and the audience kind of be like, ah, I'm not, hmm, they said it was small, hmm, I guess it is. No, instead we're having a more robust, more transparent discussion of what the numbers mean. And that's a good thing.
PJ (53:21.566)
Dr. Weber, one, thank you so much for coming on today. Besides reading your book, which everyone should, I'm a little sad. I have it on Kindle, so I can't show the cover off. That doesn't have the same effect. But besides reading your book, what is one takeaway you would leave to our audience as they listen to this episode and they carry it with them this week?
Jeremy Weber (53:25.092)
Thank you.
Jeremy Weber (53:52.632)
The takeaway is...
Make peace with statistics, that is, summaries of observations. And you want to make peace with it because statistical claims are with us. They're like the poor. They're with us always, meaning we can't help but make statements about what is common or what is rare.
Oh, nobody does that. That's not a problem. Nobody's using their food stamps to buy beer. We always make these statements. Our belief vacuums, they fill with an understanding about what's generally true. So that's happening. It's better that those claims are tethered to actual observations, systematic observations, statistics in other words, because then
you're less likely, then they're likely to be more reliable than if it's just, I talked with some neighbors and now I'm talking with other people and making assertions about what's generally true, about the effectiveness of a vaccine or about political opinions or about whatever. But I have no statistics behind it. I've just had a conversation with a neighbor and maybe I watched a movie or saw an ad. So make peace with statistics.
Because they're with us always.
PJ (55:28.439)
Dr. Weber, it's been a pleasure. Thank you.
Jeremy Weber (55:30.968)
Thank you, PJ. I really appreciated the opportunity.