WEBVTT NOTE This file was generated by Descript 00:00:00.000 --> 00:00:03.520 Judith: Welcome to Berry's In the Interim podcast, where we explore the 00:00:03.520 --> 00:00:07.160 cutting edge of innovative clinical trial design for the pharmaceutical and 00:00:07.160 --> 00:00:09.810 medical industries, and so much more. 00:00:10.580 --> 00:00:11.300 Let's dive in. 00:00:13.389 --> 00:00:14.019 Scott Berry: All right. 00:00:14.019 --> 00:00:20.859 Welcome everybody to, in the Interim, this is a podcast of Berry Consultants 00:00:20.859 --> 00:00:28.089 on all things science of clinical trials, medical decision making, drug development, 00:00:28.089 --> 00:00:33.279 and we, we are statisticians typically talking about the science of this. 00:00:33.279 --> 00:00:34.464 Today we have a really. 00:00:34.694 --> 00:00:39.584 Cool topic and a fun one for me and I know a fun one for my guest. 00:00:39.884 --> 00:00:41.384 Uh, for the first time on. 00:00:41.384 --> 00:00:44.214 In the interim, my guest is Dr. 00:00:44.214 --> 00:00:48.014 Nicholas Berry, who also happens to be my son. 00:00:48.404 --> 00:00:55.154 And the topic for today is Lessons for drug Developers from the World of Sports. 00:00:56.049 --> 00:01:01.719 So we're gonna talk about what clinical trials drug developers could learn 00:01:01.719 --> 00:01:04.299 from examples in the world of sports. 00:01:04.299 --> 00:01:07.509 So Nick, welcome to in the Interim. 00:01:08.234 --> 00:01:08.474 Nick Berry: Thanks. 00:01:08.649 --> 00:01:10.034 Yeah, happy to be here. 00:01:11.139 --> 00:01:13.359 Um, yeah, I'm happy to talk about this. 00:01:13.359 --> 00:01:16.389 I'm, I'm, I'm glad to be talking about sports because, you know, 00:01:16.389 --> 00:01:19.629 a lot of our relationship when I was young was based on sports. 00:01:19.629 --> 00:01:23.709 You were my coach for, I dunno, 15 years playing baseball. 00:01:23.709 --> 00:01:28.659 And a lot of how I started to like perceive statistics was through 00:01:29.169 --> 00:01:33.099 sports, uh, through watching twins games, through watching baseball, 00:01:33.099 --> 00:01:37.569 through watching other sports, and sort of learning about all of the. 00:01:38.334 --> 00:01:41.964 The weird things that would happen and how a, a skeptical statistician 00:01:41.964 --> 00:01:46.344 like you perceives sort of home run races in the late nineties and 00:01:46.344 --> 00:01:48.474 batting titles and things like that. 00:01:48.474 --> 00:01:53.694 And so a lot of the way I learned to infer about statistics came from sports. 00:01:53.694 --> 00:01:56.814 So this is a sort of near and dear topic, and I, I wanted 00:01:56.814 --> 00:01:57.834 to be a sports statistician. 00:01:58.389 --> 00:02:02.559 For, for a long time, um, sort of when you were at Texas a and m, you wrote 00:02:02.559 --> 00:02:07.029 this column, um, the statistician reads the sports pages in chants. 00:02:07.029 --> 00:02:09.849 And so I would read those, I would look at those and sort of, uh. 00:02:10.584 --> 00:02:15.084 Um, it, it got me interested in that and I, I tr I sort of searched 00:02:15.084 --> 00:02:16.284 out a sports path too, right? 00:02:16.284 --> 00:02:18.474 I, I applied for some internships in sports. 00:02:18.474 --> 00:02:22.434 I worked with Hal Stern who wrote a lot of sports stuff back in the day, and so 00:02:22.884 --> 00:02:27.054 I, I, I was on a statistics path for a while before I veered to clinical trials. 00:02:27.054 --> 00:02:30.204 So, uh, this is a perfect sort of merging of the two worlds for me. 00:02:30.864 --> 00:02:30.954 Scott Berry: Yep. 00:02:31.254 --> 00:02:31.494 Yep. 00:02:31.854 --> 00:02:32.544 Okay. 00:02:32.754 --> 00:02:36.474 So, and, and many of those experiences were my same experiences with 00:02:36.474 --> 00:02:40.554 my father as well, who's also a statistician and, and loves sports. 00:02:40.884 --> 00:02:43.554 Uh, and so I. 00:02:43.554 --> 00:02:45.699 We're, we're gonna talk about various concept in sports, and 00:02:45.699 --> 00:02:49.989 this is the first of, of, uh, the first topic we're gonna talk about. 00:02:50.259 --> 00:02:52.959 And we have multiple other topics that I think are really 00:02:52.959 --> 00:02:55.899 valuable, uh, for drug developers. 00:02:55.899 --> 00:02:59.919 And I think the examples will, will bring home some of the concepts. 00:02:59.919 --> 00:03:02.769 And so that's what we're also gonna try to make clear here. 00:03:03.129 --> 00:03:06.579 Uh, we're not gonna get too deep in the sports, not too deep 00:03:06.579 --> 00:03:08.919 in the drug development, but make sure we tie 'em together. 00:03:09.549 --> 00:03:09.819 So 00:03:10.079 --> 00:03:10.299 Nick Berry: So. 00:03:10.359 --> 00:03:12.459 Scott Berry: topic is regression to the me. 00:03:13.089 --> 00:03:17.649 And it was talking about the, the family love of sports and the family, 00:03:17.649 --> 00:03:19.779 of course the family love of statistics. 00:03:20.259 --> 00:03:26.049 If, if somebody in our family brings up an interesting thing that 00:03:26.049 --> 00:03:30.699 happened in the world and says to my mother, your grandmother, gee, what, 00:03:30.699 --> 00:03:31.929 what, what do you think that is? 00:03:31.929 --> 00:03:37.299 She will 90% just say regression to the mean, not necessarily knowing 00:03:37.299 --> 00:03:40.389 what it is, but knows that that's the answer to most of the questions 00:03:40.389 --> 00:03:41.499 are it's regression to the mean. 00:03:42.419 --> 00:03:44.759 So, so what is regression to the mean? 00:03:44.789 --> 00:03:48.389 So we'll talk about it within sports, uh, and then we'll talk about how 00:03:48.389 --> 00:03:53.639 it, it, it, uh, what, what it means in the world of clinical trials 00:03:53.639 --> 00:03:55.529 and statistics and, and science. 00:03:56.429 --> 00:04:00.239 So, my first experience in regression to the mean. 00:04:00.984 --> 00:04:06.654 That I remember, um, it was the 1977 baseball season. 00:04:06.804 --> 00:04:08.154 I was 10 years old. 00:04:08.514 --> 00:04:10.854 I'm a little bit older than Nick, not surprisingly. 00:04:10.884 --> 00:04:18.594 Um, and in 1977 being 10, I loved baseball and I could read the box scores and I 00:04:18.594 --> 00:04:25.524 understood how to extrapolate statistics at one point in the season and say, what 00:04:25.524 --> 00:04:27.204 is going to be the end of the season? 00:04:27.984 --> 00:04:28.674 Within that. 00:04:28.824 --> 00:04:31.524 And the first of those was home runs. 00:04:32.364 --> 00:04:35.304 George Foster was on my favorite team at the time. 00:04:35.304 --> 00:04:38.514 The Cincinnati Reds, the, the Minnesota Twins were terrible 00:04:38.514 --> 00:04:39.294 at the time, by the way. 00:04:39.294 --> 00:04:43.554 So, uh, the big red machine where, where the Cincinnati Reds and 00:04:43.554 --> 00:04:48.834 George Foster, uh, halfway through the season had 31 home runs. 00:04:50.034 --> 00:04:55.014 And he was leading the league in home runs and he had 31 home runs, and I 00:04:55.014 --> 00:04:59.964 was sophisticated enough to double that and say, okay, that's a pace of 00:04:59.964 --> 00:05:02.634 62 home runs by the end of this season. 00:05:03.729 --> 00:05:07.239 At the time, the record was 61. 00:05:07.659 --> 00:05:09.999 The famous story of Roger Mars's. 00:05:09.999 --> 00:05:12.339 61 home runs he beat Babe Bruce. 00:05:12.369 --> 00:05:14.349 Babe Ruth's record of 60 home runs. 00:05:14.349 --> 00:05:18.819 But that's the best that had ever been done in a season before that. 00:05:18.819 --> 00:05:24.009 And George Foster was on pace to break that, and I knew that halfway 00:05:24.009 --> 00:05:25.809 through the season, players have. 00:05:26.214 --> 00:05:29.184 Half of their, their, their expected number. 00:05:29.184 --> 00:05:31.794 And so was on pace to break it. 00:05:31.794 --> 00:05:34.014 And as a 10-year-old, I thought he was gonna break it. 00:05:34.674 --> 00:05:38.874 And I even mentioned this to my father, who's a statistician, said 00:05:38.874 --> 00:05:40.644 he's gonna break Mars's record. 00:05:40.824 --> 00:05:45.324 And we made a wager based on that where bet, whether or 00:05:45.324 --> 00:05:46.704 not he would break the record. 00:05:47.064 --> 00:05:53.604 And I thought at the time my dad was nuts, that he would, not only would he make 00:05:53.604 --> 00:05:58.764 that bet, but he gave me, um, 54 or more. 00:05:59.304 --> 00:06:03.624 He said, I'll bet he doesn't even hit 54 home runs by the end of the season. 00:06:04.524 --> 00:06:08.094 at that point, you know, does he hit another 23 home runs when he 00:06:08.094 --> 00:06:10.194 hit 31 halfway through the season? 00:06:10.194 --> 00:06:13.644 That seemed like a this, this was a sucker bet for my father 00:06:13.644 --> 00:06:14.844 that of course I'd make that 00:06:14.949 --> 00:06:15.239 Nick Berry: Yeah. 00:06:16.554 --> 00:06:19.554 Scott Berry: And of course, he didn't hit 54. 00:06:19.554 --> 00:06:20.814 He didn't hit 62. 00:06:21.084 --> 00:06:22.254 He hit 52. 00:06:23.229 --> 00:06:25.089 is actually very good number. 00:06:25.449 --> 00:06:29.769 Um, from his 31, he hit 20 more home runs, which is on pace. 00:06:30.189 --> 00:06:35.439 That 21 is a 42 home run hitter, which is an incredibly good home run hitter. 00:06:35.439 --> 00:06:35.769 So 00:06:35.929 --> 00:06:36.129 Nick Berry: Sure. 00:06:36.279 --> 00:06:39.549 Scott Berry: did really, really well, better than than average, 00:06:39.999 --> 00:06:43.629 but not the 31 pace in that. 00:06:44.004 --> 00:06:47.364 Okay, and this, this happens baseball season. 00:06:47.364 --> 00:06:48.714 After baseball season. 00:06:49.194 --> 00:06:56.244 A really fun thing about that season was my Minnesota twins, rod Caru, was 00:06:56.244 --> 00:07:01.944 flirting with batting 400, a batting average of 400, which is the number 00:07:01.944 --> 00:07:06.624 of hits divided by the number of attempts, and that's easy to project. 00:07:06.819 --> 00:07:11.919 It's the batting average and at, that point in the season, 00:07:11.919 --> 00:07:13.599 he was hitting over 500. 00:07:13.599 --> 00:07:18.459 And so we might, in this 1977 season have a 400 hitter and break the 00:07:18.459 --> 00:07:20.319 home run record all in one season. 00:07:20.409 --> 00:07:21.909 On my two favorite teams at 00:07:22.734 --> 00:07:24.114 Nick Berry: He was hitting over 400, 00:07:24.684 --> 00:07:25.074 right? 00:07:25.164 --> 00:07:27.439 And yeah, which is an astronomical number. 00:07:27.469 --> 00:07:31.319 Like this has happened less than, you know, a handful of 00:07:31.319 --> 00:07:32.479 times in history of the game. 00:07:33.124 --> 00:07:36.669 Scott Berry: The, the last time before 1977 that it had happened, 00:07:36.669 --> 00:07:40.899 which is still the last time it has not happened even since 1977, was, 00:07:41.199 --> 00:07:45.009 uh, Ted Williams hit 4 0 6 in 1941. 00:07:46.214 --> 00:07:50.564 so this has become a, a, a, a number that baseball fans know 00:07:50.564 --> 00:07:52.004 and know this hasn't happened. 00:07:52.004 --> 00:07:53.744 And so that's a very rare thing. 00:07:53.954 --> 00:07:57.914 And he had the best batting average in the league at the time, and I 00:07:57.914 --> 00:08:02.924 didn't make a wager on this, but he ended up hitting 3 88 actually, is 00:08:02.924 --> 00:08:05.444 one of the highest numbers since 1941. 00:08:05.444 --> 00:08:08.564 An incredibly high number rod crew, a hall of fame. 00:08:08.754 --> 00:08:11.964 Baseball player, uh, because of his batting average. 00:08:11.964 --> 00:08:13.194 Incredibly high number. 00:08:13.194 --> 00:08:16.224 And we've had several people since then that have done similar 00:08:16.524 --> 00:08:19.014 flirting with that 400 number. 00:08:19.014 --> 00:08:20.304 George Brett did. 00:08:20.634 --> 00:08:26.004 Uh, actually Lenny Dykstra did Tony Gwynn, uh, even Joe Mauer flirted 00:08:26.004 --> 00:08:29.119 with it in another Minnesota twins, and nobody has accomplished that. 00:08:30.354 --> 00:08:35.274 So this is something we're very familiar with in sports, uh, uh, of 00:08:35.274 --> 00:08:37.494 this kind of phenomenon happening. 00:08:39.234 --> 00:08:39.804 So 00:08:39.884 --> 00:08:40.104 Nick Berry: So 00:08:40.534 --> 00:08:41.344 what does it mean? 00:08:41.484 --> 00:08:46.824 Scott Berry: Uh, in, in that scenario were, was, was, was 00:08:46.824 --> 00:08:49.794 Rod Caru, truly a 400 hitter? 00:08:50.964 --> 00:08:56.814 was George Foster truly a 62 home run hitter at that point? 00:08:57.879 --> 00:08:58.149 Nick Berry: Yeah. 00:08:59.679 --> 00:09:00.969 Scott Berry: Uh, and they weren't. 00:09:02.229 --> 00:09:05.739 within that scenario, and they were the extremes. 00:09:05.829 --> 00:09:10.959 And so if we were to say how good are they really, we would never estimate them. 00:09:10.959 --> 00:09:15.489 And my father knew that George Foster was not a 31 home run hitter, which is 00:09:15.489 --> 00:09:17.379 why he was very comfortable betting. 00:09:17.439 --> 00:09:19.869 He knew he probably wasn't even a. 00:09:19.884 --> 00:09:24.174 21 home run or a 23 home run hitter, which is why he took that 00:09:24.174 --> 00:09:25.974 bet that that's an extreme number. 00:09:26.394 --> 00:09:29.304 His, his estimate was regressed. 00:09:29.334 --> 00:09:33.234 Now, George Foster had previous performances, but also regressed 00:09:33.234 --> 00:09:35.934 towards the middle, uh, in that setting. 00:09:36.579 --> 00:09:39.219 Nick Berry: Yeah, I was gonna say, I'm sure that Don, your 00:09:39.219 --> 00:09:41.379 dad looked up, George Foster's. 00:09:42.084 --> 00:09:46.524 Previous three years and saw you hit 35 home runs to 40 home runs every 00:09:46.524 --> 00:09:48.534 year, and, and, and made a deduction. 00:09:48.984 --> 00:09:49.194 Scott Berry: Yep. 00:09:49.824 --> 00:09:55.164 So we'll come back to sort of how we might estimate, um, what they really are. 00:09:55.164 --> 00:09:57.714 But this phenomenon happens in every sport. 00:09:58.164 --> 00:10:01.554 It, it happens in hockey, it happens in baseball. 00:10:01.554 --> 00:10:03.054 It happens in basketball. 00:10:03.054 --> 00:10:06.054 We have, uh, people in teams. 00:10:06.259 --> 00:10:09.949 We have teams halfway through a season that are on pace to break the, the 00:10:09.949 --> 00:10:11.899 record number of wins in basketball. 00:10:11.899 --> 00:10:16.219 We've had some, some do that, uh, and they fall short of that. 00:10:16.489 --> 00:10:18.199 Uh, and we always talk about that. 00:10:18.259 --> 00:10:20.359 We've actually recently had teams break. 00:10:20.659 --> 00:10:24.019 Uh, and we might have a team, uh, this season, break the record 00:10:24.019 --> 00:10:25.519 for most losses in a season. 00:10:25.759 --> 00:10:29.989 You know, these extreme sort of things, uh, that, that we've talked about. 00:10:30.229 --> 00:10:31.699 And then it doesn't happen. 00:10:32.189 --> 00:10:36.269 And in the sports world, we hear reasons why that happens. 00:10:36.689 --> 00:10:37.439 We hear 00:10:37.464 --> 00:10:37.754 Nick Berry: Yeah. 00:10:38.404 --> 00:10:43.649 Scott Berry: the, the, the scrutiny of the media, uh, everybody asking 00:10:43.649 --> 00:10:45.539 about it day in and day, day out. 00:10:45.539 --> 00:10:48.359 It's really hard to continue such a pace. 00:10:48.819 --> 00:10:51.969 Nick Berry: Yeah, they got in their heads and yeah, the whole, yeah. 00:10:52.254 --> 00:10:52.524 Scott Berry: Yep. 00:10:53.334 --> 00:10:57.534 let me pro, let me provide another example of this in a, in a sport, Nick 00:10:57.534 --> 00:11:00.804 and I both, uh, love, uh, which is golf. 00:11:00.864 --> 00:11:04.614 Uh, we play golf, uh, and if you're looking on video, we have golf shirts on. 00:11:05.069 --> 00:11:08.484 Uh, so, within this we love to play golf. 00:11:08.484 --> 00:11:10.584 So let me provide another example. 00:11:10.584 --> 00:11:14.724 Moneymaking gig, uh, for us potentially is. 00:11:15.684 --> 00:11:22.134 a golf tournament, and I'm gonna quote numbers from the 2017 US Open, 00:11:22.134 --> 00:11:24.324 but this happens week in, week out. 00:11:24.384 --> 00:11:27.654 Uh, very, very similar numbers within it. 00:11:27.654 --> 00:11:30.504 The US Open might be a little bit more extreme because there's more 00:11:30.504 --> 00:11:34.134 of a variety of players in the US Open than, than the run of the 00:11:34.134 --> 00:11:36.384 mill Weekly PGGA tour tournament. 00:11:37.134 --> 00:11:40.734 So in the 2017 US Open. 00:11:41.004 --> 00:11:45.864 If I take the baseline score, and I'm calling it baseline, but we're gonna 00:11:45.864 --> 00:11:47.064 start talking about clinical trials. 00:11:47.064 --> 00:11:54.864 But the first day, the golfers shot on their first day in the US Open, and I'm 00:11:54.864 --> 00:11:59.304 gonna break them up into four uh groups. 00:11:59.724 --> 00:12:07.254 Uh, four Quantiles, the top 25% of golfers, second 25% 00:12:07.254 --> 00:12:08.904 of what they shot on day one. 00:12:10.119 --> 00:12:15.609 The third and the fourth, the bottom in terms of the worst score within that. 00:12:17.079 --> 00:12:17.829 So I'm gonna 00:12:18.024 --> 00:12:19.039 Nick Berry: Gonna take those. 00:12:19.359 --> 00:12:23.529 Scott Berry: and let's talk about the worst quantile, uh, uh, 00:12:23.529 --> 00:12:26.019 at first, uh, quartile, sorry. 00:12:26.019 --> 00:12:29.619 The worst quartile of this, the 25% that shot the worst score, 00:12:29.619 --> 00:12:32.319 they shot an average of 78. 00:12:33.354 --> 00:12:37.824 On day one, US opens traditionally very, very hard, and this 00:12:37.824 --> 00:12:39.174 round was a hard round. 00:12:39.474 --> 00:12:41.004 The average was 78. 00:12:41.934 --> 00:12:45.354 I'm going to do an intervention on that group. 00:12:46.374 --> 00:12:50.184 gonna go around to each one of them and I'm gonna think really good 00:12:50.184 --> 00:12:55.434 thoughts about them, and I'm gonna give them encouragement in that. 00:12:56.439 --> 00:13:00.669 And so what happened to them on day two when I intervened 00:13:00.669 --> 00:13:05.739 on them is they averaged 3.04 00:13:06.189 --> 00:13:09.519 shots better on day two than they did day one. 00:13:10.719 --> 00:13:16.239 They, their change from baseline from day one to day two was three shots better. 00:13:17.649 --> 00:13:18.579 I did that. 00:13:19.929 --> 00:13:23.589 I caused them to shoot three shots better. 00:13:25.104 --> 00:13:30.174 And is does three shots matter in a PGA tour tournament? 00:13:30.204 --> 00:13:33.624 Three shots around is multimillions of 00:13:33.954 --> 00:13:34.244 Nick Berry: Yeah. 00:13:34.464 --> 00:13:39.744 Scott Berry: It's the difference between, uh, uh, being on, uh, 00:13:39.804 --> 00:13:44.874 uh, uh, uh, one tour, another tour winning major golf tournaments. 00:13:45.084 --> 00:13:46.224 It's millions of dollars. 00:13:46.224 --> 00:13:47.694 It's an enormous number 00:13:48.339 --> 00:13:48.519 Nick Berry: Yeah. 00:13:48.644 --> 00:13:49.134 Scott Berry: in that. 00:13:49.494 --> 00:13:52.434 And think about a four round golf tournament that's 12 shots. 00:13:52.434 --> 00:13:59.349 That if I could bottle that and have a three shot effect, I'm 00:13:59.349 --> 00:14:02.409 making millions, uh, in this. 00:14:03.279 --> 00:14:03.759 Okay? 00:14:03.969 --> 00:14:09.279 Likewise, the people that shot the golfers that shot in the top the 00:14:09.279 --> 00:14:17.169 best, uh, quartile, they got three shots worse on day two on average. 00:14:17.949 --> 00:14:19.989 I didn't think good shots of them. 00:14:19.989 --> 00:14:21.699 And in fact, I thought, I thought, I thought. 00:14:22.164 --> 00:14:27.834 Bad thoughts about them, and they do three shots worse on day two. 00:14:29.184 --> 00:14:30.414 This is repeatable. 00:14:30.414 --> 00:14:33.144 This happens every single week in the PGA tour. 00:14:33.954 --> 00:14:37.434 happens all the time, uh, in this circumstance. 00:14:38.099 --> 00:14:42.119 Um, so what does it mean? 00:14:43.899 --> 00:14:49.194 did, did, did my thinking about them affect them in any way, shape or form? 00:14:50.604 --> 00:14:50.934 No. 00:14:51.759 --> 00:14:52.049 Nick Berry: Yeah. 00:14:52.404 --> 00:14:57.714 Scott Berry: not, um, in that, that doesn't mean other things are, 00:14:59.874 --> 00:15:02.754 not understood whether they had an effect, 00:15:03.279 --> 00:15:03.429 Nick Berry: Yeah. 00:15:03.429 --> 00:15:09.639 The bottom quanti core quartile of people didn't feel no pressure and go straight at 00:15:09.639 --> 00:15:11.859 pins because they had a tough first day. 00:15:11.919 --> 00:15:16.034 It wasn't some conscious, concerted effort by the players to to, 00:15:16.039 --> 00:15:18.069 to do better play freer, right? 00:15:18.069 --> 00:15:20.799 This is variability, it's randomness that led to it. 00:15:21.334 --> 00:15:23.434 Scott Berry: So, uh, announcers will tell you that, 00:15:23.769 --> 00:15:24.059 Nick Berry: Yeah. 00:15:24.274 --> 00:15:28.414 Scott Berry: I, I had a bad day and I just, I was so super aggressive and I 00:15:28.414 --> 00:15:30.094 wasn't worried about my positioning. 00:15:30.094 --> 00:15:33.574 And you shoot much better when you do that, uh, or vice versa. 00:15:33.574 --> 00:15:34.684 Now you're near the lead. 00:15:34.684 --> 00:15:36.154 You didn't sleep well. 00:15:36.394 --> 00:15:40.054 Uh, you kept thinking about hosting the trophy and, and, and 00:15:40.054 --> 00:15:41.944 you shot worse with it, with it. 00:15:41.944 --> 00:15:44.944 So people attach reasons to that. 00:15:44.944 --> 00:15:47.374 Something that is nothing other than randomness. 00:15:47.464 --> 00:15:48.064 And this is 00:15:48.084 --> 00:15:48.144 Nick Berry: Yeah. 00:15:48.454 --> 00:15:49.234 Scott Berry: other than randomness. 00:15:49.854 --> 00:15:52.224 Nick Berry: And this isn't just like casual fans. 00:15:52.224 --> 00:15:56.364 I mean, the players themselves say, this announcers that played for 15 years say 00:15:56.364 --> 00:16:00.714 this, and this is a, this is a, you know, people who have spent their life playing 00:16:00.714 --> 00:16:07.614 the game still contribute a lot of, you know, variability to physical aspects 00:16:07.614 --> 00:16:10.734 rather than the fact that there is just a lot of variability in database scores. 00:16:11.734 --> 00:16:14.624 Scott Berry: So what is, we have this term regression to-the-mean 00:16:14.644 --> 00:16:18.814 what do we mean by regression-to the-mean Uh, within a setting like 00:16:18.814 --> 00:16:25.024 this, the, there's the average score on day one, which I think was about 00:16:25.024 --> 00:16:27.334 73 and a half, something like that. 00:16:27.664 --> 00:16:31.774 Uh, and we have the average of those golfers, all these golfers are 00:16:31.774 --> 00:16:37.444 participants, and the average is that 73 and a half and somebody shoots 78. 00:16:38.424 --> 00:16:45.834 On day one, we, and let's ignore previous tournaments that they come into the US 00:16:45.834 --> 00:16:47.904 Open, we're interested in, in, in that. 00:16:48.504 --> 00:16:55.584 Um, you want, if you estimate that that individual's true average score 00:16:55.584 --> 00:17:01.914 on that golf course is 78, you are shocked when you find out the average 00:17:01.914 --> 00:17:05.814 of the people in that quartile got better by three shots on day two. 00:17:06.309 --> 00:17:08.319 And there must be a reason for that. 00:17:10.509 --> 00:17:14.469 You could think everybody is a 73.5 00:17:15.669 --> 00:17:20.079 and all golfers are identical, and there's no difference between them. 00:17:20.199 --> 00:17:25.209 You'd estimate those golfers to shoot 73 and shoot five shots better on day two. 00:17:26.469 --> 00:17:28.869 Uh, not all golfers are all the same. 00:17:29.949 --> 00:17:30.669 they're different. 00:17:30.699 --> 00:17:32.439 Some golfers are better than the other ones. 00:17:32.439 --> 00:17:35.619 We know that, and especially the US open, there's more 00:17:35.619 --> 00:17:37.179 heterogeneity in those players. 00:17:37.209 --> 00:17:39.249 There's variability across players. 00:17:40.059 --> 00:17:44.049 There's variability on a day score a setting. 00:17:44.049 --> 00:17:47.109 So what does it mean if somebody shoots 78? 00:17:47.169 --> 00:17:48.999 They, they, they, they've shot a high number. 00:17:50.529 --> 00:17:54.099 They're probably not as good as the average, but they're 00:17:54.099 --> 00:17:56.829 not as extreme as the 78. 00:17:57.879 --> 00:18:03.429 If we were estimating them statistically, we would take an average of their 78, 00:18:04.179 --> 00:18:09.819 their, their, their score, and the average of the average of all the golfers. 00:18:10.149 --> 00:18:15.759 If this is all we knew, trying to estimate them, which is 73, and we would 00:18:15.759 --> 00:18:18.399 do some estimate midway between that. 00:18:18.669 --> 00:18:22.719 Nick Berry: Yeah, and how far you go from one to the other 00:18:22.719 --> 00:18:24.939 depends how well you think. 00:18:24.939 --> 00:18:27.879 You actually know how good golfers are and how. 00:18:28.269 --> 00:18:31.929 Maybe if a, if a golf round was a thousand holes, you think you know 00:18:31.929 --> 00:18:34.719 more about the players, so you know the amount of data you have and the amount 00:18:34.719 --> 00:18:38.199 of belief you have and how good the players are goes into this weighting 00:18:38.199 --> 00:18:39.369 of where you are between the two. 00:18:40.024 --> 00:18:40.264 Scott Berry: Right. 00:18:40.324 --> 00:18:44.644 So in, in only using the 2017 US Open, 00:18:45.369 --> 00:18:45.639 Nick Berry: Yeah. 00:18:45.854 --> 00:18:48.994 Scott Berry: The key things about how much regression you would do 00:18:48.994 --> 00:18:54.964 towards the mean is the between subject variability, how variable are 00:18:54.964 --> 00:19:01.354 golfers, and then the within subject variability in a, in an 18 hole score. 00:19:02.229 --> 00:19:05.319 And by the way, spent some time in this. 00:19:05.319 --> 00:19:09.159 It's, it's a little less than three, is a standard deviation of within a 00:19:09.159 --> 00:19:10.809 professional golfer on any one day. 00:19:11.439 --> 00:19:17.019 Uh, within that, the between variability in this UO US Open 00:19:17.019 --> 00:19:18.189 is quite a bit bigger than that. 00:19:18.489 --> 00:19:21.789 Um, within the setting, this, the, the, the, the bottom qua 00:19:21.849 --> 00:19:25.929 Quantile was 78, the top was 69. 00:19:25.929 --> 00:19:29.889 There's probably a five or six standard deviation across golfers. 00:19:30.189 --> 00:19:38.469 So we would use those, uh, in that setting now to provide that estimate of that. 00:19:38.469 --> 00:19:42.819 So we're not at all surprised when the next day scores. 00:19:43.149 --> 00:19:49.149 For all of them shrink in towards the mean it's, it's inevitable because the first 00:19:49.149 --> 00:19:55.509 day score variability pushes them outward and their truth is somewhere in the middle 00:19:55.509 --> 00:19:57.609 of that variability always goes outward. 00:19:57.729 --> 00:19:59.199 That's what variability does. 00:19:59.709 --> 00:20:04.989 So we would always say when somebody does better on day two, they go from 78 00:20:04.989 --> 00:20:11.184 to 75, we say, oh, that's regression-to the-mean Our estimate, if I was betting 00:20:11.184 --> 00:20:15.894 on somebody that shot 78 on day one, I would probably estimate 75 on day two. 00:20:16.464 --> 00:20:19.614 And I would bet and I would behave that way, uh, in the 00:20:19.614 --> 00:20:21.144 particular setting like that. 00:20:21.354 --> 00:20:26.334 Meanwhile, somebody that shot 67, I would probably guess 70 or 71. 00:20:26.334 --> 00:20:32.484 It's a regression to the mean, statistically a common way we do this 00:20:32.514 --> 00:20:35.994 in Bayesian approach is something called the hierarchical modeling. 00:20:38.394 --> 00:20:44.484 And the hierarchies are, the players is a hierarchy. 00:20:44.784 --> 00:20:49.554 And within players, their different scores is a hierarchy and that's 00:20:49.554 --> 00:20:51.624 used to provide these estimates. 00:20:51.624 --> 00:20:54.834 And we do it in sports, all the time. 00:20:55.494 --> 00:20:57.414 We also do it in drug development. 00:20:59.454 --> 00:21:05.964 Now, the is, this is a natural phenomenon. 00:21:06.834 --> 00:21:08.454 this is a natural phenomenon. 00:21:08.454 --> 00:21:11.424 If you're rolling dice, this is a natural phenomenon. 00:21:11.454 --> 00:21:13.824 If you're flipping coins, if you're playing golf, if you're playing 00:21:13.824 --> 00:21:18.204 baseball, if teams are having outcomes, it's pretty well understood. 00:21:18.204 --> 00:21:23.244 In sports, it's sometimes understood in drug development, and this is the 00:21:23.244 --> 00:21:29.964 part that drives us nuts, is people attach to it that are not right, 00:21:30.834 --> 00:21:34.674 wrong, then they behave that way. 00:21:35.559 --> 00:21:43.299 To it, is, which creates bad decision making, which, uh, bad betting, bad 00:21:43.299 --> 00:21:48.399 decision making, bad paying of baseball players, expecting them to do the same. 00:21:48.609 --> 00:21:49.959 And there are some you may heard of. 00:21:49.959 --> 00:21:54.699 So, uh, I, uh, you know, there's something I don't even know if people talk about 00:21:54.699 --> 00:21:57.039 anymore, the Sports Illustrated Jinx. 00:21:58.404 --> 00:22:03.324 recognized it that players or people or teams got on the cover of Sports 00:22:03.324 --> 00:22:09.654 Illustrated, a weekly magazine, and what happened is they tended to do worse and 00:22:09.654 --> 00:22:12.264 people referred to it as a jinx of being 00:22:12.484 --> 00:22:12.774 Nick Berry: Yeah. 00:22:12.894 --> 00:22:13.404 Scott Berry: Illustrate, 00:22:13.854 --> 00:22:16.314 Nick Berry: Sports Illustrated doesn't exist anymore, but it's 00:22:16.314 --> 00:22:18.054 now the Madden Jinx or whatever. 00:22:18.054 --> 00:22:19.374 You know, the video game that comes out. 00:22:19.374 --> 00:22:21.804 You're on the cover of Madden, and then the next year you do worse. 00:22:21.804 --> 00:22:25.674 But it also exists crazy coincidence that there's also a Madden Jinx, 00:22:25.674 --> 00:22:26.904 like a Sports Illustrated jinx. 00:22:27.249 --> 00:22:29.979 Scott Berry: And people believe that's a real thing, 00:22:30.294 --> 00:22:30.624 Nick Berry: Yeah. 00:22:31.179 --> 00:22:32.229 Scott Berry: is a real thing. 00:22:32.619 --> 00:22:35.079 There's, you've heard the sophomore jinx. 00:22:35.319 --> 00:22:38.409 People have a great freshmore freshman season. 00:22:38.619 --> 00:22:41.829 They don't do quite as well in their sophomore year, and it's a jinx. 00:22:42.009 --> 00:22:42.969 Nick Berry: Yeah, sophomore slump. 00:22:43.239 --> 00:22:45.339 Scott Berry: you know, we don't hear this anymore because if you have 00:22:45.339 --> 00:22:46.869 a great freshman year, you go pro. 00:22:47.859 --> 00:22:48.579 Nick Berry: Oh, and yeah. 00:22:49.599 --> 00:22:51.309 Scott Berry: there's a rookie of the ear jinx. 00:22:51.894 --> 00:22:52.044 Nick Berry: Yep. 00:22:52.089 --> 00:22:55.809 Scott Berry: you go look at players who win rookie of the year in baseball, in 00:22:55.809 --> 00:22:58.659 any sport, they do worse the next year. 00:22:59.379 --> 00:23:00.729 They're complacent. 00:23:01.059 --> 00:23:02.319 It's a jinx. 00:23:02.529 --> 00:23:06.189 Uh, and and it's nothing but regression to the mean largely. 00:23:06.264 --> 00:23:08.184 Nick Berry: overperformed in their first year. 00:23:08.619 --> 00:23:08.919 Scott Berry: yeah. 00:23:09.129 --> 00:23:09.339 Yeah. 00:23:09.339 --> 00:23:12.729 And if you believe they're gonna repeat that performance, 00:23:13.509 --> 00:23:15.759 that's the mistake you made. 00:23:16.569 --> 00:23:20.529 then you think, okay, we have to attach a, a, a, an effect to this, uh, 00:23:20.699 --> 00:23:20.859 Nick Berry: Yeah. 00:23:21.324 --> 00:23:26.634 Scott Berry: Um, media pressure scrutiny, uh, team chemistry. 00:23:26.844 --> 00:23:29.214 Uh, the team chemistry that great was seasoned. 00:23:29.214 --> 00:23:31.524 The next, the next year was the chemistry. 00:23:31.524 --> 00:23:32.334 Wasn't very good 00:23:33.084 --> 00:23:33.144 Nick Berry: Yeah. 00:23:33.144 --> 00:23:34.854 Scott Berry: means we, we don't know how to explain it. 00:23:34.854 --> 00:23:36.534 So we, we invent chemistry for it. 00:23:37.254 --> 00:23:40.194 This, now this happens in all kinds of things. 00:23:40.374 --> 00:23:44.814 Golf equipment, who goes, you know, you try a new putter when 00:23:44.814 --> 00:23:47.814 you're not playing well, a new driver and you're doing better. 00:23:48.354 --> 00:23:48.624 Nick Berry: Yeah. 00:23:49.149 --> 00:23:52.029 Scott Berry: Golf teachers benefit from this greatly. 00:23:52.059 --> 00:23:53.949 Who goes to see a golf professional? 00:23:53.949 --> 00:23:54.309 Somebody 00:23:54.384 --> 00:23:54.714 Nick Berry: Yeah. 00:23:55.479 --> 00:23:57.369 Scott Berry: I've, I, I'm on that left side. 00:23:57.369 --> 00:24:01.539 I'm I, you know, shooting the higher scores and I play better, and I 00:24:01.539 --> 00:24:04.899 want to pay more money to that golf professional in that setting. 00:24:05.259 --> 00:24:08.619 Now, lots of people benefit from this. 00:24:08.829 --> 00:24:11.049 Chiropractors benefit, your back hurts. 00:24:11.049 --> 00:24:15.309 You go in, doctors benefit, I'm not feeling well, and then I feel better. 00:24:15.759 --> 00:24:20.799 In the setting, and we attribute it to that rehab therapists. 00:24:20.979 --> 00:24:27.519 We, we, we use trinkets, we put magnetic bracelets on our, on our arms, uh, to 00:24:27.519 --> 00:24:29.889 make pain go away and it goes away. 00:24:29.889 --> 00:24:31.089 And you know that, this sort of thing. 00:24:31.089 --> 00:24:35.229 So it's all kinds of, this, this shows up in every walk of life. 00:24:36.109 --> 00:24:36.309 Nick Berry: Yeah, 00:24:36.339 --> 00:24:37.929 Scott Berry: let's the, so let's 00:24:37.954 --> 00:24:39.094 Nick Berry: you've been so sorry. 00:24:39.159 --> 00:24:39.399 Scott Berry: trials. 00:24:39.399 --> 00:24:39.699 Yeah. 00:24:40.239 --> 00:24:41.709 Nick Berry: You've been really pessimistic. 00:24:41.709 --> 00:24:45.309 Like everything is fake. 00:24:45.339 --> 00:24:46.569 That's not necessarily what you're saying. 00:24:46.569 --> 00:24:50.559 You're just saying that every time you observe something extraordinary, 00:24:50.589 --> 00:24:56.439 especially in the circumstance where the population that underwent this procedure 00:24:56.439 --> 00:24:59.244 or something like that was different than the norm population, but But you're 00:24:59.244 --> 00:25:01.084 just saying shrink back the results. 00:25:01.084 --> 00:25:01.764 You're not saying that. 00:25:02.259 --> 00:25:04.179 It's fake golf teachers don't work. 00:25:04.209 --> 00:25:07.929 It's just that if you see immediate benefits after you go see a golf 00:25:07.929 --> 00:25:11.409 teacher, some of that benefit is due to regression, to the mean. 00:25:11.859 --> 00:25:15.459 There are benefits of some of these things, but believing at face 00:25:15.459 --> 00:25:19.839 value, the change you see right away is naive when we know that 00:25:19.839 --> 00:25:21.099 there's going to be some regression. 00:25:21.249 --> 00:25:21.429 Yeah, 00:25:21.834 --> 00:25:22.824 Scott Berry: Yeah, exactly. 00:25:22.824 --> 00:25:26.724 And, and, um, doctors are phenomenally important. 00:25:26.724 --> 00:25:27.624 I'm not saying don't go 00:25:27.699 --> 00:25:28.659 Nick Berry: yeah, I know exactly. 00:25:28.659 --> 00:25:29.289 Yeah, exactly. 00:25:29.784 --> 00:25:34.884 Scott Berry: Uh, your uncle and my brother is a golf professional, so, uh, they, 00:25:34.884 --> 00:25:38.514 they, they absolutely play a role and they will help you shoot better scores. 00:25:38.514 --> 00:25:38.724 No 00:25:38.949 --> 00:25:39.189 Nick Berry: Yeah, 00:25:39.969 --> 00:25:40.119 but 00:25:40.119 --> 00:25:42.489 it's hard to figure out how much of it goes to each thing. 00:25:42.489 --> 00:25:46.064 Like it's really hard to, to assign how much it should shrink. 00:25:47.064 --> 00:25:49.314 Scott Berry: Okay, so let's talk about this in clinical 00:25:49.314 --> 00:25:50.754 trials and drug development 00:25:50.829 --> 00:25:50.889 Nick Berry: Yeah. 00:25:51.054 --> 00:25:53.484 Scott Berry: and now you can start to imagine some of these 00:25:53.484 --> 00:25:55.464 effects showing up within it. 00:25:55.809 --> 00:26:01.464 A, a very, one that I, I think is very misunderstood is something 00:26:01.464 --> 00:26:03.174 called the placebo effect. 00:26:04.104 --> 00:26:08.634 And people talk about clinical trials that in a clinical trial, and when people 00:26:08.634 --> 00:26:14.544 say the placebo effect, what they mean is somebody takes a treatment that is inert. 00:26:15.564 --> 00:26:19.914 But they, they're taking a treatment and the, the act of taking 00:26:19.914 --> 00:26:21.774 that makes them perform better. 00:26:23.754 --> 00:26:26.214 And you can see where I'm going with this. 00:26:26.664 --> 00:26:32.334 I was labeling the, the, my thinking about golfers in that situation. 00:26:33.069 --> 00:26:36.969 As kind of a placebo and the golfer doesn't know I'm thinking about them 00:26:37.199 --> 00:26:37.489 Nick Berry: Yeah. 00:26:38.169 --> 00:26:38.949 Scott Berry: and they do better. 00:26:39.129 --> 00:26:41.079 So let's take a clinical trial. 00:26:41.079 --> 00:26:45.699 And what happens in uh, in most clinical trials is people who 00:26:45.699 --> 00:26:52.329 enter a clinical trial like that golfer who shot 78, they're doing 00:26:52.329 --> 00:26:55.209 poorly, relatively on their scale. 00:26:55.569 --> 00:26:57.039 And it could be. 00:26:57.504 --> 00:27:02.784 It could be high blood pressure, it could be high weight, it could be pain. 00:27:03.174 --> 00:27:04.854 I'm not sleeping well. 00:27:05.034 --> 00:27:12.204 Um, in that scenario, and this happens in Alzheimer's trials, that people, that 00:27:12.204 --> 00:27:14.514 their memory has been particularly bad. 00:27:14.724 --> 00:27:16.824 I'm, I'm, I'm, I'm doing worse. 00:27:16.884 --> 00:27:24.984 They enter in trials and in many of those scenarios, patients who get no treatment. 00:27:26.784 --> 00:27:32.844 Or an inert treatment, they do better, and everybody calls this a placebo effect. 00:27:34.194 --> 00:27:41.124 Now, in many circumstances, that's the exact same thing we 00:27:41.249 --> 00:27:41.469 Nick Berry: Yeah. 00:27:41.634 --> 00:27:43.134 Scott Berry: It's a regression to the mean. 00:27:44.124 --> 00:27:49.459 Now, there are certainly some cases where I believe that the act of 00:27:49.459 --> 00:27:51.594 taking an intervention may help. 00:27:52.479 --> 00:27:57.729 you could imagine sleep, you could imagine potentially pain trials is 00:27:57.729 --> 00:28:00.399 where they've talked about it and there's actually been trials where they 00:28:00.399 --> 00:28:04.479 give placebo or they give nothing, and there's a difference between those. 00:28:04.929 --> 00:28:11.604 But most clinical trials, this is what happens And yeah. 00:28:12.204 --> 00:28:12.924 Nick Berry: Placebo effects. 00:28:12.924 --> 00:28:13.314 Interesting. 00:28:13.314 --> 00:28:17.904 It's like you take statistics in high school and you don't know anything 00:28:17.904 --> 00:28:20.784 about statistics, but they're teaching you about placebo effects and it's 00:28:20.784 --> 00:28:27.444 like a immediate thing, and so you, you just assign sort of improvement. 00:28:27.864 --> 00:28:33.384 Without, uh, getting an active drug to the placebo effect without assessing 00:28:33.564 --> 00:28:36.414 the population or thinking about sort of the populations going in. 00:28:36.414 --> 00:28:41.004 It's just, oh, getting the drug makes you think you're gonna get better, 00:28:41.004 --> 00:28:43.014 so you get better in the placebo. 00:28:43.014 --> 00:28:45.384 Getting the placebo makes you think you're gonna get better, so you get better, 00:28:45.384 --> 00:28:47.634 which is, is usually not what's happening. 00:28:49.419 --> 00:28:54.819 Scott Berry: And so, um, I, I, the misunderstanding of this has an effect. 00:28:54.819 --> 00:28:58.029 And, and let me give you an example that happened to me a couple weeks. 00:28:58.479 --> 00:29:03.339 A company collected single arm data. 00:29:03.714 --> 00:29:08.994 And what that means is they don't have a placebo, that they run a trial where 00:29:08.994 --> 00:29:16.344 everybody gets the treatment and they have a baseline marker of severity. 00:29:17.184 --> 00:29:20.364 And by the way, that's the outcome measure, but that's 00:29:20.364 --> 00:29:22.254 also measured at baseline. 00:29:22.524 --> 00:29:27.264 It's similar to their first round in a golf tournament. 00:29:27.624 --> 00:29:33.144 Within the setting and they collected it across a wide range of patients and 00:29:33.144 --> 00:29:42.354 they noticed, wow, patients that have more higher baseline, they improved 00:29:42.354 --> 00:29:44.574 more when we gave them our drug. 00:29:46.809 --> 00:29:51.894 Now, now in, in, in the golf example that's taking that top quant quartile. 00:29:52.989 --> 00:29:56.889 noticing they regressed three points in better, by the way, the people in that 00:29:56.889 --> 00:30:01.509 golf tournament in the second to the worst quartile got better by one shot. 00:30:03.774 --> 00:30:08.304 The data from this company looks almost identical to the golf tournament. 00:30:08.304 --> 00:30:13.584 Remember, in a, in a clinical trial, we record your baseline before you get the 00:30:13.584 --> 00:30:15.294 intervention, and what's your score? 00:30:15.474 --> 00:30:17.514 There's variability in those. 00:30:17.664 --> 00:30:19.914 There's variability across time. 00:30:20.034 --> 00:30:24.444 A single patient goes up and down on almost every measure. 00:30:25.689 --> 00:30:30.519 Weight, blood pressure, uh, uh, cognitive scores. 00:30:30.789 --> 00:30:34.389 Uh, this cardiovascular endpoint they were looking at goes up and 00:30:34.389 --> 00:30:35.919 down within a patient over time. 00:30:36.759 --> 00:30:42.129 so they're now running a randomized trial in that particular population. 00:30:42.129 --> 00:30:44.919 They're gonna run a randomized trial, so we get to find out, 00:30:45.639 --> 00:30:47.264 do the placebo get better? 00:30:47.439 --> 00:30:51.369 And I told them, you're gonna get a very large placebo effect in your trial. 00:30:51.909 --> 00:30:53.379 And they looked at me like. 00:30:53.874 --> 00:30:55.194 What are you talking about? 00:30:55.914 --> 00:31:01.254 And they, their perception was what the investigator tells them about how good 00:31:01.254 --> 00:31:04.224 the drug is or what we know about it. 00:31:04.224 --> 00:31:08.934 All those things are what caused the placebo effect and not the inclusion 00:31:08.934 --> 00:31:13.314 exclusion criteria at the beginning of the trial, which causes a huge amount of 00:31:13.314 --> 00:31:16.974 regression-to-the mean, which we don't understand and we call the placebo effect. 00:31:18.069 --> 00:31:20.859 Nick Berry: Yeah, the placebo effect's not a psychological. 00:31:21.459 --> 00:31:23.199 Thing that messes with patients. 00:31:23.199 --> 00:31:26.949 It's a statistical process that you just described, right? 00:31:27.669 --> 00:31:27.939 Yeah. 00:31:27.939 --> 00:31:32.019 Scott Berry: regression to the mean is the statistical process that we 00:31:32.019 --> 00:31:36.069 confuse for placebo effect, which in some cases placebo effect is 00:31:36.069 --> 00:31:36.869 Nick Berry: real thing 00:31:37.629 --> 00:31:40.479 Scott Berry: said, we don't understand very well, which is which. 00:31:40.869 --> 00:31:41.079 Nick Berry: Yeah. 00:31:42.219 --> 00:31:44.169 Scott Berry: we don't understand very well at all, which is 00:31:44.169 --> 00:31:46.149 which, uh, within that setting. 00:31:46.419 --> 00:31:51.879 So they made huge drug development decisions to enroll that particular 00:31:51.879 --> 00:31:56.379 population from a single arm trial without a control, not understanding 00:31:56.379 --> 00:31:58.839 that by the way, the people on the good side might actually be the ones 00:31:58.839 --> 00:32:03.099 that have the bigger benefit relative to a control, and now they're jumping 00:32:03.099 --> 00:32:05.049 into a very, very large trial. 00:32:05.049 --> 00:32:09.369 Do they understand this particular effect or are they making a bad decision? 00:32:09.674 --> 00:32:12.674 The beauty of it is the randomized trial is gonna tell them, 00:32:12.909 --> 00:32:13.199 Nick Berry: Yeah. 00:32:13.274 --> 00:32:14.954 Scott Berry: and we're gonna figure that out. 00:32:14.954 --> 00:32:17.174 But are, do they understand it enough? 00:32:17.174 --> 00:32:19.274 Enough to make good decisions at this point? 00:32:22.574 --> 00:32:27.104 Other examples that show up that regression to the mean 00:32:27.104 --> 00:32:28.784 is critically important. 00:32:29.834 --> 00:32:35.024 We run a lot of trials and we look at subgroups of patients. 00:32:35.874 --> 00:32:36.714 We have a trial. 00:32:36.744 --> 00:32:39.774 We have a trial designed now called a basket trial. 00:32:40.584 --> 00:32:44.874 the baskets in a basket trial is, they're different kinds of patients. 00:32:45.624 --> 00:32:51.774 And a very common one of these is we have a treatment for, uh, cancer 00:32:52.524 --> 00:32:55.494 and we enroll, uh, different. 00:32:56.619 --> 00:32:57.399 of cancer. 00:32:57.429 --> 00:33:03.069 It might be head and neck cancer, breast cancer, lung cancer, GI cancer, 00:33:03.429 --> 00:33:06.189 uh uh, all of these different types. 00:33:06.189 --> 00:33:08.919 Or they may be subsets of kinds of cancer. 00:33:09.249 --> 00:33:14.649 And we run trials in them and we go into eight different types 00:33:14.649 --> 00:33:20.589 and we find out, aha, these two types we had the best effect in. 00:33:22.089 --> 00:33:22.899 And. 00:33:23.799 --> 00:33:29.379 look at the, uh, estimate from the two best outta eight, and we run 00:33:29.379 --> 00:33:33.759 a trial after that, or we try to estimate the effect in the eighth. 00:33:34.059 --> 00:33:39.459 The best one of the eight and the data were, um, uh, we, the 00:33:39.459 --> 00:33:41.289 number of responses we got. 00:33:41.289 --> 00:33:44.679 Suppose we're looking at a cancer trial and we look at how many patients 00:33:44.679 --> 00:33:49.959 responded, and we got 10 out of 20 patients, and the best responded. 00:33:50.454 --> 00:33:54.894 And all the other types had worse responses than 50%. 00:33:57.264 --> 00:34:01.884 I, as a statistician, don't believe the response rate's 50% that one. 00:34:02.094 --> 00:34:05.334 It's exactly the same as the golfer that shut 78. 00:34:05.334 --> 00:34:06.654 I don't believe they're 78. 00:34:06.654 --> 00:34:07.464 They're better than that. 00:34:08.454 --> 00:34:09.649 This 10 out of 20. 00:34:10.794 --> 00:34:14.514 I would estimate their response rate to be closer to the mean 00:34:14.514 --> 00:34:16.434 response across all tumor types. 00:34:16.974 --> 00:34:21.984 Depending on the variability of that response and the variability of 10 out of 00:34:21.984 --> 00:34:24.144 20, which is statisticians we understand. 00:34:24.864 --> 00:34:27.684 Nick Berry: Yeah, In your hierarchy model, like we talked about with the golf 00:34:27.684 --> 00:34:33.564 example now says there is some variability across the cancer types and we know that 00:34:33.564 --> 00:34:36.684 it might actually work better in some, but there's also a lot of variability 00:34:36.684 --> 00:34:41.004 in how many responses you're gonna observe about 20, uh, on 20 subjects. 00:34:41.004 --> 00:34:42.939 And so 20 is not that many. 00:34:43.974 --> 00:34:47.394 We don't know a lot about how well each of these cancers or how 00:34:47.394 --> 00:34:48.564 bad each of these cancers are. 00:34:48.564 --> 00:34:52.494 So we shrink back and we say a lot of the variability in this was due to there 00:34:52.494 --> 00:34:54.864 only being 20 patients per cancer type. 00:34:54.864 --> 00:34:57.144 So just pull everything back towards the middle. 00:34:57.444 --> 00:35:00.084 And if you're predicting what's gonna happen in phase three, 00:35:00.474 --> 00:35:01.884 you're not gonna predict 50%. 00:35:01.944 --> 00:35:05.484 And if you do, you're gonna cost yourself, you know, millions of dollars potentially. 00:35:05.484 --> 00:35:07.374 So you just shrink it all the way back. 00:35:07.374 --> 00:35:08.304 Really close to that. 00:35:08.304 --> 00:35:09.774 Mean probably if you only have 20. 00:35:10.164 --> 00:35:13.014 Subjects, you, you probably don't vary very much from the, the 00:35:13.014 --> 00:35:14.844 overall mean of all the types. 00:35:15.399 --> 00:35:15.639 Scott Berry: Yep. 00:35:16.419 --> 00:35:16.719 Yep. 00:35:16.839 --> 00:35:20.679 Uh, we see lots of this happening where we have units. 00:35:20.739 --> 00:35:26.349 Uh, in the case like this, we have trials with multiple arms, many doses, 00:35:27.129 --> 00:35:32.859 um, and the effect on one dose, should never estimate the effect on one 00:35:32.859 --> 00:35:34.989 dose without using the other doses. 00:35:35.709 --> 00:35:38.319 be a hierarchical model, it could be a dose response model. 00:35:39.039 --> 00:35:40.149 We have endpoints. 00:35:40.149 --> 00:35:45.819 We collect a lot of endpoints in clinical trials, and we analyze 12 endpoints 00:35:45.819 --> 00:35:48.999 in a particular disease where we think a treatment may have an effect. 00:35:49.869 --> 00:35:56.409 this one did the best out of the 12, and this is like the 1977. 00:35:56.409 --> 00:35:58.449 Halfway through the season we see this effect. 00:35:58.629 --> 00:36:00.879 Now we're gonna run the rest of the season. 00:36:01.734 --> 00:36:05.184 Do I think that effect in the 12th endpoint's gonna 00:36:05.184 --> 00:36:07.194 continue at the effect I see. 00:36:07.194 --> 00:36:11.574 Or is it gonna shrink towards the other 11 endpoints within that? 00:36:12.624 --> 00:36:18.204 It's very similar scenarios to sports where it seems almost obvious in that 00:36:18.204 --> 00:36:23.244 setting, but a lot of these things aren't naturally done in clinical trials. 00:36:23.244 --> 00:36:27.924 Publications provide this single estimate, and it's up to the consumer 00:36:27.924 --> 00:36:29.724 to do that themselves, which. 00:36:30.414 --> 00:36:34.404 I think experienced people in this industry do that, and they understand 00:36:34.404 --> 00:36:40.524 that there's this whole d, there's this whole, uh, I don't, it's almost 00:36:40.524 --> 00:36:46.794 controversy about the failure for phase two trials to replicate in phase three, 00:36:47.574 --> 00:36:50.994 and a lot of the things we just talked about are reasons why that doesn't happen. 00:36:50.994 --> 00:36:54.084 But the phase two process itself. 00:36:54.849 --> 00:36:58.899 Is relatively small sample sizes, so the variability in that 00:36:58.899 --> 00:37:00.429 measurement, it can be large. 00:37:00.849 --> 00:37:03.729 And we run hundreds of phase two trials. 00:37:04.989 --> 00:37:07.419 What phase three trials do we run? 00:37:07.479 --> 00:37:09.279 Those ones that are doing really well. 00:37:10.074 --> 00:37:15.384 And, and there's a natural part to this, and there's many phase two trials, and 00:37:15.384 --> 00:37:20.814 you see the extreme right tail of those that do well, they run phase three. 00:37:20.994 --> 00:37:24.744 Lo and behold, the effect is not reproduced in phase three. 00:37:25.794 --> 00:37:26.994 it's completely gone. 00:37:27.444 --> 00:37:27.734 Nick Berry: Yeah. 00:37:27.834 --> 00:37:29.394 Scott Berry: sometimes it's just smaller. 00:37:29.394 --> 00:37:32.334 It's regressed into the, the average of the effects. 00:37:32.754 --> 00:37:36.294 And people think that this is a con, controversy. 00:37:36.294 --> 00:37:39.894 The irre, we're not, this isn't reproducible science. 00:37:39.894 --> 00:37:41.814 What's wrong with what we're doing? 00:37:42.804 --> 00:37:44.124 It's regression to the mean. 00:37:44.544 --> 00:37:45.804 Nick Berry: A huge problem with that. 00:37:46.199 --> 00:37:47.334 Yeah, yeah, yeah. 00:37:47.419 --> 00:37:48.384 Scott Berry: a regression to the mean. 00:37:48.894 --> 00:37:50.634 Nick Berry: A huge problem with that though is that now we're 00:37:50.634 --> 00:37:55.014 powering our phase three studies based on the effect observed on 00:37:55.014 --> 00:37:56.484 the best dose in phase two, and. 00:37:57.579 --> 00:38:01.509 If we're regressing our estimate, the truth is probably not as good as that. 00:38:01.509 --> 00:38:05.229 You have an underpowered study and when you inevitably observe, you know, 00:38:05.229 --> 00:38:09.819 80% of the phase two effect in phase three, you have a p value of 0.07 00:38:09.819 --> 00:38:10.809 or something like that. 00:38:10.809 --> 00:38:14.649 And so it's, it's not just a, oh, our estimate was off. 00:38:14.649 --> 00:38:17.529 I mean, this can have huge implications in the, this sort 00:38:17.529 --> 00:38:19.089 of life cycle of a, of a drug. 00:38:20.394 --> 00:38:24.174 Scott Berry: Oh, I, I hope there were lessons there, but let me 00:38:24.174 --> 00:38:28.524 turn it back to you, Nick, because I know this is a topic we talk 00:38:28.524 --> 00:38:31.554 about with a lot of people and, um. 00:38:31.939 --> 00:38:36.889 Your friends, uh, heard this topic and they're in personal finance 00:38:37.124 --> 00:38:37.734 Nick Berry: Yeah. 00:38:38.329 --> 00:38:41.629 Scott Berry: what does the regression to the mean mean to them? 00:38:41.989 --> 00:38:46.579 So what would you say to, to people outside of drug development? 00:38:46.849 --> 00:38:51.169 Uh, outside of sports, what does regression to the 00:38:51.414 --> 00:38:51.714 Nick Berry: Yeah, 00:38:51.979 --> 00:38:55.909 Scott Berry: to somebody who's doing personal finance and buying stocks? 00:38:56.184 --> 00:38:56.484 Nick Berry: yeah. 00:38:57.024 --> 00:38:57.174 Yeah. 00:38:57.174 --> 00:39:01.734 I think the general lesson that I would would say to this sort of lay person 00:39:01.734 --> 00:39:04.194 is that anytime you observe something. 00:39:04.704 --> 00:39:06.174 Extraordinary. 00:39:06.174 --> 00:39:10.434 And you get, uh, something that's, you know, you've never seen before, 00:39:10.434 --> 00:39:12.174 like this is an amazing result. 00:39:12.564 --> 00:39:16.344 You just smooth over that and you realize that there were maybe a lot 00:39:16.344 --> 00:39:19.464 of opportunities from other places to observe this amazing result. 00:39:19.464 --> 00:39:22.704 There's a lot of stocks that could have performed really well, and when 00:39:22.704 --> 00:39:30.894 you start to, to base decisions off of recent amazing things that happened. 00:39:31.704 --> 00:39:35.274 You are inevitably setting yourself up to be disappointed if you expect that to 00:39:35.274 --> 00:39:37.254 be reproduced and to happen over again. 00:39:37.254 --> 00:39:41.004 So when you observe something amazing, just smooth over it, realize that it 00:39:41.004 --> 00:39:45.144 might be good, but it's probably not actually a world beater in that case. 00:39:45.144 --> 00:39:49.044 And so just temper your expectations every time you, you, you start to 00:39:49.044 --> 00:39:50.694 make predictions based on past data. 00:39:51.449 --> 00:39:56.759 Scott Berry: So if, if there are 50 stock brokers in a company and in a particular 00:39:57.179 --> 00:40:03.569 year, one person is the best stockbroker and they earn a per certain percent. 00:40:04.559 --> 00:40:04.839 Nick Berry: Yeah. 00:40:04.854 --> 00:40:05.424 Scott Berry: best would be. 00:40:05.424 --> 00:40:07.254 30, 30% on 00:40:07.269 --> 00:40:07.869 Nick Berry: Huge. 00:40:08.094 --> 00:40:09.294 Scott Berry: over a course of a year. 00:40:10.014 --> 00:40:12.444 And then the next year they don't do 30%. 00:40:13.449 --> 00:40:15.069 Nick Berry: Yeah, they fell off. 00:40:15.344 --> 00:40:15.634 Yeah. 00:40:15.744 --> 00:40:16.464 Scott Berry: they lazy? 00:40:16.464 --> 00:40:18.954 They, you know, the, the media scrutiny got 00:40:19.094 --> 00:40:19.384 Nick Berry: Yeah. 00:40:20.064 --> 00:40:20.304 Scott Berry: Yeah. 00:40:21.054 --> 00:40:26.214 It, that they actually were never that good and they, the data got 00:40:26.214 --> 00:40:27.654 them there through randomness. 00:40:27.744 --> 00:40:29.604 Now they may be better than average. 00:40:29.754 --> 00:40:29.964 How 00:40:30.159 --> 00:40:30.429 Nick Berry: Yep. 00:40:30.624 --> 00:40:33.114 Scott Berry: depends on the variability in stock pickers and the 00:40:33.114 --> 00:40:34.644 variability in a particular year. 00:40:35.349 --> 00:40:35.649 Nick Berry: Yep. 00:40:36.264 --> 00:40:39.114 Scott Berry: the lessons are in drug development, the lessons are 00:40:39.114 --> 00:40:41.334 in sports, in in picking stock. 00:40:41.334 --> 00:40:45.624 So this was our lessons learned for drug developers from the world of Sports. 00:40:45.804 --> 00:40:47.034 Episode one we 00:40:47.139 --> 00:40:47.349 Nick Berry: Yeah. 00:40:47.349 --> 00:40:48.024 We'll see you later. 00:40:48.444 --> 00:40:49.674 Scott Berry: Yeah, we have more to come. 00:40:49.674 --> 00:40:49.914 So. 00:40:50.329 --> 00:40:52.069 Uh, Nick, thanks for joining me. 00:40:52.069 --> 00:40:52.339 And 00:40:52.504 --> 00:40:53.184 Nick Berry: Thanks for having me. 00:40:53.449 --> 00:40:54.769 Scott Berry: we are in the interim.