WEBVTT NOTE This file was generated by Descript 00:00:00.000 --> 00:00:03.500 Judith: Welcome to Berry's In the Interim podcast, where we explore the 00:00:03.500 --> 00:00:06.739 cutting edge of innovative clinical trial design for the pharmaceutical and 00:00:06.739 --> 00:00:08.819 medical industries, and so much more. 00:00:09.229 --> 00:00:09.930 Let's dive in. 00:00:11.359 --> 00:00:13.039 Scott: Alright, welcome everybody. 00:00:13.039 --> 00:00:18.529 Back to In the Interim, I'm your host, Scott Berry and I'm joined 00:00:18.529 --> 00:00:20.509 with by Nick Berry today, Dr. 00:00:20.509 --> 00:00:21.169 Nick Berry. 00:00:21.439 --> 00:00:26.809 And we are going to do our se our second episode. 00:00:27.559 --> 00:00:31.429 Lessons for drug developers from the world of sports. 00:00:32.179 --> 00:00:36.769 You can go back to our first time we did this episode 14 of, of in the 00:00:36.769 --> 00:00:40.849 interim, and we had a really nice episode on regression to the mean. 00:00:41.779 --> 00:00:47.989 Today we're gonna talk about the 10 run rule and futility, so maybe bear 00:00:47.989 --> 00:00:52.849 with us a little bit as we get as we get to that, but I do wanna revisit. 00:00:53.404 --> 00:00:57.484 The regression to the mean and a prediction that Nick made. 00:00:58.294 --> 00:01:06.874 We, we did this episode last year and May of last year, Aaron Judge was hitting 400. 00:01:07.654 --> 00:01:12.874 And for those of you who are not baseball fans, uh, I, I, last week 00:01:12.874 --> 00:01:17.884 had somebody from Iceland send me a, uh, uh, an email about the 00:01:17.884 --> 00:01:21.034 episode on, uh, the Panther trial. 00:01:21.034 --> 00:01:21.724 So we have. 00:01:21.969 --> 00:01:27.579 People globally listening to this, A 400 success rate in baseball 00:01:27.579 --> 00:01:29.979 is a, is a very, very high bar. 00:01:29.979 --> 00:01:32.559 Nobody's done it since 1941. 00:01:32.949 --> 00:01:37.359 And it's, it's one of these mythical, uh, targets that 00:01:37.359 --> 00:01:39.189 somebody could be a 400 hitter. 00:01:39.189 --> 00:01:40.269 Well, Aaron Judge was. 00:01:40.319 --> 00:01:41.549 Batting 400. 00:01:41.549 --> 00:01:47.819 It's just his hits divided by at bats, uh, in May, and that's early in the season. 00:01:48.089 --> 00:01:50.039 His, his rate was 400. 00:01:50.519 --> 00:01:52.769 We were talking about regression to the mean. 00:01:52.769 --> 00:01:57.509 So Nick predicted that by end of year, he would be hitting three 30. 00:01:58.939 --> 00:02:03.619 Interestingly, Jim Albert, who we did another episode, I did an episode 00:02:03.619 --> 00:02:07.399 with Jim Albert about Bayesian statistics and sports statistics. 00:02:07.399 --> 00:02:13.639 His, his, his career in those, he guessed three 20 and Aaron Judge's 00:02:13.639 --> 00:02:16.639 final batting average was 3 31. 00:02:17.089 --> 00:02:21.259 So Nick, you were off, um, in your guess. 00:02:21.953 --> 00:02:22.163 Nick Berry: of 00:02:22.194 --> 00:02:22.484 Scott: Yeah. 00:02:23.594 --> 00:02:23.884 Yeah. 00:02:24.563 --> 00:02:25.433 Nick Berry: 0.1% 00:02:25.433 --> 00:02:25.763 off. 00:02:25.763 --> 00:02:25.973 But 00:02:26.054 --> 00:02:26.344 Scott: Yeah. 00:02:27.803 --> 00:02:32.003 Nick Berry: yeah, I think that was basically weighted average of 00:02:32.003 --> 00:02:37.433 what he's done with some random, informed guess of his true batting 00:02:37.433 --> 00:02:39.113 average to get me to three 30. 00:02:40.223 --> 00:02:40.493 Yep. 00:02:41.059 --> 00:02:41.269 Scott: Yeah. 00:02:41.329 --> 00:02:43.669 And by the way, it was an incredible season. 00:02:43.669 --> 00:02:47.839 3 31 with the home runs he had was a phenomenal season. 00:02:48.409 --> 00:02:48.979 Uh. 00:02:49.223 --> 00:02:49.883 Nick Berry: thing now. 00:02:49.913 --> 00:02:54.683 I think he's hitting like 2 25 this year, uh, almost a month into the season. 00:02:54.683 --> 00:02:55.793 So the exact opposite. 00:02:56.153 --> 00:02:59.783 We can, we can him to the mean in the other way this year. 00:03:00.334 --> 00:03:03.064 Scott: Ah, that, that actually would've, uh, made for a good thing. 00:03:03.064 --> 00:03:04.714 Maybe we should figure out what he is. 00:03:04.714 --> 00:03:08.104 And, and you have to post another prediction, but let me talk about 00:03:08.104 --> 00:03:09.814 a different prediction you made. 00:03:09.814 --> 00:03:13.984 And this was a, a, a prediction we are coming off of. 00:03:13.984 --> 00:03:17.884 We are, we are recording this episode coming off of the 00:03:17.884 --> 00:03:18.989 Master's Golf Tournament. 00:03:19.899 --> 00:03:21.669 So that was last weekend. 00:03:21.939 --> 00:03:26.889 And, uh, we, we are, we are, we're a big golf family. 00:03:27.189 --> 00:03:28.689 Uh, we enjoy golf. 00:03:28.719 --> 00:03:33.489 Even, uh, Nick's sister Lindsay, who doesn't play golf at 00:03:33.489 --> 00:03:35.619 all, she loves to watch golf. 00:03:36.099 --> 00:03:43.449 Uh, so we were watching the Masters and halfway through the Masters, 36 holes, two 00:03:43.449 --> 00:03:47.169 rounds, Rory McElroy had a six shot lead. 00:03:47.689 --> 00:03:52.549 So he, the second place, and there were multiple golfers that were six shots 00:03:52.549 --> 00:03:57.049 back, but a six shot lead halfway through the tournament is a quite large lead. 00:03:57.259 --> 00:04:02.869 I, I believe it was the largest lead ever in the masters at that. 00:04:02.929 --> 00:04:08.839 And of course, our family text, uh, goes to predictions, goes to 00:04:08.839 --> 00:04:14.929 statistics, and the question was from, from Nick's sister, Lindsay. 00:04:15.519 --> 00:04:22.989 What's the chance that Rory McElroy wins the Masters and multiple of us guess. 00:04:22.989 --> 00:04:26.979 But Nick, you, you made a prediction of the chance that he win the 00:04:26.979 --> 00:04:29.014 Masters and what was your prediction? 00:04:29.414 --> 00:04:29.534 I. 00:04:30.643 --> 00:04:32.858 Nick Berry: I think I said 70%. 00:04:33.293 --> 00:04:39.053 Basically my, my math was, I think he probably has the best expected 00:04:39.053 --> 00:04:42.293 value of the whole field just based on the fact that he won last year. 00:04:42.293 --> 00:04:45.803 He's playing really well this year, so I said he's probably the best 00:04:45.803 --> 00:04:47.603 player in the field this week. 00:04:48.383 --> 00:04:55.913 a six shot lead and I, he that, you know, his distribution was good 00:04:55.913 --> 00:04:58.313 enough that even though there was a lot of people with the opportunity to 00:04:58.313 --> 00:05:00.923 catch him 70% chance of him winning. 00:05:02.749 --> 00:05:05.509 Scott: And my, my prediction was 60%. 00:05:05.509 --> 00:05:12.409 And interestingly, uh, Nick's, uh, mom, uh, prediction was 20%. 00:05:12.619 --> 00:05:13.609 It was quite different. 00:05:14.269 --> 00:05:14.929 Uh, and 00:05:15.218 --> 00:05:17.498 Nick Berry: golf and she knows how variable it is, and that was 00:05:17.498 --> 00:05:18.638 the reason for her prediction. 00:05:18.638 --> 00:05:20.228 It was like weird stuff happens. 00:05:20.228 --> 00:05:20.378 I 00:05:20.664 --> 00:05:20.884 Scott: yep. 00:05:21.038 --> 00:05:23.753 Nick Berry: you know, it's one guy has to hold onto the lead and so 00:05:23.828 --> 00:05:26.473 she asked 20% because of all the variability and what could happen. 00:05:29.314 --> 00:05:34.564 Scott: So at that point, uh, to, to, to sort of come to this, 00:05:34.564 --> 00:05:41.644 interestingly, Scotty Scheffler, after 36 holes was 12 shots back. 00:05:42.934 --> 00:05:46.264 And he's the number one ranked golfer in the world. 00:05:46.414 --> 00:05:50.734 Rory, I think was the number two ranked golfer in the world, uh, in it. 00:05:50.734 --> 00:05:55.024 And Scotty Scheffler was 12 shots back, essentially. 00:05:55.774 --> 00:05:59.704 Very little chance he can win less than 1% chance. 00:05:59.704 --> 00:06:01.624 He would, he would win the golf tournament. 00:06:02.164 --> 00:06:06.154 Uh, how many, how much less, but less than 1%. 00:06:07.114 --> 00:06:10.984 Interestingly, if you were 17 shots back. 00:06:11.944 --> 00:06:16.294 Meaning after 36 holes, you were five over par. 00:06:16.294 --> 00:06:18.244 Rory was 12 under par. 00:06:18.634 --> 00:06:24.814 So if you were five over par or more, you had a 0% chance of 00:06:24.814 --> 00:06:29.584 winning because you were quote unquote cut from the tournament. 00:06:31.249 --> 00:06:35.899 The, those players stopped playing and they reduce, and the, and the, the, the 00:06:35.899 --> 00:06:41.089 rule of the cut is the top 50 players, and I think it started with 91 players. 00:06:41.449 --> 00:06:47.599 The top 50 players and ties continue on into the weekend, the last two rounds. 00:06:48.019 --> 00:06:54.229 But they s they, they, they cut and, and sure, I'll say the word futility, 00:06:54.559 --> 00:06:58.554 that these players no longer have an opportunity to win the golf tournament. 00:06:59.104 --> 00:07:03.694 If you were 17 shots back, happened to be in that tournament where Scotty Shuffler 00:07:03.694 --> 00:07:06.004 was 12 back, but he got to keep playing. 00:07:07.984 --> 00:07:10.804 Now, what happened in the golf tournament, Nick? 00:07:13.268 --> 00:07:20.378 Nick Berry: Uh, so round three, Rory struggled, I think shot one over par. 00:07:20.588 --> 00:07:24.128 Um, so he went to 11 under, and the rest of the field played great. 00:07:24.518 --> 00:07:25.538 You know, Al it seemed like almost everyone that was. 00:07:27.563 --> 00:07:29.423 Six or seven shots behind him. 00:07:29.813 --> 00:07:36.023 Shot five under six under, and he went into Sunday to hide for the lead. 00:07:36.653 --> 00:07:38.183 he blew, he blew his lead, right? 00:07:38.183 --> 00:07:38.613 It, it 00:07:38.614 --> 00:07:38.764 Scott: Right 00:07:39.293 --> 00:07:40.193 Nick Berry: after day three. 00:07:41.183 --> 00:07:45.563 Um, Scotty Scheffler shot seven under par and moved into contention. 00:07:46.043 --> 00:07:52.463 Um, I think that, you know, the number one player that was 12 shots back on Sunday. 00:07:53.123 --> 00:07:54.353 whole field kind of struggled. 00:07:54.353 --> 00:07:57.593 Nobody shot seven under, nobody had a, a huge round. 00:07:58.493 --> 00:08:02.453 Rory eked out a win, um, by one stroke. 00:08:03.353 --> 00:08:04.398 Uh, so he, he did. 00:08:05.753 --> 00:08:08.783 He did technically retain his lead, even though he blew it and 00:08:08.854 --> 00:08:09.074 Scott: Yes. 00:08:09.293 --> 00:08:09.503 Nick Berry: at the 00:08:09.504 --> 00:08:09.794 Scott: Yeah. 00:08:10.313 --> 00:08:13.253 Nick Berry: Um, but the, the most important part is that I was the most 00:08:13.253 --> 00:08:17.033 right in our family group chat because I said 70%, which was the highest number. 00:08:17.333 --> 00:08:20.363 And it doesn't matter how he got there, there's no pictures on the scorecard. 00:08:20.393 --> 00:08:23.063 Uh, I picked the highest probability Rory went on to win. 00:08:23.063 --> 00:08:26.213 So I, uh, I claimed victory in the family group shop. 00:08:28.609 --> 00:08:32.494 Scott: An interesting part of it was Scotty Scheffler ended up finishing 00:08:32.494 --> 00:08:34.624 second place, one shot behind. 00:08:36.259 --> 00:08:41.749 And yes, he, he, he missed a birdie putt on 17, that that eventually could 00:08:41.749 --> 00:08:44.299 have led him to a tie, uh, within it. 00:08:44.449 --> 00:08:46.339 So he finished one shot back. 00:08:46.339 --> 00:08:52.249 He had a very, very small chance of winning the golf tournament in a, 00:08:52.249 --> 00:08:54.589 not in a, in a different setting. 00:08:54.679 --> 00:08:55.519 We could. 00:08:56.319 --> 00:08:57.669 Said, you're done playing. 00:08:57.669 --> 00:08:59.019 You have no chance to win. 00:08:59.019 --> 00:08:59.589 You're cut. 00:08:59.589 --> 00:09:01.929 You can't play, uh, in this setting. 00:09:01.929 --> 00:09:04.869 But he came back and, and, and had a legitimate chance 00:09:04.869 --> 00:09:06.219 of, of winning the tournament. 00:09:07.119 --> 00:09:12.579 And, and so I won't, I won't address your 70% being the right answer when, uh, 00:09:12.579 --> 00:09:14.649 we know more about the golf tournament. 00:09:14.739 --> 00:09:16.329 But, but you're technically right. 00:09:16.389 --> 00:09:20.109 If you evaluate the likelihood function of what happened, you had the 00:09:20.109 --> 00:09:21.909 highest likelihood function in that. 00:09:22.894 --> 00:09:30.424 But coming to this question about in a sports competition, do we stop the 00:09:30.424 --> 00:09:32.974 sports competition and what does it mean? 00:09:33.124 --> 00:09:35.464 And you won't be surprised to, to see that. 00:09:35.464 --> 00:09:40.774 We'll turn this into thinking about stopping clinical trials and 00:09:40.774 --> 00:09:42.259 what are the similarity of that. 00:09:43.609 --> 00:09:48.589 So Nick's brother Cooper, and if you're on video here, you can see 00:09:48.799 --> 00:09:54.619 Nick is wearing the team shirt for the Pomona Pitzer Sage Hens. 00:09:55.369 --> 00:10:00.169 Nick's brother Cooper plays on Pomona Pitzers baseball team. 00:10:01.294 --> 00:10:06.754 And in a recent game we were watching, he was playing up in Oregon, up in 00:10:06.754 --> 00:10:12.454 Portland, Oregon, and they were playing Lewis and Clark and in game three of 00:10:12.454 --> 00:10:15.064 their series, they were tied at one each. 00:10:15.544 --> 00:10:21.664 The, the sage hens scored six runs in the sixth inning, and they went up 11 runs. 00:10:21.724 --> 00:10:23.644 They were up 11 to zero. 00:10:24.424 --> 00:10:28.294 And in the bottom of the inning, Lewis and Clark didn't score. 00:10:29.659 --> 00:10:34.279 So after six innings, and it's a nine inning game, so you're two 00:10:34.279 --> 00:10:41.149 thirds of the way through the game, the sage hens were up 11 runs 00:10:43.459 --> 00:10:44.119 now. 00:10:44.209 --> 00:10:48.379 The game kept going, so the game did not stop at that point. 00:10:50.959 --> 00:10:55.879 In the seventh inning, they scored one more run to go up, 12 to zero. 00:10:55.969 --> 00:11:01.489 Lewis and Clark, uh, the Sea Otters, I think are their, their mascot, uh, 00:11:01.489 --> 00:11:05.629 scored zero and the game was 12 to zero. 00:11:05.629 --> 00:11:09.169 And by rule, the game ends at that point. 00:11:11.239 --> 00:11:12.679 It's futile. 00:11:13.699 --> 00:11:17.929 And there's a, you know, there's a classic 10 run rule and there's 00:11:17.929 --> 00:11:22.789 a 10 run rule after seven innings that if one team is ahead by 10 00:11:22.789 --> 00:11:25.339 runs or more, the game is over. 00:11:25.969 --> 00:11:30.164 So the game ended in seven innings at that point, and we stopped 00:11:34.339 --> 00:11:34.999 now. 00:11:36.424 --> 00:11:37.954 We have futility rules. 00:11:37.954 --> 00:11:42.334 In the masters, we stop players who are not in the top 50. 00:11:42.934 --> 00:11:45.844 In youth sports, we do this quite a bit. 00:11:46.204 --> 00:11:48.574 We have a 10 run rule in baseball. 00:11:48.724 --> 00:11:52.324 It's a little bit different in sports with a clock, baseball has 00:11:52.324 --> 00:11:56.584 no clock, it plays nine innings, and that's kind of the clock. 00:11:56.584 --> 00:11:59.374 So games can go long in that setting. 00:11:59.374 --> 00:12:03.034 They did, they, you know, they do have travel and they're dealing with that. 00:12:03.784 --> 00:12:04.234 Uh. 00:12:04.424 --> 00:12:11.054 In games like American football and in hockey and basketball, they do 00:12:11.054 --> 00:12:13.004 something called a running clock. 00:12:13.154 --> 00:12:17.714 So if a team goes up by five goals in a hockey game, they don't 00:12:17.714 --> 00:12:19.994 stop the clock between whistles. 00:12:19.994 --> 00:12:22.349 They let it run, they, they shorten the game. 00:12:24.139 --> 00:12:30.019 So thi this happens, um, Nick used to play select baseball and they did 00:12:30.019 --> 00:12:33.979 something kind of interesting where after three innings, if you were up by 00:12:33.979 --> 00:12:37.369 15, they might stop it four innings. 00:12:37.399 --> 00:12:39.919 It was 10 or uh, eight or more. 00:12:40.189 --> 00:12:45.289 And after five innings, it was, sorry, after four it was 10, 00:12:45.289 --> 00:12:46.759 and after five it was eight. 00:12:46.819 --> 00:12:51.349 So they had a graduated rule within those games, and the game is over. 00:12:52.654 --> 00:12:55.504 So why do we do this in sports, Nick? 00:12:55.534 --> 00:12:59.099 Why do we have these rules that stop the competition? 00:13:01.623 --> 00:13:04.628 Nick Berry: Oh, I think the reasons behind them are confusing. 00:13:04.628 --> 00:13:07.538 There's a lot of, uh, we'll call 'em stakeholders to this, but I 00:13:07.538 --> 00:13:13.538 think the implication of all the rules is that there's, I'll say no 00:13:13.538 --> 00:13:15.518 point in continuing to play because. 00:13:16.928 --> 00:13:17.678 The 00:13:19.688 --> 00:13:22.838 you know, result is, is determined at that point. 00:13:22.838 --> 00:13:27.578 And, uh, I think in reality it's probably, everyone understands 00:13:27.578 --> 00:13:31.328 it's just a sufficiently small probability of something other 00:13:31.328 --> 00:13:32.738 than the obvious thing happening. 00:13:32.798 --> 00:13:35.853 Um, we're, we're willing to say, 00:13:37.893 --> 00:13:42.728 for the benefit of all involved, let's truncate the game at this place. 00:13:42.728 --> 00:13:43.988 We know where this is going. 00:13:43.988 --> 00:13:45.638 I think in like youth sports. 00:13:46.523 --> 00:13:49.733 People got places to be, you know, parents gotta get home. 00:13:49.733 --> 00:13:51.413 There's another game right after this. 00:13:51.413 --> 00:13:52.973 Uh, the tournament's running late. 00:13:52.973 --> 00:13:55.553 We have only have the fields until nine o'clock. 00:13:55.973 --> 00:14:01.283 Um, a lot of different reasons to go to it for professional sports or, you know, 00:14:01.283 --> 00:14:05.483 I'll call, uh, what Cooper does something where, you know, the school's paying 00:14:05.483 --> 00:14:06.953 for this and, and there's more money. 00:14:06.953 --> 00:14:11.153 I mean, um, have a stake in this. 00:14:11.153 --> 00:14:12.923 They're, they're wasting resources. 00:14:12.983 --> 00:14:13.368 Uh, if you just. 00:14:14.378 --> 00:14:17.948 You're pitching guys late in the game that you don't want to throw, 00:14:18.008 --> 00:14:22.568 or silly things happen like, oh, we lost, we have to play two more innings. 00:14:23.198 --> 00:14:23.978 put an outfielder 00:14:24.034 --> 00:14:24.334 Scott: an out. 00:14:24.398 --> 00:14:25.748 Nick Berry: 'cause I don't wanna waste a pitcher. 00:14:25.748 --> 00:14:29.018 And, and you know, things start to get wacky in that regard. 00:14:29.018 --> 00:14:31.208 And so maybe we should just call the game at this point. 00:14:31.208 --> 00:14:35.858 But I think it all boils down to everyone knows what's gonna happen. 00:14:36.338 --> 00:14:39.698 What's the point in continuing to play sort of is, is the idea behind all of 'em. 00:14:42.139 --> 00:14:46.399 Scott: There's also a concern in youth sports that if one team has such a lead, 00:14:46.939 --> 00:14:51.589 it becomes almost unsportsmanlike for that team to be trying hard to win. 00:14:51.859 --> 00:14:52.429 Are they gonna win by. 00:14:53.884 --> 00:14:58.864 Absurd amounts, for example, uh, potential injuries. 00:14:59.104 --> 00:14:59.944 There's aspect. 00:14:59.944 --> 00:15:02.044 We don't have futility rules in sports. 00:15:02.044 --> 00:15:06.634 Now we, we, we do have some, and I'll come to those, but largely in an 00:15:06.634 --> 00:15:13.564 American baseball game, if one team is ahead by 25, runs after eight innings 00:15:13.564 --> 00:15:14.819 or seven innings, they keep going. 00:15:16.114 --> 00:15:19.864 Uh, uh, in, in those sports, they don't stop them. 00:15:19.864 --> 00:15:23.374 Now people have paid to go watch that game. 00:15:23.718 --> 00:15:24.008 Nick Berry: Yeah. 00:15:24.934 --> 00:15:28.534 Scott: they're selling beer, uh, though they don't sell beer after 00:15:28.534 --> 00:15:31.894 the seventh inning, but, but they're selling beer at an NFL game. 00:15:32.074 --> 00:15:37.174 At a, at an NBA game, a team could be ahead by 40 points in 00:15:37.174 --> 00:15:38.704 an NBA game, and they keep going. 00:15:39.459 --> 00:15:40.359 In those games. 00:15:40.359 --> 00:15:42.159 So pro sports are a little bit different. 00:15:42.159 --> 00:15:45.789 They do worry about injuries, but there are no futility rules in pro 00:15:45.789 --> 00:15:48.144 sports, which is somewhat interesting. 00:15:49.268 --> 00:15:50.288 Nick Berry: Well, you mentioned the fans. 00:15:50.288 --> 00:15:51.878 The fans do have futility rules, and this is the. 00:15:52.868 --> 00:15:55.898 If you're at a, a baseball game and a team's up 20 runs the 00:15:55.898 --> 00:15:58.478 stand's empty people to them. 00:15:58.478 --> 00:16:03.188 I think it's more important to, uh, be in the car before traffic hits than it is to 00:16:03.188 --> 00:16:06.038 watch your team get smothered by 25 runs. 00:16:07.778 --> 00:16:10.088 I'm, I'm definitely a stick it out to the end. 00:16:10.088 --> 00:16:11.138 I paid to be there. 00:16:11.138 --> 00:16:13.478 I went through the, the effort of getting there. 00:16:13.478 --> 00:16:17.618 I'm gonna watch the ninth inning of this game, but, uh, but the stadium does empty 00:16:17.618 --> 00:16:19.358 in those cases, so fans have their own. 00:16:20.393 --> 00:16:23.273 Utility rule that, or sorry, utility rule that they've created 00:16:23.273 --> 00:16:24.533 in their mind for when it's over. 00:16:26.554 --> 00:16:31.204 Scott: So can we make a mistake with a futility roll? 00:16:32.269 --> 00:16:34.159 Could we have stopped that? 00:16:34.159 --> 00:16:38.209 That game that Cooper's team played up in Oregon, they were ahead 00:16:38.209 --> 00:16:40.219 12 to zero after seven innings. 00:16:41.029 --> 00:16:43.579 They could have lost that game if they kept playing. 00:16:44.329 --> 00:16:50.359 Uh, the rules are a nine inning game, and the sea otters could have scored 13 runs 00:16:50.389 --> 00:16:53.059 in the ninth inning and won that game. 00:16:53.059 --> 00:17:00.199 It's possible in those circumstances, just like it's possible that we could have cut. 00:17:01.474 --> 00:17:06.004 Scotty Scheffler from that golf tournament and said, you don't have a chance to win. 00:17:06.004 --> 00:17:08.074 You can you no longer get to play. 00:17:08.854 --> 00:17:11.494 And he could have won that tournament. 00:17:11.524 --> 00:17:12.664 By the end, he didn't. 00:17:12.994 --> 00:17:18.094 But in another setting, we have stopped competitions that would've flipped in. 00:17:18.094 --> 00:17:19.174 The other team would've won. 00:17:23.813 --> 00:17:25.073 Nick Berry: Yeah, for sure. 00:17:28.313 --> 00:17:31.973 I think there's, there's a, a heuristic. 00:17:32.033 --> 00:17:32.423 This is. 00:17:32.828 --> 00:17:37.658 Bill James, the famous analytics, uh, the father of analytics. 00:17:37.658 --> 00:17:39.068 I think people give him credit for that. 00:17:40.118 --> 00:17:43.748 he has this, um, I, I called it a heuristic. 00:17:43.748 --> 00:17:46.388 I think it's a good, good word for it college 00:17:46.644 --> 00:17:47.134 Scott: College. 00:17:47.318 --> 00:17:52.658 Nick Berry: but when he thinks the game is over, um, it's, it's a simple 00:17:52.658 --> 00:17:56.588 calculation like you take the winning, how much you're winning by minus three. 00:17:58.613 --> 00:18:01.883 Add a little bit if you have the ball and then I don't even 00:18:01.883 --> 00:18:02.903 know what you do from there. 00:18:03.233 --> 00:18:03.863 Something like, 00:18:04.069 --> 00:18:04.744 Scott: Something like that. 00:18:04.883 --> 00:18:08.873 Nick Berry: weird, a little algebra scheme you do and it says this game is over. 00:18:08.873 --> 00:18:10.073 This game is not over. 00:18:11.603 --> 00:18:15.173 um, I bring this up because partially 'cause I wanted to talk about 00:18:15.173 --> 00:18:18.563 this example, but when I was in college, Texas a and m was playing 00:18:18.563 --> 00:18:20.963 in March in the the NCA tournament. 00:18:20.993 --> 00:18:22.883 We just finished March, Matt, March Mattis as well. 00:18:23.303 --> 00:18:25.253 Uh, and. 00:18:26.033 --> 00:18:30.083 Text a m was playing northern Iowa in I think the round of 32. 00:18:32.153 --> 00:18:37.193 they lost, they were futile, futile, according to Bill James' algorithm. 00:18:37.613 --> 00:18:41.873 Um, we were down 12 points with 42 seconds to go or something like that. 00:18:42.713 --> 00:18:46.943 and doing the calculation that's, that's impossible to, to, to, to come back from. 00:18:47.393 --> 00:18:49.883 And I have two good friends that were at the game. 00:18:50.033 --> 00:18:50.783 They left. 00:18:51.668 --> 00:18:53.918 They left, you know, it's obvious it's over. 00:18:53.918 --> 00:18:54.968 Gotta beat traffic. 00:18:54.968 --> 00:18:57.038 They're in the concourse walking away. 00:18:57.038 --> 00:19:06.518 And um, um, and my alma mater came back and won, um, Alex Caruso of now a two time 00:19:06.518 --> 00:19:08.408 NBA championship fame was on that team. 00:19:08.408 --> 00:19:12.428 And it was, you know, the comeback of a lifetime for me in school. 00:19:12.428 --> 00:19:12.788 And so. 00:19:13.703 --> 00:19:14.753 happened, you know, this was futile. 00:19:14.753 --> 00:19:15.503 This would've stopped for 00:19:15.764 --> 00:19:15.984 Scott: Yep. 00:19:16.043 --> 00:19:19.013 Nick Berry: according to a, a popular rule, and it switched. 00:19:19.073 --> 00:19:20.003 Weird things happened. 00:19:20.003 --> 00:19:21.833 The game reversed index a M1. 00:19:24.229 --> 00:19:29.119 Scott: So there, there are some cases where you can't come back, but in 00:19:29.119 --> 00:19:31.189 all those situations it's possible. 00:19:31.279 --> 00:19:35.179 Uh, the probability may be very, very small and we'll get to that question 00:19:35.179 --> 00:19:37.009 of the probability of winning to that. 00:19:37.279 --> 00:19:44.239 There are other cases that we do have futility rules in, in pro sports, when 00:19:44.239 --> 00:19:46.099 you're playing a seven game series. 00:19:47.479 --> 00:19:50.479 And one of the team goes up four to one. 00:19:50.959 --> 00:19:53.419 They've won four of the first five. 00:19:54.439 --> 00:19:55.699 They stop playing. 00:19:56.629 --> 00:19:59.509 They don't play the sixth and seventh games. 00:20:00.169 --> 00:20:04.579 The, and in that situations, it's impossible for the other team to win. 00:20:04.579 --> 00:20:05.809 So they stop playing. 00:20:06.379 --> 00:20:09.139 So they don't say, well, people have bought tickets. 00:20:09.139 --> 00:20:12.709 We're gonna sell food, we're gonna play game six and seven. 00:20:12.709 --> 00:20:14.869 They, they do stop playing and they go on. 00:20:14.869 --> 00:20:16.069 So there are. 00:20:16.489 --> 00:20:22.399 Some binding, uh, binding's the wrong word, but there are cases 00:20:22.399 --> 00:20:24.829 where it's impossible to come back and they stop playing. 00:20:25.189 --> 00:20:28.939 Another example of that is golf, where you, when you do match play, 00:20:28.939 --> 00:20:34.309 like in the Ryder Cup, you play who wins the most holes on 18 holes. 00:20:34.549 --> 00:20:40.189 If you're up four with three holes to play, the golfers stop playing. 00:20:40.219 --> 00:20:42.529 They don't play the last three holes. 00:20:42.679 --> 00:20:43.549 The match is over. 00:20:44.918 --> 00:20:45.158 Nick Berry: You're 00:20:45.424 --> 00:20:46.114 Scott: those are 00:20:46.838 --> 00:20:46.988 Nick Berry: And 00:20:47.344 --> 00:20:50.884 Scott: your, you're mathematically eliminated. 00:20:50.884 --> 00:20:51.184 Yep, 00:20:51.548 --> 00:20:53.468 Nick Berry: it's an obvious way to do futility, right? 00:20:53.468 --> 00:20:53.978 Like you don't 00:20:54.244 --> 00:20:54.454 Scott: yep, 00:20:55.418 --> 00:21:00.188 Nick Berry: a, a model or any complex reasoning to understand why that happens 00:21:00.188 --> 00:21:02.198 or, or to determine if it's a good rule. 00:21:02.243 --> 00:21:02.333 Right. 00:21:03.399 --> 00:21:03.619 Scott: yep. 00:21:04.744 --> 00:21:10.144 So let's come to questions of how we might suppose. 00:21:11.569 --> 00:21:16.579 A professionally league came to you and said, we want to institute 00:21:17.089 --> 00:21:23.539 stopping rules that when the game, you know, reaches a particular point. 00:21:23.539 --> 00:21:25.009 Bill James had a rule. 00:21:25.909 --> 00:21:27.679 We, we should stop it. 00:21:27.679 --> 00:21:30.949 For safety, for sportsmanship. 00:21:32.059 --> 00:21:33.049 How would you do it? 00:21:35.758 --> 00:21:37.558 Nick Berry: Uh, um, 00:21:39.568 --> 00:21:42.748 carefully I would start with a bunch of data. 00:21:43.828 --> 00:21:49.618 I think, I think step one is get every single game that's ever been 00:21:49.618 --> 00:21:53.818 played at that professional level and specifically with data about. 00:21:55.073 --> 00:21:57.203 Probably just the score over time. 00:21:57.653 --> 00:22:03.203 I think one thing you can't do in this is put, and we'll, we'll come 00:22:03.203 --> 00:22:07.283 to this later, but you can't put any preconceived notion about the quality 00:22:07.283 --> 00:22:08.783 of the teams into this calculation. 00:22:08.783 --> 00:22:08.993 Right? 00:22:08.993 --> 00:22:16.283 I think it has to be a hard rule based on some, just like, you know, uh, observable 00:22:16.283 --> 00:22:17.723 score difference or something like that. 00:22:17.723 --> 00:22:18.773 So yeah, you start by 00:22:18.874 --> 00:22:19.504 Scott: Hey, start by. 00:22:19.643 --> 00:22:22.988 Nick Berry: all of these score differences over time, overlay 'em on a plot. 00:22:24.008 --> 00:22:28.148 And then I would probably choose a reasonable time in the game, maybe 00:22:28.148 --> 00:22:29.558 three quarters of the way through. 00:22:30.278 --> 00:22:34.388 Uh, I don't, you don't wanna stop anything, you know, while there's still an 00:22:34.448 --> 00:22:36.308 immense amount of variability in the game. 00:22:36.308 --> 00:22:39.098 So I may start two thirds or three quarters of the way through, or some 00:22:39.098 --> 00:22:43.718 number and, and I draw a cut there on that graph that I was making and say, 00:22:43.718 --> 00:22:52.448 okay, what point on the Y axis, what score differential can I choose that, um. 00:22:53.213 --> 00:22:57.233 First, no one has ever come back from, I'd probably start there 00:22:57.233 --> 00:23:00.533 and say, okay, if you're up 37 points with 10 minutes to play in a 00:23:00.533 --> 00:23:03.083 basketball game, the game is over. 00:23:03.863 --> 00:23:06.833 I think is a useful starting place. 00:23:06.833 --> 00:23:11.873 It's probably not the rule I would end up with, but I would start there and 00:23:11.873 --> 00:23:13.973 then maybe I say, okay, what about 1%? 00:23:14.963 --> 00:23:18.443 Um, I say, okay, what, what point on this line, do you have 00:23:18.443 --> 00:23:20.213 a 1% chance of coming back from? 00:23:20.363 --> 00:23:22.133 And that's probably too big of a number. 00:23:22.613 --> 00:23:27.803 Um, in baseball, every team plays 162 games, so multiply that by, 00:23:28.763 --> 00:23:33.683 you know, what, 15 to get the total number of games played, uh, a year. 00:23:33.743 --> 00:23:38.603 That might not be the right math, but, you know, 1% of those games is still a lot. 00:23:38.603 --> 00:23:40.763 You don't want to, you don't want to have that many flip flops 00:23:40.763 --> 00:23:41.663 of, of things that you stop. 00:23:41.663 --> 00:23:44.363 So I probably choose a number, like, uh, 0.1% 00:23:44.363 --> 00:23:48.653 or something like that, or 0.05% 00:23:48.653 --> 00:23:50.573 of games and draw a line there. 00:23:51.323 --> 00:23:52.493 do this with a curve as well. 00:23:52.493 --> 00:23:56.783 Maybe I do two third, three quarters of the way through five six of the 00:23:56.783 --> 00:24:02.153 way through, and seven eighths of the way through or something like 00:24:02.153 --> 00:24:05.603 that, and draw that same point at those three lines, and you could stop 00:24:05.603 --> 00:24:07.043 for futility in any of those points. 00:24:08.884 --> 00:24:12.574 Scott: So you're getting at something, you're, you're interested in this, at 00:24:12.574 --> 00:24:18.184 looking at different points in the, in the competition where, what is the chance 00:24:18.184 --> 00:24:20.229 the team that's trailing can win the game? 00:24:21.099 --> 00:24:25.929 And you mentioned when that's below 1% or 0.1%, 00:24:26.349 --> 00:24:30.669 that the team that's trailing could come back and win, that this 00:24:30.669 --> 00:24:34.809 is a candidate game for stopping that, that we've reached it. 00:24:35.289 --> 00:24:38.229 Getting to this point of predictive probabilities. 00:24:38.574 --> 00:24:41.994 So you're building this model and you talked about getting all of 00:24:41.994 --> 00:24:47.574 this historical data we have on, on competitions, whatever sport it is 00:24:47.784 --> 00:24:52.374 that you would build that what is the probability this team can win at this, 00:24:52.404 --> 00:24:54.729 uh, from, from this state in the game? 00:24:56.089 --> 00:25:03.049 We have something like that in most of our sports, for example, if, and, and gambling 00:25:03.049 --> 00:25:05.929 is a huge part of sports, uh, within this. 00:25:05.929 --> 00:25:10.849 So you can get this our, our favorite baseball team, Nick and I are 00:25:10.879 --> 00:25:12.829 our fans of the Minnesota Twins. 00:25:13.129 --> 00:25:15.289 They played a game yesterday. 00:25:15.814 --> 00:25:21.334 Uh, it was Wednesday, April 15th, and Minnesota was trailing the 00:25:21.334 --> 00:25:23.554 Red Sox throughout the game. 00:25:23.554 --> 00:25:24.784 They got behind. 00:25:24.964 --> 00:25:28.834 They actually were ahead one to nothing, but they got behind four to, uh, 00:25:28.864 --> 00:25:31.144 sorry, nine to one by the sixth inning. 00:25:32.704 --> 00:25:39.364 ESPN provides win probabilities, and you can graph this across the game. 00:25:39.484 --> 00:25:43.714 So they're doing this, so they're providing win probabilities, and it was 00:25:43.714 --> 00:25:49.444 not long by about the seventh inning in a nine to one game that the probability that 00:25:49.444 --> 00:25:51.599 the twins were gonna win dropped below 1%. 00:25:52.789 --> 00:25:56.899 Now interestingly, they scored four runs in the bottom of the ninth and 00:25:56.899 --> 00:26:02.209 they lost nine to five, but that probability never wavered from 99%. 00:26:02.419 --> 00:26:04.279 That game could have been a candidate. 00:26:04.279 --> 00:26:06.529 Maybe it doesn't reaches the threshold. 00:26:07.669 --> 00:26:13.129 These are really natural things within games to, to look at. 00:26:13.189 --> 00:26:16.639 And then you balance why are we stopping? 00:26:16.639 --> 00:26:18.739 What, what, what are the purpose? 00:26:18.859 --> 00:26:25.759 Uh, to, to understand the risk because you could stop a game that the team 00:26:25.759 --> 00:26:28.579 would've come back and won and. 00:26:28.884 --> 00:26:31.554 That's, that's a mistake to some extent. 00:26:31.554 --> 00:26:33.744 You've made a mistake by stopping it. 00:26:34.734 --> 00:26:40.524 You could also not stop a game that the trailing team lost. 00:26:42.144 --> 00:26:47.784 Now, that's a different kind of mistake where maybe the resources used in the 00:26:47.784 --> 00:26:51.139 remainder of that game were for Naugh. 00:26:53.404 --> 00:26:59.164 Because the game didn't change in that setting, we might consider that if we're 00:26:59.164 --> 00:27:05.044 trying to look at a rule for its ability to stop trials that are gonna lose anyway, 00:27:05.489 --> 00:27:10.744 we, we, we would evaluate that those, those criteria, those errors that we might 00:27:10.744 --> 00:27:18.484 make with a, with a rule to stop sporting competitions, which of course have very 00:27:18.484 --> 00:27:22.019 different goals than clinical trials, which of course we're gonna get to. 00:27:25.324 --> 00:27:26.914 I also have a friend. 00:27:27.029 --> 00:27:27.799 Yeah, go ahead. 00:27:28.103 --> 00:27:31.463 Nick Berry: that, that quantity that you keep talking about, the, well, you've, 00:27:31.523 --> 00:27:34.913 you've referenced both probability of winning, but also predictive probability. 00:27:35.333 --> 00:27:36.083 I think it's interesting. 00:27:36.083 --> 00:27:39.173 The first podcast we did about regression to the mean, I think we 00:27:39.173 --> 00:27:44.603 kept telling people, at least non statisticians, that that's a hard thing 00:27:44.603 --> 00:27:46.913 to, to get your mind around, right? 00:27:46.913 --> 00:27:50.003 You see data and you're like, okay, that's what I'm gonna believe going forward. 00:27:50.003 --> 00:27:51.263 So we were trying to tell people. 00:27:52.538 --> 00:27:56.198 You need to regress this towards the population average, even 00:27:56.198 --> 00:27:57.668 though it feels weird to do. 00:27:57.968 --> 00:28:00.218 I think the predictive probability is the exact opposite. 00:28:00.608 --> 00:28:05.468 Everyone that is making decisions about these games intuitively 00:28:05.468 --> 00:28:06.908 understands what you're talking about. 00:28:07.268 --> 00:28:12.338 They understand like the variability left in the game, and they know why a four run 00:28:12.338 --> 00:28:15.488 lead in the fourth inning is different than a four run lead in the eighth inning. 00:28:15.863 --> 00:28:20.783 And they don't need to describe it as information fractions and variability 00:28:20.783 --> 00:28:22.613 and things like that to make that point. 00:28:22.613 --> 00:28:25.463 But I like this topic because it's 00:28:25.473 --> 00:28:28.853 so intuitive and it's just like the root of decision making in general 00:28:28.853 --> 00:28:31.613 is this like predictive framework that, that we're working under. 00:28:32.063 --> 00:28:32.333 Um. 00:28:34.549 --> 00:28:38.089 Scott: So, so you brought up this idea, and this is gonna be in sports. 00:28:38.089 --> 00:28:39.619 It seems like such a natural thing. 00:28:39.619 --> 00:28:43.669 And by the way, that's the goal of these episodes that you, you take something 00:28:43.669 --> 00:28:47.149 in sports that seems so natural and understand and then you flip it to 00:28:47.149 --> 00:28:51.229 clinical trials, which, you know, that's sort of how I think, and it's not 00:28:51.229 --> 00:28:55.669 uncommon in a clinical trial scenario that I explains something like a sporting 00:28:55.669 --> 00:29:00.049 competition, but being trailing by five. 00:29:00.299 --> 00:29:04.379 Early in the game means something very different than trailing 00:29:04.379 --> 00:29:05.654 by five late in the game. 00:29:07.129 --> 00:29:14.029 Because you have less time to have a large differential to reverse that deficit 00:29:14.029 --> 00:29:16.669 that you have, uh, in, in this scenario. 00:29:16.669 --> 00:29:18.949 It's such a natural thing in sports. 00:29:19.159 --> 00:29:25.609 Where in clinical trials do we bid build futility rules off of the observed effect? 00:29:26.149 --> 00:29:26.809 Do you know? 00:29:26.809 --> 00:29:27.709 How do we do that? 00:29:28.069 --> 00:29:29.929 The wind probability. 00:29:30.329 --> 00:29:32.579 In sports incorporates that. 00:29:33.029 --> 00:29:38.639 So in that game where the Red Sox were playing the twins, the Red Sox were 00:29:38.639 --> 00:29:41.759 up by five runs after five innings. 00:29:43.349 --> 00:29:48.629 The win probability for them under that scenario, uh, after 00:29:48.629 --> 00:29:50.909 the fifth inning is about 90%. 00:29:52.789 --> 00:29:58.669 When the game goes to the sixth inning and the score doesn't change, it's still five. 00:29:58.789 --> 00:30:03.109 The, the probability the Red Sox win goes up because there's 00:30:03.109 --> 00:30:05.389 less time left to flip that. 00:30:05.989 --> 00:30:09.949 So the same observed effect at different times have quite 00:30:09.949 --> 00:30:11.449 different predictive probabilities. 00:30:11.449 --> 00:30:14.779 It's a very natural thing in sports to understand that part 00:30:15.499 --> 00:30:17.149 in in the wind probability. 00:30:19.699 --> 00:30:20.329 Okay. 00:30:21.169 --> 00:30:26.989 So let's flip this to clinical trials, and we've already set this up a little bit. 00:30:27.139 --> 00:30:29.419 So what does this have to do with clinical trials? 00:30:30.349 --> 00:30:31.999 Many clinical trials. 00:30:33.259 --> 00:30:36.649 Are set up much like a sporting competition, 00:30:37.309 --> 00:30:40.159 especially phase three trials. 00:30:40.549 --> 00:30:41.299 Earlier. 00:30:41.299 --> 00:30:47.959 Stage trials can have, uh, uh, goals that are largely learning estimation. 00:30:47.959 --> 00:30:49.519 Those may be a little bit harder. 00:30:50.184 --> 00:30:52.884 You can, you should have futility rules in those settings. 00:30:52.884 --> 00:30:56.874 But let's think about phase three trials because they're the largest, the most 00:30:56.874 --> 00:31:02.634 expensive, but they have very well-defined rules of quote unquote winning that trial. 00:31:03.444 --> 00:31:07.404 They're targeted on demonstrating one treatment is better 00:31:07.404 --> 00:31:07.974 than another treatment. 00:31:08.759 --> 00:31:12.929 In a sporting complex, uh, contest, it's almost like a player has to 00:31:12.929 --> 00:31:16.589 play, you know, the best player in the world to show their better. 00:31:16.589 --> 00:31:19.919 They have to, to to show that they, they can win that game. 00:31:20.759 --> 00:31:24.809 We have very well-defined rules in sports about who wins the game. 00:31:25.799 --> 00:31:29.879 Now, in clinical trials, we also have really well-defined rules that, 00:31:29.879 --> 00:31:34.409 one, we're trying to see that this treatment is statistically significantly 00:31:34.409 --> 00:31:35.879 greater than another treatment. 00:31:38.749 --> 00:31:44.059 So again, thinking about this question, like before we talk about how to do 00:31:44.059 --> 00:31:50.779 this, I asked you the question about why we might stop a sporting competition. 00:31:51.439 --> 00:31:59.569 Why might we stop a clinical trial before the end for futility? 00:32:01.793 --> 00:32:02.543 Nick Berry: Again, I 00:32:02.719 --> 00:32:02.899 Scott: Yes. 00:32:02.933 --> 00:32:04.013 Nick Berry: a lot of reasons. 00:32:04.133 --> 00:32:04.883 Um, 00:32:07.313 --> 00:32:10.193 I, there, there are patient. 00:32:11.918 --> 00:32:19.748 Ethical, um, of the trial and if it's looking like the treatment is, has 00:32:19.748 --> 00:32:24.248 no chance of being, being successful in a clinical trial, you know, is 00:32:24.248 --> 00:32:27.788 it ethical to be randomizing these patients to be experimenting on them? 00:32:28.508 --> 00:32:34.088 Um, there's, for a sponsor, there's certainly monetary constraints. 00:32:34.088 --> 00:32:36.848 Some of these drugs are extremely expensive to administer. 00:32:36.848 --> 00:32:39.698 It's extremely expensive to follow subjects up. 00:32:40.148 --> 00:32:46.388 Um, often can roll over into some open label extension where a drug that 00:32:46.388 --> 00:32:50.048 might be, you know, better for these subjects can be administered to them 00:32:50.048 --> 00:32:51.968 after a trial stops for futility. 00:32:52.028 --> 00:32:55.418 Um, uh, I think there's, there's a ton of different reasons. 00:32:55.448 --> 00:33:00.758 Uh, safety and cost are probably two of the most prevalent. 00:33:04.159 --> 00:33:04.369 Scott: Yep. 00:33:04.459 --> 00:33:08.449 So that there, there are strong reason that if you're running a 00:33:08.449 --> 00:33:13.249 trial and it's very, very unlikely the trial's gonna be successful. 00:33:13.249 --> 00:33:15.409 There are benefits of stopping that trial. 00:33:15.979 --> 00:33:18.949 Uh, and the ethical parts of it are, are huge. 00:33:18.949 --> 00:33:23.809 You're asking a patient to contribute to science, and if you look at the data, 00:33:23.809 --> 00:33:25.849 they're no longer contributing to science. 00:33:25.849 --> 00:33:27.409 The answer's largely been. 00:33:28.069 --> 00:33:31.789 Uh, the, the question's been answered at that point now. 00:33:34.099 --> 00:33:39.859 You can, uh, any futility rule in that trial that, by the way, there are some 00:33:40.099 --> 00:33:44.809 strange trials where mathematically the trial cannot be successful. 00:33:44.809 --> 00:33:49.819 But tho those are very, very rare where it's integer valued outcomes and no 00:33:49.819 --> 00:33:52.039 number of events can flip the outcomes. 00:33:52.069 --> 00:33:54.409 Those, those are different trials. 00:33:54.409 --> 00:33:57.889 Most of these trials, you're now addressing similar type 00:33:57.889 --> 00:34:01.219 questions, and so we are. 00:34:01.504 --> 00:34:07.354 As clinical trial designers asked to create futility rules. 00:34:08.404 --> 00:34:13.804 So you're the NHL is not asking you this or major league baseball, 00:34:13.804 --> 00:34:18.544 but how do we create futility rules for a clinical trial? 00:34:21.608 --> 00:34:21.998 Nick Berry: Yeah. 00:34:21.998 --> 00:34:23.258 Um, let's see. 00:34:23.318 --> 00:34:27.098 So I think it's largely the same as what I described for like the 00:34:27.098 --> 00:34:31.268 NHL rule with probably one major difference is that I wouldn't 00:34:31.268 --> 00:34:32.978 start by collecting a ton of data. 00:34:32.978 --> 00:34:35.618 I don't think I'm gonna start with all the clinical trials that have ever 00:34:35.618 --> 00:34:37.028 been run and looking at their curves. 00:34:37.268 --> 00:34:39.248 I know the data generating process or at. 00:34:39.998 --> 00:34:43.598 have assumptions about the data generating process and what I think is likely, so 00:34:44.558 --> 00:34:51.218 I would probably start by, could think of it as simulating 1 million different 00:34:51.248 --> 00:34:54.398 paths through this, and maybe I think of this at the beginning as I simulate 00:34:54.398 --> 00:34:58.868 every patient or, or I look really often I take benchmarks throughout 00:34:58.868 --> 00:35:05.468 this trial, maybe a hundred benchmarks, and at each of those points I can can. 00:35:06.308 --> 00:35:11.228 I fit a t test, maybe I, you know, do fit some simple test for now, uh, to say, 00:35:11.228 --> 00:35:13.328 how much better is the active arm doing? 00:35:13.328 --> 00:35:14.888 How much worse is the active arm doing? 00:35:14.888 --> 00:35:16.448 And then I go through the same exact pro. 00:35:16.478 --> 00:35:20.258 Now I have those spaghetti plots exactly like I had for sports. 00:35:20.438 --> 00:35:22.238 You know, I had the score differential over time. 00:35:22.238 --> 00:35:23.648 I have treatment effect over time. 00:35:24.368 --> 00:35:26.228 Um, I would probably start again. 00:35:26.348 --> 00:35:28.988 Maybe I don't wanna start until halfway through the trial. 00:35:29.198 --> 00:35:31.898 I would say, okay, at what point is it impossible? 00:35:32.378 --> 00:35:40.148 Did none of my 1 million simulations come back to, to the you know, how, 00:35:40.148 --> 00:35:43.598 where the treatment was so bad that it couldn't get back to being successful? 00:35:43.838 --> 00:35:48.428 And again, like I said with the, the other example, that's not necessarily, 00:35:48.518 --> 00:35:50.108 it's not a good futility rule, right? 00:35:50.108 --> 00:35:53.858 Something that never allows a reversal 00:35:53.984 --> 00:35:54.274 Scott: Yeah. 00:35:54.458 --> 00:35:56.348 Nick Berry: a little too conservative for my liking. 00:35:57.248 --> 00:36:00.368 But that probability of reversing is really important. 00:36:00.368 --> 00:36:05.618 So I again, wanna limit it to maybe something like 1%, or in sports I use 0.1% 00:36:05.618 --> 00:36:07.838 'cause we're doing this over and over and over and over and over 00:36:07.838 --> 00:36:09.008 and I didn't want a bunch of these. 00:36:09.008 --> 00:36:15.968 But, um, in clinical trials, I think a predictive probability of 1% is still 00:36:15.968 --> 00:36:17.738 fairly conservative for futility role. 00:36:17.738 --> 00:36:22.208 But something in the one to 5% range is probably where I would start. 00:36:22.703 --> 00:36:26.513 Trying to cut that probability of reversing after thing. 00:36:26.963 --> 00:36:31.283 And again, I'm, I'm looking at trials that I simulated from start to finish. 00:36:31.583 --> 00:36:35.453 So in reality, I know when I stop a one of those simulated trials for 00:36:35.453 --> 00:36:36.683 fertility, what would've happened. 00:36:36.683 --> 00:36:41.153 And I think that is a really important aspect of the simulations that I get 00:36:41.153 --> 00:36:44.903 this counterfactual look at all of the simulations and, and can do that. 00:36:44.903 --> 00:36:47.213 And so again, I'm drawing a spaghetti plot. 00:36:47.213 --> 00:36:50.333 I'm placing benchmarks that may be, uh. 00:36:51.458 --> 00:36:56.648 three quarters away through the trial and, and myself utility rules at those points. 00:36:56.918 --> 00:37:00.283 So the same exact process with the data generating me mechanism being different. 00:37:02.464 --> 00:37:07.624 Scott: So there's, there's this same quantity that was this win probability 00:37:07.624 --> 00:37:10.384 that we, where you can go to espn.com 00:37:10.384 --> 00:37:11.074 and look at. 00:37:12.199 --> 00:37:16.879 We build that in clinical trials, and you can build that through statistical 00:37:16.879 --> 00:37:19.759 modeling on what is the outcome? 00:37:19.789 --> 00:37:22.249 Is it a, is an event outcome? 00:37:22.519 --> 00:37:25.039 Is it a change from baseline outcome? 00:37:25.039 --> 00:37:27.079 Is it a responder outcome? 00:37:27.559 --> 00:37:31.309 You, you can take that outcome and build. 00:37:31.984 --> 00:37:37.054 A mechanism for calculating what's the probability that the 00:37:37.054 --> 00:37:40.834 treatment is gonna demonstrate benefit by the end of the trial? 00:37:41.284 --> 00:37:43.264 What is its win probability? 00:37:43.564 --> 00:37:46.834 That's something in clinical trials called a predictive probability. 00:37:48.499 --> 00:37:51.199 You, you may have heard of a conditional probability. 00:37:51.199 --> 00:37:53.389 Conditional probability is similar. 00:37:53.599 --> 00:37:58.129 It's a little bit different in that it a conditional probability. 00:37:58.129 --> 00:38:03.709 You assume you know the truth about the team or or what the treatment 00:38:03.709 --> 00:38:05.689 is, and then you calculate that. 00:38:06.169 --> 00:38:10.129 A predictive probability estimates it from the data at 00:38:10.129 --> 00:38:11.989 that point with its uncertainty. 00:38:12.754 --> 00:38:16.414 In calculating the probability of winning the trial going forward. 00:38:16.894 --> 00:38:18.604 So there are different mechanisms. 00:38:18.604 --> 00:38:23.404 Bayesian tends to be predictive probability, where conditional power 00:38:23.404 --> 00:38:28.084 is a pretty simple, uh, taking the observed rate that you see right now, 00:38:28.144 --> 00:38:30.274 usually underrepresents variability. 00:38:30.274 --> 00:38:34.864 So I don't love the number that comes out for conditional probability, which is why 00:38:34.864 --> 00:38:37.384 we use Bayesian predictive probabilities. 00:38:37.954 --> 00:38:39.934 And then we look at that for, its. 00:38:39.994 --> 00:38:42.124 Potential stopping the trial. 00:38:42.124 --> 00:38:47.284 When that probability gets low, you can look at stopping the clinical trial 00:38:49.098 --> 00:38:52.853 Nick Berry: To, to keep our link to sports here, the, you're 00:38:52.853 --> 00:38:55.583 talking about conditional power or conditional probability of 00:38:55.583 --> 00:38:57.263 success versus predictive power. 00:38:57.263 --> 00:38:58.403 Predictive probability of success. 00:39:00.653 --> 00:39:01.433 calculation 00:39:01.444 --> 00:39:02.104 Scott: calculation. 00:39:02.723 --> 00:39:06.623 Nick Berry: is different than the calculation done in a clinical 00:39:06.623 --> 00:39:13.253 trial because in sports, you know, we know a lot about. 00:39:14.108 --> 00:39:18.818 The two teams going in and we that information in a way that's not 00:39:18.818 --> 00:39:20.498 common in phase three clinical trial. 00:39:20.648 --> 00:39:24.158 As Lisa, you were saying, so if I was doing a predictive probability in 00:39:24.158 --> 00:39:30.248 sports and a team's winning, uh, nine to one, like the twins game you were 00:39:30.248 --> 00:39:35.828 describing, um, I am going to shrink my. 00:39:38.123 --> 00:39:42.983 Posterior probability of, you know, the quality of those teams massively 00:39:43.004 --> 00:39:43.494 Scott: Massive 00:39:43.553 --> 00:39:45.653 Nick Berry: being more of a 50 50 game going forward. 00:39:46.133 --> 00:39:47.663 Something that conditional power 00:39:47.819 --> 00:39:48.169 Scott: power. 00:39:48.713 --> 00:39:49.283 Nick Berry: naturally. 00:39:49.309 --> 00:39:49.919 Scott: Naturally. 00:39:51.533 --> 00:39:51.923 Nick Berry: I'm gonna 00:39:52.299 --> 00:39:52.519 Scott: I'm 00:39:53.063 --> 00:39:56.063 Nick Berry: probability of the twins winning, given that the teams are 00:39:56.063 --> 00:39:59.783 equal, is something you could do with a conditional power and get 00:39:59.783 --> 00:40:00.983 that number out in a predictive. 00:40:01.388 --> 00:40:02.558 Probability sense, you would 00:40:02.704 --> 00:40:03.124 Scott: center. 00:40:03.518 --> 00:40:06.158 Nick Berry: probability that the teams are much, are 00:40:06.259 --> 00:40:07.104 Scott: Much our predict. 00:40:07.148 --> 00:40:08.558 Nick Berry: because they're professional sports teams, 00:40:08.558 --> 00:40:09.608 they are pretty close together. 00:40:09.938 --> 00:40:12.668 And even though you observe nine to one doesn't really change what you 00:40:12.668 --> 00:40:14.438 think about the team as a whole. 00:40:14.438 --> 00:40:17.618 It's just that the game is random, so your predictive probability in that 00:40:17.618 --> 00:40:21.668 sense, it's actually probably pretty close to a conditional probability 00:40:21.668 --> 00:40:23.738 of success at that same number. 00:40:23.738 --> 00:40:26.558 And so the, the huge amount of prior information, I think kind 00:40:26.558 --> 00:40:28.838 of changes the decision making of that predictive probability. 00:40:30.349 --> 00:40:34.669 Scott: That, but that can be incorporated in clinical trials where you might 00:40:34.669 --> 00:40:39.889 even want to build a stopping rule that says, we wanna stop when an optimist. 00:40:41.368 --> 00:40:41.658 Nick Berry: Yeah. 00:40:41.989 --> 00:40:47.719 Scott: the drug thinks the probability of winning is below 1% or 5% kind of 00:40:47.719 --> 00:40:54.709 thing, and you use a Bayesian prior, that gives a reasonable probability, an 00:40:54.709 --> 00:41:00.349 optimist probability, uh, um, estimate of the drug's effect, and even when. 00:41:00.724 --> 00:41:04.924 Optimist thinks the trial is unlikely to be successful, you stop. 00:41:04.924 --> 00:41:11.584 Is is by the way, it, it alludes to this in the, the Bayesian, the Bayesian 00:41:11.584 --> 00:41:16.234 draft guidance that you can use this design prior, uh, sort of thing. 00:41:16.234 --> 00:41:17.794 You can do this for the futility. 00:41:17.794 --> 00:41:21.574 So it's something that can be incorporated in clinical trials for sure. 00:41:23.738 --> 00:41:24.038 Nick Berry: Yep. 00:41:24.158 --> 00:41:24.968 That makes total sense. 00:41:26.044 --> 00:41:27.124 Scott: Now, some of the. 00:41:27.158 --> 00:41:27.398 Nick Berry: idea. 00:41:28.654 --> 00:41:33.964 Scott: Uh, yeah, I, Jacob Jay Cade's idea actually is a really nice idea that 00:41:33.964 --> 00:41:42.364 you take multiple, uh, uh, clinicians with varying views across the spectrum. 00:41:42.724 --> 00:41:48.489 And when all of those clinicians are convinced of it, then you stop. 00:41:49.519 --> 00:41:53.029 So you might need to go longer to demonstrate that you think the 00:41:53.029 --> 00:41:55.579 drug is effective to a pessimist 00:41:55.973 --> 00:41:56.183 Nick Berry: Yeah. 00:41:56.569 --> 00:42:01.459 Scott: likewise to stop when, when you think it's not effective as well. 00:42:01.549 --> 00:42:03.499 It, it's a really neat idea actually. 00:42:04.129 --> 00:42:04.429 Um. 00:42:05.404 --> 00:42:08.014 Now this the, this characteristic. 00:42:08.374 --> 00:42:13.834 We spend a great deal of time simulating different rules and we calculate those 00:42:13.834 --> 00:42:17.074 error rates that we talked about before. 00:42:17.074 --> 00:42:19.084 So you can make a mistake with a futility rule. 00:42:20.389 --> 00:42:27.319 You could have a futility rule that stops a trial that would've gone on and flipped 00:42:27.319 --> 00:42:31.669 and reversed, and you say There's very little chance for the treatment to win. 00:42:32.089 --> 00:42:34.099 And we calculate that's 1%. 00:42:34.099 --> 00:42:37.969 For example, we, we've done trials where the predictive probability of 00:42:37.969 --> 00:42:44.419 success of that trial is 2% and it stops for futility has a 5% futility rule. 00:42:44.599 --> 00:42:49.009 The trial stops by, by, by interpretation That. 00:42:49.154 --> 00:42:52.124 There's a 2% chance that trial could have come back and been 00:42:52.124 --> 00:42:53.684 successful under that scenario. 00:42:54.104 --> 00:42:55.694 It's just deemed that. 00:42:56.614 --> 00:43:01.054 Taking into account the pluses and minuses of this, that it's 00:43:01.054 --> 00:43:02.584 a good thing to stop that trial. 00:43:02.584 --> 00:43:06.694 Now that decision is made ahead of time, uh, in those circumstance, 00:43:06.694 --> 00:43:09.274 and we simulate multiple rules. 00:43:09.274 --> 00:43:10.774 We look at error rates. 00:43:11.344 --> 00:43:15.574 We also look at how, you know, what happens to power when you have a 00:43:15.574 --> 00:43:19.984 futility rule for a treatment, you. 00:43:20.404 --> 00:43:24.814 You may decrease its chance of success because some of those trials, those 00:43:24.814 --> 00:43:29.314 spaghetti plots that Nick talked about, they hit the futility rule and they 00:43:29.314 --> 00:43:31.264 would've reversed and gone on to win. 00:43:31.264 --> 00:43:36.634 You reduce power by that if you have a very, very aggressive futility rule. 00:43:37.744 --> 00:43:41.764 An example of this would be halfway through the trial. 00:43:42.004 --> 00:43:46.114 If the predictive probability of the treatment winning is less than 00:43:46.114 --> 00:43:48.039 50%, we're gonna stop the trial. 00:43:50.099 --> 00:43:53.134 That would be a strikingly aggressive futility rule. 00:43:53.134 --> 00:43:56.674 Essentially, it means that the treatment is observing something very 00:43:56.674 --> 00:44:01.294 close to what it needs to win by the end, which gives it kind of 50 50. 00:44:01.294 --> 00:44:02.704 If it stays there, it wins. 00:44:02.704 --> 00:44:07.444 If not, it loses and you stop the trial there, you're gonna reduce 00:44:07.444 --> 00:44:10.024 power by very large amounts. 00:44:10.324 --> 00:44:13.084 Uh, Nick and I are looking at example when that. 00:44:13.084 --> 00:44:18.484 That probability can drop power by 25% in your, your, your effect size 00:44:18.874 --> 00:44:22.684 because you're being so aggressive at stopping it and you don't give 00:44:22.684 --> 00:44:26.194 the treatment that, that chance to, to, to go on and be successful. 00:44:27.289 --> 00:44:27.409 I. 00:44:28.309 --> 00:44:33.529 A 20% rule, by the way, and I'm gonna say 20% because there's an interesting story. 00:44:33.529 --> 00:44:37.159 You know, futility story to this in, in the example we're looking 00:44:37.159 --> 00:44:41.809 at can have somewhere a five to 10% reduction in power, 20%. 00:44:41.809 --> 00:44:46.759 If you think about a sports competition where the team only 00:44:46.759 --> 00:44:48.739 has a 20% chance of winning. 00:44:49.489 --> 00:44:50.689 That happens a lot. 00:44:51.289 --> 00:44:55.369 Uh, those are exciting games by the way, where a team scores two 00:44:55.369 --> 00:44:57.589 touchdowns very close to the end to win. 00:44:57.949 --> 00:45:02.629 Uh, golfer comes back from five shots back on the last nine holes. 00:45:03.259 --> 00:45:07.699 I that's not even 20% coming back by three shots is sort of 20%. 00:45:08.089 --> 00:45:10.339 That's an, that's a common occurrence. 00:45:11.149 --> 00:45:13.399 Well, that rule is used in futility. 00:45:13.759 --> 00:45:15.289 For futility in trials. 00:45:15.679 --> 00:45:21.259 And there's a famous example when that rule was used, and it's a famous example 00:45:21.259 --> 00:45:25.399 where people think futility might not be a good thing in clinical trials. 00:45:25.729 --> 00:45:27.109 And we'll kind of get to that. 00:45:27.109 --> 00:45:34.309 It's the Biogen example of Aducanumab It was an Alzheimer's treatment and 00:45:34.309 --> 00:45:40.129 this, this was, you know, 5, 6, 7 years ago, uh, Alzheimer's treatment, uh. 00:45:41.134 --> 00:45:43.444 A disease modifying therapy. 00:45:43.444 --> 00:45:49.414 They're running two phase 3 trials and they did a futility analysis 00:45:49.414 --> 00:45:54.964 where they looked at both trials, a predefined futility analysis, and 00:45:54.964 --> 00:46:00.514 they looked at both trials and they stopped both trials for futility. 00:46:03.064 --> 00:46:06.244 Interestingly, they stopped those trials, but they got. 00:46:07.024 --> 00:46:12.784 Follow up data on one of the trials turned out to be statistically 00:46:12.784 --> 00:46:15.514 significant in Alzheimer's. 00:46:15.514 --> 00:46:17.044 That's a huge deal. 00:46:17.044 --> 00:46:22.984 The other trial was not, and there was a lot of discussion about this was a bad 00:46:22.984 --> 00:46:26.374 futility rule, the these types of things. 00:46:26.824 --> 00:46:31.474 Their rule for futility was that if either of the two trials had 00:46:31.474 --> 00:46:34.624 less than a 20% chance of success. 00:46:35.029 --> 00:46:36.469 They would stop both. 00:46:38.299 --> 00:46:42.709 And if you think about that in the sports comp, uh, context that's a 00:46:42.709 --> 00:46:45.169 really, really aggressive futility rule. 00:46:46.249 --> 00:46:50.089 Now, it was a bit more complicated because one of the trials, the 00:46:50.089 --> 00:46:54.414 conditional probability of the one that won was 60% when they stopped. 00:46:56.194 --> 00:47:01.654 So it was a controversial application of futility rules in this scenario. 00:47:01.654 --> 00:47:02.854 I mean, in some extent it was right. 00:47:02.854 --> 00:47:04.834 The other trial was not successful. 00:47:05.344 --> 00:47:08.494 But in hindsight, I think Biogen would've done it differently 00:47:08.494 --> 00:47:10.954 and they were not necessarily pleased with the futility rule. 00:47:12.289 --> 00:47:13.489 In, in that scenario. 00:47:13.489 --> 00:47:22.009 Now, I did a debate with Paul Eisen from, uh, USC on whether we should be doing 00:47:22.009 --> 00:47:25.399 Al Futility rules in Alzheimer's trials. 00:47:26.629 --> 00:47:31.669 Now you can go get that and you can find that debate and I did win the debate. 00:47:31.849 --> 00:47:36.499 Uh, 80% to 20% people agreed that we should be doing futility 00:47:36.499 --> 00:47:38.539 rules in Alzheimer's trials. 00:47:39.019 --> 00:47:39.704 Uh, I feel like. 00:47:40.474 --> 00:47:44.194 If I would've lost that, I, I feel like it's such a yes, we 00:47:44.194 --> 00:47:46.054 should absolutely be doing that. 00:47:46.204 --> 00:47:51.634 We should be doing futility rules in every phase three trial that, uh, if 00:47:51.634 --> 00:47:56.854 I'd have lost that, that would've been like Rory losing, uh, leading by six 00:47:56.854 --> 00:48:00.364 after 36 holes, uh, sort of scenario. 00:48:03.103 --> 00:48:05.948 Nick Berry: The, the Aducanumab example is. 00:48:06.893 --> 00:48:09.383 Interesting because earlier while we were talking, I was thinking about 00:48:09.383 --> 00:48:14.093 how the value you choose for your fu utility rule almost creates this sort 00:48:14.093 --> 00:48:18.923 of implied utility on what you're valuing in your clinical trial. 00:48:18.953 --> 00:48:24.023 Um, you know, if you have an aggressive FU utility rule, you're probably saying 00:48:24.023 --> 00:48:25.403 it's really expensive to continue. 00:48:25.403 --> 00:48:28.103 I only want to spend this extra money if it's likely that we can win. 00:48:28.343 --> 00:48:34.673 So the, the implied utility of that futility rule is something like. 00:48:35.843 --> 00:48:41.393 We need two successful phase three trials or bust, is sort 00:48:41.393 --> 00:48:42.503 of the implication of that. 00:48:42.503 --> 00:48:44.423 And that was probably what was going through their mind while 00:48:44.423 --> 00:48:45.473 they're constructing the rule. 00:48:45.473 --> 00:48:48.293 Like, we need, in order to get this approved, we need two phase 00:48:48.293 --> 00:48:51.323 three trials to meet this threshold and the FDA will approve us. 00:48:51.323 --> 00:48:54.413 And you know, clearly that's not what happened at the end 00:48:54.413 --> 00:48:57.443 of the day, but uh, you can see how they got there a little bit. 00:48:57.443 --> 00:49:01.433 And I think the utility that comes out of this is out of your 00:49:01.433 --> 00:49:03.413 futility rule is interesting. 00:49:03.593 --> 00:49:04.193 Um, and. 00:49:04.973 --> 00:49:08.753 You know, you say a lot by not actually saying anything on purpose. 00:49:09.769 --> 00:49:10.339 Scott: Yeah, no. 00:49:10.339 --> 00:49:14.749 We spend a great deal of time building these predictive probabilities in that 00:49:14.749 --> 00:49:20.209 scenario where you have two phase three trials running at exactly the same time. 00:49:20.749 --> 00:49:24.949 The idea that you do conditional power only in that trial and you 00:49:24.949 --> 00:49:26.809 ignore the data from the other trial. 00:49:27.148 --> 00:49:27.438 Nick Berry: Yeah. 00:49:27.539 --> 00:49:31.469 Scott: The fact that the other trial had a 60% chance of winning meant 00:49:31.469 --> 00:49:33.329 that it had a good effect size. 00:49:33.779 --> 00:49:37.289 If you would've done regression to the mean and shrunk over the two 00:49:37.289 --> 00:49:40.229 trials, you probably would've got a much higher probability for the 00:49:40.229 --> 00:49:44.429 second trial of being successful, and then maybe that would've eventually 00:49:44.429 --> 00:49:46.349 been successful had they run it out. 00:49:46.349 --> 00:49:46.979 To the end. 00:49:47.789 --> 00:49:52.259 So a huge part of what we do as statistical consultants is build good 00:49:52.259 --> 00:49:57.869 predictive probabilities so that we're making good decisions in almost every 00:49:57.869 --> 00:50:04.709 clinical trial, like Alzheimer's, like stroke, uh, uh, it, uh, oncology. 00:50:05.889 --> 00:50:10.659 You have information on patients that start of the primary endpoint. 00:50:11.019 --> 00:50:14.739 They're not all as simple as success and failure and you know everything 00:50:14.739 --> 00:50:16.089 about everything in the trial. 00:50:16.359 --> 00:50:18.759 You have incomplete information on this. 00:50:18.759 --> 00:50:21.699 So we're building predictive probabilities that are using 00:50:21.999 --> 00:50:23.949 maybe dose response modeling. 00:50:23.954 --> 00:50:27.759 They're using longitudinal modeling of early clinical 00:50:27.759 --> 00:50:29.139 outcomes to later clinical items. 00:50:29.759 --> 00:50:33.929 Based on their predictability, we spent a ton of time getting those 00:50:33.929 --> 00:50:39.119 predictive probabilities to be really good so that they make really 00:50:39.119 --> 00:50:41.309 good decisions in clinical trials. 00:50:41.369 --> 00:50:46.619 And I suspect that was not optimally done in the Biogen example. 00:50:50.344 --> 00:50:50.974 Okay. 00:50:51.184 --> 00:50:56.404 Um, I, I do wanna make reference to a nice paper by our colleagues, Roger 00:50:56.404 --> 00:51:01.624 Lewis and Barbara Berger In jama, there's a, a futility in clinical trials. 00:51:01.804 --> 00:51:07.024 There's a JAMA series for, uh, methodology in clinical trials, 00:51:07.024 --> 00:51:08.434 and they write about futility. 00:51:08.434 --> 00:51:10.204 You can read more about those. 00:51:11.524 --> 00:51:16.384 But unfortunately Nick, I think we've hit the futility for, for this episode. 00:51:17.608 --> 00:51:17.828 Nick Berry: Yep. 00:51:18.424 --> 00:51:25.144 Scott: Uh, uh, uh, uh, and, and that's more time and resources based here, uh, 00:51:25.144 --> 00:51:26.794 that than the fact that we're losing. 00:51:27.563 --> 00:51:28.253 Nick Berry: Yeah, right. 00:51:28.624 --> 00:51:29.134 Scott: Yes. 00:51:29.164 --> 00:51:30.454 We, we are not losing. 00:51:30.454 --> 00:51:32.494 Yes, yes, yes. 00:51:32.914 --> 00:51:38.884 Uh, so we appreciate you all joining Nick and me in this deeper dive into 00:51:38.884 --> 00:51:44.524 the intersection of sports statistics and the science of clinical trials. 00:51:45.094 --> 00:51:47.434 And until the next time, we'll be here in the interim.