WEBVTT NOTE This file was generated by Descript 00:00:00.000 --> 00:00:03.279 Judith: Welcome to Berry's In the Interim podcast, where we explore the 00:00:03.279 --> 00:00:07.639 cutting edge of innovative clinical trial design for the pharmaceutical and 00:00:07.639 --> 00:00:09.920 medical industries, and so much more. 00:00:10.819 --> 00:00:11.460 Let's dive in. 00:00:12.535 --> 00:00:16.045 Scott Berry: Well welcome everybody back to In the interim. 00:00:16.135 --> 00:00:18.835 I am Scott Berry, your host for today. 00:00:19.345 --> 00:00:21.685 I am flying solo today. 00:00:22.360 --> 00:00:24.700 I have a topic I thought I'd fly solo with. 00:00:24.700 --> 00:00:27.970 First I'm gonna, I'm gonna jump in from the sports world and then we're 00:00:27.970 --> 00:00:29.590 gonna talk about clinical trial. 00:00:29.590 --> 00:00:30.490 Simulation. 00:00:31.360 --> 00:00:36.550 I, it, it's, it's something that everybody views differently. 00:00:36.550 --> 00:00:41.380 I think there's a lot of misunderstanding of what clinical trial simulation is 00:00:41.380 --> 00:00:43.900 within clinical trial design world. 00:00:43.900 --> 00:00:48.460 And I, I want to talk about that because I think there's sufficient misunderstanding. 00:00:49.030 --> 00:00:53.830 Even among statisticians about what it is as a tool, how it's 00:00:53.830 --> 00:00:56.260 used, what that looks like. 00:00:56.260 --> 00:00:57.700 So I want to talk about that. 00:00:58.135 --> 00:01:02.185 And, but first, I, I have to, as a statistician, as we, 00:01:02.605 --> 00:01:04.555 we observe things going on. 00:01:04.555 --> 00:01:09.505 I, I do want to talk about the sports world and those that, um, 00:01:09.565 --> 00:01:11.875 uh, know, me, know I love sports. 00:01:11.965 --> 00:01:16.015 Uh, my, my initial interest in statistics was in sports. 00:01:16.525 --> 00:01:21.445 Uh, when I was in academia, I, I, I wrote about it for Chance Magazine. 00:01:21.445 --> 00:01:22.375 I wrote a quarterly. 00:01:22.865 --> 00:01:28.205 Article, uh, called a statistician reads the sports pages, where I got to 00:01:28.205 --> 00:01:32.255 pick a different topic and write about it, explore it from a, a statistical 00:01:32.255 --> 00:01:36.815 perspective, and I loved it and I wish I had the time to do it again now, but 00:01:37.355 --> 00:01:43.085 from the sports world, I want to bring up the, the oddity that is Scotty Scheffler. 00:01:44.155 --> 00:01:45.985 So I, I enjoy golf. 00:01:45.985 --> 00:01:49.015 I play golf, and I follow golf. 00:01:49.285 --> 00:01:56.275 And hey, I, I, I want you to appreciate how stunningly good he is as an 00:01:56.275 --> 00:01:59.635 outlier, a statistical oddity within it. 00:01:59.635 --> 00:02:06.355 And, and I, I am particularly interested in those elite athletes that 00:02:06.355 --> 00:02:08.845 separate themselves from the curve. 00:02:09.235 --> 00:02:11.830 The babe Ruths the Wayne Gretzkys, the. 00:02:12.260 --> 00:02:18.200 The, the Tiger Woods, Jack Nicholas, these athletes, and, and, and there, 00:02:18.205 --> 00:02:23.480 there, there, there are not a lot of them by, by definition, but 00:02:23.480 --> 00:02:25.580 that really separate themselves. 00:02:25.790 --> 00:02:31.730 And what's so stunning to me is how good he is within a game that it. 00:02:32.545 --> 00:02:35.785 It's darn near impossible to be as good as he is. 00:02:35.785 --> 00:02:40.945 So I want you to think about the, the Tour Championship is coming up this weekend 00:02:41.065 --> 00:02:43.165 and let you know when we're taping this. 00:02:43.555 --> 00:02:49.855 And there are 30 golfers in that, that perhaps the best, 30 golfers in the world. 00:02:49.855 --> 00:02:53.515 And you can argue that there's a few missing, but largely it's the best. 00:02:53.515 --> 00:02:54.775 30 golfers in the world. 00:02:55.105 --> 00:02:57.295 Every one of these golfers. 00:02:58.180 --> 00:03:01.180 Is dedicated to this game. 00:03:01.180 --> 00:03:02.440 They are all elite. 00:03:02.500 --> 00:03:05.890 They were all the best in college when they were there. 00:03:05.890 --> 00:03:08.860 They, they all had phenomenal years. 00:03:09.070 --> 00:03:10.750 They all work at their game. 00:03:10.750 --> 00:03:13.945 They all train at this elite level in terms of golf. 00:03:15.435 --> 00:03:17.265 And there are 30 of them. 00:03:17.265 --> 00:03:20.565 So, and, and they, they play in the tour championship and 00:03:20.565 --> 00:03:23.535 they all start from, from zero. 00:03:24.225 --> 00:03:29.355 And estimates of Scotty Scheffler, odds of winning this tournament 00:03:29.385 --> 00:03:32.445 are 31%, something like that. 00:03:32.985 --> 00:03:37.785 And among 30 golfers, where the average percent here is 3%. 00:03:38.565 --> 00:03:43.575 And all of these elite golfers, he has something like a 31%. 00:03:43.795 --> 00:03:45.115 Quantitative estimate. 00:03:45.115 --> 00:03:48.805 I mean, you can argue about what this is, but the odds of him winning 00:03:49.075 --> 00:03:53.875 this are related to when Tiger Woods was dominating golf at this level. 00:03:54.295 --> 00:03:59.695 It is stunning to me that somebody in, in that game can be that 00:03:59.695 --> 00:04:01.525 much better than everybody else. 00:04:01.705 --> 00:04:05.365 It's, it's hard to actually fathom him being that good. 00:04:05.875 --> 00:04:11.455 He has this unbelievable streak of finishing in the top 25, 00:04:11.755 --> 00:04:13.975 I think it's now 18 in a row. 00:04:14.215 --> 00:04:16.676 Golf tournaments where he's finish in the top 25. 00:04:17.595 --> 00:04:23.895 There are 140 professional golfers that are all elite at what they do. 00:04:24.135 --> 00:04:28.545 They're all in the right hand curve of the normal distribution, and 00:04:28.545 --> 00:04:34.275 here he is finishing in the top 25 18 straight times five wins. 00:04:34.275 --> 00:04:39.435 He's really done this for three years, so I I, if you appreciate 00:04:39.435 --> 00:04:43.875 statistical oddities and rare things. 00:04:44.815 --> 00:04:48.235 Watch Scotty Scheffler it, it, it's, it's unbelievable. 00:04:48.235 --> 00:04:54.025 And he sort of fits into this, this curve of, of elite players. 00:04:54.085 --> 00:04:56.605 So that, that was on my mind as a statistician. 00:04:57.535 --> 00:05:02.395 We can actually come back to the topic for today of simulation and maybe something 00:05:02.395 --> 00:05:04.255 I did around Tiger Woods around that. 00:05:04.675 --> 00:05:05.695 Um, uh, okay. 00:05:05.695 --> 00:05:08.125 I, I, I can't say that and not say something about it. 00:05:08.125 --> 00:05:13.075 So, when Tiger Woods was winning all those golf tournaments and he was a, a freakish. 00:05:13.495 --> 00:05:18.955 Uh, athlete on the, the, the incredible right hand scale of 00:05:18.955 --> 00:05:21.955 the player performance in that. 00:05:22.015 --> 00:05:28.495 And there was this impression that he, he, he could, will himself to win at times. 00:05:29.215 --> 00:05:36.295 So I, I simulated a golf tournaments of Tiger Woods where we, I estimated 00:05:36.295 --> 00:05:41.305 his ability relative to the other players and I simulated his career. 00:05:42.175 --> 00:05:47.545 And he won almost exactly the number of golf tournaments 00:05:47.905 --> 00:05:50.695 that Robo Tiger simulated. 00:05:50.695 --> 00:05:51.835 Tiger won. 00:05:53.005 --> 00:05:58.435 Only using the estimates of all of the PGA tour golfers, you can fit really 00:05:58.435 --> 00:06:03.805 nice normal distribution models to PGA tour golfers for their, their 18 hole 00:06:03.805 --> 00:06:05.695 scores or their tournament scores. 00:06:06.085 --> 00:06:09.655 Uh, it's slightly right skewed, but largely it fits very 00:06:09.655 --> 00:06:11.605 well for a PGA tour golfer. 00:06:11.935 --> 00:06:18.415 And when I resimulate his career, he won almost exactly the same number of 00:06:18.415 --> 00:06:21.655 tournaments simulated on average that. 00:06:22.150 --> 00:06:24.580 The, the simulation showed he should win. 00:06:24.670 --> 00:06:26.020 So what what does that mean? 00:06:26.260 --> 00:06:29.080 Largely it means he was just better than everybody. 00:06:29.320 --> 00:06:32.230 He didn't will himself to win golf tournaments. 00:06:32.290 --> 00:06:34.540 He was just better than everybody else. 00:06:34.540 --> 00:06:38.470 Now we have, we, we, he had amazing shots and the progress in all of this. 00:06:38.710 --> 00:06:41.440 And when you're better than everybody, it sort of feels like. 00:06:41.755 --> 00:06:45.505 But I, I'm not sure there was any intangible to it. 00:06:45.505 --> 00:06:50.635 And he didn't win any more than what just his raw ability said he should win. 00:06:50.815 --> 00:06:53.725 And it would be interesting to do that with Scotty Scheffler as well. 00:06:54.115 --> 00:06:59.485 So simulation, that's an interesting aspect of simulation and a lot of 00:06:59.485 --> 00:07:02.755 people see simulation in that route. 00:07:03.235 --> 00:07:08.065 The NFL season's about to start here and we could simulate 00:07:08.065 --> 00:07:09.775 what's the chance that different. 00:07:10.360 --> 00:07:14.110 Uh, teams win the Super Bowl and this is done. 00:07:14.110 --> 00:07:16.630 You can go on ESPN and read about this. 00:07:17.020 --> 00:07:22.150 You can read about Hurricane Aaron and the prediction of where 00:07:22.150 --> 00:07:23.860 Hurricane Aaron is gonna go. 00:07:23.860 --> 00:07:27.520 It's right now, it's off the East Coast and the prediction is gonna go out to the 00:07:27.520 --> 00:07:30.280 Atlantic Ocean, but how close does it get? 00:07:30.280 --> 00:07:31.180 What kind of havoc? 00:07:31.180 --> 00:07:32.650 And this is all simulation. 00:07:32.650 --> 00:07:36.430 And you can read the European model, you can read other models of it. 00:07:36.850 --> 00:07:37.030 So. 00:07:37.705 --> 00:07:46.285 The general public sees simulation as prediction and it's ingrained in people's 00:07:46.285 --> 00:07:51.745 thought that, and I talked about it for Tiger Woods, predicting what, how 00:07:51.745 --> 00:07:58.015 many wins he would have if we only simulate independent IID observations. 00:07:58.015 --> 00:08:01.075 How much should he win and does he win more or less than that? 00:08:02.065 --> 00:08:03.085 That's a prediction. 00:08:03.325 --> 00:08:04.675 Hurricanes are prediction. 00:08:04.675 --> 00:08:07.855 We use models and and simulation to, to predict them. 00:08:08.545 --> 00:08:10.555 PKPD scientists. 00:08:10.615 --> 00:08:13.975 Getting back to in the interim here where we live in the world of clinical 00:08:13.975 --> 00:08:16.045 trials and medical decision making. 00:08:16.495 --> 00:08:21.445 PKPD scientists use clinical trial simulation to predict what's gonna 00:08:21.445 --> 00:08:25.915 happen when you put a certain amount of drug with a certain amount 00:08:25.915 --> 00:08:27.775 of frequency into a human being. 00:08:28.075 --> 00:08:34.165 Of a certain size and characteristic it's prediction and they talk about simulation. 00:08:34.165 --> 00:08:38.275 You can go on and see all the PKPD tools, which is simulating what's 00:08:38.275 --> 00:08:40.075 gonna happen in the outcome of it. 00:08:41.545 --> 00:08:45.145 I use clinical trial simulation in a very different way. 00:08:45.160 --> 00:08:45.250 Right. 00:08:46.090 --> 00:08:50.650 And I use it for creating a clinical trial design, and I think 00:08:50.650 --> 00:08:52.600 it's very, very misunderstood. 00:08:53.110 --> 00:08:56.290 So, uh, when I talk about it, I'm gonna talk about how do I use it? 00:08:56.290 --> 00:09:03.040 So how do I use clinical trial simulation when there's a particular design proposed 00:09:03.040 --> 00:09:10.480 for a clinical trial, a phase two trial with four doses, 50 patients per dose. 00:09:11.005 --> 00:09:17.605 I can create software that simulates patients coming into the trial, 00:09:18.115 --> 00:09:20.125 being randomized to a dose. 00:09:20.125 --> 00:09:25.735 Just like the protocol says, I could simulate an interim analysis happening 00:09:25.735 --> 00:09:28.165 with some number of patients in the trial. 00:09:28.405 --> 00:09:35.545 You can track real time of this, so it is creating software that can simulate 00:09:35.545 --> 00:09:38.065 the outcomes of clinical trials. 00:09:38.860 --> 00:09:47.020 Exactly as designed under it, and I can record what happens in the trial 00:09:47.920 --> 00:09:50.860 now when I simulate clinical trials. 00:09:50.860 --> 00:09:57.730 I play the role of Mother Nature, where I say, assume all of the 00:09:58.240 --> 00:10:00.760 treatments are the same as placebo. 00:10:01.930 --> 00:10:03.700 What would happen in this trial? 00:10:05.155 --> 00:10:11.755 Assume that, uh, one of the doses has a strong effect and the other doses, 00:10:11.755 --> 00:10:13.735 it's linearly increasing to that effect. 00:10:14.005 --> 00:10:18.835 So I play the role of that and I plug in, in the simulation. 00:10:18.835 --> 00:10:22.195 That's how I wanna simulate the outcome of patients. 00:10:23.455 --> 00:10:28.525 Then I simulate the trial and I can record incredibly detailed 00:10:28.525 --> 00:10:30.265 information about this trial. 00:10:31.420 --> 00:10:35.380 I can record the nu amount of drug use, the time in the trial 00:10:35.380 --> 00:10:37.030 according to enrollment rates. 00:10:37.030 --> 00:10:38.530 Projected enrollment rates. 00:10:38.740 --> 00:10:43.360 I can simulate random enrollment in the trial, a process of enrollment 00:10:43.360 --> 00:10:49.690 in the trial, but I can record which dose wins the trial has, has, has the 00:10:49.690 --> 00:10:52.090 best effect on the efficacy endpoint. 00:10:53.410 --> 00:10:57.940 I can use different analysis methods of the data to estimate. 00:10:58.240 --> 00:11:03.910 The ED 90 dose in the trial, I can change the randomization 00:11:03.910 --> 00:11:09.460 probabilities and simulate the same characteristics I could simulate. 00:11:09.460 --> 00:11:12.850 What's the probability based on that outcome, that you would go 00:11:12.850 --> 00:11:14.590 forward into a phase three trial? 00:11:14.620 --> 00:11:15.940 You would meet, go, no-go. 00:11:16.360 --> 00:11:18.700 Now you have to ask those questions. 00:11:18.700 --> 00:11:20.200 You have to ask the team. 00:11:20.830 --> 00:11:23.560 Under what circumstances would you go to phase three? 00:11:24.475 --> 00:11:27.415 By the way, you have to ask them, is that a good decision? 00:11:27.415 --> 00:11:30.805 Suppose the truth of the drug, were this, do you wanna go? 00:11:30.835 --> 00:11:32.215 Would you want to go to phase three? 00:11:32.875 --> 00:11:37.285 So what's nice about this is you start asking them the real questions, 00:11:37.285 --> 00:11:41.365 by the way, in their language, when would you go to phase three? 00:11:42.115 --> 00:11:44.965 What would you have to see on the primary endpoint? 00:11:45.175 --> 00:11:48.265 And they might say, you know, I need to see that the secondary 00:11:48.265 --> 00:11:49.705 endpoint is doing the following. 00:11:49.825 --> 00:11:50.935 You can simulate that. 00:11:51.910 --> 00:11:55.960 We can simulate a tolerability or a safety endpoint and they 00:11:55.960 --> 00:11:59.230 say, you know, I, this is what I need to see to go to phase three. 00:11:59.800 --> 00:12:04.480 And then the predictive probability of phase three based on the efficacy 00:12:04.480 --> 00:12:09.370 estimate needs to be above some number for a 400 patient phase three trial. 00:12:10.060 --> 00:12:12.460 We get all kinds of different answers, and that's okay. 00:12:12.460 --> 00:12:18.130 There's no right answer to that, but we can record incredibly detailed information 00:12:18.130 --> 00:12:20.770 about the outcome of that simulated trial. 00:12:23.015 --> 00:12:27.965 And then we can repeat that simulation for many, many trials. 00:12:29.335 --> 00:12:32.365 Now if you're a statistician, this seems why. 00:12:32.365 --> 00:12:34.345 Why is, why is Scott talking this way? 00:12:34.345 --> 00:12:35.815 This is so straightforward. 00:12:35.815 --> 00:12:36.565 I understand this. 00:12:36.805 --> 00:12:39.055 This is how statisticians think. 00:12:39.475 --> 00:12:46.465 We think about the sample space and the parameters, and we understand this trial 00:12:46.465 --> 00:12:51.505 is a random of occurrence from a set of random possible outcomes of trials 00:12:51.895 --> 00:12:54.595 and simulation seems so natural to us. 00:12:54.685 --> 00:12:58.195 Remember, almost everybody you're talking to thinks it's prediction. 00:12:58.360 --> 00:13:00.460 And they're thinking it's something else. 00:13:00.610 --> 00:13:04.390 And a lot of people say that as we start to work on the design, 00:13:04.690 --> 00:13:07.690 oh, you know what's, what's the chance we win this trial? 00:13:07.840 --> 00:13:10.900 And that's not directly what we're we're addressing. 00:13:11.170 --> 00:13:14.320 By the way, I'll come back to this because you can use simulation 00:13:14.320 --> 00:13:15.940 tools to answer that question. 00:13:16.295 --> 00:13:19.205 But what we're doing is we're simulating the trial. 00:13:19.205 --> 00:13:22.925 We're recording outcomes of the trials based on different 00:13:22.925 --> 00:13:26.225 truths of the arms in the trials. 00:13:26.225 --> 00:13:28.865 The characteristics in the trial enrollment rate in the 00:13:28.865 --> 00:13:32.795 trial could matter to that if you're doing an adaptive design. 00:13:34.145 --> 00:13:39.755 But we, what's very misunderstood, I think, among statisticians that don't live 00:13:39.755 --> 00:13:44.705 this world, is we don't pick a design. 00:13:45.895 --> 00:13:49.225 And then simulate it and say, oh, we're done. 00:13:49.855 --> 00:13:53.125 You know, go, go off and simulate this and, you know, fill in the 00:13:53.125 --> 00:13:59.575 protocol, the number that's, that's not how we utilize this as a tool. 00:14:01.645 --> 00:14:06.715 We we're able to simulate the trial under different designs, different randomization 00:14:06.715 --> 00:14:11.875 ratios, different number of arms, and we have a very rich set of outcomes. 00:14:12.445 --> 00:14:15.475 We use it to compare different designs. 00:14:16.765 --> 00:14:17.155 Okay? 00:14:17.155 --> 00:14:23.965 Suppose this is the truth and I run design A or design b, I now have this 00:14:23.965 --> 00:14:27.625 ability to compare on very rich outcomes. 00:14:28.405 --> 00:14:29.785 What happens in the trial? 00:14:30.625 --> 00:14:35.215 Sometimes it's what is the probability you meet statistical significance 00:14:35.215 --> 00:14:39.385 at the end of the trial In a phase two trial, that's actually rarely 00:14:39.415 --> 00:14:41.035 something you're that interested in. 00:14:41.215 --> 00:14:44.635 What's the probability I'm going to run phase three? 00:14:44.875 --> 00:14:47.815 What dose would I have selected based on that? 00:14:47.815 --> 00:14:50.815 How often do I pick the right dose in the trial? 00:14:51.860 --> 00:14:56.420 In that scenario, incredibly rich set of scenarios. 00:14:56.420 --> 00:15:01.820 I might be doing a basket trial in oncology of five different oncology 00:15:01.820 --> 00:15:07.610 types and I, the question is, which one do I run in phase 3 Do I go 00:15:07.610 --> 00:15:09.500 into type 3 head and neck cancer? 00:15:09.500 --> 00:15:10.820 Do I go in GI cancer? 00:15:11.860 --> 00:15:14.230 Can I take two or two or three of them? 00:15:14.230 --> 00:15:18.820 Can I take all five and we can record, you know, if you'd run this 00:15:18.820 --> 00:15:23.140 design, here's the chance that you go forward in the following tumor types. 00:15:23.410 --> 00:15:24.910 And we know what the right answer is. 00:15:25.570 --> 00:15:28.870 We can show them based on this truth, what, what, what, what 00:15:28.870 --> 00:15:33.970 would you want to do if, if Mother Nature told you exactly the truth? 00:15:34.240 --> 00:15:36.730 We can then simulate different trial designs. 00:15:37.630 --> 00:15:42.190 We're using this as a tool to compare designs, and I'm gonna 00:15:42.190 --> 00:15:44.800 call that in silico design. 00:15:46.990 --> 00:15:52.480 In this scenario, it's the ability to stress, test to the design and simulate 00:15:52.480 --> 00:15:56.260 many different designs for comparing them. 00:15:57.910 --> 00:16:02.140 So I've seen a number of papers where people write about clinical 00:16:02.140 --> 00:16:05.380 trial simulation, and I think they get it all wrong actually. 00:16:06.940 --> 00:16:13.180 They want to create a simulation plan before you start, and it makes no 00:16:13.180 --> 00:16:18.880 sense because what always happens when you start to compare these different 00:16:18.880 --> 00:16:22.840 designs and you show, here's what you do with the, the, the different 00:16:22.840 --> 00:16:27.790 starting designs, you compare them is immediately somebody says, oh, I, I 00:16:27.790 --> 00:16:30.070 don't like this and I don't like that. 00:16:30.430 --> 00:16:33.640 And you introduce a new design that you didn't start with. 00:16:34.420 --> 00:16:40.480 And that's really, really good because you start to, the clinicians, the team, 00:16:40.480 --> 00:16:44.870 the regulate, the regulatory people, the design team you're working with. 00:16:45.775 --> 00:16:49.645 Start to see the outcomes of the trial, and they, they think, oh, I 00:16:49.645 --> 00:16:51.565 know I want the design to do that. 00:16:51.565 --> 00:16:55.555 I, I would never want it to do that under that scenario. 00:16:56.035 --> 00:16:57.655 And that's productive. 00:16:57.655 --> 00:17:01.735 That's part of, in silico design, you can't pre-plan that. 00:17:01.735 --> 00:17:03.625 You don't even want to pre-plan that. 00:17:04.195 --> 00:17:08.155 This ability to iterate the design and go through it. 00:17:08.605 --> 00:17:12.565 Is critically important to getting to a better design. 00:17:13.105 --> 00:17:17.965 And that's the goal of the clinical trial simulation process is 00:17:17.965 --> 00:17:20.455 getting the best design possible. 00:17:22.135 --> 00:17:26.965 And so if somebody says, oh, we need to write up a clinical trial simulation 00:17:26.965 --> 00:17:30.685 plan, here's the seven designs we're gonna compare, here's the seed we're 00:17:30.685 --> 00:17:32.905 gonna use, and here's the scenarios. 00:17:33.460 --> 00:17:34.720 It's not productive. 00:17:34.750 --> 00:17:38.510 'cause invariably when you look at the results of those, immediately you say, oh. 00:17:39.070 --> 00:17:41.110 I want this component of that design. 00:17:41.290 --> 00:17:43.180 What happens if I change randomization? 00:17:43.180 --> 00:17:44.830 Would RAR benefit this? 00:17:44.830 --> 00:17:45.910 Would this benefit it? 00:17:46.180 --> 00:17:48.100 What if we change when we do them? 00:17:48.760 --> 00:17:53.740 That's, you want that, and that's what clinical trial, simulation 00:17:53.740 --> 00:17:56.770 as a design tool is really about. 00:17:56.860 --> 00:17:59.830 It's the process of getting to the final design. 00:18:00.220 --> 00:18:01.210 It's not. 00:18:01.530 --> 00:18:06.390 A design dictated by, by, by somebody that says, I want to 00:18:06.390 --> 00:18:09.960 do this design with interims at this time, and I want to do this. 00:18:10.620 --> 00:18:14.520 Sure, you can calculate that, but that's not a design process. 00:18:14.520 --> 00:18:20.940 You're not vetting the design, you're just calculating now to back up clinical trial. 00:18:20.940 --> 00:18:25.230 Simulation is numerical integration. 00:18:26.430 --> 00:18:28.740 It's nothing other than numerical integration. 00:18:29.820 --> 00:18:30.090 It's markoff. 00:18:31.645 --> 00:18:32.605 It, it, it, sorry. 00:18:32.785 --> 00:18:36.715 It's Monte Carlo integration, which is numerical integration. 00:18:37.195 --> 00:18:42.145 Given the design, given these assumptions, what happens? 00:18:42.355 --> 00:18:47.815 The beauty of it, the what happens can be really rich set of things that normally 00:18:47.815 --> 00:18:50.185 when you're making a power calculation. 00:18:50.605 --> 00:18:52.585 You can't actually characterize. 00:18:53.065 --> 00:18:56.575 So there's also statisticians that say, oh, you don't need to simulate. 00:18:57.205 --> 00:19:01.195 I don't think they're using it as a tool the same way we're using 00:19:01.195 --> 00:19:06.985 it to vet different designs, to find optimal designs within that. 00:19:07.165 --> 00:19:07.645 So 00:19:09.790 --> 00:19:10.080 what. 00:19:11.860 --> 00:19:16.660 That, that idea, I think is, is misapplied, and I'll come back to when 00:19:16.660 --> 00:19:20.620 that could potentially be a valuable thing to do, but it's very late in 00:19:20.620 --> 00:19:22.780 the process when your design is done. 00:19:23.455 --> 00:19:26.215 And we will talk about that and how simulations are used. 00:19:26.215 --> 00:19:28.465 It's a little bit of a d different kind of beast. 00:19:28.975 --> 00:19:35.635 So we're still in this in silico design process, and this is a very standard 00:19:35.635 --> 00:19:41.035 thing that we start this process with a set of candidate designs and 00:19:41.065 --> 00:19:44.395 somebody might come to us and say, we're thinking about doing the following, 00:19:44.395 --> 00:19:46.165 seamless phase two three design. 00:19:47.200 --> 00:19:53.320 Uh, but we don't know the timing of this, the decision criteria, how big phase two 00:19:53.320 --> 00:19:58.570 should be, how big phase three should be, but let's here, here's a possible design 00:19:58.630 --> 00:20:00.460 and here's two other possible designs. 00:20:00.460 --> 00:20:01.960 So we have three skeleton designs. 00:20:02.455 --> 00:20:02.545 Yes. 00:20:03.835 --> 00:20:10.165 So given that we then simulate those skeleton designs, and this is the, 00:20:10.165 --> 00:20:13.555 the first thing you want to do in this case, and this is my warning 00:20:13.555 --> 00:20:17.395 to you, I I, I've been simulating clinical trials for 25 years. 00:20:17.395 --> 00:20:22.195 It's, it's largely what we do at Barry that if you jump straight into 00:20:22.195 --> 00:20:25.555 operating characteristics, you're gonna lose a huge number of people. 00:20:26.635 --> 00:20:28.165 They're not gonna understand. 00:20:29.185 --> 00:20:32.275 What they are for, they, they absolutely won't understand what 00:20:32.275 --> 00:20:34.585 they are, and that's your fault. 00:20:35.515 --> 00:20:38.755 You need to bring them along to make sure they understand what you're 00:20:38.755 --> 00:20:40.195 doing, and it's really simple. 00:20:40.315 --> 00:20:44.005 Clinical trust simulation is really simple, but if you skip straight 00:20:44.005 --> 00:20:46.075 to operating characteristics, you're gonna lose them. 00:20:46.765 --> 00:20:49.045 So what's, what's the right first step? 00:20:49.045 --> 00:20:51.415 You have to show them example trials. 00:20:52.765 --> 00:20:56.455 All of the people you're working with understand trial results. 00:20:56.455 --> 00:20:58.015 They understand example trials. 00:20:58.585 --> 00:21:02.095 So show them the data at the interim. 00:21:02.845 --> 00:21:07.915 Show 'em raw data, show 'em spaghetti plots of patients, show 'em Kaplan-Meier 00:21:07.915 --> 00:21:13.135 plots that that's a snapshot of data, almost what A-D-S-M-B might see. 00:21:13.705 --> 00:21:15.950 And then show them, okay, here, here are the data. 00:21:16.330 --> 00:21:16.390 It. 00:21:16.960 --> 00:21:20.770 Here's the analysis we've talked about running of that data. 00:21:21.340 --> 00:21:28.570 Uh, uh, proportional odds model, a logistic regression, a COX model. 00:21:28.810 --> 00:21:30.550 Here's the result of that. 00:21:30.550 --> 00:21:33.880 At the interim, here's what it says about the relative efficacy. 00:21:34.330 --> 00:21:37.270 Here's what happens when we fit a dose response model to it. 00:21:37.300 --> 00:21:39.100 Here's the estimate that you get. 00:21:39.100 --> 00:21:42.880 Here's the raw data, and then here's the decision. 00:21:43.300 --> 00:21:46.480 In your seamless two three trial, here's the dose you would pick 00:21:46.480 --> 00:21:51.130 according to the skeleton design, and it shows them the data and all of that. 00:21:51.130 --> 00:21:53.830 And then you show them the completion of the trial. 00:21:54.190 --> 00:21:57.190 You show them the final analysis, and you say, you know what? 00:21:57.190 --> 00:22:02.590 This trial, we're gonna record this as not meeting statistical significance. 00:22:02.595 --> 00:22:06.130 At the end of the trial, we'll call that, uh, a loss. 00:22:06.365 --> 00:22:11.255 At the end here was the dose that it picked here was the effect size at the 00:22:11.255 --> 00:22:16.655 end that didn't win, and everybody will understand that single trial result. 00:22:17.675 --> 00:22:23.825 By the way, another hint of that, don't tell them the truth that simulated 00:22:23.825 --> 00:22:29.465 that trial because as soon as you tell them the truth, they're gonna want the 00:22:29.465 --> 00:22:31.625 trial to do the right answer to it. 00:22:32.185 --> 00:22:34.855 You simulated a case where dose three was the best dose? 00:22:36.205 --> 00:22:37.975 Well, well, we just want to pick dose three. 00:22:37.975 --> 00:22:41.185 Well, in the real trial, you're not gonna know that. 00:22:41.605 --> 00:22:45.475 So you want to show them this as though they, they don't know it. 00:22:45.475 --> 00:22:47.785 They're sitting on a data safety monitoring board. 00:22:47.785 --> 00:22:49.765 This is what the algorithm learns. 00:22:50.335 --> 00:22:52.105 And if they say, well, what was the truth here? 00:22:52.105 --> 00:22:53.635 I say, well, I'm not gonna tell you that. 00:22:53.995 --> 00:22:57.085 I want you to see it for what the, the, the design is. 00:22:58.175 --> 00:23:02.440 But then when that design's over and you say, this trial results in statistical 00:23:02.440 --> 00:23:06.580 significance, here's the effect size, by the way, show 'em a positive trial first. 00:23:06.790 --> 00:23:10.120 If you show 'em a negative trial, they, they sort of get a bad vibe to that. 00:23:10.120 --> 00:23:15.460 Show 'em a positive trial, maybe show 'em an inconclusive trial, um, then, or 00:23:15.460 --> 00:23:19.990 a hard trial, show 'em a, a clearly a failed trial, and this is what happens. 00:23:20.410 --> 00:23:23.950 Then you say, okay, now that first trial here was the 00:23:23.950 --> 00:23:25.840 scenario that I simulated from. 00:23:26.025 --> 00:23:30.045 After you've shown them a couple, and now I'm gonna repeat 00:23:30.045 --> 00:23:32.025 that trial a thousand times. 00:23:32.715 --> 00:23:37.665 Here's the number of trials that met statistical significance at the end. 00:23:37.845 --> 00:23:40.575 Here's the proportion that picked dose three. 00:23:40.575 --> 00:23:46.830 Like that example trial did 20% chance you picked dose four, 10% chance, and 00:23:46.830 --> 00:23:47.910 you can show it and they'll get it. 00:23:48.610 --> 00:23:52.150 And you've walked them through it and they'll understand the simulation. 00:23:52.150 --> 00:23:53.320 It's, it's really easy. 00:23:53.620 --> 00:23:58.000 Clinical trial simulation is easy if you present it right 00:23:58.000 --> 00:23:59.440 and you walk them through it. 00:23:59.770 --> 00:24:03.460 Now as you start to iterate the design, you might be able to skip 00:24:03.460 --> 00:24:05.440 those because they're hooked. 00:24:06.580 --> 00:24:11.020 They get it and they know how to interpret the operating characteristics, 00:24:11.380 --> 00:24:15.370 which as statisticians we find very straightforward, but we think 00:24:15.370 --> 00:24:16.960 differently than everybody else. 00:24:17.350 --> 00:24:21.490 Uh, and you have to know that people don't think that way, so you have 00:24:21.490 --> 00:24:23.200 to sort of guide them through it. 00:24:24.130 --> 00:24:26.590 When we do clinical trial simulations. 00:24:27.550 --> 00:24:33.070 We spend more time on the graphs and the presentation of them than probably 00:24:33.070 --> 00:24:35.620 creating the simulations themselves. 00:24:35.950 --> 00:24:40.900 It's that important to get people to understand it again for the goal of 00:24:40.900 --> 00:24:45.340 getting the best design at the end, that they know what they're signing up for. 00:24:46.240 --> 00:24:52.330 I have simulated trials where the, the, the, it's a fixed trial and 00:24:52.330 --> 00:24:57.730 there's a target product profile, so there's a delta that if that drug is 00:24:57.730 --> 00:25:05.020 that good, we want the trial to win and in 80% power trials, 400 patients, 00:25:05.800 --> 00:25:07.960 and they're, they're okay with that. 00:25:08.380 --> 00:25:10.750 And I'll simulate that trial. 00:25:11.590 --> 00:25:18.280 From the target product profile, from their TPP, that delta, that, that's a 00:25:18.280 --> 00:25:24.400 great drug, gonna be highly effective and, and benefit a lot of patients, and 00:25:24.400 --> 00:25:31.780 I simulate 20% of the trials fail and they've said, wait, wait, wait a minute. 00:25:31.960 --> 00:25:35.770 If that drug's that good, 20% of the trials will fail. 00:25:36.925 --> 00:25:39.475 And I say that if you're a statistician and all that, you say, 00:25:39.475 --> 00:25:41.695 well, that's 80% powered for that. 00:25:42.265 --> 00:25:46.705 But sometimes they don't really understand what that means at the level that a 00:25:46.705 --> 00:25:49.435 simulation shows them what that means. 00:25:50.305 --> 00:25:54.175 And, and it's not our job to laugh and say, you don't understand it. 00:25:54.175 --> 00:25:55.525 It's, it's our fault. 00:25:55.795 --> 00:25:57.415 That's our science. 00:25:58.255 --> 00:26:02.455 To make sure they know what they're signing up for and 00:26:02.455 --> 00:26:03.865 make sure they understand. 00:26:03.865 --> 00:26:09.445 When you're running an 80% powered trial for that really good effect, 20% 00:26:09.445 --> 00:26:12.295 of trials are gonna fail Immediately. 00:26:12.295 --> 00:26:16.135 They say, wait a minute, maybe we should look at a bigger sample size. 00:26:16.825 --> 00:26:21.385 Maybe I should be 95% powered, but then the sample size is so big. 00:26:21.385 --> 00:26:22.765 Well, let's do adaptive sample size. 00:26:22.975 --> 00:26:25.645 It leads to them creating a better design. 00:26:25.645 --> 00:26:30.565 That simulation will get them there, where if they just say to you, gimme 00:26:30.565 --> 00:26:34.945 an 80% powered sample size, you haven't done, you haven't helped 00:26:34.945 --> 00:26:40.165 them with that by saying it's 400, that I'm not sure that they always 00:26:40.165 --> 00:26:41.935 understand what they're signing up for. 00:26:42.505 --> 00:26:44.905 You can also then guide them through futility. 00:26:45.205 --> 00:26:46.735 You can walk them through futility. 00:26:46.915 --> 00:26:51.235 If we do a futility analysis at 200, show me the data that we're gonna stop for. 00:26:51.805 --> 00:26:52.435 Within that. 00:26:53.035 --> 00:26:54.145 We can do this. 00:26:54.145 --> 00:26:56.155 Should we borrow historical data? 00:26:56.155 --> 00:26:57.775 What does the data look like? 00:26:57.805 --> 00:26:58.045 What? 00:26:58.045 --> 00:26:59.545 What do the effect size look like? 00:26:59.545 --> 00:27:05.215 That win a basket trial with multiple baskets in enrichment design? 00:27:05.215 --> 00:27:07.165 Suppose we're gonna drop subsets. 00:27:07.165 --> 00:27:11.785 These are all complicated things that clinical trial simulation 00:27:11.785 --> 00:27:13.195 can make relatively easy. 00:27:13.575 --> 00:27:16.185 So this is the Silical process. 00:27:16.185 --> 00:27:23.115 So we show them the results of their first three skeleton designs, and rarely do they 00:27:23.115 --> 00:27:25.845 say, oh, design three, is it, we're done. 00:27:26.865 --> 00:27:30.585 They, they, they rethink about the design, the effect. 00:27:30.585 --> 00:27:31.935 I don't want to do that. 00:27:32.365 --> 00:27:34.345 This assumption makes sense. 00:27:34.345 --> 00:27:35.815 Can you change this? 00:27:35.935 --> 00:27:39.235 And there's three new designs and they're all closer to 00:27:39.235 --> 00:27:40.705 sort of where they want to be. 00:27:41.155 --> 00:27:44.485 And there's another iteration of new simulations. 00:27:44.665 --> 00:27:46.615 They want to add some scenarios. 00:27:47.005 --> 00:27:49.435 You know, we, we, we think this is more likely. 00:27:50.005 --> 00:27:53.245 You generally have scenarios that are statistically interesting. 00:27:53.695 --> 00:27:58.555 Within that, you wanna show a range of effects from zero, maybe even harm to 00:27:58.555 --> 00:28:03.955 big effects, bigger than the TPP within that, to show them a range, then show 00:28:03.955 --> 00:28:06.205 them interesting statistical scenarios. 00:28:06.265 --> 00:28:09.865 And then they'll have scenarios that they think are likely to be true, 00:28:10.315 --> 00:28:13.495 uh, under that, and they can simulate that it leads to an iteration. 00:28:13.850 --> 00:28:18.980 That leads to a new iteration, a new step of simulations where they're getting 00:28:18.980 --> 00:28:23.015 closer and closer and they're further and further understanding their design. 00:28:24.400 --> 00:28:27.040 This is the process and it can't be scripted. 00:28:27.370 --> 00:28:31.810 You can't write a a, this isn't like a SAP that you write. 00:28:31.810 --> 00:28:34.420 You have to write ahead everything you're gonna do. 00:28:34.720 --> 00:28:35.815 This is you. 00:28:35.820 --> 00:28:37.720 You don't want this to be scripted. 00:28:37.720 --> 00:28:39.490 You want to be able to go to new designs. 00:28:39.490 --> 00:28:42.730 You want to explore things you didn't think of that the simulations 00:28:42.730 --> 00:28:46.570 brought out interesting from everybody on the team Within that. 00:28:47.800 --> 00:28:51.910 And I think I, every, every time I read a paper that says You should write down 00:28:51.910 --> 00:28:56.740 a simulation plan before you start the design process, I, I've never done that. 00:28:57.220 --> 00:29:01.240 I don't think it's a good thing, and I don't think it leads to better designs. 00:29:01.240 --> 00:29:03.610 Again, let's get to a better design within that. 00:29:04.870 --> 00:29:05.320 Okay. 00:29:06.310 --> 00:29:15.820 Now, the in, in terms of this iteration, we eventually we get to a final design. 00:29:16.945 --> 00:29:19.045 And there we have a design. 00:29:19.105 --> 00:29:19.915 We can write it. 00:29:19.915 --> 00:29:21.955 If you simulate it, you can write it down. 00:29:22.585 --> 00:29:26.425 You can write down the design, whether it's uh, the analysis method, 00:29:26.785 --> 00:29:31.375 the interim plan, what success is, all of the characteristics 00:29:31.375 --> 00:29:32.635 of the design you can write down. 00:29:34.195 --> 00:29:40.765 Now, that's the point where maybe some simulation plan makes some sense. 00:29:41.065 --> 00:29:43.915 Again, I think it's a little odd at that point. 00:29:44.710 --> 00:29:49.690 But here's where now we use simulation to fully show the 00:29:49.690 --> 00:29:51.550 characteristics of the design. 00:29:51.550 --> 00:29:57.220 We've set up what is the type one error from a range of null scenarios. 00:29:57.610 --> 00:30:02.440 What is the power from a range of scenarios, across doses, across 00:30:02.440 --> 00:30:08.920 subgroups, whatever it is, we are doing final simulations with big. 00:30:09.760 --> 00:30:14.680 Simulation number of trials, and we are putting that in a report, 00:30:14.770 --> 00:30:16.810 and we then give it to regulators. 00:30:17.170 --> 00:30:22.240 We put it as an appendix to the sap, an adaptive design report or a 00:30:22.240 --> 00:30:24.580 simulation report that becomes that. 00:30:25.240 --> 00:30:28.390 Now, that's the part that gets a little bit hard. 00:30:28.510 --> 00:30:33.370 Many times that only shows the simulations for the final design. 00:30:34.930 --> 00:30:40.390 It can be confusing if you in that report, show simulations for a design. 00:30:40.390 --> 00:30:41.080 You're not running. 00:30:42.615 --> 00:30:44.710 It, it really can be confusing. 00:30:45.070 --> 00:30:48.460 And that's not the role of the simulations at that point. 00:30:48.460 --> 00:30:51.640 They're, they're about operating characteristics at that point. 00:30:52.360 --> 00:30:55.900 Uh, and, and if you're going to regulars, you can simulate the bias of 00:30:55.900 --> 00:30:59.650 the estimate from our analysis method at the end of the trial, for example, 00:30:59.950 --> 00:31:04.030 and all of these things to evaluate the design that's being submitted. 00:31:05.350 --> 00:31:06.040 Yes. 00:31:07.090 --> 00:31:11.860 You might be asked, okay, you're doing response adaptive randomization. 00:31:12.070 --> 00:31:15.035 Can you demonstrate to us that that's a good thing to be doing? 00:31:16.120 --> 00:31:20.620 So you might have to have a section that shows what if the a different 00:31:20.620 --> 00:31:26.800 design were being run without that, or, uh, whatever thing about the trial 00:31:26.800 --> 00:31:31.540 is maybe innovative or different, and it might not be, uh, within that case, 00:31:31.810 --> 00:31:38.080 you might show comparisons to justify the design you're doing to convince 00:31:38.080 --> 00:31:39.760 somebody that it's a good design. 00:31:40.810 --> 00:31:45.910 Now what's hard about that is is I, I've been asked this question 00:31:45.910 --> 00:31:49.300 or, or this comment has come up when I've presented a design. 00:31:49.600 --> 00:31:51.940 The design has gone through this process. 00:31:51.940 --> 00:31:57.250 We come up with a final design and I might give a 20 minute talk on what the design 00:31:57.250 --> 00:31:59.440 is, and I might show simulations of that. 00:32:00.370 --> 00:32:05.620 And people have said, oh, you're just being complex to be complex. 00:32:06.265 --> 00:32:09.145 First time somebody said that it was upset. 00:32:09.475 --> 00:32:14.545 Um, I, I took that as criticism and all that, but I get what's really hard is 00:32:14.545 --> 00:32:18.295 they haven't been through that process of the design process where everything 00:32:18.295 --> 00:32:21.685 you, everything you've done in the design, you've done for a reason and 00:32:21.685 --> 00:32:23.545 you've compared it to other designs. 00:32:23.875 --> 00:32:27.175 You can't, in a 20 minute talk get up and present. 00:32:27.735 --> 00:32:31.605 47 designs you're not running and why that you might be really 00:32:31.605 --> 00:32:34.845 interested in what's the value of RAR? 00:32:35.055 --> 00:32:37.335 What's the value of borrowing? 00:32:37.485 --> 00:32:42.195 And so you might need to present to the audience to that, but that's hard 00:32:42.195 --> 00:32:43.935 and that's a different sort of thing. 00:32:43.995 --> 00:32:46.875 So clinical trial simulation's a huge tool there. 00:32:47.085 --> 00:32:50.625 It's not in silico design, you have a proposed single design. 00:32:51.255 --> 00:32:54.645 You're writing up in a report, you're providing simulations, you're 00:32:54.645 --> 00:32:57.255 showing the DSMB example trials. 00:32:58.270 --> 00:33:01.780 That at interim one, two, and three, here's what you might see. 00:33:01.780 --> 00:33:03.100 Here are the rules. 00:33:03.100 --> 00:33:04.510 Here's how that works. 00:33:04.840 --> 00:33:09.760 It's a hugely valuable tools in many ways, but it really shifts from 00:33:09.760 --> 00:33:15.975 comparing designs to showing that single designs, characteristics and behavior. 00:33:17.605 --> 00:33:21.265 And I, I think a number of people write articles about clinical trial 00:33:21.265 --> 00:33:24.985 simulation, and they're all focused on that end game where, you know, the design 00:33:25.855 --> 00:33:28.945 90% of this time and effort and work. 00:33:29.005 --> 00:33:35.815 And the, the, the really, the, the skill of the design part is this in silico 00:33:35.815 --> 00:33:38.035 design part, and it's very different. 00:33:39.670 --> 00:33:44.230 And I think misunderstood by people who don't live this space of 00:33:44.500 --> 00:33:47.260 simulating for optimizing a design. 00:33:49.030 --> 00:33:53.380 Now, it a, a couple side aspects of this. 00:33:53.380 --> 00:33:57.670 So, so one design, I, I've talked about this publicly multiple times and it's kind 00:33:57.670 --> 00:34:02.320 of an interesting time because the trial is going to read out in several months. 00:34:02.320 --> 00:34:07.300 I'm sure we will do an inter in in the interim on it, but the ice cap trial. 00:34:07.960 --> 00:34:09.325 Was a trial it. 00:34:09.330 --> 00:34:17.050 It was a trial for looking at hypothermia for the treatment of cardiac arrest. 00:34:17.560 --> 00:34:19.090 Individual suffers cardiac arrest. 00:34:20.740 --> 00:34:22.000 They're revived. 00:34:22.060 --> 00:34:25.630 The heart is revived, resuscitated. 00:34:25.720 --> 00:34:27.820 They go into the emergency room. 00:34:27.820 --> 00:34:34.540 The thought is, is that putting somebody in a state of cold hypothermia 00:34:34.750 --> 00:34:37.720 prevents the negative cascade of heat. 00:34:38.590 --> 00:34:42.850 The neurological damage that happens in heat exaggerates 00:34:42.850 --> 00:34:44.470 that from the lack of oxygen. 00:34:44.920 --> 00:34:49.210 I'm a statistician, but that's my understanding of the the mechanism of 00:34:49.210 --> 00:34:51.970 action of hypothermia post cardiac arrest. 00:34:53.055 --> 00:34:58.485 We went in this and simulated a trial for this where it was three durations. 00:34:58.485 --> 00:35:01.545 The question was, how long do you keep them in that state? 00:35:01.605 --> 00:35:04.335 It's kind of like a dose response, but it's a duration response. 00:35:04.905 --> 00:35:07.725 And was it 12, 24 48 hours? 00:35:08.175 --> 00:35:11.535 I showed them example trials in almost every one of them. 00:35:11.595 --> 00:35:12.910 They wanted a different design. 00:35:13.975 --> 00:35:16.645 Boy, I wish we had patients less than 12. 00:35:16.735 --> 00:35:18.835 I wish we had patients greater than 48. 00:35:18.835 --> 00:35:20.245 I wish we had them between there. 00:35:20.785 --> 00:35:25.585 The design grew through that process with the design team to 10 durations, 00:35:26.485 --> 00:35:35.845 6, 12, 18, 24, 30, 36, 42, all the way up to 72 hours, five days, and 00:35:35.845 --> 00:35:37.855 10 different durations with response. 00:35:37.855 --> 00:35:38.755 Adaptive randomization. 00:35:40.330 --> 00:35:44.530 And the, the duration response model became critical to 00:35:44.530 --> 00:35:46.540 driving where those patients go. 00:35:46.720 --> 00:35:50.800 It was a very restricted U-shape that it couldn't wiggle. 00:35:51.070 --> 00:35:55.300 There's no way in which it's a non monotonic relationship. 00:35:55.760 --> 00:35:59.300 Uh, within that they, they, and they wanted the design to do that. 00:35:59.510 --> 00:36:02.510 That all came through by understanding it through simulation. 00:36:02.510 --> 00:36:04.190 That trial's about to read out. 00:36:04.490 --> 00:36:08.240 Uh, they've stopped enrolling, so that'll be an interesting trial to look at. 00:36:08.240 --> 00:36:10.010 But built entirely by simulation. 00:36:10.010 --> 00:36:12.080 That's the kind of thing, the richness. 00:36:12.775 --> 00:36:16.315 Of looking at the example trials, the analysis method, comparisons, 00:36:16.615 --> 00:36:18.385 that, that, that values, that. 00:36:18.385 --> 00:36:21.535 You couldn't have come up with that design and say, Hey, let's do the 00:36:21.535 --> 00:36:23.665 following with this model and all that. 00:36:24.445 --> 00:36:27.265 It, it, it, it, it doesn't work that way. 00:36:28.585 --> 00:36:32.815 Now, the, one of the issues is if you want to go through 00:36:32.815 --> 00:36:34.315 this process, it can be slow. 00:36:34.645 --> 00:36:38.965 To custom code this if you need to write code to do all the design 00:36:38.965 --> 00:36:43.615 variants, all these things to process the data and all of that, you know, 00:36:43.620 --> 00:36:47.755 and, and the design team has timelines and you can't say, okay, gimme six 00:36:47.755 --> 00:36:49.315 weeks and I'll come back with that. 00:36:49.825 --> 00:36:54.175 This moves at a periodic process a week or two weeks to make progress. 00:36:54.175 --> 00:36:57.415 You don't need the final answer, but we need to make progress, which is 00:36:57.415 --> 00:37:03.535 why we created Facts And Fax is fixed and adaptive Clinical trial simulator. 00:37:03.925 --> 00:37:09.735 And it allows you to do a wide range of design ideas, analysis, 00:37:09.735 --> 00:37:13.035 ideas, types of datas and simulations within a user interface. 00:37:14.350 --> 00:37:17.650 And you can within an hour have clinical trial simulation results. 00:37:17.650 --> 00:37:19.690 You can vary the design, you can do that. 00:37:19.990 --> 00:37:21.850 And this was set up to do that. 00:37:22.360 --> 00:37:26.410 Now our, the, the product is, is kind of interesting because there are other 00:37:26.410 --> 00:37:30.400 simulation products out there, but they're almost all driven that you go 00:37:30.400 --> 00:37:34.120 into the product, you say, oh, I want this design, simulate that design. 00:37:35.230 --> 00:37:37.600 But it doesn't allow you to vary it in ways. 00:37:37.600 --> 00:37:39.640 It doesn't allow you to change it in this way. 00:37:39.910 --> 00:37:42.490 Fax doesn't have any kind of name designs. 00:37:43.450 --> 00:37:46.660 Uh, you know, I want dragline design number seven. 00:37:47.050 --> 00:37:48.670 Uh, it doesn't do that. 00:37:49.150 --> 00:37:53.380 Uh, within it, you create it, you create all of the parts to it to allow 00:37:53.380 --> 00:37:56.080 you to do this in silico process. 00:37:56.260 --> 00:37:59.680 So it's kind of a very different design, and it's grown out of 25 00:37:59.680 --> 00:38:01.720 years of doing that it silico process. 00:38:01.960 --> 00:38:05.830 So when people jump into the product, it's, it, it can look funny at first. 00:38:06.160 --> 00:38:09.730 And if you don't do that in silico process, you might not 00:38:09.730 --> 00:38:11.320 find facts valuable at all. 00:38:12.235 --> 00:38:15.895 Uh, within the setting, if you want to do this process and compare 00:38:15.895 --> 00:38:20.365 designs and contrast them and look at alternatives and, and vet them, 00:38:20.755 --> 00:38:23.395 it's a way to do it very, very fast. 00:38:23.395 --> 00:38:24.685 Custom coding is great. 00:38:24.685 --> 00:38:28.705 We as statisticians love to custom code, but this is a way to do that and 00:38:28.705 --> 00:38:30.235 we have a number of people to use that. 00:38:30.235 --> 00:38:35.185 So it was, it, it, as people ask about that, it's understanding, 00:38:35.185 --> 00:38:39.655 it's a tool for getting to the right design is really its role. 00:38:40.005 --> 00:38:42.225 Now, it could also be used by regulators. 00:38:42.225 --> 00:38:45.345 If somebody submits a design, they could go in and simulate that and 00:38:45.345 --> 00:38:49.035 verify operating characteristics and all of that within it. 00:38:49.575 --> 00:38:52.995 One of the really cool things in simulations I've found is 00:38:52.995 --> 00:38:54.975 counterfactual simulation. 00:38:56.235 --> 00:38:57.675 What do, what do I mean by that? 00:38:58.155 --> 00:39:03.465 You could simulate a phase 3 trial that always goes to 800 patients. 00:39:04.990 --> 00:39:08.920 And see what would happen if you run a fixed 800 patient trial. 00:39:09.370 --> 00:39:14.140 Then you could add in an interim for early success and futility. 00:39:15.580 --> 00:39:20.860 You can go in at 400 patients and say, we would stop for futility here 00:39:21.730 --> 00:39:24.250 and or we would stop for success. 00:39:24.250 --> 00:39:27.460 And you can say, what would've happened if you didn't stop? 00:39:28.180 --> 00:39:32.080 We stopped for futility and it would've failed in the end. 00:39:32.140 --> 00:39:32.800 That's a good thing. 00:39:33.700 --> 00:39:37.120 Because we, we saved 400 patients to the same conclusion, 00:39:37.930 --> 00:39:39.970 or we stopped for futility. 00:39:40.030 --> 00:39:42.460 But if you'd let that trial go, it would've won. 00:39:43.510 --> 00:39:47.290 And you can see that you can't see that any other way, but you can see 00:39:47.290 --> 00:39:49.060 that through clinical trial simulation. 00:39:49.060 --> 00:39:51.040 Then you can vary the futility rule. 00:39:51.310 --> 00:39:55.300 You can see what, how, if you lower your power, how much you lower your power. 00:39:55.300 --> 00:39:59.440 With futility, you can see the the, the how your sample size changes. 00:39:59.440 --> 00:40:00.565 The effect sizes and all this. 00:40:01.060 --> 00:40:02.830 So counterfactual simulation. 00:40:03.640 --> 00:40:06.700 Is something that's incredibly powerful in simulation. 00:40:06.700 --> 00:40:11.020 When you're comparing two designs, what would've happened on the exact same 00:40:11.020 --> 00:40:13.030 trial with a different analysis method? 00:40:13.870 --> 00:40:18.640 If I assume a a, a linear slope, or if I do an MMRM, you could take 00:40:18.640 --> 00:40:21.910 a single data set and compare the analysis method, how many win over 00:40:21.910 --> 00:40:25.690 here, but lose over here, lose over here, but win over here, for example. 00:40:26.260 --> 00:40:30.310 So these counterfactual simulations become a hugely valuable 00:40:30.310 --> 00:40:32.320 thing in trial simulation. 00:40:33.685 --> 00:40:39.775 Uh, within that, now the approach is quite different. 00:40:40.015 --> 00:40:45.205 So this is my view of clinical trial simulation, the role of it in creating 00:40:45.205 --> 00:40:48.115 de designs a sort of space I live in. 00:40:48.475 --> 00:40:52.345 But a lot of people ask about this and I, I think there's a misunderstanding of it. 00:40:52.345 --> 00:40:56.335 So it's a whole different viewpoint on that, uh, in the design. 00:40:58.105 --> 00:40:58.555 So. 00:40:59.530 --> 00:41:03.940 I hope you enjoyed this from Scotty Scheffler to Tiger Woods, to robo 00:41:03.940 --> 00:41:12.130 Tiger to in silico design, where we are here in the interim, unless 00:41:12.280 --> 00:41:16.360 this has all been simulated and we're not here in the interim. 00:41:18.040 --> 00:41:18.490 Hmm. 00:41:19.630 --> 00:41:23.620 Food for thought until next time where we're either 00:41:23.620 --> 00:41:26.080 simulated or not in the interim.