WEBVTT NOTE This file was generated by Descript 00:00:00.000 --> 00:00:03.200 Judith: Welcome to Berry's In the Interim podcast, where we explore the 00:00:03.200 --> 00:00:07.369 cutting edge of innovative clinical trial design for the pharmaceutical and 00:00:07.369 --> 00:00:09.739 medical industries, and so much more. 00:00:10.270 --> 00:00:10.939 Let's dive in. 00:00:12.584 --> 00:00:13.304 Scott Berry: All right. 00:00:13.304 --> 00:00:14.444 Welcome everybody. 00:00:14.444 --> 00:00:19.244 Back to In the interim, I am your host, Scott Berry, and I'm joined 00:00:19.244 --> 00:00:22.514 today by my co-host, uh, Kurt Veley. 00:00:22.514 --> 00:00:22.904 Dr. 00:00:22.904 --> 00:00:27.644 Kurt Veley is here, and, uh, Kurt needs no introduction, uh, as 00:00:27.644 --> 00:00:30.404 he is a, a regular on the show. 00:00:31.004 --> 00:00:35.714 Uh, we have a interesting topic and, and the topic I'll, I'll, 00:00:35.714 --> 00:00:40.394 I'll introduce the topic a little bit by telling you a true story. 00:00:41.249 --> 00:00:44.039 Uh, and Kurt is related to this story. 00:00:44.039 --> 00:00:46.889 For those that didn't know, Kurt and I were fellow graduate 00:00:46.889 --> 00:00:48.269 students at Carnegie Mellon. 00:00:49.289 --> 00:00:55.619 And, um, when I was a 23-year-old, first year graduate student at 00:00:55.619 --> 00:01:00.269 Carnegie Mellon, I, uh, called my dad. 00:01:00.749 --> 00:01:04.889 And of course, um, the, the aspect I called him is I 00:01:04.889 --> 00:01:06.899 called him and informed them. 00:01:07.304 --> 00:01:14.174 Him that my wife had taken a pregnancy test and it was positive. 00:01:15.164 --> 00:01:21.434 And uh, I don't know if many of you in the world have similar interactions with 00:01:21.434 --> 00:01:23.774 your father in when such a thing happens, 00:01:24.254 --> 00:01:24.824 but 00:01:24.989 --> 00:01:26.519 Kert Viele: I think, the answer is no, Scott. 00:01:26.579 --> 00:01:28.109 Scott Berry: I think the answer's no. 00:01:28.379 --> 00:01:33.929 Uh, the, the question, the first thing he said after I said, hi, I said 00:01:33.929 --> 00:01:38.729 that, uh, by the way, I did say, uh, we took the pregnancy test and the 00:01:38.729 --> 00:01:42.329 pregnancy test says it's 99% accurate. 00:01:43.649 --> 00:01:47.999 And I tell him this, and his first question to me was, what 00:01:47.999 --> 00:01:49.709 was your prior probability? 00:01:50.669 --> 00:01:52.889 Honestly, that's, that's what he asked me. 00:01:53.279 --> 00:02:00.209 And, um, uh, I, I, my response was, and uh, and, and Kurt knows this 00:02:00.209 --> 00:02:04.499 well, we, we were, I was a married graduate student making a stipend. 00:02:04.919 --> 00:02:07.469 I said the test was $15. 00:02:07.469 --> 00:02:07.979 Dad. 00:02:08.819 --> 00:02:13.289 The prob, the prior probability was pretty high, and his next 00:02:13.289 --> 00:02:15.779 comment was, oh, congratulations. 00:02:15.779 --> 00:02:17.069 That's fantastic. 00:02:17.369 --> 00:02:21.449 Uh, and he was excited, uh, for, for this whole thing. 00:02:22.019 --> 00:02:29.009 So they, the, the point to that is if our prior probability is low. 00:02:29.849 --> 00:02:34.049 The 99% means something different about the the, at the end of the 00:02:34.049 --> 00:02:38.549 day, the question is, is Tammy, is my wife pregnant or not? 00:02:38.879 --> 00:02:43.919 Um, in the, in that, based on this experiment where the prior probability 00:02:43.919 --> 00:02:47.759 makes a huge difference, and many of you have probably seen the rare 00:02:47.759 --> 00:02:51.599 disease examples where if you have a low prior probability of something 00:02:51.599 --> 00:02:55.019 and you take a diagnostic test and it says positive, you're still 00:02:55.019 --> 00:02:57.089 probably unlikely to have the disease. 00:02:57.839 --> 00:03:03.569 Uh, so in this circumstance, if, if, if I had a 1% chance that she was 00:03:03.569 --> 00:03:06.839 pregnant and the test was 99% effective. 00:03:07.739 --> 00:03:14.639 Meaning if she's pregnant, 99% chance it says yes, if not 99% chance it says no. 00:03:14.939 --> 00:03:17.009 I think they're actually more accurate than that. 00:03:17.009 --> 00:03:21.419 But, um, if I had a 1% chance, I still, it's only 50 50. 00:03:21.419 --> 00:03:24.959 She's pregnant in that circumstance where if I started with a 00:03:24.959 --> 00:03:27.689 10% chance, it was over 90%. 00:03:27.689 --> 00:03:28.409 She's positive. 00:03:30.164 --> 00:03:36.944 The point to that is the prior probability of that changes my interpretation 00:03:36.944 --> 00:03:38.924 of the result of the experiment. 00:03:41.744 --> 00:03:48.554 So the topic for today is using Bayesian borrowing. 00:03:49.199 --> 00:03:53.879 And that's a case of Bayesian borrowing of what my dad did faced 00:03:53.879 --> 00:03:56.099 with the result of a clinical trial. 00:03:56.549 --> 00:03:59.159 Uh, and the diagnostic test is a clinical trial. 00:03:59.159 --> 00:04:02.729 A clinical trial is the same thing based with that information. 00:04:02.729 --> 00:04:09.329 The prior probability was a huge part to his interpretation of, uh, of the result. 00:04:10.049 --> 00:04:15.239 There are cases now, and the Bayesian guidance is out where we use Bayesian. 00:04:16.124 --> 00:04:20.654 Probabilities going into the trial to interpret the result of, of the 00:04:20.654 --> 00:04:24.224 clinical trial working with the FDA. 00:04:24.944 --> 00:04:27.704 And today's topic is doing that. 00:04:27.704 --> 00:04:28.634 What does it look like? 00:04:28.634 --> 00:04:29.954 What are the characteristics? 00:04:29.954 --> 00:04:31.154 What's the behavior? 00:04:31.454 --> 00:04:33.434 Have we done this before? 00:04:34.214 --> 00:04:36.344 So that's the topic for today, Kurt. 00:04:38.174 --> 00:04:38.834 Kert Viele: All right. 00:04:38.894 --> 00:04:43.604 Sounds like a good topic and it sounds like we're talking about, um. 00:04:44.699 --> 00:04:47.429 So we, we've done this in a couple contexts, and it sounds like you're 00:04:47.429 --> 00:04:48.689 gonna talk about the second one. 00:04:48.719 --> 00:04:53.009 We've done this in context where we're only borrowing on, 00:04:53.009 --> 00:04:54.029 you know, a clinical trial. 00:04:54.029 --> 00:04:56.789 There are two arms, there's a control, there's novel treatments. 00:04:56.999 --> 00:05:00.869 We've done situations where we've borrowed only on the control arm. 00:05:01.169 --> 00:05:03.719 So you're comparing to an existing drug. 00:05:03.719 --> 00:05:08.459 There's lots of data on the existing drug, and you're trying to bring that in. 00:05:08.459 --> 00:05:10.859 That's one problem you might solve. 00:05:11.429 --> 00:05:12.029 Um. 00:05:12.474 --> 00:05:17.124 It sounds like today we're gonna talk about the, um, what we'll actually 00:05:17.124 --> 00:05:19.434 consider the more complex version. 00:05:19.464 --> 00:05:24.534 Kind of a newer version of we're borrowing on both arms where we have 00:05:24.534 --> 00:05:30.054 some idea about what the treatment effect might be based on history. 00:05:30.414 --> 00:05:37.164 And now the question is, what does our current trial have to do in order to. 00:05:37.464 --> 00:05:40.884 You know, this is a journey from we know nothing in order to, we know the 00:05:40.884 --> 00:05:45.324 drug works if we're partially there based on what we've already seen. 00:05:45.744 --> 00:05:48.114 Um, what else do we need to know? 00:05:48.144 --> 00:05:51.204 How much more evidence do we need to get over that bar? 00:05:51.684 --> 00:05:55.314 And I know that we've seen this in the context of. 00:05:56.149 --> 00:05:58.429 You know, completely external trials. 00:05:58.429 --> 00:06:04.699 We've seen it in the context of, you know, people run one trial and get p equal 0.06. 00:06:04.879 --> 00:06:07.069 And now the question is what do we do? 00:06:07.609 --> 00:06:11.119 Um, so where, where do you wanna head from here, Scott? 00:06:11.124 --> 00:06:11.534 Scott Berry: Yeah. 00:06:11.594 --> 00:06:12.014 Yeah. 00:06:12.074 --> 00:06:12.614 Uh, good. 00:06:12.614 --> 00:06:17.954 So first of all, just to, uh, and, and by the way, this is a, this is a shout out 00:06:17.954 --> 00:06:24.674 to anybody who, uh, listens, watches, uh, in the interim, if you have questions. 00:06:25.379 --> 00:06:26.399 Throw 'em our way. 00:06:26.399 --> 00:06:28.919 We're, we're, we're looking for interesting topics. 00:06:29.219 --> 00:06:35.249 This was something that somebody, uh, contacted me and asked us, have we ever 00:06:35.249 --> 00:06:38.399 gone to FDA using Bayesian, borrowing? 00:06:39.899 --> 00:06:41.579 And it was a little bit like, whoa. 00:06:41.849 --> 00:06:43.619 Yeah, this is kind of what we do. 00:06:43.949 --> 00:06:46.979 Uh, and I was a little, almost surprised by the question, but then 00:06:47.194 --> 00:06:51.449 it, it made a ton of sense that, that now with the Bayesian guidance out 00:06:51.509 --> 00:06:55.889 and they were interested in the, this interaction and what does it look like? 00:06:56.339 --> 00:07:01.259 The, the other part to this is the Bayesian guidance make, ref makes 00:07:01.259 --> 00:07:06.029 reference to Bayesian trials where you're controlling type one error or 00:07:06.029 --> 00:07:07.589 you're not controlling type one error. 00:07:07.589 --> 00:07:13.769 Her and I was asked by somebody else, a brilliant researcher, actually, uh, Dr. 00:07:13.769 --> 00:07:19.529 Bijoy Manan about, uh, he's a stroke neurologist in Calgary. 00:07:19.799 --> 00:07:21.089 Uh, what's the difference? 00:07:21.209 --> 00:07:22.199 Uh, that sort of thing. 00:07:22.199 --> 00:07:24.989 So I think we can tackle all of those things today. 00:07:25.439 --> 00:07:29.909 So maybe we should set it up with a case where this may make sense or 00:07:29.909 --> 00:07:32.114 cases where we've done this before. 00:07:33.224 --> 00:07:37.244 Uh, just to, just to give people an example of, of it. 00:07:37.274 --> 00:07:43.274 Now first maybe we should set up what is done in a Frequentist standard clinical 00:07:43.274 --> 00:07:48.884 trial is you calculate for that trial the probability of a type one error. 00:07:49.904 --> 00:07:54.974 Um, uh, and it's only data in that trial that goes into that calculation. 00:07:54.974 --> 00:07:59.654 And we say, if the probability of achieving the data in that trial. 00:08:00.239 --> 00:08:04.109 Given the null is less than 2.5%, 00:08:04.109 --> 00:08:05.879 we're gonna call it a successful trial. 00:08:06.899 --> 00:08:12.389 By the way, in the circumstance of the pregnancy test, the assuming she's not 00:08:12.389 --> 00:08:17.609 pregnant, the probability of it saying pregnant is 1% statistically significant. 00:08:18.779 --> 00:08:23.249 In that case now, that that might be how we interpret it. 00:08:23.279 --> 00:08:29.009 Now, here's a circumstance and we're, it's not uncommon that a company, a sponsor, 00:08:29.729 --> 00:08:34.954 has run a trial and the trial is, uh, 00:08:37.289 --> 00:08:43.439 not convincing for approval, but has some positivity to it. 00:08:43.439 --> 00:08:45.359 Maybe it's a p value of 0.07. 00:08:46.244 --> 00:08:50.174 Uh, some cases it's actually been a p value of 0.025, 00:08:50.174 --> 00:08:52.934 but yet FDA is not convinced. 00:08:52.934 --> 00:08:54.794 Regulators are not convinced. 00:08:55.214 --> 00:09:00.404 So let's take a case where there's positive data and a a p value 00:09:00.404 --> 00:09:04.874 of point, uh, a one-sided 0.04, 00:09:05.359 --> 00:09:08.119 Kert Viele: And I, I think Scott, Scott, to this, we should add 00:09:08.449 --> 00:09:12.049 another situation this happens is somebody runs a phase two. 00:09:12.834 --> 00:09:16.554 And gets really positive results, but it's not quite there. 00:09:16.824 --> 00:09:20.124 And their question is, Hey, haven't we already fulfilled at least 00:09:20.124 --> 00:09:21.804 part of the bar for approval? 00:09:22.014 --> 00:09:23.454 Same kind of idea. 00:09:23.454 --> 00:09:25.644 We're partway there, but we're not fully there. 00:09:27.224 --> 00:09:31.994 Scott Berry: So company A has run a trial that's not statistically significant, 00:09:31.994 --> 00:09:36.439 but it's close, and maybe they even went to FDA and say, will you approve us? 00:09:37.394 --> 00:09:42.404 Now, I've never been at the FDA, but I can imagine them thinking hard about 00:09:42.404 --> 00:09:47.594 it, actually thinking the drug probably works, but it hasn't met the thing. 00:09:47.594 --> 00:09:48.914 We can't approve it yet. 00:09:49.544 --> 00:09:52.124 So company A is in that situation. 00:09:52.454 --> 00:09:57.704 Do you tell them you have to go and run a trial ignoring all of 00:09:57.704 --> 00:10:02.684 the things we know from that first trial and you have to jump a 0.025 00:10:02.714 --> 00:10:04.874 hurdle and we, we ignore it. 00:10:05.774 --> 00:10:10.364 Company B has no data and they have a treatment for the same disease. 00:10:10.364 --> 00:10:11.714 They wanna run the same trial. 00:10:11.714 --> 00:10:15.044 They're asked to run a trial with a 2.5% 00:10:15.164 --> 00:10:15.884 type one error. 00:10:16.364 --> 00:10:19.934 Um, are, are the companies asked to do exactly the same thing? 00:10:20.834 --> 00:10:27.014 Because a ton of scientific sense that that first company a can utilize the 00:10:27.014 --> 00:10:32.444 information from that trial and maybe they don't need as large of a trial 00:10:32.444 --> 00:10:34.544 to demonstrate that the drug works. 00:10:36.254 --> 00:10:40.514 So that's a circumstance that we're talking about today where you have prior 00:10:40.514 --> 00:10:46.004 probability going into a new trial that the treatment's effective and you want 00:10:46.004 --> 00:10:49.634 to use that to ana analyze the new trial. 00:10:51.194 --> 00:10:55.094 Uh, a and it could be that first trial is a phase two trial. 00:10:55.094 --> 00:10:59.414 It could be that first trial, uh, you know, was a biomarker trial that just 00:10:59.414 --> 00:11:01.334 happened to show clinical benefit there. 00:11:01.334 --> 00:11:04.124 There's a lot of ways in which that information comes about. 00:11:05.354 --> 00:11:11.774 Another circumstance and we did a complex innovative design trial where this was a, 00:11:11.834 --> 00:11:18.254 a public presentation of a circumstance where a company had demonstrated 00:11:18.254 --> 00:11:27.224 benefit in two types of epilepsy, and they wanted to investigate a third. 00:11:27.659 --> 00:11:30.089 There's a commonality in the diseases. 00:11:30.089 --> 00:11:33.929 There's a commonality in the endpoint, the mechanism of the disease. 00:11:34.289 --> 00:11:38.129 You're taking a treatment that's been shown effective in a related 00:11:38.309 --> 00:11:42.989 but slightly different disease, and IT benefits, clearly benefits. 00:11:42.989 --> 00:11:48.809 These other two, and now you're going into a third, should the standard be. 00:11:49.679 --> 00:11:54.539 A trial in that subtype of disease 0.025 00:11:54.539 --> 00:11:59.009 type one error, or should there be some sense of borrowing from the others 00:11:59.549 --> 00:12:03.329 and, and for example, in this case, it was showing about a 30% reduction in 00:12:03.329 --> 00:12:05.429 seizures in both of the other types. 00:12:05.609 --> 00:12:08.789 If all of a sudden relatively early in this third type you're seeing a 00:12:08.789 --> 00:12:13.679 30% reduction in seizures, you're pretty, pretty sure the drug works 00:12:13.679 --> 00:12:15.599 because of that prior probability. 00:12:16.259 --> 00:12:18.239 Yes, these are cases. 00:12:18.239 --> 00:12:24.539 The third case, uh, talks a lot about it is you might have a conclusive proof 00:12:24.539 --> 00:12:27.059 of benefit in an adult population. 00:12:27.539 --> 00:12:31.259 There are rare circumstances where pediatrics have the same disease. 00:12:31.259 --> 00:12:32.009 It's very rare. 00:12:32.009 --> 00:12:33.149 It's hard to run a trial. 00:12:34.139 --> 00:12:35.609 Do we just ignore them? 00:12:35.609 --> 00:12:38.519 You might run a trial and borrow from the adult in some way. 00:12:38.519 --> 00:12:44.549 So there's a lot of circumstances that you go into a phase three confirmatory trial 00:12:44.909 --> 00:12:49.169 with prior information about the efficacy of the treatment that's in that trial. 00:12:50.414 --> 00:12:53.414 Kert Viele: And I, you know, one thing to add to this, you know, these kind 00:12:53.414 --> 00:13:00.194 of situations, they've come up in the past and there are often ways that 00:13:00.404 --> 00:13:06.704 regulators have made the bar a little bit easier given this prior information. 00:13:06.704 --> 00:13:09.374 And that may be, you know, one trial versus two. 00:13:09.374 --> 00:13:10.364 It may be, hey. 00:13:10.934 --> 00:13:14.324 We'll change something about the, you know, what you have to 00:13:14.324 --> 00:13:16.034 do in your phase three trial. 00:13:16.454 --> 00:13:20.294 What we're really talking about today is trying to formalize this process. 00:13:20.294 --> 00:13:25.244 So the question is, if you wanna apply all the statistical rigor 00:13:25.244 --> 00:13:27.434 that we usually apply to a trial. 00:13:28.544 --> 00:13:31.724 You know, ordinarily this is, you gotta do 0.025 00:13:31.784 --> 00:13:32.534 all the time. 00:13:32.564 --> 00:13:36.014 Is there a middle ground where we could basically formalize and say, 00:13:36.014 --> 00:13:37.844 here is how we're gonna analyze it. 00:13:38.174 --> 00:13:42.434 We're still gonna keep all the statistical rigor, but we're gonna recognize that 00:13:42.434 --> 00:13:44.414 you do have this, your partway there. 00:13:46.609 --> 00:13:49.814 Scott Berry: Okay, so suppose we have this prior probability. 00:13:51.119 --> 00:13:54.599 And we have a prior distribution about a treatment. 00:13:55.049 --> 00:14:01.469 And, um, we're going in and we're running a new experiment now, and we're gonna 00:14:01.469 --> 00:14:03.479 collect data in the new experiment. 00:14:03.869 --> 00:14:08.009 And the Bayesian approach is, we're gonna use this prior in some way, and we'll talk 00:14:08.009 --> 00:14:12.629 about different priors in that dynamic discounting and the, those aspects of it. 00:14:12.959 --> 00:14:16.679 But we're gonna use that prior in some way with the new data. 00:14:18.179 --> 00:14:19.859 Now explain. 00:14:20.594 --> 00:14:25.454 Why that can inflate type one error and what type one error means and why. 00:14:25.454 --> 00:14:30.524 It's a little bit disconnected in the, the way we're going about doing this. 00:14:33.194 --> 00:14:36.014 Kert Viele: So what's gonna happen here is suppose that, you know, my, 00:14:36.044 --> 00:14:41.564 my, whatever my prior evidence says is I think I have a 10% effect, 10% 00:14:41.564 --> 00:14:46.784 beneficial effect on something when I come in and compute type one error. 00:14:47.234 --> 00:14:52.994 What I am saying is, let's assume that the control and treatment arms 00:14:52.994 --> 00:14:55.004 are exactly equal to each other. 00:14:55.394 --> 00:14:55.904 Tell me 00:14:55.929 --> 00:14:57.584 Scott Berry: In in the new experiment, 00:14:57.599 --> 00:14:58.739 Kert Viele: In the new experiment. 00:14:58.919 --> 00:14:59.249 Yeah. 00:14:59.609 --> 00:14:59.849 Yep. 00:14:59.879 --> 00:15:02.099 So we got our historical evidence on one side. 00:15:02.099 --> 00:15:05.579 We got our current trial on the other, which I'll refer to it if there's 00:15:05.579 --> 00:15:07.379 no effect in the current trial. 00:15:08.159 --> 00:15:09.929 Now, this is something we don't think is true. 00:15:09.929 --> 00:15:14.039 We have prior evidence, so we at least are less likely to think it's true, 00:15:14.039 --> 00:15:15.449 is the more precise way to say it. 00:15:15.869 --> 00:15:21.029 But if it is true, what's gonna happen is that prior evidence, 00:15:21.359 --> 00:15:23.399 it's gonna bias things upward. 00:15:23.429 --> 00:15:28.139 It's gonna say, Hey, if I didn't see an effect in the current trial, I'm 00:15:28.139 --> 00:15:32.459 gonna average this in some way with the beneficial effect from the past. 00:15:32.819 --> 00:15:33.269 I'm gonna. 00:15:33.349 --> 00:15:36.109 Pull things up a little bit, I'm gonna basically say, you know, 00:15:36.109 --> 00:15:38.089 maybe I have a two or 3% effect. 00:15:38.869 --> 00:15:44.509 The net result of that is that little bit of bias makes you more likely 00:15:44.509 --> 00:15:50.119 to claim efficacy if, if, if, if all of this is, if you don't have 00:15:50.119 --> 00:15:51.589 an effect in the current trial. 00:15:52.814 --> 00:15:54.859 So the key issue is. 00:15:55.394 --> 00:15:57.344 What is the value of that assumption? 00:15:57.644 --> 00:16:01.454 And this is where we get into the guidance and you know, 00:16:01.454 --> 00:16:03.044 do we control type one error? 00:16:03.044 --> 00:16:03.914 Do we not? 00:16:04.544 --> 00:16:09.164 We will often do that calculation and say, you know, if this is 00:16:09.164 --> 00:16:12.014 true, here is the type one error. 00:16:12.404 --> 00:16:16.934 The the results of the type one error for various kinds of designs, and they're 00:16:16.934 --> 00:16:20.864 gonna be hired two point half percent because of this bias from the past. 00:16:21.884 --> 00:16:27.314 The argument that you make is this assumption that I'm worried about. 00:16:27.314 --> 00:16:29.234 The null hypothesis is true. 00:16:29.804 --> 00:16:32.114 I've already partially disproven it. 00:16:32.114 --> 00:16:37.514 So the value of that type one error calculation is less in that case. 00:16:37.514 --> 00:16:39.824 And that's where we get into, I know where you want to go. 00:16:39.824 --> 00:16:42.164 Discussions of with and without type one error. 00:16:43.019 --> 00:16:43.099 Scott Berry: Hmm. 00:16:44.159 --> 00:16:44.864 So if, uh, and. 00:16:46.289 --> 00:16:50.489 If you are gonna use that prior distribution, uh, to it, and, 00:16:50.489 --> 00:16:54.899 and what might that look like is we're gonna use the prior, use the 00:16:54.899 --> 00:16:58.949 new data and say we need a 97.5% 00:16:58.949 --> 00:17:03.779 probability that the treatment's beneficial after the combination of the 00:17:03.779 --> 00:17:06.629 two, that that's gonna be successful. 00:17:06.629 --> 00:17:09.449 We may go to FDA, and that's the criterion. 00:17:10.319 --> 00:17:13.769 Now, if we weren't using prior distributions. 00:17:14.609 --> 00:17:18.419 That probably controls type one error at 2.5% 00:17:18.419 --> 00:17:19.919 for using a flat prior 00:17:19.979 --> 00:17:23.669 that that Bayesian analysis is similar to a frequentist analysis. 00:17:24.269 --> 00:17:27.419 Uh, and so the type one error would be controlled for the new 00:17:27.419 --> 00:17:32.189 experiment, and that would satisfy in the guidance where it says you 00:17:32.189 --> 00:17:33.959 are controlling type one error. 00:17:34.229 --> 00:17:36.604 Because really all we're using is the new experiment. 00:17:36.914 --> 00:17:38.564 Uh, because we're not bringing 00:17:38.564 --> 00:17:41.594 any prior, once we bring that prior 00:17:42.194 --> 00:17:50.294 and we combine them together now, uh, as, as Kurt says, to get 97.5% 00:17:50.294 --> 00:17:55.214 with the combination, the new trial doesn't have to be 97.5 00:17:55.274 --> 00:17:56.144 by itself. 00:17:56.774 --> 00:17:59.924 That might be 93% is enough. 00:18:00.194 --> 00:18:05.684 Combined with the prior to say overall combined, we're 97.5. 00:18:07.064 --> 00:18:12.014 If you evaluate only the new experiment, it might have a 7% type 00:18:12.014 --> 00:18:13.664 one error in that circumstance, 00:18:14.089 --> 00:18:14.309 Kert Viele: Yep. 00:18:14.714 --> 00:18:18.374 Scott Berry: and that's where the FDA might agree. 00:18:18.374 --> 00:18:18.794 Okay? 00:18:18.794 --> 00:18:24.104 The data you're using is reliable, it's relevant, it's appropriate. 00:18:24.104 --> 00:18:29.654 We agree that you can use that for approval as a combination. 00:18:30.629 --> 00:18:37.319 Then they're approving within only this modular new experiment, the type 00:18:37.319 --> 00:18:39.989 one errors above the traditional 2.5 00:18:40.319 --> 00:18:43.739 and maybe it's 7% in that circumstance. 00:18:43.979 --> 00:18:50.459 So that's kind of the difference between a Bayesian analysis that controls 00:18:50.459 --> 00:18:54.479 type one error for the new modular experiment and one that doesn't, 00:18:55.909 --> 00:18:57.079 Kert Viele: Yeah, and I, I think we're. 00:18:57.119 --> 00:18:57.689 Scott Berry: about a lot. 00:18:57.689 --> 00:18:57.899 Yep. 00:18:58.304 --> 00:19:00.044 Kert Viele: we're, we're talking about the combination. 00:19:00.074 --> 00:19:03.854 I, I always tend to view this if, if I were doing things from scratch 00:19:03.854 --> 00:19:08.984 and I'm gonna do two trials, um, is that combination good enough? 00:19:09.224 --> 00:19:12.044 And the question is, you know, I'm basically designing my new 00:19:12.044 --> 00:19:17.114 experiment when the original, the historical data is historical. 00:19:17.114 --> 00:19:18.944 It's in the past so I can condition on it. 00:19:21.044 --> 00:19:25.424 Scott Berry: So let's, let's, um, and, and we'll, we'll leave as a separate topic. 00:19:26.099 --> 00:19:28.739 Where we're borrowing only on the control arm. 00:19:29.849 --> 00:19:32.159 Yes, that can inflate type one error. 00:19:32.159 --> 00:19:37.199 It's a different kind of thing about the new temporal differences in the new trial. 00:19:37.229 --> 00:19:40.799 And so we're, we're, we're talking about the circumstance where we're 00:19:40.799 --> 00:19:45.569 going in with an informative prior about the treatment effect, and you're 00:19:45.569 --> 00:19:48.599 having discussions with FDA about that. 00:19:49.184 --> 00:19:52.454 Kert Viele: So I'm gonna ask you, I'm gonna derail this slightly, and 00:19:52.454 --> 00:19:57.554 I haven't told you I'm about to ask this, but what about situations, let's 00:19:57.554 --> 00:20:04.994 Alzheimer's sepsis, where nothing has worked in the past that potentially 00:20:04.994 --> 00:20:06.074 should give us a negative prior? 00:20:07.844 --> 00:20:09.104 How should people view that? 00:20:10.799 --> 00:20:14.609 Scott Berry: Uh, I think it's appropriate and I think it's, it's been an 00:20:14.609 --> 00:20:17.639 unspoken thing probably at the agency. 00:20:17.909 --> 00:20:22.349 And there was a time not long ago where there were 25 straight 00:20:22.349 --> 00:20:24.359 failed phase three trials. 00:20:24.914 --> 00:20:26.114 Kert Viele: Which is unlucky, by the way. 00:20:26.114 --> 00:20:27.254 They should have won one by 00:20:27.719 --> 00:20:28.859 Scott Berry: Somebody should have. 00:20:29.129 --> 00:20:29.579 Yes. 00:20:29.879 --> 00:20:34.589 Uh, and that, by the way, the 26 trial, the FDA may be weary of 00:20:34.589 --> 00:20:38.849 that, that if all of a sudden the 26 trials shows P value of 0.02, 00:20:39.179 --> 00:20:40.409 maybe they're the lucky one. 00:20:41.174 --> 00:20:45.494 Uh, and you can imagine a, a skeptical prior, which is 00:20:45.494 --> 00:20:47.024 talked about in the guidance, 00:20:47.024 --> 00:20:48.284 in that circumstance. 00:20:48.734 --> 00:20:53.714 Uh, and maybe they even ask for a higher threshold of that, uh, circumstance. 00:20:53.714 --> 00:20:58.424 So that's kind of a different prior going in where historically the 00:20:58.424 --> 00:21:04.059 scenario is, is, is, has been nothing works, uh, uh, in that scenario. 00:21:05.039 --> 00:21:05.189 Kert Viele: Yep. 00:21:05.384 --> 00:21:09.044 Scott Berry: so you carry the burden of the disease with you as you, 00:21:09.044 --> 00:21:12.434 as you go into the agency, which I think is quite, quite reasonable, 00:21:12.899 --> 00:21:15.839 Kert Viele: And you don't wanna discourage research in these areas and so on. 00:21:15.839 --> 00:21:17.339 It's just a really hard problem. 00:21:17.339 --> 00:21:21.869 I wasn't expecting an answer for it, but we, we should be honest about it 00:21:21.869 --> 00:21:23.429 exists in the other direction too. 00:21:24.044 --> 00:21:24.224 Scott Berry: yep. 00:21:25.699 --> 00:21:25.989 Okay. 00:21:26.639 --> 00:21:31.709 So what this may look like, um, in the interaction. 00:21:31.709 --> 00:21:37.829 So we're in a circumstance, and by the way, we get the challenges of this. 00:21:38.189 --> 00:21:43.629 So, for example, we might get sponsors that come to us that ran a phase 3 00:21:43.649 --> 00:21:48.629 trial, and it was a P value of 0.3. 00:21:49.484 --> 00:21:52.964 Barely observed positive differences. 00:21:53.294 --> 00:21:57.584 They find a subset of 30% of the population that, wow, 00:21:57.584 --> 00:21:59.504 it's, it's nearly significant. 00:22:00.104 --> 00:22:05.504 And they want to take that subset and ignore everything else and go 00:22:05.504 --> 00:22:11.024 to the FDA and borrow from that subset in a new trial, and it gives 00:22:11.024 --> 00:22:12.839 them a jumpstart for approval. 00:22:13.244 --> 00:22:13.274 Okay. 00:22:14.744 --> 00:22:20.144 And now they've cherry picked from the previous trial, and by 00:22:20.144 --> 00:22:21.494 the way, they can explain why. 00:22:21.494 --> 00:22:25.064 Of course, that's the relevant subgroup and all of that, and they want to 00:22:25.064 --> 00:22:26.834 go to FDA and borrow from that. 00:22:27.284 --> 00:22:30.794 And there are circumstances where you and I, they come to us. 00:22:30.794 --> 00:22:34.064 We'd say if we were at FDA, we wouldn't accept that. 00:22:35.369 --> 00:22:40.829 Because we believe this is largely about multiplicity, and now you 00:22:40.829 --> 00:22:44.579 can't pull that out and create a prior that's only based on that. 00:22:45.269 --> 00:22:50.939 If you wanna use a prior, based on the entire trial, that might be something 00:22:50.939 --> 00:22:53.759 to discuss and, and have that. 00:22:53.759 --> 00:23:00.569 But to self-select data and only use that data going to the FDA, 00:23:00.779 --> 00:23:02.309 you're probably gonna get a no. 00:23:03.179 --> 00:23:06.929 And by the way, historically that's happened to companies and they come 00:23:06.929 --> 00:23:08.989 back and say, FDA doesn't do Bayes 00:23:09.749 --> 00:23:11.249 Kert Viele: Yeah, that's always frustrating. 00:23:11.639 --> 00:23:12.119 Scott Berry: right, right. 00:23:12.119 --> 00:23:17.009 And it's had nothing to do with Bayes It's, you're, you're misusing it in ways 00:23:17.219 --> 00:23:21.854 that we, as Bayesians would say, no, we, we, I would not accept that circumstance. 00:23:23.114 --> 00:23:25.334 So there's a huge part of the data. 00:23:25.334 --> 00:23:26.864 You're going into the FDA. 00:23:27.134 --> 00:23:28.574 What are you using? 00:23:28.874 --> 00:23:32.324 Uh, has it been, you know, self-selected? 00:23:32.534 --> 00:23:35.354 Is it all of the data out there? 00:23:35.564 --> 00:23:37.604 Is it relevant for the Nutra? 00:23:37.604 --> 00:23:40.214 So there's a huge part of this, of the science. 00:23:40.214 --> 00:23:42.409 Science is hard and, and you go in. 00:23:42.414 --> 00:23:47.024 So if we go in with a circumstance where we believe we're bringing 00:23:47.024 --> 00:23:49.784 to them a, a relevant prior. 00:23:49.854 --> 00:23:53.034 Based on the previous information that becomes part of the 00:23:53.034 --> 00:23:54.324 discussion with the FDA. 00:23:54.324 --> 00:23:55.494 Is this reasonable, 00:23:55.994 --> 00:23:58.184 Kert Viele: Well, and it becomes a, it becomes a major part of 00:23:58.184 --> 00:23:59.504 the discussion with the FDA. 00:23:59.534 --> 00:24:04.994 I think we spend as much time on data as we do on the methodology at this point. 00:24:05.414 --> 00:24:07.364 It certainly is the main reason. 00:24:07.364 --> 00:24:09.464 As you said, FDA says no. 00:24:09.989 --> 00:24:14.369 Um, a lot of times we're not told where this subset came from. 00:24:14.369 --> 00:24:18.179 It's, Hey, I want to do a trial in this population, and this is the 00:24:18.179 --> 00:24:20.159 data I have on this population. 00:24:20.549 --> 00:24:23.339 And, you know, at some level that's a red flag at this point 00:24:23.339 --> 00:24:24.869 that I need to ask the question. 00:24:25.229 --> 00:24:28.229 You know, you said this is the data we have on this population. 00:24:28.229 --> 00:24:29.219 What's the data? 00:24:29.279 --> 00:24:29.889 What's the, what's the. 00:24:30.479 --> 00:24:34.859 What's the world actually look like so that I can assess whether 00:24:34.859 --> 00:24:38.189 this makes, whether this, you know, is it cherry picked? 00:24:38.339 --> 00:24:40.199 How much is it cherry picked? 00:24:40.409 --> 00:24:42.059 All of these things come into play. 00:24:42.374 --> 00:24:42.644 Scott Berry: Hmm. 00:24:42.764 --> 00:24:43.034 Yeah. 00:24:43.034 --> 00:24:44.714 The, the dog that didn't bark. 00:24:44.714 --> 00:24:46.574 What are you not telling us? 00:24:46.694 --> 00:24:49.664 Uh, because we get told the rosy part of it. 00:24:49.664 --> 00:24:52.814 And what are you not telling us and what, what data are we not using? 00:24:53.534 --> 00:24:54.044 Um. 00:24:54.464 --> 00:24:58.964 And so for example, in the case of the complex innovative design program 00:24:58.964 --> 00:25:01.904 where we've got these two other indications and they don't have a 00:25:01.904 --> 00:25:05.024 third indication that failed, for example, if they did, you'd want to 00:25:05.024 --> 00:25:06.764 include that in a hierarchical model. 00:25:07.484 --> 00:25:12.164 Or there's a, a large phase three and we're using the entire estimate that comes 00:25:12.164 --> 00:25:14.144 from that to borrow for the new trial. 00:25:14.204 --> 00:25:18.014 And we go to the agency and we're presenting them 00:25:18.014 --> 00:25:19.424 how we're doing the borrow. 00:25:20.579 --> 00:25:25.079 So I'll give you one example of a case that the device was approved. 00:25:25.319 --> 00:25:29.579 The Watchman device is a left atrial appendage device that's 00:25:29.579 --> 00:25:33.569 been approved by CDRH, uh, at FDA. 00:25:33.749 --> 00:25:40.619 And they or originally ran a trial that was inconclusive and the name 00:25:40.619 --> 00:25:44.279 of that trial, the original trial, uh, I'm gonna get this wrong. 00:25:44.579 --> 00:25:48.869 Um, the original trial was. 00:25:49.334 --> 00:25:50.834 Uh, protect af. 00:25:51.869 --> 00:25:55.799 And then they ran a second confirmatory trial called Prevail. 00:25:56.339 --> 00:26:01.139 And the Prevail trial borrowed where we discounted the first 00:26:01.139 --> 00:26:03.899 trial by 50% in a static way. 00:26:03.899 --> 00:26:05.699 We'll just discount it 50%. 00:26:05.969 --> 00:26:10.229 And by the way, this, this went through FDA, uh, went 00:26:10.229 --> 00:26:11.879 through Bram Zuckerman's group. 00:26:12.179 --> 00:26:13.409 Uh, Bram is. 00:26:13.739 --> 00:26:19.319 Is a brilliant scientist and understands what it means to borrow 00:26:19.319 --> 00:26:21.719 the implications for the new trial. 00:26:22.019 --> 00:26:28.049 More than once, these types of things have gone through the cardiovascular device 00:26:28.049 --> 00:26:33.599 division at the FDA absolutely brilliant scientist who was involved in this 00:26:33.599 --> 00:26:36.179 particular case in, in that borrowing. 00:26:36.509 --> 00:26:42.059 So we go in and we propose that level of borrowing and show. 00:26:42.629 --> 00:26:44.399 What happens in the new trial? 00:26:45.869 --> 00:26:50.999 The most important thing to show, and I had a I, I had a client discussion 00:26:50.999 --> 00:26:59.549 yesterday with the same thing is what example trials of the new trial, when 00:26:59.549 --> 00:27:03.689 is it successful with the borrowing? 00:27:04.109 --> 00:27:07.079 And then how does that compare to if we didn't borrow? 00:27:07.229 --> 00:27:09.764 So for example, if we're, we, we've got a. 00:27:10.694 --> 00:27:14.264 Uh, time to event analysis where we're borrowing data. 00:27:14.774 --> 00:27:17.924 If we don't borrow, we need a hazard ratio of 0.7 00:27:17.924 --> 00:27:19.964 or better to win with borrowing. 00:27:19.964 --> 00:27:22.124 We show that if it's between 0.7 00:27:22.124 --> 00:27:23.594 and 0.77 00:27:23.924 --> 00:27:27.674 with the borrowing from the historical data that showed 0.7, 00:27:28.394 --> 00:27:30.344 those are still successful. 00:27:31.079 --> 00:27:31.979 Kert Viele: The field goal. 00:27:33.149 --> 00:27:35.969 Scott Berry: That, yes, the Bayesian field goal. 00:27:36.329 --> 00:27:40.889 And so you show them the, I think the most of we, sometimes we, we 00:27:40.889 --> 00:27:42.269 revert to operating characteristics. 00:27:43.379 --> 00:27:44.999 If the truth is 0.7, 00:27:44.999 --> 00:27:47.249 our trial's 93% powered. 00:27:47.249 --> 00:27:52.199 We, we, we would tell the FDA or we would say, if the device doesn't work, 00:27:52.199 --> 00:27:54.869 the new trial has an 8% type one error. 00:27:55.319 --> 00:28:00.599 And we, you, we talked about what that means is that additional trials are being 00:28:00.599 --> 00:28:05.279 demonstrated with the final analysis at the end being Bayesian as a combination 00:28:05.279 --> 00:28:07.259 of the trials being successful. 00:28:07.929 --> 00:28:12.459 But I, I think that's hard for the clinicians at the FDA That's hard for a, 00:28:12.819 --> 00:28:15.429 a proxy for Bram Zuckerman to understand. 00:28:15.429 --> 00:28:16.719 Okay, what am I agreeing to? 00:28:17.669 --> 00:28:24.179 In that show 'em a example of a final trial that the final 00:28:24.179 --> 00:28:27.389 trial shows an estimate of 0.76. 00:28:27.629 --> 00:28:31.529 The Bayesian analysis set gives amine of 0.72 00:28:32.009 --> 00:28:36.149 and a confidence interval that's less than one, that this would now be approvable. 00:28:36.149 --> 00:28:39.934 They can see exactly what they're agreeing to that. 00:28:40.964 --> 00:28:44.984 So that's one of the things we show them, is depending on the result, we 00:28:44.984 --> 00:28:51.014 also wanna show them if the new trial shows a hazard ratio of one, it doesn't 00:28:51.014 --> 00:28:53.534 say, oh, the the treatment works. 00:28:54.104 --> 00:28:56.954 They would not, they, they would say, wait a minute, wait a minute. 00:28:56.984 --> 00:28:59.234 You know, we, we don't agree to that. 00:28:59.234 --> 00:29:01.814 We don't want this Bayesian thing to flip that. 00:29:01.994 --> 00:29:03.524 So what is it flipping? 00:29:04.319 --> 00:29:07.799 Kert Viele: So we've had a couple situations, I can think of one in 00:29:07.799 --> 00:29:13.199 oncology with rare cancers where this was like your epilepsy example where 00:29:13.199 --> 00:29:15.179 we're trying to extend the label. 00:29:15.419 --> 00:29:17.789 And so we're borrowing across indications. 00:29:18.059 --> 00:29:21.329 And one of the things the FDA was particularly concerned about. 00:29:21.979 --> 00:29:23.749 Is, remember, these are all rare. 00:29:23.749 --> 00:29:25.339 We may not enroll very many. 00:29:25.699 --> 00:29:29.419 They never wanted a situation, and I should say ultra rare. 00:29:29.419 --> 00:29:33.079 I mean, we're gonna get a handful, but they didn't want us coming in 00:29:33.079 --> 00:29:37.489 and saying, you know, we got zero responses in four patients, and 00:29:37.489 --> 00:29:39.379 we're gonna say you should approve. 00:29:39.659 --> 00:29:43.769 On the basis of the prior, we need some, we need it to work 00:29:43.769 --> 00:29:45.329 somewhere, which makes perfect sense. 00:29:45.329 --> 00:29:47.309 We need some evidence that it's doing something. 00:29:47.609 --> 00:29:49.709 And so we've had those discussions. 00:29:49.709 --> 00:29:52.349 You know, show us the minimum that we've had. 00:29:52.859 --> 00:29:58.109 I remember one a while back it was, we were borrowing adult data 00:29:58.289 --> 00:30:02.789 to pediatrics and again, we were gonna get small sample sizes. 00:30:03.119 --> 00:30:06.299 And the FDA actually said explicitly. 00:30:07.079 --> 00:30:12.869 We would be willing if you got 15 responses out of, and I don't remember 00:30:12.869 --> 00:30:16.709 how many, uh, kids we were gonna be able to enroll, but basically if you 00:30:16.709 --> 00:30:22.829 can get 15 out of 25, we'll be happy to accept that, that we think that's enough 00:30:22.829 --> 00:30:24.629 given the adult data and everything. 00:30:24.749 --> 00:30:28.979 And so we basically back solved, you know, what would this do, and 00:30:28.979 --> 00:30:32.489 then showed that this had reasonable operating characteristics as well. 00:30:32.759 --> 00:30:36.149 But that was the way the interaction went in terms of. 00:30:36.609 --> 00:30:39.344 Here's the data that we think would be convincing to us. 00:30:42.389 --> 00:30:45.749 Scott Berry: What, and and so through this interaction, we're 00:30:45.749 --> 00:30:47.069 showing them the new power. 00:30:47.069 --> 00:30:49.644 We're showing them the type one era of this new. 00:30:50.429 --> 00:30:55.229 Modular experiment with the Bayesian analysis that happens at the end. 00:30:55.229 --> 00:30:57.119 We're showing them example trials. 00:30:57.629 --> 00:31:04.319 We've had people at the FDA say, okay, your new experiment has a 00:31:04.319 --> 00:31:07.139 8% type one error rather than 2.5. 00:31:07.589 --> 00:31:13.619 In that, what if we just say you can run a a, a new phase three 00:31:13.619 --> 00:31:18.419 trial without Bayesian borrowing and you get an alpha of 0.08. 00:31:19.724 --> 00:31:26.534 That, that, that we go to, that, you know, in, in many circumstances, that 00:31:26.534 --> 00:31:32.114 gives the same success and failure of approval that the Bayesian analysis 00:31:32.114 --> 00:31:33.854 creates at the end of the day. 00:31:36.324 --> 00:31:37.794 Kert Viele: Okay, you're asking me a trick question. 00:31:37.824 --> 00:31:39.984 So this, this one gets a little nuanced here. 00:31:40.284 --> 00:31:44.934 Um, and I think the, the answer does depend on are you borrowing only control? 00:31:44.934 --> 00:31:46.674 Are you borrowing control and treatment? 00:31:47.034 --> 00:31:50.934 Are you borrowing dynamically or are you borrowing statically? 00:31:51.504 --> 00:31:55.044 Um, I'm gonna, I'm gonna certainly focus on the dynamic borrowing, 00:31:55.044 --> 00:31:56.994 which I think we prefer to do. 00:31:57.444 --> 00:32:00.024 Um, and I'll have to talk about what that is 'cause I don't think 00:32:00.024 --> 00:32:01.074 we've really brought that up. 00:32:01.344 --> 00:32:05.184 The dynamic borrowing the idea is if you're walking in with prior 00:32:05.184 --> 00:32:09.444 information from a previous trial, suppose your current trial, it's 00:32:09.444 --> 00:32:12.804 ongoing and it just looks different. 00:32:13.239 --> 00:32:16.869 So the rates are different than what you saw in history, the treatment 00:32:16.869 --> 00:32:18.429 effects, what you saw in history. 00:32:18.999 --> 00:32:25.089 The notion is, you know, should I continue to believe this prior or should I go, Hey, 00:32:25.089 --> 00:32:27.099 something different is going on there. 00:32:27.189 --> 00:32:31.299 And if you got into the math, what you're really saying is you have a 00:32:31.329 --> 00:32:36.369 prior where maybe the current trial matches history and maybe it doesn't. 00:32:36.399 --> 00:32:38.769 Is there, there's an aspect of that to your prior. 00:32:39.164 --> 00:32:41.174 So what happens when things don't match? 00:32:41.534 --> 00:32:43.244 You would borrow less. 00:32:43.304 --> 00:32:46.454 So you start to give less weight to history, and this 00:32:46.454 --> 00:32:47.984 is a mitigation strategy. 00:32:48.044 --> 00:32:51.854 When you have mismatch, you borrow less, you're less prone 00:32:51.854 --> 00:32:53.464 to make these false conclusions. 00:32:54.809 --> 00:33:00.239 Doing that is far better than just increasing the alpha because increasing 00:33:00.239 --> 00:33:04.889 the alpha, basically you're saying, I'm always willing to take this extra error 00:33:04.889 --> 00:33:09.239 regardless of what happens and you don't have any of these mitigation strategies. 00:33:09.599 --> 00:33:14.399 So I think it's really important that you go, you know, I, I'm, I'm changing 00:33:14.399 --> 00:33:19.769 the alpha because I have evidence that it works and I should be able to 00:33:19.769 --> 00:33:21.929 discount that evidence appropriately. 00:33:22.664 --> 00:33:24.764 If the new data says it's not right. 00:33:27.019 --> 00:33:30.289 Scott Berry: The, the other nice thing that come, well, two parts 00:33:30.289 --> 00:33:32.779 that comes from the Bayesian part that's a little bit different 00:33:32.779 --> 00:33:34.819 than just saying, oh, you get 0.08 00:33:34.849 --> 00:33:40.129 alpha, is that there was an explicit incorporation of the strength of the 00:33:40.129 --> 00:33:43.159 previous data that generated that eight. 00:33:44.089 --> 00:33:44.309 Kert Viele: Yep. 00:33:44.774 --> 00:33:47.594 Scott Berry: what you got in the new experiment that gave 00:33:47.594 --> 00:33:50.894 you a final answer of 97.5% 00:33:50.894 --> 00:33:52.454 or 99, whatever that is. 00:33:52.904 --> 00:33:58.454 So it's not arbitrarily, oh, we think you get 8%, you get 11%. 00:33:58.814 --> 00:34:04.004 It's the combined data that tells you what that implied value is. 00:34:04.574 --> 00:34:09.554 But the other part is if you just take 8%, one sided, your 00:34:09.554 --> 00:34:11.324 estimate at the end of the day. 00:34:12.029 --> 00:34:14.789 Is not utilizing that information. 00:34:15.089 --> 00:34:17.999 The confidence interval doesn't match the 00:34:17.999 --> 00:34:23.249 result of the trial, uh, in that, and the estimate can be as important 00:34:23.249 --> 00:34:26.519 as the final result in the trial, which is a combined estimate. 00:34:26.579 --> 00:34:31.529 So there, there are, I, I, I believe there's lots of advantages of 00:34:31.529 --> 00:34:37.649 using the Bayesian approach to, to generate the final combined analysis 00:34:37.649 --> 00:34:39.689 of all of the data that we know. 00:34:40.229 --> 00:34:42.629 Jumping a regulatory hurdle. 00:34:43.274 --> 00:34:47.714 Kert Viele: And a lot of times that can be even borrowing good information. 00:34:48.069 --> 00:34:50.859 You may actually pull the point estimates down. 00:34:50.859 --> 00:34:53.979 We did that actually, it was the same rare cancer trial. 00:34:54.399 --> 00:34:55.959 It had nine groups in it. 00:34:56.319 --> 00:35:01.479 One of the early data sets it, it's a blockbuster success, 00:35:01.749 --> 00:35:04.359 but it estimated a 60% rate. 00:35:04.719 --> 00:35:08.439 And you know, the, the control was estimated to be about 10. 00:35:08.439 --> 00:35:10.449 So this is a huge treatment effect. 00:35:10.929 --> 00:35:12.999 But when we did all the borrowing. 00:35:13.379 --> 00:35:16.619 It didn't raise the 60% our prior. 00:35:16.619 --> 00:35:17.369 It certainly moved it. 00:35:18.119 --> 00:35:18.389 It. 00:35:18.389 --> 00:35:23.039 It was positive information, but it only estimated it to be about 48. 00:35:23.519 --> 00:35:27.059 'cause it pulled it back and said, look, we don't think it's that good. 00:35:27.329 --> 00:35:30.689 There were also aspects, it was the highest group among many, and we've 00:35:30.689 --> 00:35:32.549 talked about that in other podcasts. 00:35:32.639 --> 00:35:36.389 But anyway, what was really cool about that is they continued to 00:35:36.389 --> 00:35:40.049 do follow up and they continue to enroll a few more patients. 00:35:40.239 --> 00:35:44.259 And lo and behold, the final data came out pretty close to 48%. 00:35:44.889 --> 00:35:48.399 So I mean, it, it, it will pull things down. 00:35:50.804 --> 00:35:51.464 Scott Berry: Okay. 00:35:51.764 --> 00:35:55.574 So, uh, uh, we've touched on a number of these issues. 00:35:55.634 --> 00:35:57.284 Uh, I'll give you an example. 00:35:57.929 --> 00:36:00.029 Of a case where this happened. 00:36:00.029 --> 00:36:03.629 Now, we, we have a number of examples of trials running right now at 00:36:03.629 --> 00:36:06.899 the FDA where there's been agreed upon Bayesian analysis of this. 00:36:06.899 --> 00:36:09.959 We we're going to them, uh, I'll give you an example of one 00:36:09.959 --> 00:36:11.309 where there was this agreement. 00:36:11.309 --> 00:36:14.459 The trial ran, and actually the data's been publicly disclosed, 00:36:14.849 --> 00:36:19.349 um, uh, because it went through an advisory committee meeting. 00:36:19.799 --> 00:36:25.734 So it, the, the treatment is called REBYOTA® And it was a Ferring 00:36:26.444 --> 00:36:32.654 company, Rebiotix And the treatment is for c Diff infection, and this 00:36:32.664 --> 00:36:34.874 REBYOTA® is now approved by Seaberg. 00:36:35.699 --> 00:36:39.509 In it and they had run a phase two trial. 00:36:39.509 --> 00:36:41.939 All of this information is public. 00:36:42.299 --> 00:36:45.179 Uh, it's a wonderful thing about the advisory committee meetings 00:36:45.179 --> 00:36:48.749 is that a huge amount of this is made public and we're doing less 00:36:48.749 --> 00:36:50.429 advisory committee meetings now. 00:36:50.459 --> 00:36:53.549 Now, and this is sort of negative of that, a whole separate topic. 00:36:53.849 --> 00:36:56.789 Um, but so the information's available. 00:36:57.194 --> 00:36:59.234 In the public register on this. 00:36:59.594 --> 00:37:03.584 So they ran a phase 2 trial, and this is a treatment for 00:37:03.974 --> 00:37:07.094 recurrent c difficile infection. 00:37:07.424 --> 00:37:12.374 This happens when individuals might be treating for cancer 00:37:12.374 --> 00:37:13.784 or a different disease. 00:37:14.084 --> 00:37:16.814 Um, they take antibiotics. 00:37:17.054 --> 00:37:20.084 It kind of destroys the gut biome. 00:37:20.084 --> 00:37:24.974 And c diff, which might live in all of us, takes over and, and now flourishes. 00:37:24.989 --> 00:37:31.499 Is, and it causes it, it causes debilitating GI issues. 00:37:31.739 --> 00:37:35.519 Diarrhea can actually be fatal, uh, in it. 00:37:35.759 --> 00:37:40.289 Now what they might do is give you another course of antibiotics 00:37:40.289 --> 00:37:41.699 and hope it sort of cures it. 00:37:41.699 --> 00:37:43.594 The recurrent aspects is, it doesn't. 00:37:44.744 --> 00:37:49.874 And the, and it's, this is a horrible, a horrible disease. 00:37:50.174 --> 00:37:53.384 The treatment is actually a fantastic treatment. 00:37:53.384 --> 00:37:56.144 It's a fecal matter transplant. 00:37:56.594 --> 00:38:01.814 They, they take feces from a healthy gut biome and through an enema. 00:38:01.814 --> 00:38:05.684 They, they give it to the individual, and the hope is that 00:38:05.684 --> 00:38:10.364 this restores the GI tracted healthy. 00:38:10.709 --> 00:38:16.259 Probiotics, healthy gut biome to to fight off the c diff infection. 00:38:17.099 --> 00:38:21.479 So they ran a phase 2 trial that was not statistically significant, 00:38:21.479 --> 00:38:27.779 showed a benefit of about 15% in preventing recurrent disease. 00:38:28.139 --> 00:38:31.889 This is a fantastic trial where there's only one endpoint. 00:38:33.089 --> 00:38:36.779 Uh, sure Mortality's looked at, but the only endpoint is 00:38:36.929 --> 00:38:38.189 does the disease come back? 00:38:38.879 --> 00:38:39.029 Yeah. 00:38:39.029 --> 00:38:42.209 You know there, there isn't a vast scale, there's nothing. 00:38:42.329 --> 00:38:44.249 There's one endpoint in the disease. 00:38:44.249 --> 00:38:48.029 Does the disease come back and the proportion of time it comes back, 00:38:48.419 --> 00:38:51.659 and if it doesn't come back within two months, you're in great shape. 00:38:51.689 --> 00:38:53.249 It's very unlikely to come back. 00:38:53.249 --> 00:38:56.549 It's almost a new circumstance of it, but if it does come back, 00:38:56.549 --> 00:38:58.049 then you're dealing with this. 00:38:58.049 --> 00:39:01.589 This case, they ran a phase two trial and showed benefit. 00:39:01.949 --> 00:39:06.659 They're now running a phase three trial is a rare disease and. 00:39:07.184 --> 00:39:08.054 These. 00:39:08.054 --> 00:39:11.654 Interestingly, it became very challenging to enroll this trial 00:39:11.894 --> 00:39:18.614 because there were outfits using FMT Fecal Matter, transplant Open Biome, 00:39:18.614 --> 00:39:25.694 for example, that patients could get, and this was not being shut down. 00:39:26.069 --> 00:39:30.329 The FDA could have shut down all these things, but they decided to, 00:39:30.329 --> 00:39:35.189 at their discretion, not, uh, but it never had never been approved before. 00:39:35.189 --> 00:39:38.159 So there's a circumstance where there's a belief this is beneficial. 00:39:38.159 --> 00:39:41.459 This phase two trial is showing now they're trying to run a phase three 00:39:41.459 --> 00:39:45.149 trial and challenging to get to 0.025 00:39:45.329 --> 00:39:48.089 in this circumstance where I think largely the scientific 00:39:48.089 --> 00:39:49.799 community believes this works. 00:39:50.819 --> 00:39:54.449 So they agreed to borrow from the phase 2 trial. 00:39:54.899 --> 00:40:04.049 In the phase 3 trial, and have a combined analysis of those two trials meeting 97.5 00:40:04.229 --> 00:40:09.179 with the combination of the two trials, the phase three trial 00:40:09.179 --> 00:40:14.549 read out, and it by itself, if you analyze only the phase three trial 00:40:14.549 --> 00:40:17.279 would've been about 93% probability. 00:40:17.279 --> 00:40:20.639 The treatment works in and of itself. 00:40:20.639 --> 00:40:22.109 That experiment didn't jump. 00:40:22.844 --> 00:40:24.254 The 0.025 00:40:24.254 --> 00:40:30.794 hurdle combined with the phase 2 trial in the pre-specified analysis, 00:40:30.794 --> 00:40:39.974 pre-specified to the phase 3 trial, but not phase 2 Um, said 99.1% 00:40:39.974 --> 00:40:44.654 probability of the treatment is beneficial when you combine together the two trials. 00:40:45.404 --> 00:40:51.644 In the way that we worked with the FDA, um, uh, in creating this combined 00:40:51.854 --> 00:40:55.574 prior with the phase three was 99.1, 00:40:55.904 --> 00:40:59.174 which had jumped the 97.5 00:40:59.204 --> 00:40:59.864 hurdle. 00:40:59.984 --> 00:41:02.359 That's the standard for clinical development. 00:41:02.684 --> 00:41:06.194 It given the rareness of the disease. 00:41:06.254 --> 00:41:08.954 It's not a two trial hurdle or the hurdle. 00:41:09.134 --> 00:41:13.274 We, we don't know whether we're in the one trial, two trial world of FDA in this 00:41:13.274 --> 00:41:18.014 circumstance, but given the rare disease, it very much fit into the, the one trial 00:41:18.014 --> 00:41:20.954 level of, of, of ev, of evidence there. 00:41:22.274 --> 00:41:23.834 This went to panel. 00:41:24.599 --> 00:41:26.129 Uh, public disclosure. 00:41:26.129 --> 00:41:29.819 And of course, the, the interesting thing is the statisticians were a bit of the 00:41:29.819 --> 00:41:31.919 hurdle as to whether this is appropriate. 00:41:32.219 --> 00:41:36.929 The statisticians split their vote one to one for it, but it, it was voted 13 00:41:36.929 --> 00:41:39.569 to four that it demonstrated efficacy. 00:41:39.989 --> 00:41:43.469 Uh, in this circumstance, the treatment has been approved 00:41:43.469 --> 00:41:45.959 by FDA, it's on the market. 00:41:46.229 --> 00:41:55.154 And an interesting thing about it is that the FDA label for REBYOTA® Gives the 99.1 00:41:55.334 --> 00:41:57.524 that came from the Bayesian analysis. 00:41:57.524 --> 00:42:02.924 There are no P values in the FDA label, but it has Bayesian posterior 00:42:02.924 --> 00:42:07.934 probabilities that it describes the combination of phase two and phase 00:42:07.934 --> 00:42:12.314 three as the estimate of efficacy that comes out of that, the treatment 00:42:12.314 --> 00:42:14.714 effect, relative difference estimate. 00:42:15.254 --> 00:42:17.774 Um, is based on that. 00:42:18.284 --> 00:42:24.824 And interestingly, at the FDA advisory committee meeting, the FDA statistician 00:42:25.214 --> 00:42:31.304 presented the posterior distribution based on the combined, showed a picture 00:42:31.304 --> 00:42:35.834 of the posterior and talked about the probability it's above zero, the 00:42:35.834 --> 00:42:41.204 probability it's at least 5% better, the probability it's at least 7% better. 00:42:41.204 --> 00:42:42.884 And this, this was. 00:42:43.289 --> 00:42:48.599 Based on FDA's analysis, the best summary of efficacy we have to present to you. 00:42:48.599 --> 00:42:50.009 Advisory committee meeting. 00:42:50.129 --> 00:42:56.699 And that picture of, of that stood on the screen for, uh, uh, a long time 00:42:56.699 --> 00:42:58.899 as here's what we're trying to, to. 00:42:59.744 --> 00:43:04.364 To make a decision on today, which was, which was sort of cool, so you can go 00:43:04.364 --> 00:43:09.344 forward and they do make reference in the new draft guidance of Rebi OTA as 00:43:09.344 --> 00:43:14.234 a case study of that, and the data's publicly disclosed for your scrutiny. 00:43:16.399 --> 00:43:17.894 Kert Viele: And the publications available. 00:43:18.614 --> 00:43:20.024 Scott Berry: And the publications available. 00:43:20.024 --> 00:43:21.944 It spends more time talking 00:43:21.944 --> 00:43:25.244 about, uh, other aspects of it than the, the pure Bayesian part. 00:43:25.544 --> 00:43:30.584 Uh, the adaptive design reports in there, the Bayesian, uh, uh, models in there. 00:43:30.734 --> 00:43:33.104 All of the pieces available, Uh, 00:43:33.104 --> 00:43:33.579 for it, which. 00:43:33.629 --> 00:43:36.364 Kert Viele: a lot of our best stuff is in the supplementary material, right? 00:43:36.629 --> 00:43:37.109 Scott Berry: Yes. 00:43:37.109 --> 00:43:37.979 Yeah, exactly. 00:43:38.219 --> 00:43:42.014 I would've loved to have stood up and, and explained to them, uh, 00:43:42.019 --> 00:43:44.369 uh, all of the Bayesian machinery, 00:43:44.369 --> 00:43:46.769 but, uh, that was not deemed to be the right thing to 00:43:46.769 --> 00:43:48.419 do, uh, in the circumstance. 00:43:49.679 --> 00:43:53.609 Uh, by, by the way, we're, uh, it'll be another podcast that we talk 00:43:53.609 --> 00:43:58.349 about, that particular example, a lot of, of, uh, Anna McLaughlin, Joe 00:43:58.349 --> 00:44:00.179 Marion here at Berry Consultants. 00:44:00.479 --> 00:44:06.509 Uh, Lindy Banky at, at rebi was the, the, the clinical lead of this and the one who 00:44:06.509 --> 00:44:10.949 spoke to the advisory committee meeting about she's fantastic, uh, sort of thing. 00:44:10.949 --> 00:44:16.534 So lots of people involved in, in that particular case, um, uh, with it. 00:44:20.489 --> 00:44:27.989 Okay, so have we, we missed any parts to what it looks like 00:44:28.079 --> 00:44:31.049 going to FDA good circumstances. 00:44:31.574 --> 00:44:34.244 Circumstances by the way that we say no to a client. 00:44:34.244 --> 00:44:36.794 No, we're not, we're not gonna go present that we, we would 00:44:36.794 --> 00:44:38.864 say no if we were at the FDA. 00:44:38.864 --> 00:44:41.414 We're not gonna present a Bayesian analysis of this. 00:44:41.804 --> 00:44:45.734 Uh, but we've done this probably couple dozen times. 00:44:46.064 --> 00:44:48.644 Um, over the years, probably more than that. 00:44:48.644 --> 00:44:50.774 We've been told no by the FDA by the way. 00:44:51.224 --> 00:44:53.894 Uh, we've presented what we thought was a very reasonable 00:44:53.894 --> 00:44:55.394 scientific argument for that. 00:44:55.394 --> 00:44:57.119 We've been told no, we've been told yes. 00:44:57.939 --> 00:45:03.279 We've been told, uh, can you change it in the following ways, uh, in that, so 00:45:03.279 --> 00:45:05.619 interaction is not uncommon in those, 00:45:05.909 --> 00:45:08.099 Kert Viele: We, we've been told to do it when we didn't propose it. 00:45:08.589 --> 00:45:09.009 Scott Berry: Yeah. 00:45:09.129 --> 00:45:09.459 Yeah. 00:45:09.549 --> 00:45:13.569 Uh, the rebi actually circumstances was a suggestion of the FDA, 00:45:14.049 --> 00:45:16.149 uh, Bram has made that suggestions. 00:45:16.149 --> 00:45:20.589 Other, other, uh, uh, scientists at the FDA had have suggested, 00:45:20.589 --> 00:45:24.339 oh, you should explore Bayesian methods, um, in the circumstance. 00:45:26.804 --> 00:45:30.194 Of course, going back to the original introduction of this, 00:45:30.194 --> 00:45:35.474 the, the home pregnancy test, uh, is now one of your colleagues 00:45:35.474 --> 00:45:37.184 that was, uh, that was Nick Berry, 00:45:37.334 --> 00:45:41.324 who is now a 34-year-old scientist here at Berry Co Consultants. 00:45:41.324 --> 00:45:42.764 So, uh, the 00:45:42.854 --> 00:45:44.594 Kert Viele: Thank you for making me feel old, Scott. 00:45:44.594 --> 00:45:45.314 Appreciate that. 00:45:45.344 --> 00:45:46.424 Scott Berry: I know, I know. 00:45:46.664 --> 00:45:49.664 Uh, as you, you, you babysat the youngster, 00:45:49.994 --> 00:45:53.774 Kert Viele: so so we should, I think the new podcast, we should always start with 00:45:53.774 --> 00:45:55.784 a story of view of Brad from grad school. 00:45:56.449 --> 00:45:56.569 I 00:45:58.049 --> 00:45:59.609 Scott Berry: Yeah, we have some of you too. 00:45:59.609 --> 00:46:02.549 So, uh, turnabout is, is fair play, 00:46:02.879 --> 00:46:04.799 uh, uh, in all of this. 00:46:04.799 --> 00:46:05.279 Yes. 00:46:06.299 --> 00:46:12.929 Alright, well we hope you enjoyed this little, uh, uh, Bayesian interlude 00:46:12.929 --> 00:46:17.849 here, which we hope is a more common thing with the FDA guidance. 00:46:18.569 --> 00:46:20.819 And until next time, we will be here. 00:46:21.374 --> 00:46:22.244 In the interim. 00:46:23.354 --> 00:46:24.014 Kert Viele: Thank you, Scott.