WEBVTT NOTE This file was generated by Descript 00:00:00.000 --> 00:00:02.770 Judith: Welcome to Berry's In the Interim podcast, where we explore the 00:00:02.770 --> 00:00:06.190 cutting edge of innovative clinical trial design for the pharmaceutical and 00:00:06.190 --> 00:00:08.039 medical industries, and so much more. 00:00:08.490 --> 00:00:09.180 Let's dive in. 00:00:12.437 --> 00:00:12.947 Scott Berry: All right. 00:00:12.947 --> 00:00:14.027 Welcome everybody. 00:00:14.027 --> 00:00:20.357 Back to, in the Interim, a podcast where we explore clinical trial 00:00:20.357 --> 00:00:24.437 science, the innovations in it, uh, usually a statistics. 00:00:24.437 --> 00:00:25.217 Ben to this. 00:00:25.217 --> 00:00:31.097 I'm Scott Berry, your host, and I'm joined by, uh, of course, none other than Dr. 00:00:31.097 --> 00:00:31.907 Don Barry. 00:00:32.477 --> 00:00:32.867 Uh. 00:00:33.557 --> 00:00:35.177 Today we are gonna talk. 00:00:36.047 --> 00:00:39.107 Uh, he's waving for those of you on the podcast, he's waving. 00:00:39.167 --> 00:00:41.297 Uh, some of you get, get visual. 00:00:41.897 --> 00:00:46.217 Um, we, we actually had people, and by the way, I'd love to hear 00:00:46.217 --> 00:00:49.547 from, from the audience requests. 00:00:50.222 --> 00:00:52.022 Things you'd like to hear about. 00:00:52.022 --> 00:00:58.622 We had requests to go back to some trials and talk about, uh, issues that came 00:00:58.622 --> 00:01:01.622 up, barriers, how you overcame them. 00:01:01.772 --> 00:01:06.542 A little bit of maybe, um, the sausage making, if you will. 00:01:06.842 --> 00:01:09.542 So Don and I are going to revisit. 00:01:09.847 --> 00:01:16.957 Um, in older trial, uh, talk about the design, talk about the building of 00:01:16.957 --> 00:01:22.747 it, the different aspects of the trial design, a number of barriers, uh, in it. 00:01:23.497 --> 00:01:25.837 So this trial is. 00:01:26.047 --> 00:01:27.877 Uh, a fascinating story. 00:01:27.877 --> 00:01:31.867 It's, it's, the name of the trial is the Award five trial, but the, the 00:01:31.867 --> 00:01:34.657 trial was for a drug on Eli Lilly. 00:01:34.837 --> 00:01:36.427 All of this is public. 00:01:36.907 --> 00:01:41.317 Uh, dulaglutide was the name of the drug while we were working on it. 00:01:41.497 --> 00:01:43.537 It's a Glip one inhibitor 00:01:44.167 --> 00:01:45.817 Don Berry: No, it's a Glip one agonist. 00:01:46.352 --> 00:01:47.492 Scott Berry: clip one agonist. 00:01:47.552 --> 00:01:52.352 Um, and they were interested in running a phase two trial. 00:01:52.922 --> 00:01:58.232 Uh, initially they came to us and were interested in a four dose phase two 00:01:58.232 --> 00:02:00.812 trial for the treatment of diabetes. 00:02:01.877 --> 00:02:07.937 And the, the drug, the, the efficacy of the drug was going to be h hba one c. 00:02:08.477 --> 00:02:12.137 I'll sort of jump to the end of this and we'll come back and talk about 00:02:12.137 --> 00:02:13.937 the different aspects of the trial. 00:02:14.447 --> 00:02:21.257 We ended up helping them with the team build a seamless 2-3 trial that 00:02:21.257 --> 00:02:24.527 started with seven doses of Dulaglutide. 00:02:25.472 --> 00:02:29.402 It had an active comparator sitagliptin and it had placebo. 00:02:29.402 --> 00:02:31.202 So it was a nine arm trial. 00:02:32.072 --> 00:02:39.842 The trial had a phase two component, with adaptive randomization favoring 00:02:39.842 --> 00:02:43.652 the doses of Dulaglutide that were doing better, and we'll say a 00:02:43.652 --> 00:02:45.362 lot more about what better means. 00:02:45.647 --> 00:02:46.817 Don Berry: It's Dulaglutide 00:02:47.642 --> 00:02:49.562 Scott Berry: Dula Glide. 00:02:49.562 --> 00:02:49.862 Okay, 00:02:50.057 --> 00:02:50.477 Don Berry: time. 00:02:50.552 --> 00:02:54.632 Scott Berry: we can jump to its trade name perhaps, which is now Trulicity. 00:02:55.022 --> 00:02:59.342 Uh, is the Nate trade name of the drug, which, uh, gives a little bit of way 00:02:59.342 --> 00:03:01.682 of, of whether the trial was successful. 00:03:02.702 --> 00:03:08.972 It had a component where it did interim analysis every two weeks. 00:03:08.972 --> 00:03:13.382 During the course of the trial, during the first part of the 00:03:13.382 --> 00:03:15.452 trial, this was for response. 00:03:15.452 --> 00:03:20.012 Adaptive randomization over the doses of Trulicity. 00:03:20.762 --> 00:03:28.382 At when it enrolled 200 patients in between 200 and 400, it could shift 00:03:28.442 --> 00:03:38.312 seamlessly to phase three and if it met certain criteria on the efficacy of, 00:03:38.312 --> 00:03:44.372 HbA1c but also additional parameters we'll talk about, it could move to. 00:03:44.777 --> 00:03:49.517 The phase three components, which was fixed, randomized, which could 00:03:49.517 --> 00:03:55.247 be one or two doses, uh, with sitagliptin, the active comparator 00:03:55.247 --> 00:04:00.677 and placebo, that size of phase. 00:04:00.677 --> 00:04:02.657 The the, the phase three component. 00:04:02.657 --> 00:04:07.727 The second stage of this trial was adaptively selected, and that 00:04:07.817 --> 00:04:10.637 represented a phase three trial. 00:04:11.267 --> 00:04:16.457 The patients from both com, both parts of the trial would go into the primary 00:04:16.457 --> 00:04:18.557 analysis at the end of phase three. 00:04:19.577 --> 00:04:22.367 We controlled type one error as part of that. 00:04:22.367 --> 00:04:24.377 That was part of the trial design. 00:04:25.217 --> 00:04:29.087 The trial was in, was run entirely by Bayesian algorithms. 00:04:29.717 --> 00:04:34.757 The final analysis at the end of the trial would be a frequentist analysis, an 00:04:34.757 --> 00:04:39.512 ANCOVA analysis of HbA1c So what happened? 00:04:39.542 --> 00:04:43.862 The trial at the first time, it could possibly go to phase three. 00:04:43.862 --> 00:04:44.522 It did. 00:04:44.822 --> 00:04:48.242 It selected two doses, the 0.75 00:04:48.242 --> 00:04:49.952 milligram and the 1.5 00:04:49.952 --> 00:04:55.352 milligram doses moved to phase three, enrolled phase three. 00:04:55.352 --> 00:04:59.282 At the same time, it spawned additional phase three trials 00:04:59.282 --> 00:05:01.292 with the doses that were selected. 00:05:02.627 --> 00:05:07.547 The trial, the trial and the additional phase three trials ran out. 00:05:08.387 --> 00:05:13.667 the, it ended up, it was a non-inferiority trial to the active comparator. 00:05:13.667 --> 00:05:15.677 It ended up showing superiority. 00:05:16.427 --> 00:05:20.627 It was very effective in weight loss, not supply, not, not surprisingly. 00:05:22.312 --> 00:05:31.522 Um, the drug ended up getting approved by the FDA in 2014 and in 2024 00:05:31.522 --> 00:05:34.642 it was a $6 billion selling drug. 00:05:35.302 --> 00:05:41.332 and it's been a multi-billion dollar selling drug since the trial. 00:05:41.752 --> 00:05:46.582 Eli Lilly talked about saving 12 to 18 months of development time 00:05:46.582 --> 00:05:48.952 with the seamless phase 2-3 trial. 00:05:50.387 --> 00:05:55.067 So that I'm gonna pause and throw it to Don to add to any compartments. 00:05:55.067 --> 00:05:56.957 So that's the trial design. 00:05:56.957 --> 00:06:05.597 This really, quite, elegant, reasonably complex phase 2-3 trial with interim 00:06:05.597 --> 00:06:10.457 analysis every two weeks in the trial resulting in a successful trial. 00:06:10.457 --> 00:06:13.487 And we thought we would go back and revisit the trial in 00:06:14.327 --> 00:06:18.107 different, aspects of the building of this trial and what happened. 00:06:19.397 --> 00:06:23.422 So Don, anything to add to the overall structure of the trial? 00:06:24.747 --> 00:06:29.397 Don Berry: Yeah, I, I, and I, I'm going to say one thing and then 00:06:29.937 --> 00:06:31.587 pass it to you for the other. 00:06:31.647 --> 00:06:37.197 Um, and, and the, the one thing that I want to pass to you to say 00:06:37.197 --> 00:06:42.507 something about the CUI, the clinical utility index and how that came about. 00:06:43.287 --> 00:06:46.797 Um, or maybe they're reserving that for the later time point 00:06:46.997 --> 00:06:49.307 Scott Berry: I, I was reserving that, so we'll come to 00:06:49.667 --> 00:06:53.417 Don Berry: And are you also reserving the uh, DSMB? 00:06:54.227 --> 00:06:59.927 Because the, you know, the picking the doses is, there's, there are lots of nice, 00:07:00.017 --> 00:07:02.447 interesting stories associated with that. 00:07:02.447 --> 00:07:02.837 Scott Berry: yep. 00:07:02.897 --> 00:07:04.487 We'll come to that as well. 00:07:04.547 --> 00:07:04.937 Don Berry: okay. 00:07:04.937 --> 00:07:06.377 So I have nothing more to add. 00:07:06.887 --> 00:07:07.397 Scott Berry: Okay. 00:07:07.487 --> 00:07:07.877 Okay. 00:07:08.267 --> 00:07:08.717 Alright. 00:07:09.077 --> 00:07:12.407 So we, um, uh, several aspects. 00:07:12.407 --> 00:07:14.147 So what were the barriers of this? 00:07:14.147 --> 00:07:18.797 We originally, by the way this, this took place, we were building this. 00:07:18.797 --> 00:07:23.027 I went back and looked 2007, 2008 ish. 00:07:23.417 --> 00:07:27.797 Uh, we were building, so this predated even the original FDA 00:07:28.067 --> 00:07:30.797 draft guidance on adaptive designs. 00:07:31.097 --> 00:07:33.647 Uh, this was early stages of. 00:07:34.082 --> 00:07:39.122 You know, in complex innovative designs, uh, within the setting, we, 00:07:39.122 --> 00:07:47.702 the original trial design was, uh, meant to be a short term, four months, 00:07:47.702 --> 00:07:51.302 four doses, 50 patients on each dose. 00:07:51.302 --> 00:07:57.302 Phase two trial, let that trial read out and then make development decisions. 00:07:57.677 --> 00:07:59.077 Don Berry: Endpoint A1C. 00:08:00.002 --> 00:08:03.392 Scott Berry: The endpoint hba one C That's right. 00:08:04.247 --> 00:08:08.807 So as we started to look at the potential for a seamless two, three 00:08:08.807 --> 00:08:15.467 trial design, that that trial could immediately move to phase three, several. 00:08:15.707 --> 00:08:21.527 The initial parts that were really interesting about this were by doing 00:08:21.527 --> 00:08:25.007 that and including those patients in phase three, they could enroll 00:08:25.007 --> 00:08:30.347 perhaps more patients in phase two potentially, but also more doses. 00:08:30.707 --> 00:08:38.987 So the issue became could we explore more doses a a, as part of the phase two trial? 00:08:39.707 --> 00:08:44.537 But in order to do that, we couldn't necessarily do fixed sample size 00:08:44.537 --> 00:08:49.127 on all the doses because then it grows linearly and now it's, 00:08:49.127 --> 00:08:51.137 you know, seven fourths is big. 00:08:51.137 --> 00:08:52.787 If we're gonna go to more doses. 00:08:59.057 --> 00:09:03.887 So the discussion of the number of doses in that tied to response, 00:09:03.887 --> 00:09:11.987 adaptive randomization, the trial ended up we started to explore, uh, 00:09:12.857 --> 00:09:15.527 using response adaptive randomization. 00:09:15.527 --> 00:09:19.307 The ent, the eventual trial did fixed randomization for 00:09:19.307 --> 00:09:21.407 only the first 50 patients. 00:09:23.102 --> 00:09:27.992 But then did response adaptive randomization at that point. 00:09:31.022 --> 00:09:35.762 The one of the problems, or one of the issues with that is you don't just 00:09:35.762 --> 00:09:41.342 want to do one uh, of those analyses because you want to continually learn 00:09:41.342 --> 00:09:48.062 as the datas was getting more and more rich, and so now we had to do multiple. 00:09:48.392 --> 00:09:53.702 Uh, analyses of this, which eventually grew to doing analyses every two 00:09:53.702 --> 00:09:55.562 weeks during the course of the trial, 00:09:57.172 --> 00:10:00.802 Don Berry: And the definition of response became much more complicated. 00:10:02.272 --> 00:10:02.602 Right. 00:10:03.632 --> 00:10:04.142 Scott Berry: right? 00:10:04.262 --> 00:10:11.132 So we started to simulate this trial with hba one C, and one of their 00:10:11.132 --> 00:10:14.732 concerns was that the higher doses. 00:10:15.737 --> 00:10:21.467 Could end up having really good hba one C effects, but at the same time 00:10:22.007 --> 00:10:25.097 could have potential safety issues. 00:10:27.467 --> 00:10:35.447 And so it couldn't, the algorithm couldn't only be H hba one C, so at that point it 00:10:35.447 --> 00:10:44.837 had to incorporate other endpoints into the trial, and it became clear that. 00:10:46.562 --> 00:10:50.762 Blood pressure and heart rate, and regulators wanted those. 00:10:51.152 --> 00:10:58.142 Had to be not elevated too high for risk of cardiovascular events. 00:10:58.592 --> 00:11:03.002 So this became a big question in the design is how do you do 00:11:03.002 --> 00:11:05.132 response Adaptive randomization? 00:11:05.912 --> 00:11:11.072 How do you do dose selection when you're worried about those endpoints? 00:11:11.102 --> 00:11:15.482 And later on down the story, weight loss became a really important part to that. 00:11:15.482 --> 00:11:16.952 We maybe we can come to that. 00:11:17.742 --> 00:11:23.892 So that became sort of barrier number one, is how to incorporate these multiple 00:11:23.892 --> 00:11:26.202 endpoints in the design parameters. 00:11:33.612 --> 00:11:36.942 I don't know if you remember how this started. 00:11:37.697 --> 00:11:43.307 But one of the, the really interesting things that happened is we came into 00:11:43.307 --> 00:11:51.287 this and they were very against a clinical utility index initially. 00:11:52.427 --> 00:11:56.207 That they had tried to do this in some other settings, and 00:11:56.207 --> 00:11:58.097 they, they, they were against it. 00:11:58.637 --> 00:12:06.167 So we started creating a number of rules that we'll take hba one C, but if the 00:12:06.167 --> 00:12:12.917 posterior probability of, uh, heart rate increase was above some level, 00:12:13.247 --> 00:12:15.527 we would stop randomization to that. 00:12:16.457 --> 00:12:19.097 But if this was true, we would do this. 00:12:19.097 --> 00:12:21.317 And if this was true, we would do that. 00:12:21.797 --> 00:12:25.127 And we ended up creating, um, something. 00:12:25.127 --> 00:12:30.557 And I, I, I learned the terminology at the time, a Rube Goldberg 00:12:30.617 --> 00:12:36.407 machine, and I think you referred to the design at one point as that. 00:12:36.437 --> 00:12:37.697 Do you remember that? 00:12:38.247 --> 00:12:38.697 Don Berry: I do. 00:12:40.127 --> 00:12:42.767 Scott Berry: What, first of all, what's a Rube Goldberg machine? 00:12:46.607 --> 00:12:51.137 Don Berry: Uh, it's a very complicated machine. 00:12:51.842 --> 00:12:52.352 Scott Berry: Yeah. 00:12:52.997 --> 00:12:56.147 Don Berry: It's, um, you mentioned Post-it notes before. 00:12:56.297 --> 00:13:03.887 Uh, to me, uh, it's a very complicated machine that essentially does nothing. 00:13:04.907 --> 00:13:07.097 Uh, it just, uh, machinate. 00:13:09.122 --> 00:13:13.292 Scott Berry: So, so he created these huge things where a ball would drop, it would 00:13:13.292 --> 00:13:15.902 hit something else, a lever would fly off. 00:13:16.232 --> 00:13:19.772 47 things would happen, and at the end it squeezed it. 00:13:20.522 --> 00:13:22.292 Uh, toothpaste or something. 00:13:22.652 --> 00:13:28.442 Um, and, and our design sort of became that, uh, aspect of it. 00:13:28.472 --> 00:13:33.212 And I remember you and I talking that we, we would show simulations 00:13:33.212 --> 00:13:34.952 in every, and the, and the issue was. 00:13:35.342 --> 00:13:40.802 They would bring a new scenario and it wouldn't necessarily do what they wanted 00:13:40.802 --> 00:13:45.572 it to do because of the complexity of all these rules that were kind 00:13:45.572 --> 00:13:47.882 of custom to every scenario we got. 00:13:47.882 --> 00:13:50.912 We'd create a new rule, but every time we got a new one, it 00:13:50.912 --> 00:13:52.322 wasn't doing the right thing. 00:13:52.802 --> 00:13:55.232 So you and I were reviewing this. 00:13:55.232 --> 00:13:57.122 We were talking with the statisticians. 00:13:57.122 --> 00:14:01.232 By the way, Brenda Gatos said Eli Lilly was a huge hero in this. 00:14:01.707 --> 00:14:04.107 In her work and the, the work and the design. 00:14:04.107 --> 00:14:08.787 So you and I may talk about our, our discussions, but, uh, the, the team at 00:14:08.787 --> 00:14:11.667 Lilly was fantastic in, in all of this. 00:14:11.862 --> 00:14:16.512 Don Berry: Just a shout out to Mary Jane Geiger, who was an MD. 00:14:16.952 --> 00:14:23.132 I mean, one of the barriers in these, in that we're gonna talk about are MDs and 00:14:23.132 --> 00:14:28.742 working with MDs and getting them, you know, they're used to, they, they know 00:14:28.742 --> 00:14:34.712 how to run clinical trials 'cause they've done a gazillion, um, and to say, well, 00:14:35.222 --> 00:14:37.292 have you thought about this or this? 00:14:37.892 --> 00:14:41.642 Uh, so Mary Jane was, uh, uh, also critical with Brenda. 00:14:42.227 --> 00:14:42.647 Scott Berry: Yeah. 00:14:43.832 --> 00:14:45.362 Don Berry: And the FDA was critical. 00:14:45.362 --> 00:14:47.522 You know, we couldn't have done this without the FDA 00:14:47.927 --> 00:14:48.347 Scott Berry: Yeah. 00:14:48.902 --> 00:14:53.762 Don Berry: uh, uh, uh, we, they had some input into the clinical 00:14:53.762 --> 00:14:57.272 utility index that we're gonna talk about and we should mention that. 00:14:58.202 --> 00:14:58.742 Scott Berry: Yeah. 00:14:58.747 --> 00:15:02.882 And, and they had given feedback that they were concerned. 00:15:02.882 --> 00:15:09.002 So we, we talked to the FDA about potentially, um, seamless phase 00:15:09.002 --> 00:15:12.332 two, three, that we wouldn't have an end of phase two meeting. 00:15:13.007 --> 00:15:17.597 Where we showed the dose, but we would show the algorithm to them that would 00:15:17.597 --> 00:15:22.097 pick the dose and they weighed in that heart rate and blood pressure were 00:15:22.097 --> 00:15:24.857 really important and gave parameters that they didn't want to see. 00:15:24.857 --> 00:15:27.407 Heart rate and blood pressure go above certain values. 00:15:28.427 --> 00:15:35.027 So we were looking at simulations of a number of these designs, and we saw this 00:15:35.027 --> 00:15:39.947 as a Rube Goldberg machine and we were really concerned, and you and I thought 00:15:39.947 --> 00:15:45.407 the only way forward was gonna be to create a clinical utility index to create 00:15:45.407 --> 00:15:50.747 a weight function of these different values that if heart rate went above a 00:15:50.747 --> 00:15:53.267 particular value, that dose had to be. 00:15:53.882 --> 00:15:57.872 You know, pushed down the value of that dose was lower. 00:15:59.042 --> 00:16:05.312 So we took the scenarios that they had created for us and what dose they 00:16:05.312 --> 00:16:10.562 wanted to pick, and we, at this point, we were probably simulating 30 different 00:16:10.562 --> 00:16:15.632 scenarios over the different endpoints, and we knew what they wanted to pick. 00:16:15.632 --> 00:16:19.022 And so we started to create mathematical functions that 00:16:19.022 --> 00:16:21.212 represented their decisions. 00:16:22.487 --> 00:16:24.617 And we thought this was the only way forward. 00:16:24.617 --> 00:16:31.067 So we went and presented them simulations of a new approach to pick the dose, and 00:16:31.067 --> 00:16:36.617 we showed 'em how it, how it behaved, and how often it picked different doses 00:16:37.037 --> 00:16:43.817 before we kind of jammed the clinical utility, uh uh, and the function on them. 00:16:44.567 --> 00:16:45.707 And they loved it. 00:16:46.127 --> 00:16:50.717 They love the choices of this, and they became somewhat advocates 00:16:50.717 --> 00:16:52.397 for the utility function. 00:16:54.812 --> 00:16:59.612 Don Berry: And so, for example, the utility was zero if the heart 00:16:59.612 --> 00:17:02.072 rate increase was more than X. 00:17:03.002 --> 00:17:10.292 And, um, the FDA, the only thing they contributed to this, except for their 00:17:10.292 --> 00:17:11.702 concern that Scott talked about. 00:17:12.377 --> 00:17:17.687 Uh, in terms of the nitty gritty of what the, the clinical utility Index was, 00:17:18.017 --> 00:17:22.727 is they wanted, uh, a decrease in X. 00:17:22.727 --> 00:17:26.297 They wanted a more conservative, uh, choice. 00:17:27.077 --> 00:17:31.307 Uh, and that turned out to be germane in, in, in what the doses were. 00:17:31.397 --> 00:17:34.547 Uh, were, uh, eventually decided. 00:17:35.777 --> 00:17:42.617 Scott Berry: So the utility function ended up being, on four endpoints, the mean 00:17:42.617 --> 00:17:47.177 change on HbA1c relative to Sitagliptin. 00:17:47.807 --> 00:17:51.887 So for example, if it was within the non-inferiority boundary 00:17:51.887 --> 00:17:53.957 that had a value of say, one. 00:17:54.647 --> 00:17:57.887 To, to, uh, therapeutic benefit. 00:17:57.917 --> 00:18:02.177 If it had better values, it had more and more value. 00:18:02.177 --> 00:18:07.937 So it was a, it was an increasing function as you decreased hba one 00:18:07.937 --> 00:18:13.577 c up to a particular limit and then it actually leveled off because 00:18:13.577 --> 00:18:15.887 they worried about too high of that. 00:18:17.277 --> 00:18:20.367 Don Berry: And I am sure there's some people, uh, out there that have tried 00:18:20.367 --> 00:18:24.357 to do this and they're interested, well, how you put these things together? 00:18:24.357 --> 00:18:25.947 You've got four dimensional space. 00:18:26.667 --> 00:18:29.187 Um, and are you assuming independence? 00:18:29.277 --> 00:18:33.357 Are you just multiplying, uh, the individual utilities? 00:18:33.957 --> 00:18:36.777 Uh, and the answer is, you know, nearly. 00:18:37.307 --> 00:18:46.697 Uh, we had a multiplicative, uh, CUI, um, with the exception of one joint 00:18:46.877 --> 00:18:54.377 distribution or one joint function of two of the end points that were, uh, sort 00:18:54.377 --> 00:18:58.277 of the, if then, uh, that were related. 00:18:58.787 --> 00:19:01.187 So, except for that, it was multiplicative. 00:19:01.742 --> 00:19:01.962 Scott Berry: Yep. 00:19:02.492 --> 00:19:08.402 Where if the, uh, there was a parameter in the model, which was the, the mean change 00:19:08.402 --> 00:19:11.042 in blood pressure relative to placebo. 00:19:11.762 --> 00:19:17.732 The mean change in heart rate relative to placebo, and there was a region 00:19:17.732 --> 00:19:23.882 where if it was within a, a, a couple x, that you got full value of the hba 00:19:23.882 --> 00:19:28.802 one C, but as those values increased, it took away from the value of 00:19:28.832 --> 00:19:30.932 hba one C in a multiplicative way. 00:19:30.962 --> 00:19:35.792 Both of those parameters, if it got below, above the value, FDA said, 00:19:35.792 --> 00:19:36.962 we don't want to see it there. 00:19:36.962 --> 00:19:40.892 It had zero value, so it wiped out that dose as a value. 00:19:41.732 --> 00:19:46.772 The fourth endpoint that came into it wasn't really safety and it wasn't the 00:19:46.772 --> 00:19:51.632 primary efficacy endpoint, but was weight loss, which of course we understand 00:19:51.632 --> 00:19:57.782 now, uh, plays a critically important role for Glip one agonists, um, in this. 00:19:57.782 --> 00:20:03.212 But they knew that if the drug increased weight, it was actually 00:20:03.212 --> 00:20:06.572 one of the, the, the early parts that they worried if it increased 00:20:06.572 --> 00:20:08.612 weight, nobody would take the drug. 00:20:09.992 --> 00:20:16.112 And if it actually decreased weight, which they had hopes for, but you know, in, in 00:20:16.112 --> 00:20:21.362 many ways, they were wanting to prevent weight gain that had additional value 00:20:21.632 --> 00:20:25.022 as a, as a therapeutically good dose. 00:20:25.742 --> 00:20:30.362 So we showed them the utility function and we simulated a bunch of trials and showed 00:20:30.457 --> 00:20:32.822 'em the selection and they ended up. 00:20:33.227 --> 00:20:37.787 Looking at these and, largely agreeing and then they tinkered with a little 00:20:37.787 --> 00:20:41.927 bit when they disagreed with the rule, with the outcome of the trial, 00:20:42.287 --> 00:20:46.127 they changed the utility function as they molded it around what they 00:20:46.127 --> 00:20:48.272 thought was the the risk benefit. 00:20:49.247 --> 00:20:54.527 Uh, of these endpoints, and it got to the point where we were simulating these 00:20:54.527 --> 00:20:58.007 trials and when they, when we showed them results and said, this is what 00:20:58.007 --> 00:21:01.967 happens in this example trial, they said, oh, I don't know if I'd do that. 00:21:01.967 --> 00:21:03.347 I think they'd pick the other dose. 00:21:03.347 --> 00:21:07.187 They actually got to the point where they, thought they were wrong, and the 00:21:07.187 --> 00:21:13.157 algorithm was right to the point that they believed this utility function was 00:21:13.157 --> 00:21:15.707 absolutely the right way to go forward. 00:21:16.667 --> 00:21:21.677 And they instructed the DSMB eventually on that. 00:21:25.397 --> 00:21:31.757 So the utility function was then used to do response, adaptive randomization 00:21:31.757 --> 00:21:34.577 over the one that had the best profile. 00:21:35.657 --> 00:21:41.687 It drove whether or not a drug would, would graduate and go to the second stage. 00:21:42.182 --> 00:21:45.992 Whether it would bring a second dose along into the trial. 00:21:46.412 --> 00:21:51.512 And the algorithm drove all of this, which was a Bayesian model 00:21:51.512 --> 00:21:57.632 on those four endpoints, a Bayesian model that calculated the posterior 00:21:57.632 --> 00:21:59.612 distribution of the utilities. 00:22:00.152 --> 00:22:04.862 Uh, and it drove, uh, all of those decisions. 00:22:06.152 --> 00:22:12.422 So the, the, the, the one other, uh, uh, another really interesting part to this 00:22:12.542 --> 00:22:21.122 was the longitudinal modeling within this, um, which we described the original trial 00:22:21.122 --> 00:22:26.642 was maybe four months they were gonna look at, but now the primary endpoint 00:22:26.642 --> 00:22:28.532 in phase three was gonna be 12 months. 00:22:29.837 --> 00:22:30.557 Within it. 00:22:30.797 --> 00:22:34.517 And so we wanted to make our inferences based on the 12 00:22:34.517 --> 00:22:36.167 month results for hba one C. 00:22:36.167 --> 00:22:40.937 It was six months for, uh, heart rate, blood pressure and weight loss because 00:22:40.937 --> 00:22:44.447 that was a comparison to placebo, which they could only give for six months. 00:22:45.767 --> 00:22:51.647 Um, and so we built longitudinal models of the outcomes over time. 00:22:52.007 --> 00:22:58.397 That predicted the six month for three of them in the 12 month for hba one c to 00:22:58.397 --> 00:23:05.087 drive the decision making in the trial became a huge aspect of, of being able 00:23:05.087 --> 00:23:09.497 to make these decisions continually during the course of the trial. 00:23:11.772 --> 00:23:15.102 Don Berry: So tell her about the pharmacologists and our discussion 00:23:15.102 --> 00:23:19.032 with them and what they said we should be looking at, and we said. 00:23:19.757 --> 00:23:20.327 Prove it. 00:23:21.842 --> 00:23:24.362 Scott Berry: Yeah, so originally. 00:23:25.502 --> 00:23:31.202 We were trying to forecast HBA one C, which is a blood sugar value. 00:23:31.562 --> 00:23:38.312 They actually expected that it would be better to predict that using fasting 00:23:38.312 --> 00:23:44.522 blood glucose than using hba one c, say at one month or two months, that you 00:23:44.522 --> 00:23:46.232 would look at fasting blood glucose. 00:23:47.132 --> 00:23:50.822 The fabulous thing was they actually had data on previous drugs. 00:23:50.822 --> 00:23:55.142 They had done quite a bit of work in diabetes, and so they were able to provide 00:23:55.142 --> 00:24:02.042 us data on fasting blood glucose, and hba one C, and so we fit the longitudinal 00:24:02.042 --> 00:24:09.692 models to the previous data and hba one C greatly outperformed fasting blood 00:24:09.692 --> 00:24:12.782 glucose as a predictor of hba one c. 00:24:13.547 --> 00:24:19.547 And so we were able to abandon fasting blood glucose and use only the hba one 00:24:19.547 --> 00:24:21.617 C values during the course of the trial. 00:24:25.247 --> 00:24:34.607 So, um, we now, this is in the age of si, we were simulating control of 00:24:34.607 --> 00:24:37.427 type one error with the regulators. 00:24:38.612 --> 00:24:45.872 With, um, uh, uh, in the scenario where we're carrying forward, the doses selected 00:24:45.872 --> 00:24:52.772 from the first part to the second part, we ended up sending them over 300 null 00:24:52.772 --> 00:25:02.072 scenarios to simulate to go to regulators and working with them to, to approve the 00:25:02.252 --> 00:25:04.862 seamless phase two, three part of it. 00:25:05.702 --> 00:25:12.122 Now we made some concessions to the regulators on this where at least 70% of 00:25:12.122 --> 00:25:14.252 patients had to come from the second part. 00:25:14.252 --> 00:25:15.932 They felt comfortable in that. 00:25:16.922 --> 00:25:21.752 Do you remember what Rob Hemmings said when we came back to them 00:25:21.752 --> 00:25:23.672 with the updated simulations? 00:25:23.672 --> 00:25:25.382 He was at EMA at the time. 00:25:26.797 --> 00:25:27.667 Don Berry: Yeah, of course. 00:25:30.727 --> 00:25:33.997 Uh, but I'll, I'll let you say it. 00:25:34.727 --> 00:25:37.517 Scott Berry: so we, we had gone to them with simulations. 00:25:37.517 --> 00:25:41.327 We had provided updates and gone back and 00:25:41.372 --> 00:25:46.052 Don Berry: And they, and they kept, uh, saying, you know, well, what about this 00:25:46.052 --> 00:25:47.672 and what about this and what about this? 00:25:47.672 --> 00:25:54.422 And eventually, uh, and so, uh, they were kept saying no sort of. 00:25:56.297 --> 00:26:00.707 Scott Berry: and we came back to them and eventually Rob Hemmings said, 00:26:00.947 --> 00:26:04.007 we've run out of reasons to say no. 00:26:04.967 --> 00:26:08.657 And they accepted the simulation control of type one error. 00:26:08.657 --> 00:26:11.027 And this is roundabout 2008. 00:26:11.387 --> 00:26:12.347 In this scenario. 00:26:12.992 --> 00:26:18.092 Don Berry: It wasn't easy in the, uh, FDA on this side of the Atlantic either. 00:26:18.752 --> 00:26:20.372 Uh, they in their. 00:26:21.227 --> 00:26:30.377 2010, uh, guidance in adaptive designs has said, uh, that they had, uh, uh, they 00:26:30.377 --> 00:26:36.527 were not comfortable with, um, simulations that they said that sim showing type 00:26:36.527 --> 00:26:41.957 one error control type one error, uh, by simulation is not well understood. 00:26:42.287 --> 00:26:44.297 That was in the guidance. 00:26:44.957 --> 00:26:49.067 Um, and eventually, I mean, that, that has. 00:26:49.352 --> 00:26:56.012 Has the, the reason you said, you know, the 300, uh, null scenarios 00:26:56.642 --> 00:27:02.792 is because they of course, were, uh, anal about controlling type one error. 00:27:03.212 --> 00:27:08.462 But type one error depends on, for example, what the 00:27:08.462 --> 00:27:10.382 underlying control rate is. 00:27:10.832 --> 00:27:14.642 It depends on accrual rate. 00:27:16.172 --> 00:27:16.772 Um. 00:27:17.387 --> 00:27:22.397 And so if you know, if, as Scott indicated, if the accrual rate is fast, 00:27:23.057 --> 00:27:26.747 you may not be able to take too much advantage and the type one error is 00:27:26.747 --> 00:27:29.327 affected by that or if it's too slow. 00:27:29.927 --> 00:27:34.127 So we had to do lots of accrual rate, uh, simulations. 00:27:34.847 --> 00:27:40.607 They became in the next version of the adaptive guidance, which was nine 00:27:40.607 --> 00:27:43.967 years later, uh, they had learned. 00:27:44.942 --> 00:27:49.892 Uh, to be more comfortable with simulations and they had learned, uh, 00:27:49.922 --> 00:27:52.952 how to handle this sort of a scenario. 00:27:53.552 --> 00:27:58.262 So, for example, we've designed phase three trials, uh, uh, 00:27:58.292 --> 00:28:00.092 much, you know, since then. 00:28:00.722 --> 00:28:04.562 Uh, and we don't have to run nearly that number of simulations. 00:28:04.562 --> 00:28:09.272 We have to make them comfortable and, uh, promise them that we, you 00:28:09.272 --> 00:28:12.242 know, we'll do post, uh, analyses. 00:28:12.737 --> 00:28:15.887 To make them even more comfortable, should that be necessary. 00:28:16.547 --> 00:28:20.987 Uh, and so that's actually written in the updated adaptive guidance. 00:28:21.587 --> 00:28:28.787 Just while we're on the FDA, uh, this was at the time the critical path initiative 00:28:28.787 --> 00:28:33.527 of CER, uh, uh, dates back to 2004. 00:28:34.157 --> 00:28:40.457 Uh, but you know, this was, uh, uh, Janet Woodcock's baby and, um. 00:28:40.877 --> 00:28:46.787 2006, it got somewhat red revised and uh, but this was regarded as part 00:28:46.787 --> 00:28:48.707 of that critical path initiative. 00:28:49.247 --> 00:28:54.827 Uh, even though as Scott indicated it predated the CID the complex 00:28:54.827 --> 00:28:57.977 innovative design, uh, initiative. 00:28:58.367 --> 00:29:03.677 And it was probably, I mean, I don't think there's been a CID that is more 00:29:03.677 --> 00:29:06.377 complicated than the award five trial. 00:29:06.697 --> 00:29:06.987 Scott Berry: Yeah. 00:29:07.952 --> 00:29:08.342 Yep. 00:29:10.217 --> 00:29:14.777 So I, one of the, one of the things that Lily did in this that 00:29:14.777 --> 00:29:19.697 was tremendous in, I've never seen, uh, another case of this is. 00:29:21.497 --> 00:29:24.467 of the concerns we had was we have this algorithm doing 00:29:24.467 --> 00:29:26.177 response, adaptive randomization. 00:29:26.177 --> 00:29:30.677 It was using longitudinal models, so it needed somewhat mature data to, to, 00:29:30.977 --> 00:29:36.317 to do a good job at dose selection is that we simulated the trial under a 00:29:36.317 --> 00:29:38.567 range of scenarios or concern that. 00:29:38.912 --> 00:29:44.492 You send this out to operations and they enroll gangbusters, 00:29:44.522 --> 00:29:46.232 which you can do in diabetes. 00:29:46.742 --> 00:29:52.532 So we simulated a range of scenarios and we looked at the likelihood of making 00:29:52.532 --> 00:29:57.992 good decisions, picking the right dose, uh, before it went to phase three. 00:29:58.437 --> 00:30:02.577 And Lily, they, they created boundaries and they said, we 00:30:02.577 --> 00:30:04.737 have to stay in these boundaries. 00:30:05.067 --> 00:30:08.847 And they did not want to enroll more than eight per month, for example, 00:30:08.847 --> 00:30:12.507 because they knew that the, it was going to make worse decisions. 00:30:12.927 --> 00:30:17.697 And so they told operations, you can't go above this number, which I think was 00:30:17.697 --> 00:30:20.157 a huge part of the success of the trial. 00:30:21.317 --> 00:30:21.497 Don Berry: Yeah. 00:30:21.497 --> 00:30:26.477 And so that made the trial somewhat longer, uh, but nowhere near the 00:30:26.477 --> 00:30:30.587 savings that you indicated earlier on in terms of, but it, you know, 00:30:30.587 --> 00:30:33.677 this was taken into account in their calculation about the savings. 00:30:34.292 --> 00:30:34.712 Scott Berry: Right. 00:30:35.372 --> 00:30:40.052 So now as you described this, this is a, a reasonably complicated design. 00:30:40.052 --> 00:30:44.762 It has got clinical utility index, it's got interims every two weeks. 00:30:45.302 --> 00:30:48.872 Uh, the algorithm upstate, the randomizations, it is got 00:30:48.872 --> 00:30:51.782 prospective go to stage two rules. 00:30:52.082 --> 00:30:55.382 So now we start working on the implementation of this, 00:30:55.382 --> 00:30:59.042 the operations of this, and 00:30:59.282 --> 00:31:02.522 Don Berry: along with that, you know, the, the, um. 00:31:03.332 --> 00:31:09.782 It also picked the sample size for stage two and the sample size it 00:31:09.782 --> 00:31:14.552 picked when it eventually went was the smallest possible sample size, and 00:31:14.552 --> 00:31:16.862 it was a minimum number in stage one. 00:31:17.582 --> 00:31:25.922 It made that decision, um, uh, uh, in the actual trial, uh, as 00:31:25.922 --> 00:31:27.902 soon as it was allowed to make it. 00:31:28.502 --> 00:31:29.402 And, uh. 00:31:30.662 --> 00:31:35.372 At some point I want to tell you about the DSMB and working with the DSMB 00:31:35.787 --> 00:31:37.637 Scott Berry: Yeah, so let's, let's shift to that. 00:31:37.697 --> 00:31:38.657 Let's shift to that. 00:31:38.657 --> 00:31:44.447 So we, they, they created DSMB and we spent a lot of time showing them 00:31:44.447 --> 00:31:48.617 simulations of the trial designs, explaining the utility index, the 00:31:48.617 --> 00:31:55.367 decision rules, and Lily told the DSMB, they weren't allowed to change the 00:31:55.367 --> 00:31:57.707 decision of the algorithm around doses. 00:31:58.592 --> 00:32:02.072 And, and it was how much they believed in that utility 00:32:02.072 --> 00:32:03.842 function to be the right dose. 00:32:04.352 --> 00:32:09.242 They could stop a dose for safety during the course of the trial. 00:32:09.242 --> 00:32:11.582 They could stop the whole trial for safety, so that 00:32:11.582 --> 00:32:13.022 was still a huge part of it. 00:32:13.772 --> 00:32:17.672 But they knew simulation control of type one error was a big part 00:32:17.672 --> 00:32:21.602 of the design, and the utility index was a big part of it. 00:32:21.602 --> 00:32:24.152 And so the DSMB was fully on board with this. 00:32:25.112 --> 00:32:31.802 So you and I were, uh, advisors to the DSMB and we were unblinded, and 00:32:31.802 --> 00:32:36.632 if you remember these, these would happen every two weeks on Wednesday, 00:32:36.962 --> 00:32:41.162 uh, every fortnight on Wednesdays, these reports would come out. 00:32:41.522 --> 00:32:46.262 So, uh, a number of really interesting aspects to the DSMB, 00:32:46.262 --> 00:32:47.522 which have been made public. 00:32:47.582 --> 00:32:49.982 Uh uh, so yes, jump into that. 00:32:51.102 --> 00:32:52.992 Don Berry: So you said they were on board. 00:32:53.202 --> 00:32:54.672 They were more than on board. 00:32:55.752 --> 00:33:02.862 Uh, they said, you know, you've told us that we can't change the design. 00:33:02.862 --> 00:33:03.312 We can't. 00:33:04.337 --> 00:33:07.187 Drop a, an arm for lack of efficacy. 00:33:07.187 --> 00:33:08.927 We have to follow the design. 00:33:10.007 --> 00:33:15.467 But if we were designing, if we were making all of the decisions, 00:33:16.937 --> 00:33:24.137 we would make exactly the decisions that the algorithm has made and the 00:33:24.137 --> 00:33:28.877 algorithm, you know, didn't choose the highest doses, uh, even though 00:33:28.877 --> 00:33:32.987 weight loss was, you know, incredible on the, on the highest loss doses. 00:33:33.797 --> 00:33:41.087 Um, and it, it, it toward the end of that period when they could make, they could 00:33:41.087 --> 00:33:51.287 drop a dose for, uh, safety, uh, they were worried about the highest dose and they 00:33:51.287 --> 00:33:56.117 understood they were actually tracking things and, you know, uh, how things 00:33:56.117 --> 00:34:01.757 were going and how close to making a decision to pick this dose over that dose. 00:34:02.597 --> 00:34:08.447 And so they were, um, worried about the highest dose, and I said, you can 00:34:08.447 --> 00:34:12.557 drop it now if you don't drop it now. 00:34:12.797 --> 00:34:18.107 And next week when we get, you know, the permission to pick the two doses, 00:34:19.037 --> 00:34:25.367 if you dropped, um, a, a dose that would've been picked, we still go 00:34:25.367 --> 00:34:27.827 with two doses according to protocol. 00:34:28.892 --> 00:34:33.902 If you wait until next week and the highest dose is one of the ones that's 00:34:34.022 --> 00:34:41.942 picked and you decide to drop it, then we continue on only one dose. 00:34:42.812 --> 00:34:47.852 So they understood that and so they hastily drop the, uh, the 00:34:47.852 --> 00:34:51.062 highest dose, uh, for safety. 00:34:51.452 --> 00:34:57.632 And it was, you know, the combination of, of, um, blood pressure and heart rate. 00:34:58.007 --> 00:35:06.917 Uh, considerations that, by the way, um, these doses are now part of the marketed. 00:35:07.632 --> 00:35:11.807 The, the, the marketed doses were the ones selected in these phase 00:35:11.807 --> 00:35:15.377 three trials, that the other phase three trials were the same ones. 00:35:15.377 --> 00:35:18.317 And they were happy with that and, you know, we got the right dose. 00:35:19.067 --> 00:35:24.287 Um, but they, uh, learned over time that heart rate, even though it was increased. 00:35:24.317 --> 00:35:28.607 It was not that important and didn't have clinical ramifications. 00:35:29.357 --> 00:35:37.187 And so the, the marketed doses now include higher doses than, you know, one dose 00:35:37.187 --> 00:35:41.927 that's higher than the ones even that we considered and the award five trial. 00:35:42.847 --> 00:35:43.412 Scott Berry: What, what? 00:35:43.412 --> 00:35:45.962 One of the, one of the great, the interesting things is 00:35:45.962 --> 00:35:47.937 when they drop the high dose. 00:35:49.022 --> 00:35:55.382 It had already been zeroed out in the randomization by the RAR, so it 00:35:55.547 --> 00:35:56.687 Don Berry: Well, it was zero. 00:35:56.687 --> 00:35:59.087 It was zeroed out when they were allowed to zero out, 00:35:59.702 --> 00:35:59.972 Scott Berry: right? 00:36:00.077 --> 00:36:01.457 Don Berry: for the final analysis. 00:36:01.667 --> 00:36:06.227 But it was predicted based on what the utilities were. 00:36:06.857 --> 00:36:11.057 Uh, you could see that, you know, almost certainly next week it's gonna 00:36:11.057 --> 00:36:13.277 be the same and they're gonna drop it. 00:36:13.742 --> 00:36:14.642 It's gonna be dropped. 00:36:15.177 --> 00:36:15.397 Scott Berry: Yep. 00:36:15.992 --> 00:36:16.202 Yep. 00:36:16.982 --> 00:36:17.522 Um. 00:36:18.422 --> 00:36:23.042 So the, the, and we could see, and it was the, the, the results were amazing 00:36:23.042 --> 00:36:26.942 that when it reached that 200 minimum, it was gonna jump at that point. 00:36:26.942 --> 00:36:31.892 So it was one of the designs where the design ran exactly as planned. 00:36:31.892 --> 00:36:36.932 The algorithm ran, the RAR was updated, the decision was made. 00:36:36.932 --> 00:36:40.682 The DSMB agreed with each of the decisions in the course of it. 00:36:41.132 --> 00:36:43.262 And the trial picked 1.5 00:36:43.262 --> 00:36:45.092 milligrams and 0.75. 00:36:45.092 --> 00:36:47.282 And one of the interesting parts is those weren't. 00:36:47.307 --> 00:36:52.077 The tho, neither one of those doses were part of the original 00:36:52.077 --> 00:36:54.297 four dose phase two trial. 00:36:55.077 --> 00:36:57.087 So at the time, this 1.5 00:36:57.087 --> 00:36:58.647 milligram and the 0.75 00:36:58.647 --> 00:37:03.597 milligram were, uh, you know, really good doses in terms of this and they 00:37:03.597 --> 00:37:05.787 may never actually have been used. 00:37:06.122 --> 00:37:10.142 With the other, uh, uh, with the other phase two trial. 00:37:10.502 --> 00:37:13.922 So it had a number of aspects sped up development, and by the way, 00:37:13.922 --> 00:37:15.752 then it jumped to phase three. 00:37:15.932 --> 00:37:19.652 Those doses moved on and in into the other trials. 00:37:19.652 --> 00:37:23.492 And hence going back to the start of the story, uh, it ended up showing 00:37:23.492 --> 00:37:28.772 superiority to Sitagliptin, won all of its phase three trials, showed 00:37:29.132 --> 00:37:33.842 benefit in its cardiovascular risk trial, and is now, now Trulicity. 00:37:34.917 --> 00:37:42.057 Don Berry: And, and Trulicity is the, is the initial, uh, Glip one agonist you 00:37:42.057 --> 00:37:48.987 read today about, uh, the, uh, the other diabetes drugs that are Glip one agonists. 00:37:49.527 --> 00:37:49.917 Um. 00:37:50.297 --> 00:37:52.967 But the big deal is the weight loss. 00:37:53.537 --> 00:37:58.697 And, um, the weight loss, by the way, in the award five trial in terms of 00:37:58.697 --> 00:38:06.137 the doses, was completely predictive of the weight loss in, uh, a later 00:38:06.137 --> 00:38:12.497 trial that they ran with Trulicity, uh, that showed that the, uh, dose 00:38:12.497 --> 00:38:17.867 by dose, I mean, it was almost this, uh, you know, uh, matching, uh. 00:38:18.362 --> 00:38:22.412 What, what ha what, what we would've predicted from the award five trial. 00:38:23.192 --> 00:38:26.162 Uh, but it trulicity is not approved. 00:38:26.192 --> 00:38:29.762 Uh, Lily hasn't looked for it to be approved for weight loss 00:38:29.762 --> 00:38:32.612 because they have other Glip one agonists that are, you know, next 00:38:32.612 --> 00:38:34.982 generation that are even better. 00:38:35.582 --> 00:38:35.822 Scott Berry: Yep. 00:38:36.992 --> 00:38:40.472 So if you'd like to read more, uh, about this, you can read 00:38:40.472 --> 00:38:41.942 about the award five trial. 00:38:41.942 --> 00:38:45.782 There's, there's, uh, paired publications about it in the Journal 00:38:45.782 --> 00:38:47.852 of Diabetes, science and Technology. 00:38:48.302 --> 00:38:50.162 Uh, in 2012. 00:38:50.522 --> 00:38:54.212 Uh, publications of paper before the results came out of this. 00:38:54.542 --> 00:38:56.132 Uh, you can read those. 00:38:56.372 --> 00:39:01.082 Uh, want to acknowledge Zach scr, uh, Jenny Chen. 00:39:01.082 --> 00:39:05.822 Mary Jane Geiger, who Don mentioned Andy Anderson and Brett Brenda Gatos, 00:39:06.062 --> 00:39:11.762 who are on those papers as well, doing, uh, uh, huge work and, and uh, uh, 00:39:11.762 --> 00:39:14.342 working, worked with them on the team. 00:39:14.342 --> 00:39:17.732 And this was about nine months, uh, that we built this trial 00:39:17.732 --> 00:39:19.082 and got it ready to execute. 00:39:20.027 --> 00:39:22.517 Don Berry: Just one additional aspect of the nine months. 00:39:22.967 --> 00:39:30.797 Took us nine months, but I started to consult with Eli Lilly in the 1970s 00:39:31.847 --> 00:39:34.847 and we built Scott and I over time. 00:39:35.267 --> 00:39:41.027 I don't know if Scott was pretty young in the 1970s, uh, but when he became 00:39:41.027 --> 00:39:44.627 old enough to go to Indianapolis, he went to India and we developed 00:39:44.627 --> 00:39:50.057 a really great relationship and we, we couldn't have done this without 00:39:50.057 --> 00:39:52.877 the support of the statisticians. 00:39:53.477 --> 00:39:59.297 Um, all the way back to, uh, Charlie Sampson who set up, uh, 00:39:59.327 --> 00:40:04.847 biostatistics at, uh, Eli Lilly, and also was a co-founder of the. 00:40:05.192 --> 00:40:10.232 Muncie meetings for those of you that might have been a a around back 00:40:10.232 --> 00:40:16.502 then, uh, Muncie, Indiana where, uh, the ball State University is. 00:40:16.502 --> 00:40:21.482 And, uh, I presented their adaptive designs because of, 00:40:22.142 --> 00:40:23.852 uh, Charlie and others at Lilly. 00:40:23.852 --> 00:40:29.852 But we've, we've, and, and they designed, this was pro Prozac days. 00:40:30.392 --> 00:40:33.152 They designed adaptive trials in Prozac. 00:40:34.097 --> 00:40:38.567 Uh, so they, you know, shout out to, to all of them. 00:40:38.987 --> 00:40:46.037 If you pick a random pharmaceutical company and drop us, um, by parachute on 00:40:46.037 --> 00:40:51.257 their, onto their campus and see if we can persuade them to do this kind of thing. 00:40:52.097 --> 00:40:56.897 I'm not sure of the results, but it's not a certainty that we would 00:40:56.897 --> 00:41:02.147 be able to do it because we don't have the, the trust build up and. 00:41:02.717 --> 00:41:05.267 And, and the, the great working conditions. 00:41:05.267 --> 00:41:08.417 So it was all a great story for us, and it was a great story for 00:41:08.957 --> 00:41:16.307 Eli Lilly who decided that we gotta simulate every trial and every trial 00:41:16.307 --> 00:41:17.837 that we run, we're gonna simulate. 00:41:18.407 --> 00:41:22.607 And I don't know whether they're still doing that, but it, it's not a bad thing. 00:41:22.607 --> 00:41:23.177 I'll tell you. 00:41:23.552 --> 00:41:27.482 Scott Berry: Yeah, no, Karen Price is, is there now many other statisticians 00:41:27.482 --> 00:41:28.802 and they're still simulating. 00:41:28.802 --> 00:41:33.842 And by the way, they're the largest financial, uh, uh, uh, 00:41:33.902 --> 00:41:36.247 pharmaceutical company in the world at this point, I think. 00:41:37.197 --> 00:41:40.247 Alright, well, uh, thank you, Don. 00:41:40.397 --> 00:41:43.517 Don Berry: I, I don't think that's true, but they, they are certainly 00:41:43.517 --> 00:41:46.067 the biggest on Wall Street. 00:41:46.772 --> 00:41:46.982 Scott Berry: Yeah. 00:41:46.982 --> 00:41:48.932 Their market cap or something. 00:41:48.932 --> 00:41:49.112 I, 00:41:49.397 --> 00:41:49.457 Don Berry: Yeah. 00:41:49.457 --> 00:41:49.847 Yeah. 00:41:50.252 --> 00:41:50.402 Scott Berry: yeah. 00:41:51.092 --> 00:41:51.362 Yep. 00:41:51.392 --> 00:41:53.162 That little Indiana company. 00:41:54.572 --> 00:41:56.402 Alright, well, so thank you Don. 00:41:56.552 --> 00:42:01.382 We're, we're gonna jump into some, some interesting trials as we go forward. 00:42:01.532 --> 00:42:06.332 And again, if you have, uh, any ideas for the show, holler at us. 00:42:06.902 --> 00:42:09.602 Uh, in the meantime, we are in the interim. 00:42:10.112 --> 00:42:14.687 Don Berry: Yeah, and if you have questions about what we've said or what we did, 00:42:14.777 --> 00:42:16.667 uh, please, uh, you know, send an email. 00:42:17.477 --> 00:42:19.697 Scott Berry: Yep, in the interim. 00:42:19.967 --> 00:42:21.107 Thank you everybody. 00:42:21.297 --> 00:42:21.827 Don Berry: Thank you. 00:42:21.857 --> 00:42:22.427 Thanks Scott. 00:42:23.117 --> 00:42:23.597 Scott Berry: Bye.