WEBVTT NOTE This file was generated by Descript 00:00:00.000 --> 00:00:03.119 Judith: Welcome to Berry's In the Interim podcast, where we explore the 00:00:03.119 --> 00:00:07.130 cutting edge of innovative clinical trial design for the pharmaceutical and 00:00:07.130 --> 00:00:09.260 medical industries, and so much more. 00:00:10.039 --> 00:00:10.729 Let's dive in. 00:00:11.800 --> 00:00:12.460 Scott: All right. 00:00:12.460 --> 00:00:13.750 Welcome everybody. 00:00:13.750 --> 00:00:15.790 Back to In the interim. 00:00:16.390 --> 00:00:23.665 This is actually our 60th episode of In the Interim, I'm your host, Scott 00:00:23.665 --> 00:00:29.750 Berry, and I'm joined by two other senior statistical scientists at Barry, Dr. 00:00:29.750 --> 00:00:30.849 Nicholas Berry. 00:00:30.849 --> 00:00:31.509 And Dr. 00:00:31.509 --> 00:00:32.409 Joe Marion. 00:00:32.409 --> 00:00:35.200 Welcome back to In the Interim. 00:00:35.501 --> 00:00:35.921 Nick Berry: Thanks. 00:00:36.846 --> 00:00:37.836 Joe Marion: Uh, thanks Scott. 00:00:37.866 --> 00:00:38.466 Happy to be here. 00:00:39.205 --> 00:00:45.474 Scott: So the 60th episode of, of in the interim 60 interims. 00:00:46.165 --> 00:00:53.305 Interestingly, uh, Joe, it was just recently involved in the 60th 00:00:53.305 --> 00:00:55.555 interim analysis of a clinical trial. 00:00:55.555 --> 00:01:00.295 And we won't say what trial that was, but you're doing double duty on the 60. 00:01:01.086 --> 00:01:04.416 Joe Marion: Yeah, it's, uh, it's funny how that number creeps up on you, right? 00:01:04.566 --> 00:01:07.566 People always ask me like, how many interims can you have in a clinical trial? 00:01:07.566 --> 00:01:09.246 Or like, you know, sure. 00:01:09.246 --> 00:01:11.675 Surely, you know, you can't do too many of these. 00:01:11.675 --> 00:01:14.631 In a sky's the limit as, as far as I can tell. 00:01:14.916 --> 00:01:20.136 If I think, if I stayed at Barry for another, uh, 30 years, you know, we'd be 00:01:20.136 --> 00:01:21.845 at what, how many interims would that be? 00:01:21.875 --> 00:01:22.505 It'd Be a lot. 00:01:22.900 --> 00:01:23.500 Scott: Be a lot. 00:01:23.526 --> 00:01:23.735 Joe Marion: lot. 00:01:23.735 --> 00:01:23.795 Yeah. 00:01:24.601 --> 00:01:26.146 Nick Berry: joke about how you, uh. 00:01:26.791 --> 00:01:28.651 Spend your alpha at Barry. 00:01:28.651 --> 00:01:31.231 Sometimes when you do an interim, you're spending your personal alpha. 00:01:31.231 --> 00:01:33.421 So 60 interims is a lot of alpha for you to spend. 00:01:34.190 --> 00:01:35.271 Joe Marion: I am all at Alpha man. 00:01:35.451 --> 00:01:41.151 Well, I mean, if you believe that the alpha spending literature, 00:01:41.151 --> 00:01:43.670 every time you get a failure, you get to recover some alpha. 00:01:43.670 --> 00:01:44.030 Right? 00:01:44.030 --> 00:01:46.491 So maybe I've got plenty of alpha, you just dunno. 00:01:46.491 --> 00:01:47.211 Yeah, yeah. 00:01:47.841 --> 00:01:48.110 Yeah. 00:01:49.705 --> 00:01:50.575 Scott: Alright, stay. 00:01:50.575 --> 00:01:54.985 There's nothing better than adaptive design humor, uh, for, 00:01:55.206 --> 00:01:55.416 Joe Marion: That's 00:01:55.524 --> 00:01:56.335 Scott: our episodes. 00:01:56.485 --> 00:01:56.695 Yeah. 00:01:56.795 --> 00:01:57.936 Joe Marion: People love it and 00:01:58.016 --> 00:01:58.306 Nick Berry: Yeah. 00:01:58.386 --> 00:02:00.470 Joe Marion: invite us to parties because they want to hear it. 00:02:00.586 --> 00:02:00.666 Nick Berry: Hmm. 00:02:01.541 --> 00:02:01.831 Joe Marion: Yeah. 00:02:04.375 --> 00:02:05.005 Scott: Okay. 00:02:05.274 --> 00:02:07.619 So today's episode, I, I. 00:02:08.800 --> 00:02:13.690 When we come up, we've done 60 episodes of this, and we're not running out of ideas, 00:02:13.690 --> 00:02:15.370 just like we're not running out of alpha. 00:02:15.370 --> 00:02:22.180 We have lots of ideas, and I'm asked all the time when I go out, what, how 00:02:22.180 --> 00:02:24.940 is AI affecting Berry consultants? 00:02:25.300 --> 00:02:27.040 I'm asked by statisticians. 00:02:27.100 --> 00:02:28.210 I'm asked by clinicians. 00:02:28.210 --> 00:02:31.570 I'm asked by people who don't know really what we do, but we know we have 00:02:31.570 --> 00:02:34.720 a company, uh, in, in the tech space. 00:02:34.900 --> 00:02:37.480 So today's episode is. 00:02:38.155 --> 00:02:39.355 AI at Barry. 00:02:40.555 --> 00:02:41.585 How do we use it? 00:02:42.235 --> 00:02:44.215 What, what do we know about it? 00:02:44.215 --> 00:02:46.465 What do we think where the world is going? 00:02:46.705 --> 00:02:51.775 And today, I, I have Nick and Joe on, and they, they're part of our 00:02:51.835 --> 00:02:59.275 committee that is exploring ai and it's a incredibly rapidly moving thing. 00:02:59.275 --> 00:03:01.435 And we're, we're, we're keeping track of this, so we're gonna 00:03:01.500 --> 00:03:03.655 talk about today, AI at Berry. 00:03:04.465 --> 00:03:08.095 I think it's important to set up what Barry does. 00:03:08.395 --> 00:03:14.515 So statisticians within the clinical trial space do lots of different things. 00:03:14.845 --> 00:03:16.435 Some of those things we don't do. 00:03:16.645 --> 00:03:20.635 Some things we do, and I think it's important in the role 00:03:20.635 --> 00:03:22.705 of AI in those, those things. 00:03:23.125 --> 00:03:27.475 So Berry Consultants at its heart is a clinical trial science company. 00:03:28.075 --> 00:03:31.495 We have clinical trial scientists that work at Berry. 00:03:32.525 --> 00:03:35.435 Most of them are statistically trained. 00:03:35.435 --> 00:03:40.145 We do have clinicians, we have mathematicians, computer scientists, 00:03:40.355 --> 00:03:42.995 but they are clinical trial scientists. 00:03:43.415 --> 00:03:50.735 We do really three major, major things, core competencies, and the first 00:03:50.735 --> 00:03:53.585 is trial design and trial strategy. 00:03:54.120 --> 00:03:59.040 So the science of clinical trial development and how the trial fits 00:03:59.040 --> 00:04:05.730 into that, the science of it, we most of the time were involved in designs 00:04:05.730 --> 00:04:07.560 that would be considered innovative. 00:04:07.950 --> 00:04:09.690 And from that meaning. 00:04:10.135 --> 00:04:13.195 Not typical, not fixed trials. 00:04:13.195 --> 00:04:18.685 So we do adaptive trials, seamless trials, adaptive enrichment trials. 00:04:18.954 --> 00:04:25.344 We incorporate external data within a clinical trial, more complicated analysis. 00:04:25.870 --> 00:04:30.820 In a trial platform, trials, basket trials, all of these 00:04:30.820 --> 00:04:36.460 fall under this umbrella of innovative clinical trial design. 00:04:36.460 --> 00:04:40.690 So we spend a lot of times in the design of these trials. 00:04:41.875 --> 00:04:46.375 The implementation of these innovative trials, whether it's carrying out the 00:04:46.375 --> 00:04:52.435 analysis, doing 60 interim analysis in the trial needs, additional expertise, 00:04:52.435 --> 00:04:58.104 and so we have a group that are experts in implementing innovative trial designs. 00:04:58.705 --> 00:05:03.445 So that's another core competency here at Barry in, in doing that. 00:05:04.930 --> 00:05:10.150 Clinical trial simulation plays a critical role in the design of innovative trials 00:05:10.450 --> 00:05:13.600 simulation, guided clinical trial design. 00:05:13.840 --> 00:05:15.970 We do it on individual projects. 00:05:15.970 --> 00:05:19.180 We might be writing custom code for individual projects. 00:05:19.390 --> 00:05:24.430 We also have software that we license to pharmaceutical, 00:05:24.430 --> 00:05:27.670 academics, government, CROs. 00:05:27.740 --> 00:05:30.050 For clinical trial simulation. 00:05:30.350 --> 00:05:32.630 And so we have a product that does that. 00:05:32.630 --> 00:05:37.460 It's another core competency at Barry in, in what we work on 00:05:37.460 --> 00:05:39.320 and what we do on a daily basis. 00:05:39.320 --> 00:05:43.305 So thinking about what is the role of AI in these various activities? 00:05:44.340 --> 00:05:48.120 What we don't do is we don't do a process stuff in trials. 00:05:48.120 --> 00:05:51.300 We stay out of the quote unquote CRO space. 00:05:51.660 --> 00:05:54.210 We don't create databases for trials. 00:05:54.210 --> 00:05:58.650 We don't do data management in trials, generation of the tables, 00:05:58.650 --> 00:06:03.090 listings and figures, the the SAS programs that generate these. 00:06:03.825 --> 00:06:05.625 Creation of CRFs. 00:06:05.775 --> 00:06:06.795 We don't do that. 00:06:06.825 --> 00:06:11.565 This is CROs are really good at that and that's not part of this strategy, 00:06:11.565 --> 00:06:14.055 part of clinical trial science. 00:06:14.715 --> 00:06:18.580 SAPs play an interesting role and maybe an interesting role in the whole I. 00:06:19.390 --> 00:06:20.950 Role of AI in all of this. 00:06:21.070 --> 00:06:23.950 We do contribute to statistical analysis plans. 00:06:24.370 --> 00:06:29.350 We don't create the huge tomb of the, the all of the tables, listings and 00:06:29.350 --> 00:06:33.520 figures, concomitant meds and all of that, because the, the group doing 00:06:33.520 --> 00:06:35.710 the TFS should be involved in that. 00:06:35.710 --> 00:06:36.820 But we play a really. 00:06:37.090 --> 00:06:43.090 Contributing contr critical role to the analysis, the objective within sap. 00:06:43.090 --> 00:06:48.159 So we work on SAPs, but we don't sort of generate them as a core competency. 00:06:49.000 --> 00:06:51.405 The last thing is we are not experts in ai. 00:06:52.285 --> 00:06:54.655 Uh, now we're interesting conversation. 00:06:54.655 --> 00:06:59.155 Should we be calling ourselves experts in ai, uh, in that, but 00:06:59.155 --> 00:07:00.625 we don't consider ourselves. 00:07:00.625 --> 00:07:09.925 So in this question of ai, we're not selling clients that we're ai, but we're 00:07:09.925 --> 00:07:12.235 utilizing AI and how we're using it. 00:07:12.235 --> 00:07:15.265 We're learning about that, and that's what we're gonna talk about today. 00:07:16.585 --> 00:07:20.815 So maybe I'll, maybe I'll throw it to Nick now. 00:07:20.815 --> 00:07:25.255 Nick, you're, you, you, you do design, you do implementation. 00:07:25.614 --> 00:07:29.755 Uh, you, you, you spend the majority of your time on the software team. 00:07:30.385 --> 00:07:31.830 How do you use ai? 00:07:34.381 --> 00:07:37.921 Nick Berry: Yeah, I mean this is maybe the most straightforward, um, 00:07:38.281 --> 00:07:44.551 use case where I think it's pretty generally accepted that AI is very 00:07:44.551 --> 00:07:46.591 good at coding, especially if you. 00:07:47.536 --> 00:07:52.546 Limited to things that are, you know, all over the internet. 00:07:52.546 --> 00:07:59.506 Like me, it's incredibly good at creating user interface interfaces. 00:07:59.566 --> 00:08:05.326 Uh, it's better than I am at modifying visual interfaces, connecting, 00:08:05.896 --> 00:08:11.026 um, buttons to effects and making sure all the data is propagating 00:08:11.026 --> 00:08:12.856 to the right places and tracking. 00:08:14.506 --> 00:08:17.836 Data storage models and things like that, it's incredibly good at that. 00:08:17.896 --> 00:08:20.806 Um, and I, definitely use it. 00:08:20.986 --> 00:08:21.826 Uh, for that. 00:08:22.096 --> 00:08:26.866 We have this fax that is now, I dunno, 17 years old is the name of our software Fax. 00:08:27.166 --> 00:08:29.206 It was created obviously before ai? 00:08:29.566 --> 00:08:32.266 the desktop app, the fax desktop app has. 00:08:32.326 --> 00:08:37.096 I mean essentially no, uh, LLM generated code in it, right? 00:08:37.096 --> 00:08:38.506 It's written predating it. 00:08:38.506 --> 00:08:39.286 It's is, yeah. 00:08:39.646 --> 00:08:46.366 Um, we are working on new technologies, uh, developing new fancy versions of, 00:08:46.786 --> 00:08:52.516 of facts and other applications and it's comes in handy, in that, with that 00:08:52.516 --> 00:08:57.586 said, like with the user interfaces thing that I love using it for, I. 00:08:58.696 --> 00:09:01.756 Am by now familiar with some of the pitfalls of using it. 00:09:01.936 --> 00:09:06.076 Um, probably makes me better at using it in general, but it's not 00:09:07.276 --> 00:09:11.506 incredibly good at writing technical statistical code in my experience. 00:09:11.566 --> 00:09:12.106 Um. 00:09:13.666 --> 00:09:15.196 Some things it does really well. 00:09:15.346 --> 00:09:19.336 Uh, you know, I think if you're simulating a trial and you want it to, to generate 00:09:19.336 --> 00:09:22.966 an accrual of patients for you, like it knows it's gonna do the cumulative 00:09:22.966 --> 00:09:26.626 sum of exponentially distributed patients, and it does that every time, 00:09:26.986 --> 00:09:28.546 that that's not what I'm talking about. 00:09:29.116 --> 00:09:31.216 Um, yeah. 00:09:31.216 --> 00:09:34.636 Predictive probabilities is a case where I've seen it, you know, it thinks 00:09:34.636 --> 00:09:40.766 it knows what it's doing, and, uh, even says something really, you know. 00:09:41.671 --> 00:09:45.151 Reassuring about what it's doing and then lo and behold, it's not 00:09:45.151 --> 00:09:46.171 doing what you think it's doing. 00:09:46.171 --> 00:09:50.851 And so some of the more, you know, maybe nuanced statistical concepts, 00:09:50.941 --> 00:09:52.351 I think it, it struggles at. 00:09:52.351 --> 00:09:55.081 But in general I think really good. 00:09:55.081 --> 00:09:58.621 Even, you know, in simulations writing simulation code, it will 00:09:58.621 --> 00:10:02.161 give you a backbone, it'll give you a nice flow through your app 00:10:02.341 --> 00:10:03.481 and, and help you write stuff. 00:10:03.481 --> 00:10:09.336 And so, I think we use it for a lot of like non-technical coding. 00:10:10.381 --> 00:10:16.201 I tried to, to get help our, we, we are adding analytical calculations 00:10:16.201 --> 00:10:17.701 for a lot of designs to facts. 00:10:17.701 --> 00:10:19.921 And, you know, this is a numerical integration. 00:10:19.921 --> 00:10:20.641 It's, it's a, 00:10:22.921 --> 00:10:27.391 it's really computer science heavy code, not so much like statistical 00:10:27.391 --> 00:10:28.591 heavy code in a lot of ways. 00:10:28.591 --> 00:10:30.151 And in that case. 00:10:30.946 --> 00:10:31.396 It, it 00:10:31.721 --> 00:10:31.940 Joe Marion: Is it good? 00:10:32.146 --> 00:10:34.906 Nick Berry: away, like, oh, you're using this person's integration algorithm. 00:10:34.906 --> 00:10:36.436 These are your weights, so on, so on. 00:10:36.706 --> 00:10:39.106 And then it randomly slips in something you don't want to be there. 00:10:39.106 --> 00:10:43.726 And it's hard to, to catch that in a lot of cases because, you know, those 00:10:43.726 --> 00:10:46.786 algorithms are not like intuitive to look at and say, oh yeah, okay, 00:10:46.786 --> 00:10:48.346 I see exactly where it went wrong. 00:10:48.346 --> 00:10:54.916 And so we had some, some mixed, uh, mixed results in, in like 00:10:54.916 --> 00:10:56.386 that heavy computation area. 00:10:57.471 --> 00:11:00.391 Joe Marion: Is, is the cost of debugging worth the cost of using it? 00:11:01.175 --> 00:11:03.486 Like, that's what's, what's the trade off in time? 00:11:03.486 --> 00:11:05.736 And then like, I don't know, what's the trade off in enjoyment? 00:11:05.826 --> 00:11:07.961 Like what, what would you rather be doing? 00:11:08.191 --> 00:11:08.481 Yeah, 00:11:09.181 --> 00:11:11.161 Nick Berry: for me it, it was, it was worth it. 00:11:11.191 --> 00:11:13.411 Um, the, 00:11:15.511 --> 00:11:16.021 yeah. 00:11:16.141 --> 00:11:18.571 that, that's not like a dopamine hit for me. 00:11:18.571 --> 00:11:18.871 Right. 00:11:18.871 --> 00:11:21.631 Like writing numerical integration code didn't, actually, 00:11:21.786 --> 00:11:22.026 Joe Marion: No, 00:11:22.861 --> 00:11:25.381 Nick Berry: that doesn't keep the lights on like mentally for 00:11:25.475 --> 00:11:27.605 Joe Marion: not coming over to my house, that's for sure. 00:11:28.401 --> 00:11:29.401 away on Friday nights. 00:11:29.550 --> 00:11:29.840 Yeah. 00:11:30.286 --> 00:11:33.826 Nick Berry: that interestingly, the, the UI stuff, watching it make big 00:11:33.826 --> 00:11:37.546 changes and like, oh, I, I feel like I have a really nice flow for this. 00:11:37.546 --> 00:11:39.886 Like, I'm gonna help people understand what they're inputting in 00:11:39.886 --> 00:11:41.356 our app and it's gonna be so nice. 00:11:41.416 --> 00:11:45.166 That is a dopamine hit and the amount that it can do sort of in 00:11:45.166 --> 00:11:47.446 one shot is kind of incredible. 00:11:47.536 --> 00:11:49.006 And it's low stakes, right? 00:11:49.006 --> 00:11:51.646 A lot of times I'm just, you can look at it and say, Yeah. 00:11:51.646 --> 00:11:52.576 it did the right thing. 00:11:52.816 --> 00:11:55.186 And I can look at the, the database and say, yeah, all the 00:11:55.186 --> 00:11:56.266 stuff's where it needs to be. 00:11:57.016 --> 00:12:02.446 other things that are higher stakes, like my group sequential probabilities 00:12:02.446 --> 00:12:06.706 being calculated correctly and am is the type one error that I'm reporting, 00:12:06.706 --> 00:12:07.936 actually a type one error and things like 00:12:08.106 --> 00:12:08.396 Joe Marion: Okay. 00:12:08.491 --> 00:12:08.781 Nick Berry: yeah, 00:12:10.660 --> 00:12:14.379 Scott: So let's talk a little bit about efficiency, and you sh you demonstrated 00:12:14.379 --> 00:12:18.369 to me an example the other day, and I, I think, by the way, I, I can ask, what's 00:12:18.369 --> 00:12:21.609 your favorite AI for code generation. 00:12:24.211 --> 00:12:24.811 Nick Berry: I, I use. 00:12:25.726 --> 00:12:27.316 Claude, um, I use 00:12:27.405 --> 00:12:27.694 Scott: Okay. 00:12:28.366 --> 00:12:29.296 Nick Berry: as of today. 00:12:29.446 --> 00:12:30.436 4.7. 00:12:30.556 --> 00:12:32.056 Um, is the model 00:12:32.060 --> 00:12:32.990 Joe Marion: Extra thinking. 00:12:32.990 --> 00:12:33.411 Right? 00:12:33.411 --> 00:12:35.000 Or maximum thinking How? 00:12:35.000 --> 00:12:36.320 How much thinking do you think? 00:12:36.320 --> 00:12:36.590 Yeah. 00:12:36.590 --> 00:12:37.610 Extra high thinking. 00:12:37.610 --> 00:12:38.060 Yeah. 00:12:38.060 --> 00:12:38.070 Yeah. 00:12:38.206 --> 00:12:39.016 Nick Berry: places to be. 00:12:39.016 --> 00:12:40.276 I can't use Max. 00:12:40.726 --> 00:12:41.116 Yeah. 00:12:41.875 --> 00:12:47.155 Scott: Okay, so you provided a demonstration of a pretty simple clinical 00:12:47.155 --> 00:12:52.225 trial design, but simulations it made in our shiny app, really beautiful app 00:12:52.225 --> 00:12:54.235 for exploring things and all of that. 00:12:54.280 --> 00:12:58.944 And, and I know you kind of knew what you wanted to do going into this, but largely 00:12:58.944 --> 00:13:01.225 you could write this code in 10 minutes. 00:13:01.944 --> 00:13:06.084 Uh, it was the demonstration of this with Claude, with Claude. 00:13:06.121 --> 00:13:06.511 Nick Berry: right? 00:13:06.561 --> 00:13:08.481 Joe Marion: the Claude writes the code in 10 minutes. 00:13:08.484 --> 00:13:08.694 Scott: yeah. 00:13:08.694 --> 00:13:13.944 No, I, yeah, I meant using AI where if you sat down from scratch in writing 00:13:13.944 --> 00:13:16.794 this, it's, you know, it's 10 days. 00:13:17.094 --> 00:13:19.704 Uh, I, and, you know, don't know the magnitude of that. 00:13:19.854 --> 00:13:23.994 And now, yeah, now I just, I just challenged Nick about how long he 00:13:23.994 --> 00:13:26.004 could do it for that, but, but. 00:13:27.271 --> 00:13:30.301 Nick Berry: I might never need to write reactivity code in shiny ever again. 00:13:30.301 --> 00:13:31.861 I mean, I'm not a shiny developer, right? 00:13:31.861 --> 00:13:35.041 I'm not even really a web developer, even though I do It to some extents. 00:13:35.461 --> 00:13:40.261 But I mean, I'm a statistical developer, uh, in, in most regards. 00:13:40.471 --> 00:13:42.991 so, Claude 00:13:43.074 --> 00:13:43.084 Scott: was 00:13:43.111 --> 00:13:44.486 Nick Berry: good at doing the things that I don't really. 00:13:45.331 --> 00:13:46.921 Know that well, right. 00:13:46.921 --> 00:13:49.651 These are the fringe cases of my job that I've had to learn and I've had to 00:13:49.651 --> 00:13:54.481 go and, you know, take courses online about how to do web development and now, 00:13:55.111 --> 00:14:02.851 um, passing off a lot of that skillset, um, which is great as long as it's doing 00:14:02.851 --> 00:14:04.411 a good job, which it seems to be doing. 00:14:04.861 --> 00:14:08.341 Um, but the skillset that I'm sort of an expert in, which is the more 00:14:08.341 --> 00:14:11.281 statistical computation writing MCMC code, things like that. 00:14:12.151 --> 00:14:13.441 I'm doing that still. 00:14:13.441 --> 00:14:13.651 Right. 00:14:13.651 --> 00:14:15.661 That's not something I'm passing off to, to AI at this point. 00:14:17.290 --> 00:14:20.079 Scott: So we we're putting together a development plan. 00:14:20.079 --> 00:14:23.439 So you, you mentioned fax as a desktop product, for example. 00:14:23.620 --> 00:14:26.020 We're making it a cloud-based product. 00:14:26.260 --> 00:14:28.599 We're redesigning the GUI to this. 00:14:28.599 --> 00:14:31.329 We're adding a, a, a number of really nice things. 00:14:32.199 --> 00:14:35.775 Had we done this five years ago, we would've had. 00:14:36.459 --> 00:14:40.300 More developers and it would've taken significantly longer. 00:14:40.540 --> 00:14:43.899 All of a sudden with what AI has done in the last year, 00:14:44.170 --> 00:14:47.170 we need less goey developers. 00:14:47.170 --> 00:14:52.060 Within all of this, Nick, is the productivity maybe of three 00:14:52.060 --> 00:14:54.790 developers years ago in terms of time. 00:14:54.999 --> 00:15:00.939 This is, uh, this is game changing, enormous ability for us to do 00:15:00.939 --> 00:15:02.649 this much more efficiently. 00:15:03.676 --> 00:15:05.986 Nick Berry: Well, in my mind it's like the perfect use case. 00:15:05.986 --> 00:15:08.626 I, I'm not gonna say like, oh, we 10 XD our development. 00:15:08.626 --> 00:15:09.076 That's not, 00:15:09.739 --> 00:15:10.029 Scott: Yeah. 00:15:10.696 --> 00:15:14.356 Nick Berry: a mature app that we've spent 15 years with 00:15:14.356 --> 00:15:15.751 statistical programmers developing. 00:15:16.636 --> 00:15:19.606 We've always had The engines in the background, like 00:15:19.606 --> 00:15:20.866 the, the crown jewel, right? 00:15:20.866 --> 00:15:23.446 This is the multiple amputation 00:15:23.526 --> 00:15:24.065 Joe Marion: part, right. 00:15:24.856 --> 00:15:28.066 Nick Berry: response modeling, all the good stuff that drives facts 00:15:28.066 --> 00:15:33.286 and makes it do stuff that no other software can do that is there. 00:15:33.346 --> 00:15:35.476 It's essentially unchanged. 00:15:35.656 --> 00:15:40.576 The only thing we have to do for this web development is create a facade around it 00:15:40.906 --> 00:15:44.806 that takes in inputs, converts them to parameters, and passes it into engines. 00:15:45.256 --> 00:15:47.356 So I think it's like the perfect use case, right? 00:15:47.356 --> 00:15:51.586 The, the tests that we use for current facts carry over. 00:15:51.586 --> 00:15:53.986 They go to web fax, like, you know, our testing. 00:15:53.986 --> 00:15:58.366 We can run these on the cloud, we can run them on the desktop 00:15:58.366 --> 00:15:59.416 and make sure the results match. 00:15:59.416 --> 00:16:02.836 Like we have so many protections while we develop this, that it's 00:16:02.836 --> 00:16:07.816 like the perfect use case, uh, for AI in for, for, you know, this. 00:16:08.581 --> 00:16:11.401 AI as a coding agent, at least in my mind. 00:16:11.401 --> 00:16:15.241 And so we've gotten really nice returns from it as like 00:16:15.474 --> 00:16:15.744 Scott: Yeah. 00:16:16.081 --> 00:16:17.221 Nick Berry: UI scaffolder. 00:16:18.225 --> 00:16:20.105 Joe Marion: Scott, let me, let me ask you a question, Scott. 00:16:21.605 --> 00:16:23.795 now Nick is doing the work of three nicks. 00:16:23.795 --> 00:16:25.986 I mean, we're making up the number three, but let's just say it's three 00:16:26.199 --> 00:16:26.410 Scott: Yeah. 00:16:26.619 --> 00:16:26.890 Yeah. 00:16:27.396 --> 00:16:28.626 Joe Marion: So I mean. 00:16:29.391 --> 00:16:32.420 To me that says maybe you want more K Nicks, right? 00:16:32.751 --> 00:16:35.271 Like he's, in some sense, you're getting three times the 00:16:35.271 --> 00:16:36.740 value for the, the same person. 00:16:36.740 --> 00:16:41.001 Like maybe you want even more like, like maybe the, maybe it's not that 00:16:41.001 --> 00:16:44.631 we're more efficient, but like, I mean, maybe you'd want nine, nine 00:16:44.631 --> 00:16:46.131 K Nicks worth of capacity now. 00:16:46.131 --> 00:16:46.490 Right. 00:16:46.490 --> 00:16:46.761 And 00:16:46.764 --> 00:16:47.035 Scott: Yeah. 00:16:47.360 --> 00:16:47.480 Joe Marion: I 00:16:47.634 --> 00:16:47.844 Scott: Yep. 00:16:47.930 --> 00:16:50.180 Joe Marion: the margin, it's kind of changing the value of what we can do. 00:16:50.450 --> 00:16:50.840 So. 00:16:51.114 --> 00:16:51.354 Scott: Yeah. 00:16:51.354 --> 00:16:54.624 From a business perspective perspective, this is really interesting. 00:16:54.624 --> 00:16:55.464 Is, is. 00:16:56.199 --> 00:17:01.269 Do I have the work to support nine times, nine nicks, you know, sort of 00:17:01.269 --> 00:17:03.999 thing within that in the development. 00:17:04.300 --> 00:17:09.339 And so we, as a company we've talked about, we, we are. 00:17:10.450 --> 00:17:12.369 We, we've got a good amount of business. 00:17:12.369 --> 00:17:13.839 Barry's doing very well. 00:17:13.929 --> 00:17:19.480 We could expand people, we could add more people to Barry, it's an 00:17:19.480 --> 00:17:23.980 interesting question of, in five years will all of the individuals 00:17:23.980 --> 00:17:28.060 at Barry be three times themselves, five times themselves, and all of a 00:17:28.060 --> 00:17:31.420 sudden, you know, the, the work needed? 00:17:31.420 --> 00:17:32.710 Do I need more people? 00:17:32.710 --> 00:17:35.680 I, it's a really interesting thing from a business perspective and 00:17:35.680 --> 00:17:37.450 software is different than design. 00:17:37.461 --> 00:17:40.101 Joe Marion: Yeah, software is, I'm thinking more on the software 00:17:40.101 --> 00:17:40.971 side of the house as well. 00:17:41.001 --> 00:17:41.751 This is like, 00:17:41.794 --> 00:17:42.145 Scott: right, 00:17:42.321 --> 00:17:44.871 Joe Marion: you know, you, you could, you could do things. 00:17:45.651 --> 00:17:47.901 It would've been really expensive before and now may. 00:17:47.901 --> 00:17:50.456 Now maybe they make sense, Right, Like now they're, 00:17:50.475 --> 00:17:50.825 Scott: right, 00:17:51.286 --> 00:17:56.746 Nick Berry: the skillset that makes me good for the software team right now 00:17:56.776 --> 00:18:00.916 is not ability to develop software. 00:18:00.916 --> 00:18:00.926 It's. 00:18:01.741 --> 00:18:05.461 It's a little bit of that, but mostly it's me being able to 00:18:05.461 --> 00:18:07.171 see what I want and have ideas 00:18:07.306 --> 00:18:07.426 Joe Marion: I. 00:18:07.471 --> 00:18:07.531 Nick Berry: I. 00:18:07.531 --> 00:18:10.201 want from, you know, we've, we've learned a lot of lessons 00:18:10.201 --> 00:18:11.971 with fax desktop over the years. 00:18:12.211 --> 00:18:17.101 We're trying to improve the, this cloud app to resolve all the things we 00:18:17.101 --> 00:18:22.351 wish we'd resolved, but being able to know what I think people would expect, 00:18:22.351 --> 00:18:24.481 what I want this, this new app to do. 00:18:24.481 --> 00:18:29.071 And then precisely state, and this is where my software skills 00:18:29.071 --> 00:18:30.781 come in, is tell the, the. 00:18:31.486 --> 00:18:34.816 AI agent exactly what I want it to do. 00:18:34.816 --> 00:18:38.476 And a lot of times that involves speaking in sort of software terms and 00:18:38.476 --> 00:18:45.256 using things that, uh, aid me in being precise about specifying what we want. 00:18:45.346 --> 00:18:47.626 Um, and especially for like the front end. 00:18:47.626 --> 00:18:50.596 And for things like that, it's mostly just having a vision and 00:18:50.596 --> 00:18:53.176 being extremely specific about how you want to implement it. 00:18:54.306 --> 00:18:54.456 Joe Marion: Yeah. 00:18:54.485 --> 00:18:55.476 And having good taste. 00:18:55.476 --> 00:18:56.316 Having a good vision, 00:18:56.361 --> 00:18:56.521 Nick Berry: vision. 00:18:56.740 --> 00:18:57.040 Scott: Yeah. 00:18:57.221 --> 00:18:57.501 Nick Berry: A good 00:18:57.546 --> 00:18:57.876 Joe Marion: yeah. 00:18:57.876 --> 00:18:59.586 I think both of those things matter, right. 00:18:59.806 --> 00:19:00.076 Nick Berry: Yeah. 00:19:00.346 --> 00:19:00.526 Yeah. 00:19:01.326 --> 00:19:03.606 Joe Marion: It's an incredible time to build things like. 00:19:04.990 --> 00:19:06.400 Scott: So this is gonna come back. 00:19:06.400 --> 00:19:09.340 As in you, you brought up the really important issue that 00:19:09.760 --> 00:19:14.410 you have this imagination and ability to know where to go. 00:19:14.710 --> 00:19:18.550 You're consuming lots of information and making these really good decisions. 00:19:18.820 --> 00:19:22.540 And then you can tell AI to do something and you know what it is. 00:19:22.780 --> 00:19:26.860 And AI's really, really good at taking those instructions and creating something, 00:19:26.860 --> 00:19:30.040 and you're iterating it and you're, you're on top of it and you're controlling it. 00:19:30.376 --> 00:19:34.426 Nick Berry: It's been fun because I can ask it to do three things and I 00:19:34.426 --> 00:19:35.566 can pick which one I like the best. 00:19:35.566 --> 00:19:38.296 Whereas normally I, I don't have time to do all three things and look at 00:19:38.406 --> 00:19:39.786 Joe Marion: Yeah, you're not gonna do three things, 00:19:39.796 --> 00:19:40.366 Nick Berry: things. 00:19:40.666 --> 00:19:41.356 I can look at 00:19:41.395 --> 00:19:41.605 Scott: Yep. 00:19:41.626 --> 00:19:44.266 Nick Berry: I can say, oh, this one actually turned out the best. 00:19:44.266 --> 00:19:44.896 Let's do that one. 00:19:44.896 --> 00:19:47.206 And like that's, yeah, so 00:19:47.815 --> 00:19:48.085 Scott: Yep, 00:19:48.196 --> 00:19:49.306 Nick Berry: amount is actually get 00:19:49.375 --> 00:19:49.645 Scott: yep, 00:19:49.666 --> 00:19:52.216 Nick Berry: at the options live in the app and going so, 00:19:53.004 --> 00:19:53.155 Scott: yep. 00:19:53.586 --> 00:19:53.856 Joe Marion: And I, 00:19:53.905 --> 00:19:54.715 Scott: Joe, how. 00:19:54.906 --> 00:19:56.346 Joe Marion: for general use too, right? 00:19:56.346 --> 00:19:57.036 Is like, 00:19:57.046 --> 00:19:57.436 Nick Berry: sure. 00:19:58.206 --> 00:20:00.636 Joe Marion: ask for multiple things, give it feedback on what 00:20:00.636 --> 00:20:06.006 you like, like, you know, the labor is in some sense cheap on its end, 00:20:06.250 --> 00:20:06.610 Scott: Mm. 00:20:06.906 --> 00:20:07.476 Joe Marion: uh, yeah. 00:20:08.496 --> 00:20:09.516 it and interact with it. 00:20:09.521 --> 00:20:09.641 Yeah. 00:20:10.840 --> 00:20:11.230 Scott: Okay. 00:20:11.230 --> 00:20:14.800 So that, that, maybe we should make sure we touch on how to use it well. 00:20:14.800 --> 00:20:16.629 But that sounds like that's an important role of it. 00:20:16.810 --> 00:20:22.840 So how do you, uh, Joe spends more of his time in the design of clinical trials. 00:20:22.840 --> 00:20:27.010 He does implementation 60 interims in a particular trial. 00:20:27.310 --> 00:20:31.930 Does implementation, um, what, how do you use AI at Barry? 00:20:33.201 --> 00:20:37.911 Joe Marion: Yeah, I mean, I think there's, I with what Nick does of course, right? 00:20:37.911 --> 00:20:38.856 Like I think. 00:20:39.516 --> 00:20:41.946 Coding is where these things are their very best. 00:20:42.006 --> 00:20:46.476 So, you know, we're writing simulation code. 00:20:46.866 --> 00:20:49.266 Um, we do a lot of visualization. 00:20:49.746 --> 00:20:52.956 It's good at visualization, but of course, you gotta be very specific, right? 00:20:52.956 --> 00:20:55.656 You have to know what to tell it that you want. 00:20:56.616 --> 00:20:58.896 It'll, it'll produce plots, but they're often like, you know, 00:20:58.896 --> 00:21:00.276 not quite of the right quality. 00:21:00.276 --> 00:21:05.496 And so, You know, things you can do is like, you can have like a style guide 00:21:05.496 --> 00:21:08.286 for how you want plots to look and that kind of thing, and that helps it sort of 00:21:08.286 --> 00:21:10.176 one shot those tasks a little bit better. 00:21:10.896 --> 00:21:13.716 Um, things I'm really excited about. 00:21:15.246 --> 00:21:22.566 Um, well, I, one application that I think is really cool there's 00:21:22.566 --> 00:21:26.076 this, this common like, step of regulatory interactions with. 00:21:26.436 --> 00:21:30.156 Where, where when you're preparing for a briefing with the FDA, I feel like 00:21:30.156 --> 00:21:33.276 every company ever that's doing this preparation, one of the things they 00:21:33.276 --> 00:21:37.356 do is they think about what could the FDA's response be to our questions? 00:21:37.626 --> 00:21:39.426 What should we be prepared to answer? 00:21:39.426 --> 00:21:40.956 How do we do that strategizing? 00:21:41.796 --> 00:21:43.986 I, I sort of always, I don't know. 00:21:43.986 --> 00:21:45.906 I mean, I felt it was necessary, right? 00:21:45.911 --> 00:21:47.586 But I hated writing those questions down. 00:21:47.586 --> 00:21:48.696 I kind of knew what they were. 00:21:49.836 --> 00:21:54.876 it's tremendously interesting to get the AI critique of like a sanitized document. 00:21:55.851 --> 00:21:58.311 What is the regulatory perspective gonna be? 00:21:58.701 --> 00:22:02.811 And I think it's really valuable to us at Berry because you know, you know, 00:22:02.811 --> 00:22:05.781 based on the things it's trained on, it's gonna give you a pretty traditional view. 00:22:05.781 --> 00:22:09.981 And I think that can really help you sort of understand what that is. 00:22:10.251 --> 00:22:13.011 It's not the argument we're usually articulating because of, you know, the 00:22:13.011 --> 00:22:18.081 kind of work that we do, but uh, I love that ability to get critique from it. 00:22:18.081 --> 00:22:19.611 I think it's tremendously fascinating. 00:22:21.011 --> 00:22:23.746 Nick Berry: I, in my experience when doing something like that, it's usually 00:22:23.746 --> 00:22:29.146 like a numbers game in getting a lot of different things from the, the, 00:22:29.176 --> 00:22:33.796 the AI model where you can pick and choose the things that are actually 00:22:33.796 --> 00:22:35.296 intellectually interesting and. 00:22:36.481 --> 00:22:40.051 and like you said, I kinda wanna emphasize we work at Berry Consultants. 00:22:40.081 --> 00:22:44.671 We tend to have, we're not, obviously not a homogenous population of 00:22:44.671 --> 00:22:48.691 people, but in general we're designing adaptive trials, often Bayesian 00:22:49.021 --> 00:22:50.551 interim analysis, things like that. 00:22:50.551 --> 00:22:55.111 And the literature, you know, everything that's posted online, the vast majority of 00:22:55.111 --> 00:23:00.901 it is traditional and frequentist and not as innovative as a lot of things we do. 00:23:00.901 --> 00:23:01.966 So the LLM. 00:23:03.361 --> 00:23:04.681 Thinks like that, right? 00:23:04.681 --> 00:23:07.651 So you're almost getting a, you, you are getting a conservative 00:23:07.651 --> 00:23:09.421 perspective when you ask it questions. 00:23:10.416 --> 00:23:12.366 Joe Marion: Yeah, complimentary review and then you can think 00:23:12.366 --> 00:23:13.476 through how you'd answer them. 00:23:13.866 --> 00:23:16.806 It's not very good at answering its own questions 'cause you know, 00:23:16.806 --> 00:23:19.986 we've got a particular house view that's like really underrepresented, 00:23:19.986 --> 00:23:21.311 but you know, I think that that. 00:23:22.011 --> 00:23:23.181 It's like a pretty exciting thing. 00:23:23.181 --> 00:23:27.651 And yeah, like Nick said, you want a bunch, like if you get 10 criticisms, 00:23:27.651 --> 00:23:30.441 you might like three or four of them might really be worth, worth, 00:23:30.441 --> 00:23:31.761 interesting, and worth addressing. 00:23:31.761 --> 00:23:33.231 But it's still good value. 00:23:33.446 --> 00:23:33.736 Yeah. 00:23:33.886 --> 00:23:34.546 Nick Berry: told it? 00:23:35.296 --> 00:23:38.146 Uh, give me feedback like you work at Berry Consultants. 00:23:40.176 --> 00:23:40.221 Joe Marion: Yeah. 00:23:40.281 --> 00:23:43.191 Not, well, actually, I think one of the, I've, I haven't tried this in a 00:23:43.191 --> 00:23:44.421 while, but I used to tell it to like. 00:23:45.036 --> 00:23:47.196 You know, write like Kurt Veley, right? 00:23:47.196 --> 00:23:49.506 Or like, give me the Kurt Veley arguments for this kind of thing. 00:23:49.536 --> 00:23:54.246 And like, not a very good Curt, I wonder if it's a good curt now, I 00:23:54.341 --> 00:23:54.631 Nick Berry: yeah. 00:23:54.846 --> 00:23:56.106 Joe Marion: that would be interesting to me. 00:23:56.181 --> 00:23:56.471 Nick Berry: Yeah. 00:23:56.521 --> 00:23:58.506 Joe Marion: I, I kind of think it probably isn't. 00:23:58.656 --> 00:24:03.186 Um, it, it can do other public figures reasonably well, but I don't think it 00:24:03.186 --> 00:24:05.196 does any like statistician super well. 00:24:05.366 --> 00:24:05.656 Nick Berry: Yeah, 00:24:05.885 --> 00:24:06.725 Scott: Okay, so this I, 00:24:06.991 --> 00:24:08.041 Nick Berry: on Twitter a lot, right? 00:24:08.041 --> 00:24:08.431 He has 00:24:08.645 --> 00:24:08.885 Scott: yeah. 00:24:09.031 --> 00:24:12.391 Nick Berry: Twitter about different things, and he may probably not, you 00:24:12.391 --> 00:24:17.371 know, he has a lot of public visibility on some of these topics because 00:24:17.466 --> 00:24:19.926 Joe Marion: He has some chance of being out there, right? 00:24:20.016 --> 00:24:20.286 Yeah. 00:24:21.036 --> 00:24:21.276 Yeah. 00:24:21.936 --> 00:24:25.146 Um, other stuff that I think AI is cool for. 00:24:25.266 --> 00:24:25.866 Um. 00:24:27.861 --> 00:24:29.871 I, produce way more shiny apps than before. 00:24:30.021 --> 00:24:32.601 because the, the cost of doing so is really low. 00:24:33.081 --> 00:24:36.741 And what's great is that you can pass them to clients and, and then that 00:24:36.741 --> 00:24:40.311 really gives them like a hands-on tool that, that allows them to interact 00:24:40.311 --> 00:24:43.161 with the decision they're making to tweak all the levers themselves to 00:24:43.161 --> 00:24:45.201 get that kind of firsthand experience. 00:24:46.606 --> 00:24:49.521 you know, sometimes in like a rare disease trial, they'll want to 00:24:49.521 --> 00:24:53.201 know it's, you know, I have 50 end points, which one should I use? 00:24:53.201 --> 00:24:53.651 Right. 00:24:54.101 --> 00:24:58.930 And the ability to hand a tool over instead of like a ton of 00:24:58.930 --> 00:25:01.391 plots is, is tremendously useful. 00:25:01.751 --> 00:25:03.371 Absolutely improves the communication. 00:25:03.761 --> 00:25:05.381 I'm hopeful like that. 00:25:05.381 --> 00:25:07.151 This will be good for facts too, right? 00:25:07.151 --> 00:25:11.051 You can imagine a plain, plain language version of facts where 00:25:11.051 --> 00:25:14.351 you describe in text what you want the design to be, and then it knows 00:25:14.351 --> 00:25:16.421 how to set up a FACTS file, like. 00:25:17.196 --> 00:25:20.916 You know that that's, you know, probably a little bit of a ways away, but it's 00:25:20.916 --> 00:25:23.166 not like a pie in the sky dream anymore. 00:25:23.586 --> 00:25:27.096 I think five years ago that would've been a tremendously complex endeavor. 00:25:27.501 --> 00:25:27.561 Yeah. 00:25:27.646 --> 00:25:27.916 Nick Berry: course. 00:25:27.916 --> 00:25:31.426 Every single person that we show this cloud app to their mind initially 00:25:31.426 --> 00:25:32.866 goes like, it makes so much sense. 00:25:32.866 --> 00:25:34.756 This has to be something that we can do. 00:25:34.756 --> 00:25:35.281 It's so cool and. 00:25:36.346 --> 00:25:38.596 The parameters that go into facts are known, right? 00:25:38.596 --> 00:25:42.346 There's, it's not like it has to write simulation code that 00:25:42.346 --> 00:25:44.086 can do infinitely many things. 00:25:44.236 --> 00:25:46.876 There's a set of parameters that are predefined. 00:25:47.116 --> 00:25:49.996 The storage mechanism for those parameters is predefined. 00:25:49.996 --> 00:25:51.616 We have documentation for that. 00:25:52.006 --> 00:25:56.656 Presumably, you give that to an LLM, someone can describe their trial that 00:25:56.656 --> 00:25:59.386 they want, not design me a good trial. 00:26:00.196 --> 00:26:03.676 I don't, we can get to this later, but it's not gonna do a good job. 00:26:03.676 --> 00:26:04.876 It's not gonna do what you want. 00:26:04.876 --> 00:26:06.226 It's not gonna do a good job in that case. 00:26:06.526 --> 00:26:09.076 But I want a trial with three interims. 00:26:09.076 --> 00:26:14.386 I want Goldilocks stopping style, um, decisions at interims with 00:26:14.386 --> 00:26:16.096 these predictive probabilities. 00:26:16.186 --> 00:26:19.666 Um, and I want to figure out what p value threshold gives me. 00:26:19.666 --> 00:26:20.386 A good type one error. 00:26:20.416 --> 00:26:25.696 Like now it's got the parameters it needs, and it can fill in, can say, oh, 00:26:25.696 --> 00:26:27.076 when do you want your interim scheduled? 00:26:27.076 --> 00:26:27.886 And things like that. 00:26:28.246 --> 00:26:29.326 And then all it has to do is say. 00:26:30.331 --> 00:26:35.611 Interim analysis schedule Vector comma 400, comma 500, and you 00:26:35.611 --> 00:26:37.351 know, it's, we have to teach it. 00:26:37.501 --> 00:26:40.441 It's guardrails and then I think it's like Yeah. 00:26:41.226 --> 00:26:41.931 Joe Marion: But it'll not. 00:26:41.941 --> 00:26:44.611 Nick Berry: of of doing, of, of figuring out exactly, you know, 00:26:44.611 --> 00:26:47.071 what prompt gets it to do the right thing and things like that. 00:26:47.071 --> 00:26:47.821 But, but it's 00:26:47.886 --> 00:26:48.246 Joe Marion: Yeah. 00:26:48.336 --> 00:26:52.866 W what's the right like iteration loop that, that lets it learn that, right? 00:26:53.136 --> 00:26:53.406 yeah. 00:26:54.070 --> 00:27:00.835 Scott: So as, as of this point, so AI is being used, the LLMs where. 00:27:00.975 --> 00:27:08.055 You're asking it to write code or you're asking it to do a task, and it's making 00:27:08.055 --> 00:27:13.725 us much more efficient, even better at that task, you said, uh, typically you'd 00:27:13.725 --> 00:27:18.555 walk in historically with a set of seven slides that you're presented to a client. 00:27:18.705 --> 00:27:21.225 Now you're walking in with an app that has presumably. 00:27:21.715 --> 00:27:23.965 Hundreds of potential things that they could look at. 00:27:24.205 --> 00:27:27.985 So you're, you, you're, you're increasing things, but largely you're using it 00:27:27.985 --> 00:27:32.545 to be more efficient and maybe a bit more broad about what you're doing. 00:27:33.055 --> 00:27:37.045 What you just described is a whole different thing where we're training 00:27:37.045 --> 00:27:42.865 something and a generative AI that we're now creating a tool that's using AI 00:27:43.105 --> 00:27:47.785 for somebody to use facts or somebody to use it, which we haven't done yet, 00:27:47.785 --> 00:27:50.095 but is, is an intoxicating thought. 00:27:51.751 --> 00:27:52.021 Joe Marion: Yeah. 00:27:53.221 --> 00:27:56.101 Really democratizing in some sense, right? 00:27:56.251 --> 00:27:58.861 You still have to have good taste to use that tool, right? 00:27:58.951 --> 00:28:01.441 You still, you still would have to know what to tell it, right? 00:28:01.441 --> 00:28:01.651 It. 00:28:02.391 --> 00:28:05.361 It's not a self-governing process, but I think it cuts 00:28:05.361 --> 00:28:06.621 so much of the learning curve. 00:28:06.621 --> 00:28:09.026 Some, so much of the uninteresting learning curve out maybe. 00:28:09.376 --> 00:28:09.666 Yeah. 00:28:10.006 --> 00:28:10.216 Nick Berry: Yeah. 00:28:10.495 --> 00:28:12.985 Scott: Okay, so let's, let's dive into some of that. 00:28:12.985 --> 00:28:18.115 I, I, but let's talk about things that are happening at Barry with, with ai. 00:28:18.115 --> 00:28:18.430 I think. 00:28:18.785 --> 00:28:24.425 You can't do a search engine now without it being ai the default in all of this. 00:28:24.425 --> 00:28:26.285 And that's, that, that's pretty common. 00:28:26.285 --> 00:28:27.515 I think everybody's doing that. 00:28:27.875 --> 00:28:32.975 The writing, you talked about writing, uh, the ability for it to, to read your 00:28:32.975 --> 00:28:35.435 writing, to critique it, to, to update it. 00:28:35.795 --> 00:28:40.925 Every one of these podcasts goes into AI and it creates social 00:28:40.925 --> 00:28:44.285 media, blurbs, and a blog. 00:28:44.361 --> 00:28:44.781 Joe Marion: sense. 00:28:45.010 --> 00:28:46.060 Scott: And a blog. 00:28:46.360 --> 00:28:49.690 Now, I read every single one of the social media things. 00:28:49.690 --> 00:28:52.180 I edit it, I read the blogs. 00:28:52.360 --> 00:28:56.320 I, I will go in and read some of these and there will be a paragraph that 00:28:56.320 --> 00:29:00.430 I just don't understand, and it just didn't sort of work and all of that. 00:29:00.430 --> 00:29:02.470 I don't mean that to be critical of ai. 00:29:02.740 --> 00:29:05.200 I, I think that that's, that's. 00:29:05.201 --> 00:29:06.641 Joe Marion: the technology is sometimes, right? 00:29:06.831 --> 00:29:07.121 Like 00:29:07.735 --> 00:29:07.975 Scott: Yep. 00:29:08.155 --> 00:29:11.485 Uh, and maybe it was because I did do a very good job on the podcast 00:29:11.485 --> 00:29:14.845 of explaining it, and that's what AI came out with, with the meaning of it. 00:29:15.115 --> 00:29:20.005 Uh, we did, we did it, we did one on the award, five trial, that was the 00:29:20.005 --> 00:29:24.925 acronym of the trial, and it kept saying the award winning Trulicity 00:29:24.925 --> 00:29:26.665 trial, you know, sort of thing. 00:29:26.995 --> 00:29:28.615 Um, which was great. 00:29:28.825 --> 00:29:33.415 But those things, if I were to produce them myself. 00:29:33.775 --> 00:29:38.725 Is maybe four hours to write the blog and the social media stuff. 00:29:38.875 --> 00:29:41.095 I get those and I edit them in an hour. 00:29:41.695 --> 00:29:42.025 Uh, 00:29:42.076 --> 00:29:42.366 Joe Marion: Okay. 00:29:42.595 --> 00:29:46.735 Scott: is, this is productivity and it, it does some neat stuff, 00:29:46.765 --> 00:29:50.275 uh, to it that, that, that I think makes it better than if I had to 00:29:50.275 --> 00:29:52.585 self generate it entirely by myself. 00:29:52.765 --> 00:29:54.355 It's a different sort of view on it. 00:29:54.985 --> 00:29:58.255 We're, we're exploring things like we write, we write 00:29:58.255 --> 00:29:59.665 hundreds of proposals a year. 00:29:59.725 --> 00:30:01.525 We have a bank of proposals. 00:30:01.855 --> 00:30:02.455 Um. 00:30:02.985 --> 00:30:06.915 And I, I write many of these and these take, you know, a couple 00:30:06.915 --> 00:30:08.835 hours to write each one of these. 00:30:08.835 --> 00:30:12.435 All of a sudden, AI does a really nice job, uh, at, at that. 00:30:12.825 --> 00:30:18.105 So these are natural things that, that, that this does, uh, 00:30:18.435 --> 00:30:19.935 within it and, and the code. 00:30:20.325 --> 00:30:26.265 Uh, other things currently at, at Barry that we can say we currently do with ai. 00:30:28.761 --> 00:30:29.841 Nick Berry: I think we need to talk about writing. 00:30:30.296 --> 00:30:30.836 Um. 00:30:31.251 --> 00:30:31.671 Joe Marion: Mm-hmm. 00:30:32.696 --> 00:30:36.301 Nick Berry: You, you kind of, this could be considered one of our, 00:30:36.541 --> 00:30:40.381 you said our three pillars are like design, implementation, and software. 00:30:40.381 --> 00:30:44.521 I, think you could essentially add, you know, publication and research to that as 00:30:44.521 --> 00:30:45.961 something that we do a lot of at Barry. 00:30:46.741 --> 00:30:47.131 Uh, 00:30:48.230 --> 00:30:51.865 Scott: I, I'll push back on that a little bit though, and what AI does on that. 00:30:51.985 --> 00:30:54.175 'cause I a, a simple little thing where, 00:30:54.241 --> 00:30:55.256 Nick Berry: I think too, but. 00:30:55.360 --> 00:30:55.650 Scott: okay. 00:30:56.016 --> 00:30:58.146 Joe Marion: Yeah, I think we are gonna disagree, so this should be 00:30:58.375 --> 00:31:00.295 Scott: Oh, that, that makes for a good podcast. 00:31:00.355 --> 00:31:00.625 Uh, 00:31:00.731 --> 00:31:00.991 Nick Berry: Oh 00:31:01.075 --> 00:31:03.745 Scott: an interesting blog when it comes out and describes, 00:31:03.826 --> 00:31:08.461 Nick Berry: I, I might disagree with both of you, but I'm, think AI sucks 00:31:08.461 --> 00:31:09.961 at writing things from scratch. 00:31:10.621 --> 00:31:13.471 think it's obvious when it writes something. 00:31:13.501 --> 00:31:21.186 I, I, maybe this is but I feel like I can tell, and obviously the, the. 00:31:21.991 --> 00:31:24.511 Other side of this is that, think of all the times I've read something 00:31:24.511 --> 00:31:26.161 and haven't said, oh, that's ai. 00:31:26.161 --> 00:31:27.001 And when it actually has been. 00:31:27.001 --> 00:31:29.251 So obviously I don't know my error rate, but 00:31:29.436 --> 00:31:31.296 Joe Marion: Well, there, there's some science behind that. 00:31:31.296 --> 00:31:34.416 If you're a heavier user, you tend to know to be a better identifier. 00:31:34.651 --> 00:31:35.011 Nick Berry: okay. 00:31:35.136 --> 00:31:35.946 Joe Marion: That's a, true thing. 00:31:35.946 --> 00:31:38.586 That's a, that's a, that's a studied fact, but yeah. 00:31:39.421 --> 00:31:46.081 Nick Berry: think it's, it's vacuous and non precise when it writes, especially 00:31:46.081 --> 00:31:47.191 when it's writing from scratch. 00:31:48.286 --> 00:31:53.056 even, you know, I go on LinkedIn where this a, this podcast will be 00:31:53.056 --> 00:31:54.826 advertised on LinkedIn and I'll see it. 00:31:55.906 --> 00:31:58.606 single post on LinkedIn looks the same. 00:31:58.606 --> 00:31:59.566 They're all the same. 00:32:00.406 --> 00:32:04.186 it's like header, five bullet points with emojis footer. 00:32:04.216 --> 00:32:07.936 And it's like people just over and over and over write this. 00:32:08.086 --> 00:32:10.906 So I think it's bad at writing things from scratch. 00:32:10.936 --> 00:32:12.946 It uses similar patterns over and over. 00:32:12.946 --> 00:32:16.666 And when I notice that pattern, I immediately. 00:32:17.101 --> 00:32:22.446 Start discounting the ideas in the thing that I'm reading, So I, maybe I'm, I'm 00:32:22.451 --> 00:32:27.391 being a little too negative, but this is like kind of how I go, you know, I 00:32:27.391 --> 00:32:37.741 will not as much heed in what is written if I think I detect AI being the main 00:32:37.741 --> 00:32:39.091 author of the thing that I'm reading. 00:32:39.091 --> 00:32:39.661 I think it's 00:32:39.881 --> 00:32:41.681 Joe Marion: I bet a lot of people feel that way. 00:32:41.681 --> 00:32:44.231 I think, I think that's like a, I think that's a correct view. 00:32:44.381 --> 00:32:44.591 Yeah. 00:32:44.591 --> 00:32:48.041 Would wouldn't disagree that that's how people perceive that kind of writing. 00:32:48.161 --> 00:32:48.581 Right. 00:32:51.161 --> 00:32:52.121 I mean, I think so. 00:32:52.121 --> 00:32:52.541 One. 00:32:53.661 --> 00:32:56.781 You know, there are lots of kinds of writing that aren't, that 00:32:56.781 --> 00:32:59.571 aren't necessarily about getting lots of good character or an 00:32:59.571 --> 00:33:00.951 interesting structure into it. 00:33:00.951 --> 00:33:01.251 Right? 00:33:01.581 --> 00:33:04.041 There are lots of ki, there's a lot of, there's a lot of writing that's about 00:33:04.041 --> 00:33:08.241 just like sort of clearly presenting what you're describing, laying 00:33:08.241 --> 00:33:09.801 out the facts, that kind of thing. 00:33:10.251 --> 00:33:16.371 Um, I think, you know, the idea of voice is less important in that kind of writing. 00:33:17.151 --> 00:33:21.561 Um, but I don't, I don't think it's like a standalone thing that, that you 00:33:21.561 --> 00:33:23.181 should let write something by itself. 00:33:23.181 --> 00:33:23.511 Right. 00:33:23.931 --> 00:33:25.431 Um, I. 00:33:26.166 --> 00:33:29.106 I think a really good use case of it is like, you know, you always used 00:33:29.106 --> 00:33:32.586 to get that writing advice, like just write down your ideas as fast as you 00:33:32.586 --> 00:33:35.916 can and get a first sloppy draft out there and just like make that happen. 00:33:36.306 --> 00:33:40.716 And I think this is a much more pain-free way to, to get that first draft of 00:33:40.716 --> 00:33:42.276 a paragraph or something like that. 00:33:42.726 --> 00:33:44.136 It's like definitely not the final one. 00:33:44.196 --> 00:33:46.476 But you know, getting words on the page is a huge. 00:33:46.751 --> 00:33:49.931 A huge, uh, un blocker in a lot of situations. 00:33:50.620 --> 00:33:55.181 Um, and then I like that once that's been done, I can be very critical, right? 00:33:55.181 --> 00:33:58.781 It puts me in the mode of a reviewer when I'm reading that text. 00:33:59.561 --> 00:34:01.961 you need to be right, 'cause you need to be critical of it. 00:34:02.351 --> 00:34:05.171 But when you're seeing it, then you can think, okay, don't 00:34:05.171 --> 00:34:06.251 have any buy-in in this text. 00:34:06.251 --> 00:34:07.841 There's no sentences I love in here. 00:34:07.841 --> 00:34:10.991 Like, let's, let's really dissect this and pick it apart and make it 00:34:10.991 --> 00:34:12.521 into something that's worth doing. 00:34:13.271 --> 00:34:15.581 will say like a lot of it's like. 00:34:16.911 --> 00:34:24.321 It often like Um, it gets like the subtleties way wrong when it writes 00:34:25.011 --> 00:34:29.721 and, and so that it's a kind of like sloppiness or imprecision in, in the, 00:34:29.721 --> 00:34:30.981 in the way that it's communicating. 00:34:30.981 --> 00:34:34.131 And so that I pay a lot of attention to, if you just kind of breed through 00:34:34.131 --> 00:34:36.771 it, it's like, oh yeah, all this, those words sound kind of nice together. 00:34:36.771 --> 00:34:39.561 It like kinda, it gives you a warm fuzzy, but then when you think 00:34:39.561 --> 00:34:41.991 real deep about it, like, oh, maybe I don't like that at all. 00:34:42.381 --> 00:34:42.831 So. 00:34:43.206 --> 00:34:46.026 But I love, I love the ability to do that first drafting. 00:34:46.836 --> 00:34:50.586 then I love the ability later on to do tweaking little things like, Hey, I 00:34:50.586 --> 00:34:52.716 don't, I know this sentence is wrong. 00:34:52.716 --> 00:34:53.856 Here's what I don't like about it. 00:34:54.336 --> 00:34:55.776 Gimme five different options. 00:34:56.166 --> 00:34:58.866 And then you get a sentence and then you tweak that again. 00:34:58.866 --> 00:35:02.526 And then like, I don't, I like that iterative, you know, version of writing. 00:35:02.826 --> 00:35:05.751 I'm sure it comes real natural to someone who's written a very, very large amount. 00:35:06.821 --> 00:35:07.111 Yeah. 00:35:07.786 --> 00:35:07.876 Nick Berry: I 00:35:08.020 --> 00:35:08.560 Scott: I think, 00:35:08.656 --> 00:35:10.456 Nick Berry: editing and the, the fine tuning 00:35:10.585 --> 00:35:10.875 Scott: yeah. 00:35:11.086 --> 00:35:16.366 Nick Berry: the ability to, to ask it, to edit your text and then review things 00:35:16.366 --> 00:35:19.816 as it tries to make changes where it's not just now sending you back a huge 00:35:19.816 --> 00:35:23.596 paragraph with stuff change, like, you know, it highlights where it made 00:35:23.596 --> 00:35:25.546 a change and what's new and things 00:35:25.626 --> 00:35:26.406 Joe Marion: What's different 00:35:26.446 --> 00:35:27.406 Nick Berry: walk through. 00:35:28.276 --> 00:35:30.496 see what it's actually changing without having to 00:35:30.636 --> 00:35:33.631 Joe Marion: and it'll tell you why sometimes too, which is nice, right? 00:35:33.741 --> 00:35:34.031 Yeah. 00:35:34.186 --> 00:35:36.616 Nick Berry: I think it definitely, it, it's a good editor, uh, 00:35:36.616 --> 00:35:37.786 in a lot, a lot of cases. 00:35:39.565 --> 00:35:41.755 Scott: So I don't think we have the huge disagreement 00:35:41.755 --> 00:35:42.865 that we thought we would have. 00:35:42.886 --> 00:35:43.216 Nick Berry: Okay. 00:35:44.035 --> 00:35:44.755 Scott: I agree. 00:35:44.905 --> 00:35:45.775 Agree entirely. 00:35:46.015 --> 00:35:46.315 Yeah. 00:35:46.675 --> 00:35:46.735 Um. 00:35:46.806 --> 00:35:46.956 Joe Marion: What? 00:35:46.956 --> 00:35:47.016 are 00:35:47.305 --> 00:35:47.365 Scott: Yeah. 00:35:47.701 --> 00:35:47.901 Joe Marion: for? 00:35:48.011 --> 00:35:48.301 Yeah. 00:35:48.650 --> 00:35:49.975 Scott: Yeah, that's right. 00:35:50.455 --> 00:35:50.755 Right. 00:35:50.755 --> 00:35:51.445 That, that's right. 00:35:51.445 --> 00:35:56.455 Now, but the interesting thing in all of this is not to become, 00:35:57.025 --> 00:35:59.125 uh, you know, critical of ai. 00:35:59.125 --> 00:36:03.895 And, and somebody wrote me a, a thing where they largely created, they took 00:36:03.895 --> 00:36:08.695 25 references and said, write me a paper based on these 25 references. 00:36:09.145 --> 00:36:10.915 And I, I hated it. 00:36:11.325 --> 00:36:13.755 Uh, there, but the, the, there's nothing new. 00:36:13.755 --> 00:36:14.895 It wasn't written very well. 00:36:14.895 --> 00:36:17.745 I didn't, you know, it, it was, it was, but this is gonna get 00:36:17.745 --> 00:36:18.825 better and better and better. 00:36:19.155 --> 00:36:24.225 I, the, the idea that AI will never do this, I mean, imagine how good 00:36:24.225 --> 00:36:26.235 it's gotten in the last few years. 00:36:26.235 --> 00:36:29.025 So this is a current status of it, but I agree. 00:36:29.025 --> 00:36:32.835 It's, it, I, I don't like it, but it's an, it's an amazing. 00:36:33.655 --> 00:36:34.825 Editor tool. 00:36:34.825 --> 00:36:39.025 It's amazing thing that if you are actively working with it and you're 00:36:39.025 --> 00:36:42.085 writing it, getting suggestions, this is an incredible tool. 00:36:42.816 --> 00:36:46.176 Nick Berry: Yeah, a couple years ago, which I, I mean a couple years 00:36:46.176 --> 00:36:51.486 ago, like the very onset of chat GPT, when it became public, someone sent 00:36:51.486 --> 00:36:53.586 a, uh, email to Don Don Berry, our. 00:36:54.811 --> 00:37:00.061 Founder, um, been on the podcast a bunch of times saying, if you give me $5,000, 00:37:00.721 --> 00:37:03.241 I'll write a book, a biography about you. 00:37:04.321 --> 00:37:08.371 And Don asked for a snippet, said, okay, send me a snippet, 00:37:08.401 --> 00:37:09.181 or blah, blah, blah, blah. 00:37:09.391 --> 00:37:13.051 And it wrote Don's biography and they sent it to him in a lot of 00:37:13.051 --> 00:37:17.041 things and he read it apparently and said it got a lot of things right. 00:37:17.041 --> 00:37:20.101 So he replied and said, if you can make it do this, this, this, 00:37:20.101 --> 00:37:21.391 this and this, I'll pay you. 00:37:21.841 --> 00:37:27.001 And I think they became impossible for AI to do, um, especially at that time. 00:37:27.001 --> 00:37:29.461 But, so we never paid $5,000, but you know 00:37:30.385 --> 00:37:30.745 Scott: Yeah. 00:37:30.895 --> 00:37:31.135 Yeah. 00:37:31.135 --> 00:37:31.145 Yeah. 00:37:31.231 --> 00:37:31.591 Nick Berry: yeah. 00:37:31.891 --> 00:37:32.221 So. 00:37:32.440 --> 00:37:33.370 Scott: It's interesting. 00:37:33.370 --> 00:37:39.970 For example, could I dump my 60 episodes of, in the interim into AI and say, 00:37:39.970 --> 00:37:42.730 write me a book on in the interim, or, 00:37:42.751 --> 00:37:43.591 Nick Berry: at that point, but yeah. 00:37:43.930 --> 00:37:47.920 Scott: yeah, Or, or, you know, what would Scott say? 00:37:48.591 --> 00:37:49.326 Joe Marion: out there for sure. 00:37:49.330 --> 00:37:51.940 Scott: What would Scott say to the following? 00:37:51.970 --> 00:37:53.530 Does it know who I am now? 00:37:53.530 --> 00:37:56.890 Obviously it, it's gonna do really well at knowing, you know, what would 00:37:56.890 --> 00:37:58.570 Trump say, what would Obama say? 00:37:58.570 --> 00:37:59.320 What would others say? 00:37:59.350 --> 00:38:01.510 'cause there's such a wealth of stuff out there. 00:38:01.840 --> 00:38:02.950 Uh, but that's interesting. 00:38:03.010 --> 00:38:07.210 But let's, let's move to something that, that concerns me as a company. 00:38:07.210 --> 00:38:07.480 So. 00:38:07.920 --> 00:38:10.710 Where when we're working with a client, we really need to 00:38:10.710 --> 00:38:12.120 worry about confidentiality. 00:38:12.120 --> 00:38:14.370 So what does that have to do with ai? 00:38:14.880 --> 00:38:22.170 Uh, if we use ai, we need to make sure that the confidential information that 00:38:22.170 --> 00:38:27.960 it's using to write something doesn't then go outside and become part of, 00:38:27.960 --> 00:38:30.995 its its wide world that it's using. 00:38:31.860 --> 00:38:35.550 To write other stuff where all of a sudden it's using that for somebody else. 00:38:35.880 --> 00:38:38.190 So security's a huge deal. 00:38:38.460 --> 00:38:42.120 Um, what, what, what do we know about security? 00:38:45.481 --> 00:38:46.141 Nick Berry: Do you want this, 00:38:46.410 --> 00:38:46.800 Scott: come on. 00:38:47.191 --> 00:38:47.371 Nick Berry: So 00:38:47.610 --> 00:38:48.030 Scott: Okay. 00:38:48.241 --> 00:38:48.301 Nick Berry: the 00:38:48.531 --> 00:38:49.161 Joe Marion: I'll, start. 00:38:49.161 --> 00:38:49.821 Yeah. 00:38:49.860 --> 00:38:51.030 Scott: You've looked into this. 00:38:51.256 --> 00:38:51.546 Nick Berry: Yeah. 00:38:53.331 --> 00:38:55.191 Joe Marion: So I mean, we know some things, right? 00:38:55.191 --> 00:38:59.121 So there are things that are, that are, that they say that, you know, 00:38:59.121 --> 00:39:02.991 for example, our data is encrypted in transit and it rest, that means 00:39:02.991 --> 00:39:06.111 that there's not really a point in the process when it's human readable. 00:39:06.321 --> 00:39:08.511 So you have to be able to decrypt it to read the data. 00:39:09.321 --> 00:39:11.061 Um, they talk about, 00:39:13.776 --> 00:39:16.866 Uh, they, they say, you know, the, uh, business versions of the paid 00:39:16.866 --> 00:39:20.136 versions of these softwares say that they won't train on your data. 00:39:20.586 --> 00:39:26.286 We learned yesterday that that is, has at least some known caveats around it. 00:39:27.666 --> 00:39:30.666 uh, in particular, if you give them feedback, did you like or not like 00:39:30.666 --> 00:39:33.696 a response, well then that actually gives them permission to look at that. 00:39:33.696 --> 00:39:36.456 So, you know, that's kind of concerning, you know. 00:39:37.296 --> 00:39:41.526 It's a little bit harder to parse the implications of that statement, right? 00:39:41.526 --> 00:39:44.316 Like it's, it doesn't, won't train on your data. 00:39:44.316 --> 00:39:47.376 I mean, what, what does that mean exactly? 00:39:47.376 --> 00:39:49.026 And how would a lawyer understand that? 00:39:49.026 --> 00:39:49.866 It, it's harder. 00:39:49.866 --> 00:39:50.586 I don't know. 00:39:51.326 --> 00:39:56.586 I, think in some sense I do think about this in some ways at least. 00:39:56.646 --> 00:40:00.996 Um, and there are absolutely exceptions to, there are absolutely reasons why, 00:40:00.996 --> 00:40:04.686 this is not a good analogy, um, which we can go into, but I think in some ways. 00:40:05.301 --> 00:40:09.981 It's a little bit analogous to email, right, in that you have your 00:40:09.981 --> 00:40:14.871 confidential information, uh, it's stored in someone else's, uh, it's 00:40:14.871 --> 00:40:16.521 stored in someone else's servers. 00:40:16.521 --> 00:40:17.841 It's not local. 00:40:18.261 --> 00:40:20.421 It is potentially, it's encrypted. 00:40:20.481 --> 00:40:23.721 Um, but you know, you're kind of relying on their guarantees that 00:40:23.721 --> 00:40:26.931 they're treating that data in a way that's fair and reasonable. 00:40:27.051 --> 00:40:27.381 I don't know. 00:40:27.381 --> 00:40:28.491 Nick, what do you, what do you think? 00:40:28.776 --> 00:40:31.666 Nick Berry: Yeah, I mean, I think I use that as a mental model in the same way. 00:40:31.666 --> 00:40:37.726 And, uh, there are things that we don't email right at Barry. 00:40:37.876 --> 00:40:42.646 Certainly, you know, uh, level data doesn't go through email. 00:40:42.646 --> 00:40:47.056 Interim analysis reports for trials that are still unblinded, that 00:40:47.056 --> 00:40:48.296 are still blinded to the public. 00:40:49.051 --> 00:40:51.631 Don't go through email and we have different like SFTP 00:40:51.631 --> 00:40:53.101 protocols that we use for those. 00:40:54.961 --> 00:41:00.661 there's, I mean, we are not AI policy lawyers. 00:41:01.051 --> 00:41:04.651 There are AI policy lawyers looking into these things, but that's not us. 00:41:04.741 --> 00:41:10.051 Um, I think in because of that, we. 00:41:10.921 --> 00:41:14.251 Have taken a sort of conservative approach at a lot of things. 00:41:14.251 --> 00:41:17.851 And even though we have this like business plan that says we're not 00:41:17.851 --> 00:41:20.911 gonna train on your data, like, that doesn't give us free reign to upload 00:41:20.911 --> 00:41:22.381 everything and to put everything in there. 00:41:22.381 --> 00:41:27.181 So we, we are tasked with, and we don't have an SOP yet, uh, we don't have this 00:41:27.601 --> 00:41:33.601 like put together, but Joe and I and the committee that we're on are tasked with 00:41:33.601 --> 00:41:39.541 trying to come up with, uh, strategies for this and, um, you know, they're obvious. 00:41:40.531 --> 00:41:42.061 Red line, don't do this. 00:41:42.061 --> 00:41:44.881 And there are gray areas, and we don't have all the answers 00:41:45.091 --> 00:41:47.461 to this, but, uh, conservatism 00:41:47.481 --> 00:41:47.721 Joe Marion: Yeah. 00:41:47.931 --> 00:41:50.691 and I think people generally don't have the answers. 00:41:50.691 --> 00:41:51.081 Right. 00:41:51.141 --> 00:41:53.451 And, but yeah, I agree with you. 00:41:53.451 --> 00:41:56.361 We're we're trying to take a conservative, kind of measured approach. 00:41:56.461 --> 00:41:56.671 Nick Berry: so 00:41:56.811 --> 00:41:56.871 Joe Marion: Yeah, 00:41:57.241 --> 00:41:58.171 Nick Berry: data blind. 00:41:59.281 --> 00:42:04.231 Uh, documents that you upload, um, you know, you can give the structure of a 00:42:04.231 --> 00:42:09.631 data frame without needing to give the data set to, to AI and things like that. 00:42:10.021 --> 00:42:15.001 And it's, it's hard because each, we're not all, like Barry is not 00:42:15.001 --> 00:42:16.951 a homogenous group of AI users. 00:42:16.951 --> 00:42:20.046 Every person feels differently, thinks differently, uses it differently, 00:42:20.281 --> 00:42:25.831 and at the end of the day, we don't get to, to watch people. 00:42:26.746 --> 00:42:27.946 Interact with ai. 00:42:27.946 --> 00:42:31.756 And so we kind of have to have some trust in the people at Barry, and we 00:42:31.756 --> 00:42:35.086 tried to do this at our meeting, which is sad, is we have to instill some 00:42:35.086 --> 00:42:39.376 sort of working knowledge about AI in the entire company because everyone's 00:42:39.376 --> 00:42:44.596 gonna be using it and we want everyone that's using it to be at least informed 00:42:44.596 --> 00:42:48.376 in some, in some regards about how they should and shouldn't interact with it. 00:42:48.376 --> 00:42:48.556 And so. 00:42:49.336 --> 00:42:54.946 Um, it's, it's hard, but we laid down some like obvious do not dos, 00:42:54.946 --> 00:42:58.246 and I feel like that helps a lot, uh, to, to, to clean up some of 00:42:58.270 --> 00:42:58.450 Scott: Yep. 00:42:58.696 --> 00:42:58.936 Nick Berry: But 00:42:59.680 --> 00:42:59.920 Scott: Yep. 00:42:59.975 --> 00:43:00.265 Okay. 00:43:00.376 --> 00:43:00.706 Nick Berry: areas. 00:43:01.825 --> 00:43:02.425 Scott: Oh, okay. 00:43:02.755 --> 00:43:10.225 Now another one that bothers me and is AI that's generating a statistical model. 00:43:12.040 --> 00:43:16.450 And interestingly, if you ask AI to create you a piece of code, you have 00:43:16.450 --> 00:43:18.520 the code and that that's the product. 00:43:18.520 --> 00:43:20.140 And you can, you can do that. 00:43:20.830 --> 00:43:24.850 Uh, unlearn, for example, talks about generating digital twins, and it's, 00:43:25.150 --> 00:43:26.650 they don't tell you what it does. 00:43:26.650 --> 00:43:27.015 It's ai. 00:43:27.995 --> 00:43:29.615 It's a black box. 00:43:29.735 --> 00:43:32.075 And by the way, it's also a hype thing. 00:43:32.525 --> 00:43:35.555 It can't be wrong because it's AI sort of thing. 00:43:35.555 --> 00:43:41.915 So I bristle at this a little bit because behind this, if AI helped 00:43:41.915 --> 00:43:45.875 you create a model, there's a model and we could talk about the model 00:43:45.875 --> 00:43:47.405 and the data that goes into it. 00:43:47.405 --> 00:43:50.945 So I really worry this becomes both a hype thing. 00:43:51.535 --> 00:43:55.645 A black box thing and takes away from the value of us as 00:43:55.645 --> 00:43:57.595 statisticians, as model builders. 00:43:57.955 --> 00:44:03.295 So I don't think at Barry we've ever used AI to create a model. 00:44:04.721 --> 00:44:07.831 Nick Berry: To fit a model to get results from a model, like, 00:44:07.945 --> 00:44:11.695 Scott: Yeah, I'm, I'm not even sure what that means nowadays, but it is certainly. 00:44:11.821 --> 00:44:14.611 Nick Berry: density of the the treatment effect parameter. 00:44:15.175 --> 00:44:19.045 Scott: Or predict what this patient is gonna be like in six months. 00:44:19.075 --> 00:44:22.075 Don't tell me how you did it, but, you know, predict it kind of thing. 00:44:22.686 --> 00:44:25.506 Joe Marion: because I've definitely done, here's the model I want right. 00:44:25.506 --> 00:44:26.256 In Stan. 00:44:26.256 --> 00:44:27.396 I mean that, that's no issue, 00:44:27.505 --> 00:44:28.020 Scott: yeah, yeah. 00:44:28.116 --> 00:44:29.076 Joe Marion: not what you're talking about. 00:44:29.076 --> 00:44:29.376 Right? 00:44:29.436 --> 00:44:29.676 Yeah. 00:44:29.815 --> 00:44:30.355 Scott: no, no. 00:44:30.565 --> 00:44:30.925 Uh, 00:44:31.066 --> 00:44:31.186 Nick Berry: would 00:44:31.345 --> 00:44:32.005 Scott: not that so, 00:44:32.146 --> 00:44:34.756 Nick Berry: model, fit your stand model, make you a density plot, 00:44:34.756 --> 00:44:36.046 and do all that for you now, 00:44:37.135 --> 00:44:38.965 Scott: so I, I really worry about that. 00:44:39.066 --> 00:44:39.396 Joe Marion: to 00:44:39.595 --> 00:44:40.165 Scott: Yeah. 00:44:41.181 --> 00:44:41.941 Joe Marion: separate experiment. 00:44:41.971 --> 00:44:42.261 Yeah. 00:44:42.505 --> 00:44:46.555 Scott: Though, if I ever want, if I wanted to sell Berry, I think if I changed its 00:44:46.555 --> 00:44:52.225 name to Berry ai, much like what is this shoe company, Allbirds or something? 00:44:52.225 --> 00:44:52.945 Who did this? 00:44:52.956 --> 00:44:54.096 Joe Marion: uh, birds. 00:44:54.355 --> 00:44:54.685 Scott: Yep. 00:44:55.570 --> 00:44:55.990 Yeah. 00:44:56.376 --> 00:44:56.886 Joe Marion: AI is, 00:44:57.040 --> 00:45:00.520 Scott: somehow this hype all of a sudden Barry's worth five times what 00:45:00.520 --> 00:45:02.260 it is, you know, sort of as a thing. 00:45:02.530 --> 00:45:04.270 So I, I, I bristle at that. 00:45:04.420 --> 00:45:08.920 We talked a little bit about, about, about generative ai and 00:45:08.920 --> 00:45:10.360 we haven't jumped into that. 00:45:10.360 --> 00:45:16.570 I think a real, let, let's sort of finish this a little bit with the 00:45:16.570 --> 00:45:19.450 question of will AI take our jobs? 00:45:21.250 --> 00:45:21.940 Within the Senate? 00:45:22.030 --> 00:45:23.320 It's a, it's a fair question. 00:45:23.320 --> 00:45:26.110 I think certain people in the statistical community, it's a, 00:45:26.110 --> 00:45:27.940 it's, it's, it's a, it's a worry. 00:45:28.480 --> 00:45:29.710 And it's, it's interesting. 00:45:29.710 --> 00:45:35.410 I, it, the, the people whose jobs are threatened is more than the people 00:45:35.410 --> 00:45:37.600 who think their jobs are threatened. 00:45:37.990 --> 00:45:39.460 Oh, AI can't do what I do. 00:45:39.820 --> 00:45:42.430 Uh, you know, eventually AI is gonna do what we all do. 00:45:42.430 --> 00:45:47.020 But you know, how long is it, do you feel like AI threatens our jobs? 00:45:51.171 --> 00:45:53.631 Joe Marion: I mean, I, I hope it threatens part of my job. 00:45:53.631 --> 00:45:58.161 I, I hope that pretty soon we're using AI just like fully 00:45:58.161 --> 00:45:59.271 for validation of code, right? 00:46:00.531 --> 00:46:03.981 I think we would do a lot more validating, which I think would be valuable for us. 00:46:03.981 --> 00:46:06.381 And I would personally be doing less code validation. 00:46:06.381 --> 00:46:09.741 And if it took that part of my job, I wouldn't shed any tears. 00:46:10.131 --> 00:46:10.581 Right. 00:46:11.451 --> 00:46:14.361 Um, is it gonna take my job overall? 00:46:16.191 --> 00:46:16.701 I don't know. 00:46:16.761 --> 00:46:18.831 I think it's quite hard to say where things are going. 00:46:19.251 --> 00:46:20.931 The robots are building themselves right now. 00:46:20.931 --> 00:46:21.261 Right. 00:46:21.261 --> 00:46:25.131 And hard to know like what the end state of that is. 00:46:25.401 --> 00:46:26.421 Um, yeah. 00:46:27.291 --> 00:46:28.101 Nick, what do you think? 00:46:28.101 --> 00:46:28.161 Um. 00:46:29.446 --> 00:46:32.056 Nick Berry: Yeah, I mean, I do more software development than, 00:46:32.056 --> 00:46:34.996 than both of you, so that puts my job immediately at more risk. 00:46:34.996 --> 00:46:36.316 I think, um, 00:46:36.501 --> 00:46:36.791 Joe Marion: Yeah, 00:46:38.416 --> 00:46:38.686 Nick Berry: you, 00:46:40.726 --> 00:46:43.846 Anthropic, just you, you're talking about the hype of ai. 00:46:43.846 --> 00:46:46.976 Anthropic just released that their new model mythos 00:46:47.196 --> 00:46:48.976 can't be released to the public 00:46:49.466 --> 00:46:54.256 because it's too dangerous because it found zero day security issues 00:46:54.286 --> 00:46:56.386 in every major operating system. 00:46:56.626 --> 00:46:59.206 I, it's probably hype, like right? 00:46:59.206 --> 00:47:02.446 If you want to advertise your new model called Mythos, you say it's 00:47:02.446 --> 00:47:04.576 too dangerous to be released and 00:47:04.836 --> 00:47:06.306 Joe Marion: Oh, absolutely. 00:47:06.786 --> 00:47:07.921 Just like fax Cloud. 00:47:08.861 --> 00:47:09.641 Too dangerous. 00:47:11.296 --> 00:47:14.866 Nick Berry: gotta think of a better name with Greek origins than Fax Cloud. 00:47:15.256 --> 00:47:21.286 We, we did talk about, uh, I can't remember something, some cloud 00:47:21.286 --> 00:47:25.216 name, uh, Cirrus or something like that as our fax cloud or something. 00:47:25.216 --> 00:47:25.846 That probably works, 00:47:25.881 --> 00:47:28.131 Joe Marion: What's the word for storm cloud in Greek? 00:47:28.131 --> 00:47:28.926 That's what we want. 00:47:30.306 --> 00:47:30.526 Uh. 00:47:30.755 --> 00:47:31.045 Scott: Yeah, 00:47:31.546 --> 00:47:31.846 Nick Berry: yeah. 00:47:32.235 --> 00:47:34.015 Scott: it's, yeah. 00:47:35.026 --> 00:47:36.796 Nick Berry: So my job might, at least my. 00:47:38.056 --> 00:47:39.046 Pure programming. 00:47:39.046 --> 00:47:42.316 Part of my job is probably more at risk, but like I talked about earlier, 00:47:42.520 --> 00:47:42.760 Scott: Yeah. 00:47:43.216 --> 00:47:46.066 Nick Berry: the part of my job that I'm retaining completely is 00:47:46.066 --> 00:47:48.976 the decision making part and the planning part and the vision part. 00:47:48.976 --> 00:47:55.816 And it's hard for me to see with the, you know, generation 00:47:55.816 --> 00:47:57.496 of models that being at risk. 00:47:57.736 --> 00:48:06.241 Uh, what happen eventually is that my decision making turns into more of a. 00:48:07.336 --> 00:48:07.786 Clicking. 00:48:07.786 --> 00:48:08.626 Yeah, that's good. 00:48:08.626 --> 00:48:09.676 Yeah, that's good. 00:48:09.826 --> 00:48:14.536 And it's iteratively making a bunch of stuff or you know, it makes 10 decisions 00:48:14.536 --> 00:48:18.046 and I choose the good ones and it builds itself, or something like that. 00:48:18.526 --> 00:48:22.636 So, I mean, obviously we're doing, we're stupidly conjecturing about something 00:48:22.636 --> 00:48:26.776 that we could can't possibly know and aren't that informed about, but. 00:48:27.871 --> 00:48:28.321 I don't know. 00:48:28.321 --> 00:48:33.271 I feel like the decision making that we have is safe and the, the planning 00:48:33.271 --> 00:48:37.141 and telling the story in the right way, which is something we do so well, 00:48:37.141 --> 00:48:40.141 and so and so purposefully is safe. 00:48:40.231 --> 00:48:46.261 But you know, a lot of the, you know, the technical things, eventually it's gonna 00:48:46.261 --> 00:48:48.791 be able to do the technical things that I said it's not good at doing right now. 00:48:50.016 --> 00:48:52.416 Joe Marion: The, the parts of this job that are about communication and 00:48:52.416 --> 00:48:55.236 especially relationship building, I don't think those will go away. 00:48:55.295 --> 00:48:55.585 Scott: Okay. 00:48:55.896 --> 00:48:58.776 Joe Marion: Like, they're never gonna send Claude to go talk to the 00:48:58.776 --> 00:49:00.276 FDA, they're gonna send Scott Berry. 00:49:00.276 --> 00:49:00.606 Right. 00:49:00.636 --> 00:49:04.176 Like, you know, that's, the AI is just not gonna have the same weight. 00:49:04.296 --> 00:49:04.686 Right. 00:49:04.986 --> 00:49:06.936 Same credibility, like, yeah. 00:49:07.505 --> 00:49:12.115 Scott: The, the whole strategy part that even people are using it to design trials. 00:49:12.115 --> 00:49:15.715 You could go into to one of these and say, design me a trial. 00:49:16.390 --> 00:49:19.720 But your it, it's gonna do what you tell it to do. 00:49:19.750 --> 00:49:21.700 Three arms, this and all that. 00:49:21.700 --> 00:49:24.760 Not asking the questions, is this the right thing in your development? 00:49:24.760 --> 00:49:26.140 What are your uncertainties? 00:49:26.410 --> 00:49:29.470 Asking the right questions to understand why are you doing the trial? 00:49:29.470 --> 00:49:35.080 This whole part before you get into, give me the calculations of the 00:49:35.080 --> 00:49:38.050 following are the huge part to this. 00:49:38.050 --> 00:49:40.150 And, uh, AI can't do that yet. 00:49:40.240 --> 00:49:44.170 Maybe eventually it's, it's doing those parts, but it's not there yet. 00:49:47.395 --> 00:49:48.265 Okay. 00:49:49.945 --> 00:49:57.295 The, um, uh, I, you know, I, I, I harken that to, um, I was, I went in for. 00:49:58.045 --> 00:50:03.775 Dizziness a couple days ago, and I saw a real clinician and she asked me a 00:50:03.775 --> 00:50:09.235 couple questions and she knew immediately what it was, uh, positional, vertigo. 00:50:09.625 --> 00:50:13.345 And not only that, she said, do this Epley maneuver. 00:50:13.975 --> 00:50:17.365 And it was a physical thing of moving your head into position 00:50:17.365 --> 00:50:19.555 for three minutes and it was gone. 00:50:19.836 --> 00:50:20.126 Nick Berry: Okay. 00:50:20.275 --> 00:50:21.295 Scott: It was a cure. 00:50:21.865 --> 00:50:25.645 So, you know, maybe AI's gonna get there and it could ask me the right 00:50:25.645 --> 00:50:27.385 questions and did the right thing, but. 00:50:27.531 --> 00:50:28.521 Joe Marion: That's crazy. 00:50:28.675 --> 00:50:32.065 Scott: Uh, I, I did, and, and your mom spent more of the time 00:50:32.065 --> 00:50:33.475 asking what was wrong with me. 00:50:33.475 --> 00:50:37.135 I think she, she asks Claude that quite a bit. 00:50:38.245 --> 00:50:40.195 What is wrong with my husband? 00:50:40.516 --> 00:50:41.116 Joe Marion: with my husband? 00:50:41.215 --> 00:50:41.995 Scott: Yes. 00:50:42.055 --> 00:50:42.475 Yes. 00:50:42.856 --> 00:50:43.356 Joe Marion: Oh man. 00:50:43.555 --> 00:50:44.155 Scott: All right. 00:50:44.155 --> 00:50:45.475 We will end it there. 00:50:45.685 --> 00:50:51.055 We appreciate you all joining us for our 60th episode, and until next 00:50:51.055 --> 00:50:53.485 time, we'll be here in the interim.