[00:00:14] Chris Freeland: We've been warned to expect artificial intelligence to arrive as a kind of robot apocalypse, a Skynet style break from human control. But in everyday life, AI looks a lot less like an all-knowing overlord and a lot more like a word processor on steroids. Is artificial intelligence really racing toward a intelligent break from human control, Or are we misunderstanding what kind of technology it actually is?

Hi everyone. I'm Chris Freeland. I'm a librarian at the Internet archive. I wanna welcome you to today's Book Talk in their paper, AI as Normal Technology, author Sayash Kapoor and Arvind Narayanan articulate a vision of artificial intelligence as normal technology. Not the utopian or dystopian takes we've seen in popular culture, but plain old, boring, normal technology.

Sayash is joining us today and will be in conversation with Legal scholar Kevin Frazier. Now here to set the stage for today's discussion is Dave Hansen, the Executive Director of Authors Alliance. 

[00:01:14] Dave Hansen: Thanks Chris. I'm particularly excited about the subject that we have today and our speakers, and so I was just so excited that Sayash was willing to do this talk with us because I've been a big fan of his work, not just AI as Normal Technology, but also his book that you may have also heard of. He's a co-author of AI Snake Oil, and one of the reasons why I really like it is there's just so much grifting it seems like out there, and hard to get a clear read on kind of what's really happening and what seems likely to happen in the coming months.

And years and decades, even with respect to this technology. So with that, I will introduce our speaker today and our moderator. So first I'll introduce our moderator. Kevin Frazier is the AI Innovation and Law Fellow at UT Austin School of Law. Where he studies how to design regulatory ecosystems that accelerate AI adoption and diffusion.

He's been published in numerous scholarly outlets such as the Tennessee Law Review and also places that you may read, like the MIT Tech Review. He's really active on a lot of fronts. If you Google him, you'll probably find some recent testimony that he's given before Congress about AI and state regulation.

So we're just so excited to have him. Today as a moderator, he holds a law degree from uc, Berkeley. So Kevin will be moderating today and in conversation with Sayash Kapoor. Sayash is a computer science PhD candidate at Princeton University Center for Information Technology Policy. As mentioned, he is also co-author of AI Snake Oil, and his research focuses on the societal impact of ai.

He's a recipient of numerous awards, for example, from a CM and other organization. And was included on time's inaugural list of 100 most influential people in AI. So Sayash and Kevin, all over to you. 

[00:03:15] kevin Frazier: Well, Dave, thank you so much for the opportunity. And Chris, thanks also to you and your team at the internet archive for making this possible.

Sayash, it's always a pleasure to get to talk with you. 

[00:03:26] Sayash Kapoor: Likewise, it's great to be here. Thank you so much for organizing this, and thank you, Kevin for moderating. 

[00:03:31] kevin Frazier: Sure thing. So walking and talking about AI is kind of like going into a Baskin Robbins and saying, I'd like ice cream, right? We've got 31 flavors.

Some of them are good, some of them are awful. Unless you're a kid and have a sugar addiction. And when we talk about AI, there's a temptation to use that umbrella term and assume that we are talking about one thing. And yet, in your work, AI Snake Oil, and then in your subsequent work, AI as Normal Technology, you and your co-author do a wonderful job of being a little bit more specific. So I just wanna start there and say, when we're talking about ai. What are some of the major categories, or for lack of a better phrase, flavors of AI that we should be aware of and more specific about? 

[00:04:18] Sayash Kapoor: That's a great place to start, and I think Dave mentioned earlier that there's a lot of grifting that happens in the AI community.

This is one major reason for that. So there are some kinds of ai, notably and most recently generative ai, where we have indeed made a lot of technical advances over the last decade. We've gone from models that could barely recognize what's in an image. To models that sometimes even outperform experts at tagging what's in an image.

We've gone from language models that couldn't classify whether a piece of text was humorous or was being satirical to language models that can now output this like long texts based on any specific guardrails or criteria you give them. And so generative AI is one type of AI where we have indeed made progress.

In contrast to generative AI though, is predictive ai. This is the type of AI system where its developer claim that you can use it to predict certain things about in, in many cases about individuals. For example, you started seeing the use of predictive AI in hiring app applications. So people when they apply for jobs, most often now are faced with an algorithm, with an AI system that decides whether or not that candidacy should proceed to the next stage.

As far as we can tell, it's here in the setting of predictive AI where there's a lot of snake oil. So this is what we mean by the term ai snake oil is developers claiming that AI tools work well when oftentimes they don't have any evidence of putting it, and in fact, the underlying task might even be impossible for an AI system to solve correctly.

Most often this involves making predictions about the future. And especially making predictions about individuals and when these sort of AI tools are applied to consequential decisions in healthcare or finance or hiring, I think the consequences can be dramatic. 

[00:06:02] kevin Frazier: And what really stands out to me about that great divide that you set forth is there's a difference between understanding what a technology does in of itself, and then kind of societal expectations of that technology.

I'm a lawyer by training, and so everyone in the lawyer lawyerly community has this one case in mind where early in 2023, a lawyer basically relied entirely on AI to generate a brief. Unsurprisingly, that didn't turn out very well. And when the judge asked, what the heck were you doing using this AI tool to just send it and hope that no one noticed this was entirely generated by AI.

And the lawyer to his credit said, Hey, I had seen all these headlines, that AI was fantastic, that we had experienced all these gains. And so what role does. Expectations and kind of our general media environment play in how we use and develop and deploy AI tools. 

[00:07:02] Sayash Kapoor: I think it definitely plays a very big role, and especially in the early sort of days of LLMs language models like Chat, GBT, I think expectations were really high in part because developers were claiming that their systems could do all sorts of things.

For example, when OpenAI released the GPT four model in early 2023. The, I would think one of the first lines of the technical report said that GPT four can now pass the bar exam at the 90th per as well as a 90th percentile test takeup. And you know, this sounds really impressive until you realize that what a lawyer does is not just answer bar exam questions all day.

And so the systems can simultaneously do very well at statistical pattern matching where you have this type of task that does not represent the day-to-day activity of a liar. That does not automatically mean that this competence translates onto the downstream tasks that a lawyer might actually need to do in the real world.

That said, to some extent, I'm optimistic that the fact that these tools are deployed so broadly, the fact that hundreds of millions of users are now using language models every day, I think that also means that people will encounter these edge cases much more frequently. I often hear from people that they're very impressed when they ask ChatGPT or an agent to do something outside their expertise.

They're not so impressed when they ask you to do something within their expertise because that's when you can really see where the gaps are. Now, that isn't to say that these tools aren't getting more impressive, they really are. Software engineering is something that I used to spend a lot of my time doing, and I would say at this point, the bulk of the work I do is not really in writing code anymore.

Now, of course, software engineering does not just consist of writing code. You need to think of the design principles, what the users want, and so on, and all of that remains. But the part of my work where I used to actually have to sit down in front of a computer and type code is basically completely like gone.

I, I don't spend any of my time writing code anymore. 

[00:08:54] kevin Frazier: So much more time to just fill the world with more insightful essays. I can't wait to see what you go, right. So this point is so important for understanding that ai, like any other tool, is only useful when deployed in the right circumstances. I like to tell my students here at UT Austin that if your friend said, Hey, I'm gonna go take my John Deere lawnmower and drive to Houston.

You would say, Hey, you know, I, I'm not sure that's a wise decision you might wanna recalibrate when and how you're going to use that lawnmower. And in the same way we've heard all of these ideas about what AI can do in isolation, when really it's always going to be based on that context. And so when you came out with this essay, AI is normal technology.

It really sent shockwaves across the AI community. And more generally is, whoa, we're not talking about something that is. In entirely new or entirely unanticipated based off of prior releases of technology. Can you explain why this concept is so important and perhaps has proven to be so compelling to so many diverse audiences?

[00:10:04] Sayash Kapoor: And I think the best way to do that is by looking at a previous technology that was really transformative. One of my favorite examples is the invention of the dynamo. So the dynamo was invented in around the 1880s, and you know, at the time people expected sort of this rapid impact across industries, but it wasn't until 40 years later that we had the first electrified factories.

Now why was there such a big gap? So I think the fact that we've developed this new general purpose technologies as economists like to call it, doesn't automatically mean that we know how to apply it. Well, and basically in the case of factories, factory owners through the process of trial and error, realized that you essentially needed to reorganize the entire factory.

Around the process of electrification in order to get the benefits. And this was a process that we, where you first had to develop new machines that could be electrified, then adapt those machines to existing factories, and you figure out that doesn't work and reorganize factories, and that's what the process of diffusion can sometimes look like.

So you have this lag time, not of like months or years, but oftentimes of decades between the invention of a new general purpose technology and its true impacts across different industries. So we took this concept of general purpose technologies and you know, the same had been observed for computers, for the internet, for all sorts of other general purpose technologies.

And because we think that AI is yet another example of a general purpose technology, we sort of saw where this framework would take us if we try to look at AI's impacts. And indeed, I think this framework does help explain a lot of what we've seen so far, and I think offers a useful guide as to how we should think about AI's impacts in the future.

So the first stage of this is definitely the invention of a new technology. Let's say in this case large language models, but these models are not automatically useful to end users. As so many lawyers and coders and so on have found you actually need to build applications on top of these models. For example, in the last few months, many people might have heard of Claude Code having its moment where lots of people who are not software engineers are now figuring out, and they can now write code using language models.

The capabilities to sort of do this have been around for three years. But the thing that has changed is now we have an application that uses this underlying language model in a way that appeals to the user, that the user can understand and communicate with and so on. But even here, the process doesn't stop here because even just taking the example of software engineering, I think the real gains in software engineering will not come at the level of individuals adopting, let's say, cloud code or other programming tools here and there.

They'll come when we sort of. Drastically change our practices around software engineering. We sort of need to reimagine what peer review or code review looks like, how we sort of check in code into million dollar, million dollar in million dollar companies and like millions of lines of code repositories in a way that prevents these catastrophic security failures.

And I think once that happens, we will actually start to see a lot of the benefits of AI and software engineering. And then you can think about. This process playing out in every single industry across many different applications. That's what it'll take to sort of realize AI's impact across these industries.

And I think that's the key process that's often overlooked, like the AI industry is all too happy to say that we have built a new model and that's what'll lead to sort of these widespread impacts. They oftentimes ignore the other stages of the process where you actually need to deploy these tools in order to get the real benefits.

[00:13:23] kevin Frazier: I love that because you help put in context the fact that when we talk about AI adoption, the common metric is monthly average users, and we just assume that anyone using AI. Is using it towards productive ends or correctly, right. When in actuality, if you ask my dad, for example, if he was using ai, he would say yes, and then he would tell you, I use it to check college football scores.

Is that, you know, a societally transformative use of ai? No. And so I think. The need to get to a deeper analysis of AI adoption, understanding both the intensity of your usage as well as the novelty of that usage, right? How are you using it to transform underlying processes is really what we have to get at.

But we're not necessarily measuring that as accurately as we could be. So when we talk about AI as normal technology, just to put this all in context. How would you contrast that with some of the narratives that AI is, for lack of better phrase, abnormal technology, and why is that perhaps not a productive way of seeing and framing ai?

[00:14:33] Sayash Kapoor: So when we wrote this essay, we had a sort of unconventional. Goal. We basically wanted to convey a worldview about ai. Not just talk about our predictions for what will happen or our prescriptions for what should happen, but really just talk about how one can think of AI in this new way, and in some sense in a very old way, like I think economists have been thinking of AI as a general purpose technology for a long time.

But what had happened was the popular conversation around AI had shifted so much. Into thinking of AI as potentially a new kind of being or a super intelligence or something that might sort of take over humanity. But we felt that even this sort of common sense notion in some sense was being lost and needed to be defended quite rigorously.

So that was our goal when we set out to write this paper, it wasn't to sort of break new ground. It was merely to say that, look, we've been here before and here's how we've dealt with in the past. And so what that meant also was we can look to these past examples of technologies. To see how we were able to shape them, to see where we went wrong, even in some cases, and to learn from those failures to figure out what good policymaking might look like.

And I think if you think of AI as a potential super intelligence, the sorts of policies you might come up with to regulate it would look very, very different. You might think of something like non-proliferation, which is what we used with nuclear weapons. Where you sort of constrict the resources you might want, who you might not want your adversaries to get your hands on.

You might think of policies that would sort of centralize power instead of decentralizing it so that rogue entities don't get access to ai. I think all of these analogies, like the nuclear weapons analogy and the centralization tendency would both be counterproductive if we think of AI as normal technology.

Because if we think of it as another general purpose technology, and I very much think current trends point to it being that way. Then the centralization of power and of the ability to build powerful AI systems is precisely one of the main risks. And so there's where I think there's like a folk in the road where if you adopt a certain worldview, some policies that turn out to be beneficial from that worldview might actually be really counterproductive from another.

[00:16:38] kevin Frazier: And with this sense of thinking through AI as normal technology, you flagged, even if that were the view that everyone took and regarded it as normal technology. We would still see lags in adoption. Now, framing this as yet another manifestation of a general purpose technology, what are those reasons for a sort of slow integration of this new technology into yesteryear systems?

What were the patterns we've seen that have lead to those inflection points in which suddenly that GPT general purpose technology becomes ubiquitous and becomes the thing that underpins new businesses and new institutions? 

[00:17:19] Sayash Kapoor: So I think there are a few like specific bottlenecks that have led to past GPD slowing down, and that I think will continue to lead to AI adoption slowing down.

One of them is that, especially in safety critical industries, the tolerance for errors is really, really low. What we often have is called the five nines of reliability. You need something that's 99.999% reliable before you actually deploy it in, let's say the aviation industry or the nuclear industry, and to get to that level of reliability.

We need to do a lot more application specific work. I think one of the ways in which current language models lack the most is reliability. There are these inherently stochastic systems that might offer random outputs at certain points, and so while it's like fairly straightforward to get them to like a 90% accuracy, let's say, on a number of tasks going, that last mile from 90 to 99.999% has turned out to be very challenging.

So I think in safety critical industries, this reliability bottleneck will continue to work. There are also other industries where safety is not that much of a bott link. And software engineering, for better or for worse, is one of these industries where, you know, software engineers are really enthusiastic about adopting new frameworks they've done.

So even when we are not talking about ai, it's one of the industries where software engineers spend a lot of time basically figuring out ways to automate their own job. And I think that's part of the reason why we are seeing a lot of the impacts of AI happen very quickly in software engineering. That said, even for these industries where you don't have sort of safety related bottlenecks, even here, there are other bottlenecks, for instance, related to how organizations function.

So let's say we take the example of cloud code. Now we have engineers at a company who are writing a hundred times more code than the phone. That's great, but who's reviewing all of this code? And are there enough product managers at the company to figure out what the user needs are Like in other words, like as soon as we get to a point where writing code.

Is basically free. We realized that there were all of these other bottlenecks in the system. Things like figuring out user demand, figuring out what user interfaces are testing and verification that now act as the bottlenecks on everything else that is now sped up using ai. And so this turns out to be a much more like difficult bottleneck to solve because now you need to rehaul your entire organization.

Maybe you need 10 times as many product managers as you had before because writing code is no longer a bottleneck. That is something that takes a lot of time. It takes experimentation to figure out how to make these deeper organizational changes, and I think that's the key bottleneck that sort of hinders or slows down diffusion.

It also gives society a lot of time to adapt. So because of these bottlenecks in diffusion, people have sort of talked about AI related job losses. I think these job losses will occur if they do on a scale of decades and not fears. This was similarly the case with the internet as well. We did see many sort of specific professions where there were concentrated job losses, but it took years or decades, for example, for Blockbuster to go bankrupt as a result of streaming.

And I think similar sort of patterns might hold for AI related shocks as well. 

[00:20:23] kevin Frazier: There's a reason that we have a thing called Amaras Law. The idea that we overestimate the short-term ramifications of technology and underestimate their long-term ramifications. Because if you talk to a farmer in the year 1900, when about 40% of all Americans were farmers are otherwise engaged in agricultural production, and then fast forward to the turn of the century and only 2% of Americans are doing so well.

Yeah, that's night and day. But to your point, that was quite. Gradual. And I think it's also fun to see, for example, so many organizations have all of these press releases of, we've adopted this new tool, we've done this new thing, and then you check in with them six months later and you ask, has anything changed?

No. Right, but it's often because they're not investing those critical resources. Economists besides yourself often say that there's a sort of J curve to getting the maximum benefit of a new technology, which is to say for a short period of time, you have to be spending quite a bit. You're gonna be losing some money doing that process of retooling employees, changing systems, doing all of that expensive.

Task to get ready for this new technology before you get to that other side of the J curve. But that's often not covered and some people go as far as to say that it's a nine to one ratio of you're gonna need $9 of expenditure for every $1 on that new technology like ai. Is that something you would subscribe to or buy into?

Saying that we really do have to kind of spend it before we see any real productivity or changes as a result of ai. 

[00:22:01] Sayash Kapoor: I mean, I might even go one step further to say that the spending here, or like the expenditure of capital might not just need to be at the level of individual firms, but actually at the level of entire institutions.

And so maybe to take legal services as one example, I'm writing something with Harvard law student just incur and Arvin Orion on what would happen if we get access to advanced legal AI systems? How would it change legal services? Would it make them cheaper? By default, I think our answer should be an emphatic no, that's because, let's say we do have access to, say, an advanced language model that can answer all legal queries, does not hallucinate and so on.

Even so, using this language model to actually answer questions might run into unauthorized practice of law regulations, for example, and we've seen tools like LegalZoom, which are far simpler. They basically just give people very simple instructions on how to fill out forms. They've also been gone after they've faced multiple lawsuits, arguing that they're basically conducting an authorized practice of law.

So that's one place to begin, like the institution itself needs to figure out how to adapt in the face of this new technology, without which I think these benefits won't come to the fore. My favorite example is when we sort of came up with e-discovery tools. That is tools that could be used to fetch old cases, figure out relevant past legal cases, or figure out what specific documents need to be surfaced in discovery.

This actually increased the cost of discoveries because basically in this litigation process, both the parties are adversarial. And what that meant was these parties would try to put these costs of discovery onto the other ones often by sending them a truckload full of printed documents. So I think as long as we have this adversarial legal system.

As long as we have regulations that prevent technology from being used to lower the cost of legal services, even the promise of really advanced AI would not come to the fore. And so as you can imagine, this would be a really costly process as well. We would need to update state regulation. Perhaps we would need to come up with a system to make the legal arbitration process or like litigation, much less adversarial.

And this would impose heavy costs upfront in trying to modify the institution itself. But I think those costs are eventually what will lead us to a place where AI can have this transformative impact. 

[00:24:15] kevin Frazier: So I'm gonna push you a little bit, right? You came out with this paper in April of 2025, which in the age of AI is akin to saying that you published it in the Jurassic Period, right?

It's been a minute since this paper came out, and I wonder if the advances we're seen in AI have caused you to perhaps say, huh, maybe AI is less. Of a normal technology than we previously anticipated. And just to play devil's advocate for a second here, we saw, for example, the publication of Anthropics Claude Constitution.

And in this constitution, anthropic, one of the leading labs described in many ways. A sort of development of a personality for Claude and raise the possibility that perhaps these models have some degree of welfare that we need to look out for in terms of maybe they, which in and of itself is a anthropomorphization of ai.

Maybe they don't want to do. Certain kinds of tests and so on and so forth. So have you seen advances that have nudged you at all or challenged your priors with respect to how we should regard this technology? 

[00:25:29] Sayash Kapoor: I think it's interesting to have written this essay and then like also, uh, collaborated with people who have very different worldviews.

So six months after we wrote the essay, we wrote something together with the authors of AI 2027, where we sort of got together and said, Hey, like what are our predictions for the next few years? How do we think AI will go? And we actually realized that for the next, let's say three to five years, there were many things we actually agreed on.

Including certain policy prescriptions that we think are good, regardless of which of the two worlds we live in. So for readers unfamiliar AI 2027 was this essay that argues that AI would indeed sort of turn out to be a potential super intelligence and we should act now as if it were in particular, to restrict the development of certain types of AI systems and to have much more transparency.

You know, on the latter part of these policy goals, things like transparency, I think we emphatically agree. I think whether or not AI is normal, technology has no bearing on the question that AI companies should be transparent about what practices they take and how they're training the models and so on.

That said, I think even when we wrote the essay in April, 2025, we did expect AI capabilities to sort of continue improving at a very fast pace. And I think the key difference in our view versus. Let's say people who think AI is a potential super intelligence is that we don't think the technology alone determines its fate or its impact on the world.

So in other word, we reject technological determinism, which sort of says that. Once we have AI of a certain capability that AI systems would be able to, let's say, take over the world or carry out certain harmful tasks, we completely reject that. And the distinction we make is that between capabilities, which is AI systems being able to do certain impressive things versus power, which is the ability to actually impact the world, the ability to take actions on your environment.

And I think so long as we sort of have this distinction in mind. So long as we avoid giving AI systems power over critical processes. I think the sort of risks of ai, for example, as told by people who think of it as a super intelligence, are naturally going to be limited. And in fact, there I think there will be like a natural feedback loop where people who do deploy risky AI systems will face massive pushback and as a result of simply result of market forces would be required to invest more in safety.

We've seen this happen in the case of self-driving cars. Where Waymo is basically the market leader right now because of its focus on safety and other competitors who deprioritize safety have basically fallen out of the race. And I expect this to continue happening over the next few years with AI as well.

So of course it remains to be seen how it plays out. 

[00:28:11] kevin Frazier: Just to make sure we're all on the same page. One of the key concerns that folks have kind of on the AI as abnormal technology front would be, Hey, these systems may become so sophisticated that we may see the diffusion and development of bio weapons at a scale we haven't seen previously.

Why isn't that something we should be staying up at night worrying about in addition to the myriad other things that keep us up at night? 

[00:28:35] Sayash Kapoor: That's a great question. So. This was specifically a concern that was raised quite heavily in the last few months of the Biden administration. So the Biden administration came out with an executive order that talked about the risks of releasing large language models openly models like Metas Lama or Deep C Car one, precisely because they were concerned that people might, let's say, use them for cybersecurity attacks or for developing bio weapons.

And so that was the point at which we did some research to figure out. What are the real bottlenecks for, let's say, a terrorist to develop the next bio weapon? And of course, knowledge acquisition is one of those bottlenecks. So a language model could tell you, for instance, how to prepare a bio weapon. It could give you some precise instructions, but we quickly found out that very similar instructions were available in, let's say, AP biology textbooks or on Wikipedia.

And you know, someone willing to go the last mile would be able to find other public sources for acquiring this knowledge, even if it might take them a little longer. The reason though that we weren't so concerned, at least at the time with the bio weapon potential of AI, was that it's harder, much harder to acquire the resources required to actually build that weapon.

It's much harder to acquire, let's say, a tabletop, tabletop, DNA synthesis tool. It's much harder to acquire the materials that are necessary to create a bio weapon and. That is a place where we should be enacting screening and enforcement regardless of whether AI is involved because that is something that, you know, a malicious actor could have access to regardless of whether they have access to advancing Act.

and I think I was happy to see that the executive order from the Biden Harris administration did in fact uplevel the screening requirements on certain tools and certain materials needed for the development of bio weapons as well. So I think that is a much better place to emphasize. Our defenses as opposed to emphasizing the impact of AI itself.

If anything, I think AI plays a small, marginal role, but like all of the other steps of the process are where we have much stronger capabilities already in place to both monitor who has access to these materials as well as moderate and sort of drill down on when we find anomalous actions. 

[00:30:40] kevin Frazier: And from a public policy standpoint, I think this is such a crucial distinction because.

From my own 2 cents would be that oftentimes we frame what are long stemming foundational issues that I regard as human problems, as AI problems, and it's easier to point the finger at this new boogeyman rather than addressing some of the more foundational systemic issues. For example, in a mental, mental health context.

There's no denying that AI companions. Can and have led to tragic, egregious outcomes. But I also wanna make sure from a societal level, we'll, we're still asking, well, why kid? Why don't kids have access to more extracurricular activities? Why do they feel so lonely to begin with, and why don't we have greater access to mental health professionals?

And address AI companions, right? It's taking a more holistic approach. That's really important here, rather than just pointing the finger at ai, running with that headline and forgetting all of the issues we were worried about in October of 2022, right before we got chat, GBT 3.5. So I'm a law professor. I love a good hypothetical.

Let's imagine I check my phone right now. There's a new tweet that SES Kaur has been named, the AI Czar of America. Congratulations. Or sorry, depending on how you're feeling today, what would be among the first three policy ideas you would want to see implemented? 

[00:32:10] Sayash Kapoor: Oof. If I 

[00:32:10] kevin Frazier: have, you can just pick one. You can pick one.

It doesn't have to be your priority. We'll, I'll understand that I put you on the spot, so no worries. 

[00:32:18] Sayash Kapoor: I mean like really what I think is. The sort of challenge here is figuring out how AI interacts with other domains. So in particular, I think as you mentioned, we shouldn't lose sight of the fact that AI is now interacting in very complex ways with existing problems, these wicked problems that have faced institutions for years.

In some sense, it's an opportunity because we now have like a legitimate reason to think about how we can restructure institutional practices that have been running for a long time. Part of my response would definitely be to figure out how AI impacts different domains. How can we sort of make legal services more efficient for people?

How can we make the scientific process more efficient so that scientists don't just focus on publishing more papers but actually incentivize for progress? So I think that's one part of the equation, which is like promoting beneficial uptake. I think the other part of it is like figuring out how to address risks there as well.

I think of course, AI itself has certain risks that are. Best addressed by developers being more transparent or perhaps safe harbors for third parties who are looking into the risks of AI because they're currently doing so under like the risk of legal threats, right? Like a lot of researchers, including myself when they're testing AI systems and making claims about how well they do or don't work.

Constantly sort of under the potential legal risk of companies coming after us for it's A-D-M-C-A violations. So I think that would be on the AI company sort of side particularly. And then finally, I think looking at the risks of AI in different domains would also require domain specific interventions.

For instance, we briefly talked about the mental health challenges of using AI systems. I think there's a lot that AI companies can do to address this type of risk. And I think if we do adopt the worldview or perspective of AI as being a normal technology, then this is precisely the sort of harm that becomes, comes at the forefront, which is a product safety issue.

Right. So your products as. People are interacting with these systems to have certain sort of safety requirements, and you have certain obligations to fulfill those requirements as users are using them. But this might mean, for instance, the changes that Open AI did in response to all of the tragic incidents of teenagers committing suicides.

After talking to the chatbots where they have parental controls, now they have certain restrictions, the chat bot is trained in a different way. I think we can do a lot more across different types of risks that AI systems have. 

[00:34:36] kevin Frazier: All right. Well, I think you earned the job, so congratulations when we flip the script and go instead to a kind of personal analysis of your own AI use.

I know when I talk to folks, there's often concerns around. Cognitive de-skilling, or to be a little bit more blunt, becoming dumb. As a result of over reliance on ai, there are concerns about sharing private information. There are concerns about losing certain skill sets. So how do you use AI yourself in your day-to-day capacity, both in a professional way and any personal uses you'd feel comfortable sharing with the rest of us?

[00:35:13] Sayash Kapoor: Absolutely. I mean, I think the concerns around cognitive de-skilling and so on are definitely warranted. That said, I think of it as being very, I don't know, similar to the case of people switching from, let's say individual contributors to becoming managers, right? Like to some extent, as you take on some management responsibilities, you also lose the skills that you might have earlier had to rely on.

So in some sense I think it's a choice more than it is, like something that's forced on you. And particularly as I mentioned earlier in writing code, I think I haven't lit written more than like a few lines of code each day or like the last two years at the very least. I don't think this is something that has like de-skilled me, so to say.

I mean in this, in some sense, the history of programming is one where programmers continue to move up the layers of abstraction. So we move from assembly language, to like. Object oriented programming to write using frameworks and to now using AI assisted code. And I see this as some sort of like progression that's happening here.

One thing I'm much more productive of though is like writing and using AI for developing thoughts. I find the theory of writing as thinking really. Where, you know, the first draft that you come up with is almost a struggle because you are in the process of forming this new sort of causal model of the world.

You're trying to come up with a new theory or new explanation for how the world works. And if you outsource that process to ai, then a byproduct of that is you're not developing these new mental models of the world yourself. So that's something that I try to like basically completely avoid. I have found AI systems very useful for translating what I've written.

For, let's say a different audience. So I often write the same type of content for both my academic peers as well as for policy makers. And going between a policy draft and an academic version of the same text once you've developed the ideas, is something that's made much simpler using AI systems, especially if you want like just a first draft that gives you a feel for how the final thing would look.

and then finally, in terms of more personal users, I mean, one thing that I think is underappreciated right now is just the rise of single purpose software. You can now develop applications that you might use like twice in your entire life, and that is something that, you know, I rely on all the time.

Just the other day I realized that you can't clean a MacBook without sort of having a trigger on again, like. Basically Mac does not allow you to shut it down. If as soon as you press a key, the Mac will sort of open up again. And so I developed an app that just allowed me to close the Mac until I pressed a certain key combination that took me like all of five minutes.

So that's the sort of thing that has now become like a very normal sort of routine. For me, routine task for me is developing applications to solve singular pain points where previously it might not have been worth the time. 

[00:37:57] kevin Frazier: I wanna get your 2 cents on one final inquiry here, which would be, you are one smart dude.

You've got so many different paper topics going on, and you're perhaps diving into now legal services. You mentioned. What else is on your research agenda? What other questions are you really excited to pour into? 

[00:38:18] Sayash Kapoor: I am really excited these days about the scope for institutional reform. Like I think that's a question that is.

Ignored to some extent. I think there's like a big question mark between developing better AI and having it have a transformative impact on the world. And I think that that will be answered by how we figure out how to reshape our institutions. So legal services is just one example. Um, but I'm thinking about how we should think about scientific research and many other institutions that would benefit very greatly from minor improvements that could sort of enable AI's transformative impacts within those institutions.

[00:38:53] kevin Frazier: This is such a exciting moment where we see a lot of questions about how do we redesign entire institutions. I know here at Texas Law we're asking how do we redesign legal education, for example, where there's a lot of folks on both ends of the spectrum of some professors would prefer AI play, no part.

In their class. Other folks are saying, Hey, students, take the wheel. Good luck. Send me your prompts, but otherwise go forth and prosper. And so as we all kind of acclimate to this new world, it does certainly seem like we're right for institutional change and a new transition period. And I think the pressure is on all of us to.

Find out how do we move through a new transition phase in a way that improves upon prior generations in which perhaps we've left folks behind, we've left regions behind, and we haven't done as good of a job of saying this is a technology that can improve the general welfare and advance some of our core values.

[00:39:50] Sayash Kapoor: Absolutely. I couldn't have said it better. No. 

[00:39:52] kevin Frazier: Well, thank you. I'm sure you could have, or at least AI could have. Dave, I'm keen to hear any questions that you would like to tee up for all of us. 

[00:40:02] Dave Hanson: Hi. Yes. Thank you so much. This has been fantastic. You know, one place that I thought would be interesting to start is about some of the bigger existential risks.

You know, in the paper, one of the things that we hear all the time, right, is like about catastrophic misalignment scenarios, Skynet and Terminator and that kind of stuff. And in the paper you talk about, or you make the point that catastrophic misalignment is a speculative risk, at least at this point.

Can you talk a little bit more about why you think that's so? 

[00:40:31] Sayash Kapoor: Absolutely. So I mean, what we mean here by speculative is that we actually have no way of collecting evidence to prove or disprove a certain thing. So for instance, for certain types of like risks of AI systems, we have some causal mechanism that we can hypothesize through which this type of risk would.

For the catastrophic misalignment risks that we often hear about where an AI system, for example, acts against its interest and the surrounding ecosystem is unable to contain it despite like the security measures we take, we actually have no data. And like if we do it well, we should have no data at any point.

That tells us how we can even estimate this risk. I appreciate the work that some organizations are doing to try to sort of get this on more scientific grounds. For example, an organization called Apollo, does research where they put AI systems and make believe world and see if they try to break out.

But I think this type of evaluation is still in its infancy and the systems that we are developing, the authors would agree, are sort of contrived. They don't respond to how we develop these systems in the real world. And so when it comes to like the real world misalignment risks. The risk that real world systems wouldn't have the security measures in place for AI systems to be catastrophically misaligned.

I think at least as of yet, it remains a speculative risk. 

[00:41:48] Dave Hansen: So this is actually for both of you. I'd love your thoughts on this. The question about an anthropomorphism, Kevin, you brought up the flawed constitution, which I was actually kind of surprised at how much that document sort of humanized ai, and so what's likely to happen with that?

I would layer onto that. How do you see that phenomenon affecting sort of public perception of the technology and also policymakers perception of the technology? 

[00:42:15] Kevin Frazier: Yeah, thanks for the question, Dave. I'll be sure to answer in probably like 75 minutes if that's okay. No, I'm just kidding. But it is a very, very complex and really fascinating question.

I'd say from a public policy standpoint, the. Anthropomorphization of AI to me is somewhat a self-fulfilling prophecy that I take issue with because we have a number of grave concerns about individuals establishing personal relationships, for example, with ai well. If we have leading labs referring to these models as people or giving them person like attributes and referring to them.

In that sense, that's only going to perpetuate the idea that these are indeed some sort of person or person like thing for which you would establish relationships. So I do think that. It is a matter of culture and a matter of norms and a matter of public conversation to say these are just like any other tool they can be, for example, a diagnostic tool that helps a therapist learn what your needs may be, or a diagnostic tool that helps a educator learn how you learn or study differently.

That, to me, is a much more positive framing that make sure we're emphasizing, as sesh pointed out on a number of occasions. But I don't wanna put words in his mouth. We do have agency in these conversations. I have a big concern that we're oftentimes talking about AI as if it's something that's happening to us.

For which we can't do anything about it. And there are very real tactics being used by AI companies without naming names that try to lure users to spend more time on those applications or to otherwise invest in them in perhaps deleterious ways. And those can take advantage of folks in particular who may not know how AI works or how these companies are operating.

But that to me is an education issue and a literacy issue. So I would very much like to see a greater emphasis on the fact that when, for instance, you survey parents and say, do you feel comfortable having what I refer to as the new talk with your child? Talking to them about how and when they should use ai.

Most parents will say, no, I have no clue what I would say. Well, that's a public policy issue, right? That's information we can provide to parents to make sure they can have that new talk with their kids. So that's a long way of saying I'm not wildly optimistic or in favor of that sort of language when it comes to Claude's Constitution as a public policy document.

I think it's a admirable step in acknowledging the company's values and beliefs about how this company may develop and how this model may develop and be trained over time. That sort of transparency, again, to S's point is really valuable from a pulp. Public policy position to be able to stand, understand what's being baked into the model, how it may react in certain circumstances.

I think that's all very favorable. The one thing I'll flag is just I'm not a huge fan of them using the word constitution. They do address this in the Constitution, but when you throw a word like constitution at a problem, it gives it a sense of. Legitimacy or a sense of democratic participation in formatting these values, when in actuality this was a document predominantly written by one person from one perspective, in one very idiosyncratic part of the country.

And I've lived in San Francisco. I love it, but it's not representative of the US and I can say that from here in Austin. 

[00:45:50] Sayash Kapoor: That was great. Yeah. Thank you Kevin. I, I agree with all of that. Maybe one thing I'll add is like, from a company's perspective. Of course there is the sort of risk of izing their tools and layering in customers and so on.

But like from a more pragmatic perspective, I think it's basically just a strategy that makes the model work better, and that's also why they're using it. So I think they're only just beginning to understand like the science of AI language models is catching up to the point where we understand why these things happens.

But at a high level, what we've seen is if you sort of treat AI systems. Say trade and scare codes here. If you treat AI systems as if they're responsible or they're sort of good citizens, et cetera, if you sort of prompt them with text that says, so they're more likely to behave as if they are responsible citizen or like not take harmful actions, et cetera.

And I think that is one pragmatic reason why people are optimistic about this paradigm of personality training. But I think as you said, I completely agree that we should acknowledge it as like this tool that we are shaping using certain technical methods. It lives somewhere in a data center and it is an FM thing that runs on graphics processing units, and I think that's something that we definitely should have parents talking to their kids about.

It shouldn't be thinking that talking to like an imaginary friend or something. It really is like a physical process that lives somewhere and runs on like a data center in Virginia almost probably. 

[00:47:09] Dave Hansen: So you're mentioning a data center in Virginia raises another set of questions around the environment and I think with the framing that we think that AI will develop or progress as normal technology that may be so, but I think lots of people are looking around.

And seeing the effects of that technology in a very not normal way in large data centers being put in. Some companies, you know, talking about just massive, massive power requirements, building their own nuclear reactors and crazy things, and how much of that is hype and how much of that is real. You know, it kind of pervades this conversation, but I wonder if you could talk a little bit about the environmental impact and how that kind of relates to the overall framing and perception of the technology.

[00:47:52] Sayash Kapoor: Absolutely. So, I mean, I'm by no means an expert on this, but I have read some work by people who are thinking very deeply about this, and my overall perspective is that on a sort of global level, on like the level of entire countries or entire populations or states, the impact of AI on energy requirements is actually.

Like not huge. It is still like a sizable percentage. I think data centers overall consumed around 4% of US electricity. A few years ago it was projected to go up to six to 7% by 2026, I think, but I'm not sure on the exact number. But like on this large scheme of things, the reason we are sort of building so many more data centers and power demands is because companies do expect AI systems to be very useful to a large number of people.

And I think that part of it is maybe less objectable. The thing that is objectionable though is that local communities where these data centers are built often have very little say in what the terms of these data centers are. And I think this was sort of most blatantly obvious in X AI building its data centers where they used natural gas rather than other more sustainable ways of powering these data centers.

That was like a big menace too. Many of the people who lived around these data centers, they had their like quality of life severely degrade. And I think this is what prompts a lot of concern about the environmental impact as well, is like, what if it happens in our backyard and what if we have no say over it?

So in some sense, I think it's less of an AI problem and more of a problem of democratic participation. I think if companies were to sort of agree to better terms. For building these data centers to agree to, let's say, building solar farms or to figuring out alternative ways or getting more community input.

I think this would be far less objectionable than in the specific way that some companies in particular, XAI and a couple of others, have decided to use these like very environmentally recreating ways of setting up these data centers only because they want to do it on a tight timeline. 

[00:49:48] Kevin Frazier: I'll just very briefly say ditto, and this is one of those instances in which I hope we take a holistic analysis of not only AI but all of the other contributors to climate change.

I have relatives in Arizona who tell me, oh my gosh, they're building a new data center just north of Phoenix, and on your way there, you drive past 15 golf courses. Let's just make sure we're taking everything in perspective. 

[00:50:15] Dave Hansen: Well, thank you both. This has been so interesting. This is such a great conversation.

I'm gonna hand it over to Chris to wrap us up. 

[00:50:22] Chris Freeland: Thanks, Dave. Yeah, I wanna echo my thanks to both of you, Kevin and Sayash. What a fantastic conversation. Thanks all. Have a great day.

Thanks for joining us on this journey into the future of knowledge. Be sure to follow the show. New episodes, drop every other Wednesday with bold ideas, fresh insights, and the voices shaping tomorrow.