Subscribe
Share
Share
Embed
Minnesota Law Now is a new podcast from the University of Minnesota Law School that brings together some of today’s most insightful legal scholars in conversation about law, policy, current events, and the ideas that shape our world.
Welcome to Minnesota Law Now,
the podcast from
the University of Minnesota Law School.
I'm William McGeveran,
the Dean
and William S. Pattee Professor of Law.
Artificial intelligence
is rapidly changing
how we work, make decisions,
conduct research,
and understand the world around us.
As AI becomes more powerful,
we face important questions
about what happens
to human judgment, expertise,
responsibility, and accountability
when we rely on machines
to help us think and decide.
Today, I'm joined
by three of my brilliant colleagues
at the forefront
of interdisciplinary scholarship
exploring AI
and its intersections
with legal reasoning, neuroscience,
medicine, and ethics:
Professor Daniel Schwarcz,
Professor Susan Wolf,
and Professor Francis Shen.
Daniel Schwarcz
is the Fredrikson & Byron Professor of Law
and a distinguished
university teaching professor.
He is a leading scholar
of insurance, financial regulation,
consumer protection,
and artificial intelligence,
with recent research
examining how AI is changing
legal practice and legal education.
Susan Wolf
is Regents Professor,
McKnight Presidential Professor
of Law, Medicine, and Public Policy,
Faegre Drinker Professor of Law,
and a professor of medicine.
She chairs the university's Consortium
on Law and Values
in Health, Environment,
and the Life Sciences.
Susan is a leading
interdisciplinary scholar and bioethicist
whose work examines
the legal and ethical questions
raised by advances
in medicine, science, and technology.
Francis Shen is Professor of Law,
Associate Dean
for Research and Innovation,
and Solly Robbins
Distinguished Research Fellow.
A pioneer in law and neuroscience,
his research explores the intersection
of neuroscience, AI, and the legal system,
including
ethical, legal, and social implications
of those emerging technologies.
Together, they offer
three distinct
but complementary perspectives
on a rapidly changing landscape of AI,
and an opportunity for us to step back
from some of the excitement and the hype
and the anxiety surrounding AI
and ask some of the bigger questions.
How should we think
about human judgment
in an age of increasingly capable
artificial intelligence?
What responsibilities
should we delegate to machines,
and what should remain distinctly human?
How do we ensure
that innovation is accompanied
by appropriate ethical safeguards
and public accountability?
Let's begin with the law,
and I'll turn to you, Dan.
You've been studying
how AI is changing
legal practice and legal reasoning.
Where do you see
AI making the biggest changes
in what lawyers do every day?
Well, I really think
because it's a general-purpose technology,
it really is affecting
pretty much every element of law.
I think that there's not one area
where it's better or worse
in terms of its capacity to affect change.
I do think
that what lawyers are increasingly finding
is that targeted use of AI
can make them better lawyers
and can make them
more effective and efficient
at what they're doing.
I think we're seeing that
across pretty much every practice area.
How does that change the lawyer's work
when they start to delegate to AI
things that they would have previously
either done themselves
or delegated
to another colleague of theirs?
Well, I think at the end of the day,
the most important principle
that I tell lawyers,
and I think many lawyers internalize,
is that when you're using AI,
the most important thing is
you have to be confident
about the quality of the output.
If you think about what lawyers do,
when senior lawyers are doing work,
a lot of times
what they're doing is making a judgment.
"This is good, this is bad.
This is good in this way,
but bad in this way.
This is missing this assumption."
Increasingly,
I think what we're finding
is that mid-level and senior lawyers
are actually the ones
who are best equipped to use AI.
Then the real question becomes,
what is the role for the junior lawyer?
We need to train the junior lawyers
to be senior lawyers
and have that judgment to use AI well.
At the same time,
if AI can do a lot
of what the junior lawyers are doing,
there are some risks.
I think
different firms and different lawyers
are navigating those challenges
differently.
I think that's the real tension
that we're trying to work through
as a profession
because AI is increasingly capable
of doing the work
that many junior lawyers do
in a number of different domains.
Francis, I want to pick up
on that boundary
between human judgment
and artificial intelligence.
What does neuroscience tell us
about how humans actually make decisions,
and why does that matter
for our discussion today?
Great question.
I'm glad we're having this discussion.
I remember back in 2017,
I taught our first law and AI class
here in the building.
I remember asking the students,
"Do you think anybody in the building
has any idea what's coming down the pike?"
They all said no, but now we do.
Neuroscience teaches us that
everything we do,
every decision we've made,
every emotion we've had,
every behavior we've done,
is facilitated
in a pretty straightforward,
but still complex way.
We take information in
from our eyes or our hands or our ears,
from the rest of our body.
Our brain processes that.
It produces some outcome.
We hear music, and we feel good.
We see a friend, and we hug them.
We read a legal brief,
and we figure out
what to write in response.
Artificial intelligence,
we know the intelligence parts.
That's the processing information.
The artificial means it's not being done
through our biological systems.
It's some silicon system.
I think the big question is,
for me, one of them,
whether we care
that some of this decision-making
is now being done outside the human.
The answer is that often we don't,
and we haven't for decades.
We start our class for an entire day
asking, what is artificial intelligence?
There's great debate.
What about a calculator and spell check?
Those are things
that take in information and do something,
and we don't do it anymore.
Most people are just fine with that
because it produces better outcomes,
like Dan was saying,
because we know what it's doing,
and because
it frees us up to do other stuff.
I think the big take-home point
from neuroscience is,
this is just
another type of information processing,
increasingly sophisticated,
and raising a lot of questions,
but there are a lot of parallels.
-We'll maybe get to some of those.
-Sure.
One thing that people talk about, though,
is what happens with the interaction.
How might using AI tools change the way
that humans are thinking
and making decisions?
Yes, through a lot of really cool ways.
One of the ways right now,
almost all of the artificial intelligence
is external.
We think about going to a terminal
and typing something in
in a large language model
like ChatGPT, or Gemini
gives us something back,
and then we're taking that in
and then reprocessing it,
like Dan was saying.
Okay,
here's my information I got from ChatGPT.
Now, let me figure out
what I'm going to do
based on my experience and training.
-Bill, can I ask you to fill in the blank?
-Yes.
We all live in a yellow--?
Submarine.
You could have picked any word
in the entire English language.
Pre-Beatles,
you wouldn't have known what to pick.
Why did you do that?
Because I gave you an input, you heard it,
and critically,
you were pre-trained on your life,
and then you filled in the blank.
You predicted,
"What does Francis want you to say?"
That's what
those generative models are doing.
They're trained on information,
and we've got to really think carefully
about the information that goes in,
and it comes back out.
The last thing I'll say is that
where neuroscience is going
is towards implantables.
We're already at wearables,
people in their ears, on their watches,
measuring, and then giving things back.
If we could combine
the power of information processing
Silicon Valley will give us
with our biological processing,
that's really the next frontier.
By the way, University of Minnesota,
across the bank, is leading that work.
Yes, they sure are.
Susan, you pay attention
to this boundary
between the body and the external inputs,
both on the clinical side,
in medicine and patient care,
and also
in biomedical or scientific research.
Where do you see
the greatest potential benefit
of having these tools
that Francis and Dan were describing
available to the human reasoning
that doctors and researchers have?
Well, I think we have to distinguish
different kinds of AI.
AI is this gigantic category.
Some AI integrated in a pedestrian way
to do calculation,
as Francis was suggesting,
may be non-problematic.
When you get into AI
that is doing fancier things,
that is really designing research,
generating synthetic data.
-I'm getting to some pressure points here.
-Yes, you are.
Analyzing enormous datasets,
affecting decisions
that physicians and other clinicians make
in the care of their patients.
Now, you're getting
into much more sensitive territory.
On the research side,
there are a million ways
in which AI is being used.
The 2024 Nobel Prize in Chemistry
went to AlphaFold,
which was this astonishing use of AI
to do something
that had really baffled human minds,
which is: how do you go
from a string of amino acids
to those really complicated
3D protein-folded designs?
That's what amino acids do.
They construct proteins.
It took the capability of AI
to really crack that problem.
More problematically, though, in research,
you get to the use of synthetic data,
which I mentioned,
where you're no longer using real data.
There's a spectrum of syntheticness,
but you have extrapolated
from actual measured data
to artificial data.
Why would researchers do that?
Well, they might say
we'll have fewer privacy problems.
We're no longer using real data,
but there still are privacy issues
if the synthetic data are based
on real people's information,
and you have enough of it.
That's all in the realm of research,
where you also have issues like:
when is the use of even research
as opposed to quality improvement?
When do you need consent
from the research participants?
That's all on the research side.
On the healthcare side,
it's really interesting.
I just, about a week ago,
went to the doctor.
The doctor said to me, "Would you mind--"
She pulled out her phone.
"I'm just going to use
basically an AI scribe."
-I've had this happen, too.
-Right,
"so that I can pay attention to you."
I'd never been asked it before.
I said, "Sure."
Only later did I think, "Well,
was she actually
going to read the transcript
and make sure there was nothing in it?"
I made a joke during the encounter,
and I thought later, [gasps]
"Is that going to somehow
be in my notes?"
With more consequence,
more and more clinicians are using AI
to quickly make sense
of a patient's complex presentation.
What's the latest information?
There are tools widely used
to summarize the open literature
for physicians
or to prognosticate and predict
what this patient's course will be.
All of that raises very sensitive issues,
including
back to this question of consent.
The other question
I'd like to lay on the table
-is the black box quality of AI.
-I was going to ask you about this.
Explainability, right?
Well, it's also the black box quality
to the user,
to the physician
or other clinician themselves.
Do they know what the AI system is doing?
Are they capable of evaluating it?
Can they spot errors?
Can they spot problems?
People talk a lot about the governance
provided by the human in the loop.
More and more, I think
there's grounds to question
how effective are humans
in overseeing sophisticated AI.
Right.
All of you have presented enough questions
for six podcasts, I think.
One strain that goes through all of it,
I think,
and I'm picking up
on your black box question here,
how different is it
to offload a question to AI
rather than offloading it
to a senior, experienced
specialist colleague?
You can ask follow-up questions
of your colleague,
but you can ask follow-up questions
of the AI tool frequently as well.
There is some element of that
that's similar.
Is there a meaningful ethical difference
between turning to a colleague
and turning to an AI tool?
I think a lot of it depends on context,
right?
Let's take judges, for instance.
A huge part of what judges do
is resolve cases
in a way that is fundamentally human.
If we want to say,
"Well, instead of having judges,
let's offload decision-making
in individual cases to AIs,"
we might, in some ways,
get similar results.
We might, in some ways,
get similar outputs.
I think, fundamentally,
we'd actually be undermining
the rule of law
because a huge part
of why people value the law
and why they respect it
is because of their sense
that it is rendering human judgment.
In that context, I think
that there is something
fundamentally human
about the nature
of resolving individual cases.
When it comes to, for instance,
doing more ordinary pedestrian legal work,
I think that, a lot of times,
what we care about is the output.
We care that the will is good.
We care that the contract is sound.
Frankly, we may not care a lot
about how the lawyer got from A to B.
Now, that means that,
if we can rely on the AI
to produce
output that is of sufficient quality,
that may be enough.
The difficulty is
because we don't know exactly
how it gets from A to B.
We don't know how it gets
from your inputs to the contractor,
from your inputs to the will.
We ultimately, at least for now,
need a human
that is not only going to be in the loop,
but is going to be in the loop
in a meaningful way.
It's going to exercise sound judgment
regarding the quality of the output.
That just actually turns out
to be really difficult
because it turns out
it's a much different process
to look at something and decide
whether it's good or not
and what ways it may be deficient
than to create it yourself.
A lot of times,
one of the things AIs excel at
is producing things
that look superficially good.
Then when you dig in
and really think about it,
you realize there are gaps.
I think that's a real tension
that we're working with.
We have this human-in-the-loop model
in a lot of contexts, in law especially,
but in medicine,
in other critical settings.
We assume
that if we just throw a human in there
and say, "Well, you look at this,
and make sure it's good,"
things are going to work out well.
It turns out that,
actually, a lot of times,
having a human in the loop is not enough.
We need someone
to actually really be scrutinizing it
in order to have
our assurance of quality.
You need a human
who both knows enough and does enough
in order for that
to be a meaningful constraint
-on potential errors or problems.
-Yes.
Actually, to tie it back
to some of the conversations we're having,
and actually another area of interest
that I really have
is insurance, health insurance.
A huge trend right now are
health insurers
are increasingly relying on AI
to make determinations
about medical necessity,
to make determinations about coverage.
The law says,
"Well, you can't rely on AI.
You have to have a doctor in the loop.
You need to have a human in the loop
to make sure that's right."
There's now these lawsuits
alleging that what these insurers have
is they have a doctor
who looks at it and clicks yes, yes, yes.
It's actually two seconds of review
for each person,
and so then the question is,
"Well, how do you legislate a doctor
meaningfully looking
at an AI determination
and making a clarifying--"
and is that even possible to do,
or is the right approach to ban AI,
or is the right approach
to figure out something else?
These are just real questions
about how
our health insurance system work,
about how the human mind works,
and something we all know,
which is when you're reading
someone else's work,
you don't understand it nearly as well
as when you do the work yourself.
It's just a fundamentally different thing.
The health insurance example brings up
a really important point in health care.
Let's say
you have a patient who presents--
this is based on an article,
a 60-year-old male with lower back pain.
You have to decide,
"Are you going to order an MRI,
which is expensive,
or are you going to do watchful waiting?"
Every decision practically in health care
is a decision
where there are competing values.
The patient wants to feel less pain.
The health insurer
doesn't want to spend the money.
The physician doesn't want to be liable.
There are all these competing values.
I was pulled into a project
about a year ago,
and I've talked to Dan about this,
that Isaac Kohane ran the RAISE project.
He's the inaugural chair of bioinformatics
at Harvard Med,
and he's also the editor-in-chief
of New England Journal of Medicine AI.
It was all about trying to excavate
what are the values
embedded in different AI systems,
so that when a health system
like M Health Fairview or like Kaiser
or any of the biggies
chooses which of these to use,
they know what they're getting.
Are they getting one
that favors saving money?
None of this is visible right now.
You actually need
a lot of humans in the loop,
including at adoption,
to figure out
what values are we importing
into our whole decisional framework
in adopting this and not that AI platform.
Francis, when you look at that boundary
and the ways in which the AI
will bring its own judgments to the table,
is that similar
to hiring a new doctor or a new attorney
and knowing
what skills and background and norms
they bring,
or is there something
meaningfully different
about having it be an AI system
rather than a person
that you're adding to the discussion?
I don't think at a high level,
there's much different.
That is, we probably don't ask enough
about the humans who show up.
Someone wears a white coat
and they walk in
and we think everything they say is true.
Could have been they just had
a horrible argument at home that morning,
and they're not their best self.
It could be that they're just a person
who's always more skeptical.
That's actually one of the things
that we should probably be doing anyway.
We know intuitively.
If I ask three partners
what they think of this,
there's one who always says
we're going to lose.
There's one who always says
we're going to win,
and there's somebody in the middle.
I think, as to Dan's point,
we ought to understand our models.
Where are they in that?
I add one other thing, though.
This discussion assumes a little bit
that we're choosing
between human lawyer and AI lawyer,
or some combination.
The vast majority of people
who need lawyers
can't get them at all.
We actually discussed this
in the health insurance case
of an individual
who gets their claim denied.
They're going up
against the big health care.
Now, they can write letters and memos
that look a lot like they came from
a lawyer, a pro se criminal defendant,
someone who's navigating
a deportation case.
These tools right now can provide them
with the level of expertise and output
that would be impossible for them to do
just a few years ago.
They prefer a big law firm,
but they can't afford it.
There is this possibility
of AI empowerment
and reshaping the legal field;
I'd say, a lot of ways in which
media has been totally reshaped
by the fact that anyone with an iPhone
can now get millions of views.
That just wasn't the case
all that long ago.
I guess I would add,
that then is creating all these questions
about the architecture
of our legal systems.
There are questions about,
how do court systems deal
with the fact
that they're getting
floods of pro se complaints,
which maybe have
greater rates of hallucination?
Traditionally,
court systems have relied on the fact
that there are
not that many pro se cases,
and that when you have a pro se case,
they're probably
not citing much legal authority at all,
and they're pretty sure.
We are having institutions
that need to adjust there.
In the health insurance space,
there are questions about the fact
that now that patients do at least
have some empowerment to use tools--
and by the way,
it's not just the traditional AI systems.
There are new startups like Claimable,
which is one that I've worked with,
that are actually developing
these sophisticated
retrieval-augmented generation tools
that can actually bring in
relevant documentation.
Actually,
an article I'm working on right now
with my colleague, Amy Monahan.
Do we need to restart thinking
how we think
about external review
of health insurance decisions,
and maybe making those decisions
a bit more transparent
to actually enable
those types of uses of AI?
Do we need to restart thinking
about attorney-client privilege,
and whether or not
communications with an AI system
should be privileged?
There are all these new legal questions
that are arising
as our embedded assumptions
about what individuals can do
and what lawyers can do
are shifting.
I think that makes us, in some ways,
an exciting time for lawyers
because we need to rethink
what does privilege mean.
We need to rethink all of our systems,
given the realities of the fact
that this technology is changing,
assumptions that we made
that are fundamental
to how we design them in the first place.
The healthcare analog, though,
Francis, to what you're saying is
everybody's using AI, right?
Somebody puts their symptoms
into their phone,
and maybe they can't afford
to go to the doctor.
That can be quite dangerous.
In a way,
different from getting AI
to write your letter
to protest
the denial of your healthcare coverage.
Radiologists are using AI.
Now, people are studying,
is there a degrading of their ability
to interpret scans
as they rely more and more and more?
The real impacts, pro and con,
to the use of this technology.
I want to spend a little more time
on two of those concerns
that you just raised.
One is,
we touched on it before,
how does AI change the way we think?
Then the other is
potential overconfidence in AI.
The computer said it, so it's true.
Maybe a professional
like an attorney or radiologist,
we have more confidence
that they're going to have
healthy skepticism towards those outputs.
It's very possible
people without specialized training
have no way to make that evaluation,
and AI is going to give them data.
What can law do
to respond
to some of those potential concerns?
We've talked a lot
about the potential great benefits.
That's where law might come in
to help address the maybe problems.
What do you see law starting
in this infancy phase to do
in response to those possible problems?
Well, I do think
one of the common moves has been
the human-in-the-loop requirement,
and the assumption
that if you have an AI, plus a human,
that's going to be better
than a human alone.
I think what we're increasingly learning
is just there's a lot
that depends on context there.
I think we need to empirically study it.
One of the things that I've done
with some colleagues here,
with Nick Bednar, David Cleveland,
Allan Erbsen,
is we've actually tried
to empirically evaluate.
What happens
when human lawyers use AI to produce work
versus not?
Not only have we found productivity gains,
but just in terms of the risk that
people will over-rely on the AI systems
or that will actually
undermine their reasoning,
we found some evidence
that that doesn't always happen.
In particular, what we found is
that folks who were using AI
sometimes actually were better able
to confront a legal problem
even after the AI was taken away
because the AI enabled them
to understand the law better,
and then that benefit persisted.
On the other hand,
we have pretty clear data
from a variety of contexts
that long-term use of AI
can degrade skills.
Going back to the human-in-the-loop,
if you have someone whose job it is
just to make sure
that, say,
the AI-informed health insurance decision
or the AI will check certain boxes,
you can get degradation in the long term.
I think
we're still really trying to figure out
the right governance structures here.
I don't think we have clear answers
because a lot of it
is so context-dependent.
It depends on the task.
It depends on the person.
It depends on the way you're using AI.
It depends
on what the safeguards are in place.
There's not, unfortunately,
a one-size-fits-all model
for, "Hey,
here's how to solve these problems."
There's a great new article
I was just reading yesterday.
The lead author is Margaret Mitchell,
who's an ethicist in the AI space,
arguing that more and more AI,
particularly as you get
to more agentic AI,
is designed
to defeat the human-in-the-loop
and not enable.
They make recommendations
for how to redesign AI
to enable human brains--
you may have a view on this,
human brains
to actually perform surveillance,
that it's crazy to think
that a human would be able
to maintain attention
and unpack all this stuff
and catch problems,
which could be subtle and sophisticated,
in real time
to prevent a detrimental cascade.
I think the human-in-the-loop solution,
which I agree has been the big go-to,
is probably an illusory safeguard.
Certainly, it sounds
like it's at least maybe insufficient.
For sure.
I'd just add
a little point and a big point.
The little point is that transparency
to me
is always the number one thing that I want
because it's hard, as Dan was saying,
to have a one-size-fits-all.
It's so important to do empirical work.
You can't do any of that
unless there's transparency
about what specifically is being used
and how it's being used,
and then you can evaluate it.
The big-picture thing
I think law has to do, though,
is decide with the community
and its constituents
what values
it wants to put into the legal system.
This gets to this question
that you've raised of de-skilling.
What things do we want
only humans to be able to do
or protect as humans,
and what are we totally okay
with them never doing again?
The example I use in the legal context
and the academic context
is using a card catalog.
Some of our leaders and researchers
might remember;
you've learned how to use a card catalog.
It would be academic malpractice for us
to spend even a minute
teaching our students
how to use a card catalog.
They've lost that skill.
They would look at them.
In fact, I'd show my students a picture.
They're like, "I don't know what that is."
We're like, "That's great
because it allows them
to do other things."
There are other skills where we may say,
"We always want that to be,"
or just other parts of being human.
That's the wild thing about AI
is that it's not just about a work tool.
It is about fundamentally reshaping
our human relationships.
See chatbots, see sex bots,
see this question we ask the students,
"Should the law recognize marriage
between a human and a non-human entity?"
Whoa, that's about the 37th big question
-we've raised so far in our podcast.
-The podcast of 2019.
I want to hit
a couple of more key questions
that come up a lot in this context.
One is about bias
in the traditional sense,
that AI tools are only absorbing
the material that's fed into it.
They can often
reconstitute and perpetuate human biases
rather than being
this perfect neutral source.
What are the legal responses,
and are they
in the front end around training
or in the back end
around decisions made by AI,
around discrimination?
I'll say this, actually.
I have a funny background
in insurance and AI.
You might think,
"Well, what's the connection?"
I actually got interested in AI
surrounding this precise issue
over a decade ago
because insurers,
one of the key things they try to do
is predict, right?
One of the things
that AI has been good at for quite a while
is using lots of data to make predictions.
The difficulty is
that in many areas of law,
and including in insurance,
we want to limit
how we make predictions.
We don't want predictions to be based
on factors like genetics
or race or health status
for reasons that actually are much larger
-than just the insurance system.
-The fundamental human--
Right, exactly.
The difficulty is because
of the black box nature
of these algorithms
where we feed in data, and we get results,
but we don't quite know
what's going on inside.
It's very hard to know
when they're making a prediction
whether or not they're using proxies
for these data points,
even if we don't give them the data.
An example I like to use is that, look,
if you train an AI to make predictions
about how likely
you're to be in a car accident,
and it turns out
that one of the things
that actually matters
is a certain genetic information,
but we say you can't use that information,
and you just feed them
lots of other tools,
they'll construct proxies.
They'll say, "Well,
I don't know the genetics,
but if there are information
that is representative
of, say, your reflexes
that can be discerned from, say,
your TV watching habits or your magazine,
it's entirely likely
that they'll figure out
ways to construct that."
That opaque nature
of the way that AIs construct algorithms
means it's really hard
to debias them in that context.
Again, a lot depends on context.
When a lawyer's
using a large language model
to produce a contract,
we may worry
about different types of biases,
biases about
what types of terms are in there.
A lot of it depends
on the type of AI you're using
and the context when you're doing it.
There's not one answer
to how do you limit bias.
Bias is always a problem,
both in biomedical research
and in health care.
When you design research
using human beings,
always you're thinking
in the design of your sample.
What population
are we trying to generalize to?
If you're trying to generalize,
for example,
to pick a simple example,
to people of all genders,
but you've got only men in your sample,
you have an obvious problem.
That's very well-recognized.
A subtler version of this
is the genomics database
is largely
white people of Northern European descent.
This is a very well-recognized problem.
If you just blindly go along
and say to the AI, "Okay.
Well, use the known genomic database,"
you're cooked.
You've built in bias
from the very beginning.
There has to be a way
to continually interrogate.
You can cause tremendous harm.
You can miss
because there are certain genomic variants
associated
with propensity to fatal cardiac disease.
I'm thinking of TTR.
That's more common
in traditionally minority populations.
You can just completely miss it,
-and people can die as a consequence.
-Right, if people who are like them
are excluded from the data.
Right, or if a clinician using that data
fails to recognize
and look for this particular variant
in this patient.
These are life-and-death matters
that have to be continually
checked and improved.
At the end of the day,
I want to throw in one other thing,
which is liability.
We're lawyers.
Of course, a lot of this discussion,
we're thinking
about accountability and liability.
Susan, I'll go to you first
and ask the others to chime in.
When an AI-assisted decision
causes real harm,
who is responsible?
Well, I think the answer
so far in the healthcare space
has been that the clinician,
the physician,
and sometimes the institution itself
remains accountable.
It's making people extremely nervous
because of all the elements
we've talked about.
I am relying on this AI tool.
I don't fully understand it.
You then get into meta questions.
Is the use of that tool standard of care
-within your specialty?
-Sure.
Is it well-established?
Is it well-understood?
Did the institution exercise due care
in adopting that tool,
in monitoring that tool, et cetera?
What is the nature of the problem?
If the radiologists
that I relied on the tool,
and there's this big white blob
staring at the radiologist from the scan
and they just ignored it,
going, "Well, the AI didn't say that,"
the radiologist is in trouble.
It becomes very fact-dependent.
I don't foresee a near future
where the clinician is off the hook.
In many contexts, the answer is the same,
which is the insurer
is the one who's often on the hook,
at least
in terms of the monetary consequences.
That's raising
a lot of really interesting questions
about, "Okay,
are we actually seeing changes
in the way
that insurance policies are structured?"
There's talk that
certain insurance policies
are excluding risks from AI
in order to shift the risk back
onto individuals and companies.
There's also a burgeoning market
of AI-specific insurance.
Many policies are not
specifically excluding AI.
Then that's raising
really fundamental questions of, "Okay,
is insurance actually undermining
some of the goals we might have with law?"
If the goal of malpractice law is to say,
"Well, human radiologists,
don't ignore that blob,"
but at the end of the day,
the insurance policy is going to cover it,
and it doesn't have an exclusion
and the price is not going to change,
then maybe we're actually undermining
some of the incentive structure there.
Now, that may happen less in medicine
because, frankly, doctors hate being sued
even if they don't have to pay anything.
If it's a context where, for instance,
we're talking about a corporation
and we're going to hold them
liable for a product,
then maybe at the end of the day,
we have to worry about insurance
undermining that incentive,
or we can think maybe insurers
are actually
going to go out there
and be private regulators.
This is another area
where I've done a lot of work
thinking about,
"Can we rely on insurers
to actually get these structures right?"
My bottom line on that is
some heavy skepticism
about both insurance,
but also liability,
because a lot of times,
whether liability
works to produce an outcome
depends
on how insurance intermediates that.
If we don't have confidence
that insurance is going to price the risk
correctly
and transmit the incentives,
then maybe we need to think a bit more
about how we're going to regulate
things ex ante
and set rules up front
as opposed to saying, "Well,
we're not going to regulate you much
in terms of what you do,
but if you cause harm,
we'll impose liability on you."
That's not always the best solution,
especially when insurance
stole those incentives.
I can already picture myself as a 1L
in your torts class, Dan,
hearing this spiel.
I spend a lot of time on insurance
in my tort law class.
-[laughs]
-I'm just going to pick up
on one thing Susan said.
She's absolutely right
that the standard of care
is shifting so quickly,
and the market demands
are moving so quickly
that although one could say,
"I'm going to limit liability
by really, really restricting
my use of any of these tools,"
you're going to, A, be out of business
and, B, maybe be liable
for the reason Susan said
because a court is going to say,
"It's no longer reasonable
for you not to use these tools
given what we know."
It's the speed of discovery bill
that is wild here.
The analogy I heard recently,
which I like, is
COVID hit, and everything sped up.
It wasn't like,
"Let's have a five-year committee,
and then we'll--"
No, we had to act immediately.
Of course, this law school did.
A lot of these tools, in my mind,
are moving at that speed.
You blink five years down the road?
Forget about it.
A lot of the problems we're talking about
were problems
before we were using AI systems,
problems of bias, problems of accuracy.
How much is it the case
that the use of the AI is magnifying
or intensifying these problems,
and how much is it just a new presentation
of an old problem?
I do think it's difficult
to know how much worse AI is.
I think its contextual neuroscience
and psychology have shown us,
over the last century,
a number of ways in which
how we actually process information
and produce behavior and decisions
is not the way we think we do.
That has fundamentally reshaped
eyewitness memory in the law.
It's reshaped all sorts of areas.
AI, on one hand,
we have the benefit of knowing
a little bit about how it was created.
We have no idea,
and we can't reverse-engineer
the human brain yet.
On the other hand,
this has been the conversation,
we don't know enough
about how it's processing information.
The last thing I'll say about that is,
at least with the generative models,
and we do this with our students,
be aware
that it's not like looking in a dictionary
where all of us, we go to the same page
in our different dictionary.
It's the exact same thing.
It's one book.
The generation depends
on who's asking the question.
There's a bias there as well
because,
depending on who's asking the question,
you might get a different response
trying to please you.
It's not always the case,
I think this is Dan's point,
to say human-in-the-loop is better.
It's not always the case that,
"Oh, the human did it.
Let me choose that over the AI."
It really depends.
We've covered so much ground
and talked about so many different things.
I think one useful thing before we close
might be,
if each of you could talk
about the one principle
that you think
comes out of this issue in our discussion
for how society should use and govern AI,
what's the most important thing
to take away from our conversation,
do you think?
I guess I'll start,
and I'll focus my answer on lawyers
because we probably have
many of them listening.
Of course, we're all law professors.
From my mind,
the most important thing
about how lawyers use AI
is, actually,
they need to rely on their legal skills
because it turns out lawyers
are very well-situated to use AI well.
What are we trained to do?
We're trained
to evaluate the quality of output.
We're trained
to ask for counter-arguments,
to look for hidden assumptions,
to look for biases,
and then to follow up.
A lot of times, actually,
I think lawyers worry
that they're not well-equipped
to use AI well.
At the end of the day,
lawyers are trained to be skeptical.
Lawyers are trained
to evaluate output and ask questions.
If lawyers bring that perspective
to using AI,
I think, a lot of times,
they're some of the best users
-of AI.
-"Hey, everybody, come to law school
to learn
how to be the best possible AI user."
No, I really actually think
there's a lot of truth to that
because AI works through language
and spotting assumptions in language
and asking follow-up questions
and being a skeptical reader.
Those are the core skills we teach.
I really do think
those are the core skills you need
to use many AI systems well.
I do think that's an important lesson
lawyers need to learn
and law students need to learn as well.
Susan, what's your key takeaway?
I'm really worried about alignment.
That's a term we haven't used yet,
whether AI tools,
and I think about them in the context
of health care and biomedical research,
incorporate the values
and the legal decisions we've already made
about how health care, for example,
should operate.
I remember being in a meeting last year.
Everyone was asked,
"Intubate or don't intubate?"
The scenario was a patient with anorexia
who was refusing a feeding tube.
Do you intubate?
Do you start the feeding tube?
What's the clinical decision?
I sat there,
I've talked to Dan about this,
thinking, "Wait a minute.
What does the patient want?"
There are all these procedural rules
and ethics values, patient autonomy.
Is the patient competent?
If not,
who's the surrogate decision-maker?
Unless it's a total emergency,
way before you place that feeding tube,
or way before you intubate.
I started to worry
about whether we've got
runaway technology at the door
that's going to erase a century of work
on things like informed consent,
patient decisional rights.
We have to make sure
that as we integrate these tools,
we really keep our eye
on the values
we thought we were protecting
in health care.
Patient comes first.
What are the patient's values
that should govern those kinds of things?
I think those are increasingly at risk.
Yes, I would say this,
"Don't be scared.
Read the manual."
That's the slide I put up
is read the manual.
If you don't understand
what the system is that you're doing,
you're in trouble.
Like Dan said, we're really well-trained
to read, to understand.
When we do that,
I think we'll see that these are tools,
we'll be impressed by them,
but we'll also understand
they're not nearly as good as our brains.
That, I guess, I would leave with.
We've been trying to reverse-engineer
the brain for centuries.
AI emerged in the mid-20th century,
1950s.
It's taken this long just to get this far.
Still, even the best systems
are nowhere close to understanding
even our most basic understanding.
When we all go home tonight
and hug with our loved one,
what is happening?
I'm not too worried.
I'm pretty excited.
I will put a footnote
that in maybe 15, 20 years,
we'd come back and talk about robots.
10 years ago, when you said AI,
everyone thought about robots.
That's called embodied AI.
It's slowly coming down the pike.
Save that for later,
but this has been great.
That might have to be a future podcast.
We've heard about healthy skepticism,
alignment of values,
and understanding
what's happening under the hood.
I think that's come through
in our discussion throughout.
I want to thank
all of you for being here.
AI really is already challenging
how we practice law, conduct research,
deliver health care,
and understand what's happening
when humans are making decisions.
As we've heard today,
it's both about deciding what AI can do
and deciding
what we want humans to continue doing,
and what responsibilities we are
and are not willing to delegate, and how.
This conversation's only beginning.
I'm delighted
that I have the three of you
and multiple other colleagues
who are doing really hard thinking
about these issues
as AI continues to evolve
and the questions evolve as well.
My deepest thanks
to Dan Schwarcz, Francis Shen,
and Susan Wolf
for joining us
for this fascinating conversation
and for sharing
your varied and deep expertise.
Thank you
for listening to Minnesota Law Now.
I'm William McGeveran.
If you enjoyed today's conversation,
please be sure to subscribe
wherever you get your podcasts,
and we'll see you next time.
[music]