Is 99.9% safe good enough for robotaxis? | Phil Koopman of Carnegie Mellon University (Professor Emeritus)

In this episode of The Garage, host John Heinlein interviews safety expert Phil Koopman on the complex challenges of scaling autonomous vehicle fleets from hundreds to hundreds of thousands. Koopman identifies key hurdles, including mass software failures, broad safety definitions, infinite edge cases, and legal accountability gaps, stressing that machine learning's lack of common sense requires human oversight for rare events.

Creators and Guests

Host

John Heinlein, Ph.D.

An experienced technology and marketing leader, John brings his background from startups and established companies to Sonatus. He worked for 14 years at Arm, most recently leading Automotive Partnerships for North America where he engaged OEMs, Tier-1s, and others to deploy Arm-based solutions into automotive applications, including autonomous vehicles. His team was integral to launching the SOAFEE industry initiative for software-defined vehicles of which Sonatus is a member. Earlier he served as VP and Chief of Staff to the CEO and led a group responsible for competitive strategy. For three years, he was VP of Corporate Marketing where he led centralized outbound marketing, spanning marketing campaigns, press, events, web, digital marketing, ecosystem programs, and working closely with investor relations. Prior to Arm, John had an 11-year tenure at microprocessor startup Transmeta where he held several senior roles spanning business development, marketing, and customer success, among others. John earned his B.S. in Computer Engineering from Carnegie-Mellon University and an M.S. and Ph.D., both in Electrical Engineering, from Stanford University.

What is The Garage by Sonatus?

The Garage Podcast from Sonatus brings you conversations with thought leaders from around the vehicle technology space discussing far-reaching topics about software innovation in vehicles.

These episodes will include industry experts, whether from Sonatus, or from our amazing partners across the industry, all of whom will share their Ideas and their outlook for the most important topics in vehicle technology. Episodes span from more technical topics, to business evolution, and market trends. Join us and learn about the future of vehicle in The Garage.

Today in The Garage,

we're going to look at
autonomous driving in a new way.

Everyone is excited to see
how much progress autonomous,

vehicles have made in recent
years with Waymo and Zoox and

many other companies now
commercially available.

Many people in in cities
like here in San Jose,

the Bay Area where we live,

can hail a robotaxi on their
phone and it shows up with no

driver and takes them on a ride.

And it's been incredibly
exciting and successful to see.

I, personally, in my career was working
on robotaxis about five years ago

when it wasn't quite ready
and it was a very frustrating

experience for me.

So to see how much progress
they've made is really

inspiring to see how that
technology is going and an

incredible use of AI.

Having said that,

as we're getting to higher
numbers of robotaxis,

the complexity of making
robotaxis that work well

to make them at the level of
working well that is viable to

scale to thousands and
thousands of vehicles is a

bigger challenge.

And who best to talk about that
than a real expert who's been

studying this for thirty years?

Our guest today is
professor, now newly retired,

Phil Koopman from Carnegie
Mellon University who spent

thirty years studying
robotaxis and other aspects of

safety engineering and
will tell us all about the

pluses and the challenges to make
robotaxis very viable in scale.

Let's go.

I'm John Heinlein, Chief
Marketing Officer with Sonatus.

We're so pleased to have Phil
Koopman here with us today.

Phil, welcome to The Garage.

Thanks for having me
on. Happy to be here.

So we we both have a strong
connection to Carnegie Mellon.

I went to undergrad there.

You were a professor
for a long time.

And we we I think we crossed
paths when back at when I was a

student and you
were a grad student.

Start by telling
us your background.

Sure.

My background is I started
with an undergrad in computer

engineering at RPI.

And after that, I served in the Navy
because they were kind enough to pay

for my undergrad.

That that's how it
worked back then.

I then worked well, then I went
to Carnegie Mellon University,

got a PhD, which is where
we ran into each other.

I was a teaching assistant
in a hardware design course,

bit-slice CPUs for people
who know even what that is.

And but it was a fun course.

And after I graduated there,

I went to work for Harris
Semiconductor as a CPU designer.

I spent time at United
Technologies at the Central

Research Center, got a lot of
embedded domain experience,

and then finally went back to
Carnegie Mellon as a professor,

taught there for
almost thirty years,

and just recently retired.

Well, congratulations
on an incredible career,

and you have a real
distinguished background.

But we're gonna get to that.

But before you, you gotta
tell us a fun fact about you.

Sure.

The the fun fact is I used to
drive submarines in the navy

during the cold war.

I have a combat medal, but I
can't tell you how I got it.

That's an incredible story.

I love submarine movies, Das
Boot and The Hunt for Red

October.

Such such great stories, so
I'm jealous of your experience.

Well, Hunt for Red October
is an exceptionally good one.

It's very accurate for the
types of things I was doing

back when I was doing it.

Very fun.

So tell us about your
career at CMU and especially

about safety engineering.

That was a key
specialty you had.

I wonder if you can
introduce for our audience,

what is the concept of safety
engineering and how does it

differ in some ways from
sort of, if you will,

classic engineering of of
just building something?

How does it complement that?

Sure.

I've been doing self-

driving car safety
specifically for thirty years,

and I've been doing a lot
of other types of safety

engineering for I lost count at
200 design reviews of embedded

systems, and I lost
count like 15 ago.

And and some of them
were just doesn't work,

but a lot of them were safety in
all sorts of different domains.

So I've seen lots of safety
standards, lots of safety.

And if you have to
boil safety down,

it really comes down to if
you're building a system and

you're testing it, engineers are
taught to make sure it works.

And they're really
good at the happy case,

but maybe not the unhappy case.

And safety engineering isn't
about how to make it work.

It's about what happens
when something goes wrong.

And specifically,

what you're taught to do is
create a list of hazards,

things that can cause problems.

A lot of times, it's
personal injury,

but it could be severe
equipment damage.

It could be a lot of things.
It's something bad will happen.

You make the list of here are
all the things that can that

can go wrong and here's our
plan to make sure that either

it won't happen or
if it does happen,

that the harm will
be mitigated somehow.

And so safety engineering
is listing the hazards,

figuring out how to mitigate
the risk from the hazards,

and making sure those safety
requirements go into the

engineering requirements,

which means now you have to
design the safety behavior

intentionally and you have to test
that the safety stuff really works.

Just a really simple one: On
a train, there's air brakes.

If everything goes wrong,

there's a cylinder of
compressed air and you hit the

button and it slams
on the brakes.

And and what's that for?

Well, there's lots of things
that you go on a train,

but most of them get cured
by slamming on the brakes.

So that's the plan.

It's interesting to to describe
it that way and I think that,

you know, as you're right.

As an engineer, you want to
make something work, but, know,

you have to really think about
all the failure scenarios.

And I think there's a concept
called fail safe and fail

dangerous, which I think is
is often often talked about.

I think probably your
goal, I'm assuming,

is first avoid the failure
happening at all in the first place.

But if it does happen,

have it fail in a graceful way
that safety is not put at risk.

Is that a fair way
to describe it?

Yeah.

That's well, that's the way
it used to be until recently.

And so classical systems, if you
de-energize the system, it's safe.

You you have a chemical process
and if you can get that thing

wound down so it's at room
temperature instead of superheated

steam, it's
a lot safer.

So so you wanna
avoid the problem.

You wanna have a way that if
you have an unsafe situation,

you can sort of wind it
down to to become safe,

and usually that means by
dissipating the energy.

You have a car, you
bring it to a stop.

But the catch is there's some
situations where winding it

down to no energy
isn't good enough.

So you have an airplane and you
have an engine fire at 35,000 feet.

Turning off all the engines
is the wrong move. Right?

And so there's fail safe,
which usually means fail stop.

Just wind everything de energize
everything and you're safe.

And that works in
a lot of places.

But in some places,

if there there's a high
velocity motion involved,

you need to actually keep working
long enough to wind it down.

And there's a reason why passenger
airliners have two engines,

and there's a reason why they
have two or more sets of flight

controls because when one fails,

you have to fail
not just fail stop,

but fail operational long enough to
get yourself into a safe position.

And that applies to cars too.

If you're 60 miles an hour on
a highway and your steering is,

well, the steering is a problem.

It'll shut down.

That doesn't work.

You need the steering to work long
enough to get the car shut down.

It's true.

And and you can't even just
come to a stop either because

then another car behind you
is gonna plow right into you.

So you you need to
somehow gracefully...

You have to get off the road.

You have to get out
the travel lane.

I mean, there may be a very rare case
where there's nothing you can do.

If the if the if the engine
falls out of the car,

there's only so much you can do.

But in almost all cases,

you want enough oomph to get
yourself out of the travel lane.

So you need you switch from
the normal operation mode to a

recovery operation mode with
less capability and a very

short mission time, but
that backup has to be there.

So you have to have
backups or it doesn't work.

That's fantastic.

We're getting a degree
in safety engineering

here in in the podcast today.

I think you've been such a you
mentioned you've been doing

autonomous driving for for
thirty years, and obviously,

there's been massive massive
progress over that time from a

glimmer in someone's eye to
you can hail a robotaxi right

outside my office here.

So I think let's start talking
first about some of the the

rosy things that we've seen and
then we'll talk about some of

the pragmatism that's
come up as well.

We're seeing now practical
robotaxis, Waymo's,

Zoox, a few others that
you can actually hire.

Certainly there's some
solutions in China as well that

maybe people know where
they're actually in practical

deployment in certain cities,

in certain limited
operating domains.

I'm sure we'll talk about that.

So there must be incredible progress
over the time you've been doing this.

Well, it's been thirty years,

but there has been
great progress.

So it's important to
say, yes. Absolutely.

You can go hail a robotaxi.
I've had rides in robotaxis.

It's amazing technology.

And the thing about safety
is 99.9% is a good start.

It's not the finish line. Right?

And we're it feels like we're
99% there because my goodness,

you can go hella robotaxi,
but it did take thirty years.

So thirty years ago,
thirty one years ago,

Carnegie Mellon had a car that
went from Washington DC to San

Diego 98% hands off the wheel,

and it's taken all this
time to get that last

1.999%, whatever it is. Alright?

But it's been great progress because
that's an amazing achievement.

You can actually there's
nobody in there driving it.

There's, you know,

there's no man behind the
curtain when most of the time

when these things are driving.

That's great.

And it feels like
we're almost there,

but I think what we're going
to talk about today is once you

have a 100 robotaxis on
the road, that's amazing.

But from a 100 robotaxis
to a 100,000 robotaxis,

it's a whole different thing.

And what I like to say is
99% done, 99% more to go.

I mean it's it's another way to
say that because as you know as

you mentioned when you think
about you know it's famous in

business to talk about five
nines which is 99.999 or we

know how many ever
nines you want.

The thing is really what it
depends on is what's the numerator.

Right?

If if there's a 100
robotaxis on the road,

99.9%, probably pretty good.

When there's a thousand,
when there's 10,000,

it becomes a situation where
you're going to hit some of

those those smaller nines.

Simple division.

Go ahead. Please.

Sure. So simple division.
Just to illustrate the point.

Something that happens once a
year to a 100 robotaxis happens

twice a week to
10,000 robotaxis.

And that's not saying there's
anything with a robotaxi,

but once a year, you
probably may not even

notice it happen, but when it's
happening twice a week, it matters.

And that's not that
the robotaxi got worse.

It's that it's a numbers game.

The more you have,

the more you have to pay
attention to the things that

were so rare they didn't matter.

And so the the technology to
run a 100 robotaxis is truly

impressive, and it's it's here.

It's now.

The next challenge is what do
you do with the rare events

that used to be so rare they
weren't a big deal because now

they're going to become
a big deal at scale.

That's the challenge.

Well, this is an exciting topic.

We've had guests over many
years talk about here on The

Garage talk about this this
problem from a different lens.

A senior person from Mobileye
earlier this season and others

talking about what
they're doing.

But you bring a really valuable
and unique perspective.

So let's talk about some of
the challenges you see in mass

production and mass
deployment of robotaxis.

Well, I I have four bins,

and I guess we'll probably
walk down the four bins,

but let me give a preview.

The the first one
is mass failures.

When there's one robotaxi,

if it stops in the middle of
the street, it's one robotaxi.

If you have a 100 robotaxis all
stopping in the middle of the

same street, now you
have a different problem.

So and they and software tends to
fail under similar circumstances.

So if all the robotaxis
have the same software,

that that's that's
what computers do.

They tend to fail at once.

So you you have to
have a plan for that.

The next one is there's
more to safety than crashes.

And so the industry has been
really talking about reducing

crash rates and then making
great progress at that.

But what we've learned as the
number of robotaxis has gone

from 10 to a 100 to a thousand,

that there are other things
that matter to safety that are

sort of bubbling up.

That that in the first days,
I I wouldn't have said,

I wouldn't have worried
about things like this,

but but it turns
out they matter,

so we need to learn from that.

The third one is that
edge cases, rare events,

turned out to be more more well,

they turned out to
be a big challenge.

I would say more
than anyone thought,

but I knew they were
going to be a challenge.

And the industry
has a plan for that,

but that brings new challenges.

And the last one
is accountability.

As you get scaled up,

you have to have a better accountability
plan than where we are now.

And that's not the
robotaxi industry's fault.

That's that our legal system isn't
really ready to handle what's going on.

Thank you for that taxonomy and a
good preview of where we're going.

And I think that's that covers
so many different it really

covers it from
different perspectives.

So let's start
with the first one,

which is you mentioned mass
failures or when one fails that

tends to correlate
with other failures.

Talk about that and and what's
the challenge and what should

we do about it?

Well, when you talk
about human drivers,

human drivers make
mistakes all the time.

Cars break down. All
sorts of things happen.

But it's rare for a 100 or a
thousand cars to all have the

same problem at the same
time because every human driver is

different different cars.

Sure. There are things that
stop or or clog up traffic.

If you have a blackout,

it's gonna cause some
pandemonium for all drivers.

But if you have a the same
computer driver in every single

robotaxi and that's a
significant fraction of the

traffic in a city, then
if something goes wrong,

you're gonna have
a citywide problem.

And and don't tell
me that computers,

network computers at large
scale never go wrong.

You know, CrowdStrike is
just the latest poster child.

Things happen.

Now there was a
issue in in China,

I think it was in Wuhan,

where all the taxi robotaxis
from a single company all went

dead at the same time because
there was some some sort of

communication problem.

And now these these people were
stranded on high on elevated

roadways, and and we've seen
we've all seen the the the

videos of US robotaxis caught in

just mass events like a big
party event or something like

that where there's a lot of
cloggage or during a power blackout.

You know, this is the kind
of thing that happens.

Now that doesn't mean you
can't use the technology.

It means you have to
realize it's inevitable.

You know, let's let's put all
our software as a service stuff

on the same server in the same
city and serve the world from

that really bad idea.

Right?

There well, guess what?

Robotaxis is gonna have the
same sort of vulnerabilities.

You have to have a plan.

You mentioned when a
significant fraction of the

traffic is from
a single company.

So obviously, that's not
something we have today.

But there are many
people, maybe the bulls,

the optimists
about robotaxis who

feel like they could at some
point in the future obsolete or

certainly reduce significantly
the number of cars.

So it sounds like that's a
real practical problem to think when

there's consolidation
from a certain company.

Now we're not talking
about cyber security.

That's probably a whole another
thread we could do at another time.

Whole whole thing.

Yep.

You talk about mass hacking
of a fleet of robotaxis,

you can imagine bringing
a city to a grinding halt.

So that's certainly a
theoretical problem as well.

But your point was even
that could happen without a

malevolent actor just due
to a correlated failure.

That's right.

And there are some consequences
beyond just the bunch of

robotaxis stranded.

If you heaven forbid,

there's an earthquake and if
the robotaxis all get stuck in

the earthquake now I know human
cars have trouble in earthquake,

but human drivers are pretty
good at getting their car out

of the way of
emergency vehicles.

And we've seen a lot of trouble
with robotaxis struggling with that.

And, if it's the ones, twos,
they they figure it out.

But if there are hundreds of
robotaxis stuck and they're

blocking all the emergency
vehicles in a in a disaster

situation like that, it
can be a problem.

Now is it a big problem
compared to human drivers?

Interesting question.

But those are the kind
of questions we need to ask.

If you're gonna scale this up
to be a substantial fraction of

the cars, you need to think
about disaster response.

You need to think
about, well, okay.

So there's and I guess we
should do this one next about

remote assistance.

But if you have 50 remote
assistants and 200 cars

all get stuck at the same time,
you know, what's your plan?

You need a plan.

Right. Yeah.

In a similar way, I mean,

we'll talk later in in in item
three about kind of corner cases.

But let's imagine, you know,
you said an earthquake.

Let's imagine a terrible
earthquake where a lot of the

roads are cracked.

All of a sudden now,

you could experience a
situation where many of the

corner cases literally happen
instantaneously where the roads

are great and now the roads are
cracked and robotaxis see obstacles.

And you don't even
need an earthquake.

You just need a power blackout to
take out all the traffic lights.

And you can say even if 90% of
the cars get the traffic lights

right, that 10% adds
up pretty quick,

and that this is actually
something that's happened.

Or hurricane evacuation
or or who knows?

If there's some situation
they haven't been trained for,

they're all gonna fail at
the same time if it's an

environmental thing.

Now this can be improved,

so the the question is
not not to say, well,

there's nothing you can do.

The question is, are
we looking ahead?

Are we waiting for these things
to happen before we respond,

or are we getting ahead of it?

And and what we need to do as we
scale is we need to get ahead of it.

Okay. Alright. So
that's topic one.

That's that's really super
interesting and and scary.

Topic two, you were talking about human
drivers and and failover approaches.

Right.

So the this is the technical
challenge for robotaxis since

the first day has
been edge cases.

So rare events well,
theoretically rare events,

it gets it gets sort
of tricky there.

But if you think
about something like a power

blackout, power
blackouts are rare.

They don't happen that often.
Depends where you live.

But just because it's rare,
if it's high consequence,

it doesn't mean you can ignore
it if you care about safety.

And there are things like
pedestrians walking out in the

middle of a block.

It doesn't happen all the time.

It depends what
city you live in.

I know some listeners
are going, wait a minute.

He doesn't live in my city.

But but people going across or
or school kids coming out of

their parents' car and crossing the
block to get to the school.

Well, depending where you are,

you may never have seen that
or you may see it every day.

So rare rare is
kind of relative.

So this is why it gets tricky.

You say, well, that
doesn't happen often.

Well, it doesn't matter if
it doesn't happen often.

If it's high severity
and it happens to you,

you have to deal with it.

Right?

And and the catch with machine
learning is machine learning is

good at what it's trained at,

and it tends to be very
brittle and and lack,

what for better lack
of a better word,

I'll call it common sense.

Machine learning
has no common sense.

So if it gets into a situation
that it hasn't been trained

for, it's prone
to overconfidence.

It doesn't even realize
it doesn't know.

It does something we
would say is just stupid.

And if that's a high
severity situation,

that's gonna be a problem.

So the edge cases are how
do you deal with things the

machine learning
hasn't been trained on?

And whether you think
it's rare doesn't matter.

If it doesn't know,
it doesn't know.

It doesn't it doesn't help
to say, well, I've seen that.

Well, if it hasn't been trained,

it hasn't been trained.

And so safety severe
events are very rare.

People, even if you
conclude the drunks ,

it's about one fatality
per a 100,000,000 miles.

A 100,000,000 miles
is a lot of miles.

People are impressively
good drivers,

especially if
they're unimpaired.

It's way way better than that.

And so you can say,

really, only the
rare things matter.

People will say, well, that's
rare, so it didn't matter.

Like, no. Fatalities are
extraordinarily rare.

So and, in fact, the rare
things are what control safety.

So normal engineering
is, I pressed the button.

Did it work? Yeah.

Okay. Cool.

Safety engineering is,
I know that's rare,

but it's high risk.

And if you add up all
the events over time,

it's too high a risk.

We have to mitigate it so
even the rare stuff doesn't cause a

big loss event.

Right.

And so you talked
about, you know,

strategies to to attack
these rare events.

What are what are some
strategies to to tackle them?

Well, it used to be it looked kinda
hopeless because there's an it

it's like you're
hunting for Pokemon

except there's an infinite
number of Pokemon.

You can't collect them all because
the world is full of craziness.

It's just absolutely full.

And the the saying used to be what
the industry used to say is, well,

we'll catch enough that
we're willing to suck up the

occasional weirdness.

That if we're a 100 times
safer than a human driver and

something bad happens, people,
for a factor of a 100 people,

are just gonna well, you
know, that bad things happen.

There's nothing you do about it.

But they're not anywhere near a 100
times safer at its fatality rate.

We we don't know for sure,

but there's no way
it's a 100 times safer.

And they never
actually got there.

So the the industry basically
declared defeat on edge cases.

We don't have the ability to
make AIs handle all the weird

high severity stuff.

We just don't. So what's their
plan? But they're on the road.

How did they do that?

The way they did it was
they said, you know what?

When the robot taxi
gets into trouble,

it's gonna phone home and a real
life human's gonna help it out.

And that's an ingenious
way to handle edge cases.

So now you don't have to be trained
on how to deal with everything.

You just have to be trained
well enough to know,

I have no idea what's going on.

I'm gonna go go
phone phone home.

So a lot of the incidents
you see are there's still room to

improve in knowing
when to phone home.

But when they phone home,

then they have a human
operator that says, oh, yeah.

I know you can't see the
traffic light because of road

glare or the wind spun,
trust me, bro, it's green.

Okay?

And safety is it had better
be green when they say it's green

because the robotaxis
just you know,

the robotaxis has all
these sensor inputs,

but that remote humans is sort
of casting the deciding the

vote and many times
determines what happens next.

Very interesting.

So I think that obviously there
was a bit of a dirty little

secret of the industry for
a while that a lot of these

companies weren't acknowledging
they're human backup drivers.

But I think that most of them
are acknowledging that they do

that in some situations.

Now indeed, it's
probably the rare case,

but that they are there for
those exceptional cases.

It it actually
happens all the time.

But but that's not bad because
getting hung up on autonomy

purity is is beside the point.

It I don't care if
it's fully autonomous.

In fact, I'm glad they have people
helping it out when it doesn't

know because the
alternative is scary.

Right?

It's a it's a question of how
many people do you have versus

how many vehicles.

And so if you have a
100 vehicles per person,

it's cost effective.

Actually, if you have ten
ten vehicles per person,

beyond that, it's kind
of diminishing returns.

Right? The numerator is
small. The divisor is high.

Right.

You know, at 10 to
one, at 20 to 1,

it's it's not that
big a difference.

So as long as you have a
bunch of cars per person,

the economics, which is what
matters here, works out.

So the next question is,

are the remote are the remote
assistance really safe?

And the industry's finally
come clean that they have them,

and I think that's fine.

I don't think that's a negative.

I think that's realistic.
It's no problem.

What the industry hasn't has
to come clean with still is

that those people are making
safety critical decisions,

and we need to be attentive
to the fact that they're in a

safety critical role.

The industry is still working
on on sort of addressing that.

But when when you have 10
or 20 remote assistants,

you can hire the very best,
the cream of the crop,

and maybe you won't
have a lot of problems.

Maybe it's no big deal.

But if you have a
million robotaxis,

you probably have a lot of call
centers with remote assistance

and you have to manage
the safety there.

So that's that's a remaining
scale up challenge.

Alright.

There's number two zero four
and I think that's a really

interesting topic.

I think the third is the corner
cases and we touched on that a

little bit in the
previous section.

So

we had a guest from
Mobileye on a few months

ago and we were chatting about the
one of the benefits that they have,

they're one of the largest
driving databases in the world,

is they can accumulate
the corner cases and

inject more corner cases into
the training model for their

customers and so on like that.

And definitely beneficial.

But as we were chatting earlier,

think the challenge with
that approach is that,

please you take the ball
and run with it from here,

is that that only talks about
the problems you know about.

And the trick is that there are
many problems you don't know

about and then the unknown
unknowns is I think is the

biggest challenge.

Am I right about that?

Yeah.

That's the infinite Pokemon
problem that I talked about before.

But the there's a nuance
here that matters.

Traditionally, people think
of edge cases as, oh, look.

There's a zebra in the
middle of the road,

and I don't know
what a zebra is.

They think about about crazy
objects or crazy events,

and for sure, those
are edge cases.

But intuitively, maybe you
can catch almost all of them.

The part that's more subtle and
more tricky is that there are

combinations of known things
that also form an edge case

if you haven't
been trained on it.

In particular, you could have a
circumstance that combines known things in a

way that has a different
meaning than you're trained to.

I'll give you a
super simple example.

If you have a stop sign
this is a trick question,

I'll warn you.

If you have a stop sign
on a pole, what do you do?

You stop.

Right? You stop. Okay.

Then you go. Right?

Right.

Except trick question.

If it's a stop sign in a pole
being held by a construction

worker, you stop
and you say stopped.

Well done. Well done.

More trick question, but I'm
not done. More trick question.

Stop sign in a pole and it's
in somebody's hand and it's

not in front of their face,
but it's down by their leg,

you ignore it because that's a
school crossing guard who's not

serious about having you stop.

And every you
intuitively know this.

Everyone intuitively knows this.

But stop signs are
really complicated.

And if it's a stop
sign bolted onto a bus,

whether you stop depends
if it's out or folded back.

Because one, you stop and say
stop, and the other, you ignore it.

So so my point is that edge
cases are more than stop sign.

When you're trained on stop
signs, you say you are,

but why is your robotaxi
running bicycle buses?

Well, turns out stop
signs aren't so simple,

and I took the I took the
example everyone knows.

It's there's more
to it than that.

Right.

It's such an
interesting example.

And, you know, I was I
was driving last year,

and a car decided
to completely turn

left into the oncoming traffic,
turn left into my oncoming...

and so you're just
not even used to

seeing that because he's just
where he's not supposed to be.

So what do you do?

Well, I got the hell out
of the way is what I did.

But you know, the question is you're not
even used to other people breaking the rules.

And so not only do you
have to follow the rules for yourself,

but you have to imagine
other drivers that are doing

something even extremely crazy.

And in real time, have
to use common sense.

So I'll I'll do an example that I'd
never occurred to me until recently.

I'm driving along and a fire
truck stops in front of me,

and the firefighters
are getting out.

They open the door and there's
a stop sign painted on the

inside of the door.

So and I'm like, I
know what they want.

They want to not get run over
as they get out of the truck,

so I believe I'll stop and say
stop till the door closes is

what I'm gonna do.

Right? So that and
but people are great.

I mean, bet you would
have done the same thing.

Would have figured out from
context that must be more

Right. Be what this means.
It's on the driver manual.

It I was never trained on it.

We've and and AI is
spectacularly bad.

It's good at interpolating
between known points.

That's what it does.

When they say journalize,

it means we have training points
and we do something in between.

But if that something in
between has a different meaning

than it's all its
neighbors, it has a problem.

And if it's way we we have
no idea, it has a problem.

It's so interesting.

And we we could probably have
an entire conversation about

that alone because obviously
there's a lot of talk about AI

right now in the news, you know,

and are we or aren't
we in AGI and so on.

And the reality is
these corner cases,

there's there's miles to
go before we sleep on fully

general purpose AI.

Absolutely.

But but rather than get
sucked down into that tar pit,

why don't we talk about there's
more to safety than crashes?

How's that?

I think it's great.

Let's do that and that'll be
our that'll be our wrap up.

This has been an
incredible conversation.

Okay.

So the next one is there's
more to safety than crashes.

Now this was a surprise to me.

It wasn't really
top of mind for me.

When we started, the narrative
was it's gonna save lives.

It's gonna have fewer
crashes, and and that matters.

But once you get on par,

let's say that robotaxis
today are on par for for human

drivers, maybe a little
better, maybe the same.

The the truth is for fatalities,

we don't have the data to really
know how that's gonna turn out.

Let's just say that that's
where we're going to end up.

There's no reason to believe that
won't happen with with due attention.

There's more to
safety than that.

And the more to safety
than that is, for example,

if you stop and block a fire
truck or you block you block an

ambulance with somebody
having a heart attack and that

ambulance can't get through,
that wasn't a crash,

but you can't say that
has nothing to do with safety.

Or you're running
by school buses.

The school bus has a stop sign
out and it's extended so you

know that you have to stop and
stay stopped and your car blows by.

Your robotaxi blows by.

You didn't hit a kid that time,

but there's a reason you're
supposed to stop because it's a

chaotic environment that no
driver is good enough they can

go at full speed and
stop if the kid appears.

You don't you don't have
the reaction time. Right?

So you're stopping for
the safety of the kid.

You can't say that, well, I didn't
hit a kid this time, so it's okay.

I mean, that
doesn't work. Right?

If you're not stopping,
it's a safety problem.

So there are a bunch of things
we found with safety that

ordinary folks get very
upset about safety,

and it has nothing to do
with crashes per mile,

which is the metric
the industry likes.

So as we scale up,

we're gonna see more of
these kind of incidents that matter

to stakeholders,
especially municipal

stakeholders, parents,
things like this,

that have to do with safety
that really have nothing to do

with crashes per mile.

So as the robotaxi
industry scales up,

they're gonna have
to deal with that.

And there was one more on my
list I wanna make sure we get

in, which is
accountability for harm.

So right now, if you're a human
driver and you break the law,

you get a ticket, you
could lose your license.

If you hurt someone
while breaking the law,

you can be sued wrongful death, those
kind of tort, civil tort, negligent suits.

In many states, robotaxis well,

in no state can a robotaxi
be sued for tort negligence.

It's product liability,
completely different.

But in a lot of states,

the person on the hook for traffic
tickets is not the manufacturer.

It's the vehicle owner
or the vehicle operator.

So if you go buy a fleet of 10
robotaxis instead of buying a

condo to rent it out and you
rent out your robotaxis and

they get traffic tickets,

as near as I can tell, that
goes on your personal license,

and you had no ability
to control the driver.

So there's a lot of really
funky stuff going on,

and it's the robotaxi industry
past lobbying for state laws

that blame everyone but them.

Yeah.

And and with only a few
robotaxis and they're operating

their own fleet and they can
just pay out of petty cash the

tickets in states some states
they can't even get tickets,

but it's really no big deal.

But if you want to scale
up to a million robotaxis,

you're going have to have
a plan for what happens.

They're not going to be perfect.

There are to be crashes.

That doesn't make them bad,

but you need accountability and
a way to resolve the crashes

that make sense and that
incentivizes safety in the

future instead of what we have
now, which is just a mess.

You know, really merging
those two points.

I think those are such
interesting points.

Mean, the first the first point
you made about was how crashes is

not the only failure.

And indeed, you know, the
industry says, you know,

they're much safer
than human crash rates.

And as you say, we
don't exactly know,

but it could be true
that that is the case.

And yet, there are still unsafe
things that are happening that that

just so happens they
get they get lucky.

That's kind of where we are now.

That is kind of where we're now.

The industry says
they're safer on crashes.

We can have a
discussion about that,

but even if we
stipulate that's true,

what about not stopping
for school buses?

I know. Exactly.

And I'm I'm reminded
back in CMU,

I took a class called
engineering and public policy,

which was one of the most eye
opening classes I've ever taken.

And and they were talking about
the space shuttle Challenger

for those of you
who remember that.

And the the point that they
made during the class was that

really that the the NASA at
that time was playing a game of

Russian roulette.

And the saying from from the
from the report on the space

shuttle Challenger disaster was
"when playing Russian roulette,

the fact that the first shot
got off safely is little

consolation for the next." Yeah.

And so the fact is these
unsafe things happening and yet

not causing your problems means
eventually there's going to be a bullet.

I like to say that "getting
lucky doesn't scale."

There you go.

And then the second point you
made was about accountability.

And and there's no doubt
that this is a an active

conversation that I've heard
many times that there is a well

understood and yet not
addressed gap in this liability

issue because if these
things are going to scale,

I think there's going to need to
be some reform of how this is done.

Because the model right
now is not scalable.

It's going need to be addressed
in some way or other that's

going to make this viable
because the current approach is

clearly not viable.

And it's going to take
political will and probably,

unfortunate to say,

some bad situations happening for
people to wake up to what

you've what you've what
you've pointed out.

And I think we should solve it
because I think robotaxis are

are a public good and that they
they can and will be beneficial,

but that we're not all the
way there yet to that problem.

I'd like to get ahead of the
problem so that we don't have a

lot of folks who get unlucky
early on having to bear the

brunt of the burden of
reforming the system.

I'd rather get it performed without
having to go through that process.

I fully agree and and I
hope that that's the way that the

the politics goes.

Look, this has been an
incredible conversation, Phil.

I've learned so much.

I hope our guests have enjoyed
this mini course in safety

engineering and about robotaxis.

Look, there's still plenty to to
be happy about with the incredible

progress and for you to
have been doing this thirty years

and see how much this has
come must be very exciting.

But you've showed us that
there's still more to do.

I I think that the future
can be very bright.

There are some challenges
for scaling up,

and the way to succeed is for
the industry to acknowledge and

and face the challenges.

That that's how everyone
wins. So thanks for having me.

Really appreciate it.

Thank you so much
for joining us.

If you like what you're seeing,

please like and subscribe to see more
episodes like this from The Garage,

and we look forward to seeing you
in another episode again very soon.