AI at the Wheel: Unpacking Comma.ai's Hands-Free Future

We spoke with Harald Schäfer, CTO at Comma.ai, about their driver assistance car system called Comma 3X which runs on the open source software openpilot. This driver assistance system allows drivers to equip their own cars to enable hands-free highway driving by mimicking human driving behaviors. Schäfer explained how openpilot learns entirely from human demonstrations without hardcoded rules, and why this end-to-end approach can scale better than classical approaches. He also discussed Comma's in-house hardware, their direct-to-consumer business model, and their vision to build general purpose household AI robots. Check out this fascinating interview to learn more about Comma's innovative driver assistance car product Comma 3X.


COMMA.AI

https://comma.ai/
https://discord.comma.ai/
https://github.com/commaai/
https://twitter.com/comma_ai


FRY-AI.COM

https://www.fry-ai.com/subscribe
https://twitter.com/lazukars
https://twitter.com/thefryai

Creators & Guests

Host

Ryan Lazuka

The lighthearted Artificial intelligence Journalist. Building the easiest to read AI Email Newsletter Daily Twitter Threads about AI

What is Behind The Bots?

Join us as we delve into the fascinating world of Artificial Intelligence (AI) by interviewing the brightest minds and exploring cutting-edge projects. From innovative ideas to groundbreaking individuals, we're here to uncover the latest developments and thought-provoking discussions in the AI space.

Harald Schäfer: I mean, as for my background, I'm not sure there's all that much to say. I mean, it was kind of tinkering this stuff and building stuff and then decided to get an engineering degree. And then after that, I came to America, had moved to California, and did a graduate degree in electrical engineering. And kind of always had like a vague interest in AI.

I mean, it always kind of seemed like it could be more and more relevant as time goes on. But didn't really do much with it. I mean, I was working on electrical engineering projects. I worked on some side projects and won some competitions in college that were kind of unrelated to AI. But then the first job I applied to after getting my graduate degree was to be an intern at Coma that was back in 2017. And yeah, basically, I got that job. And I was working as a GPS engineer at Coma in the beginning. I did that for probably a year or two doing classical slam algorithms and classical controls, that kind of stuff. Then graduated into machine learning and eventually became CTO here, basically running all of the machine learning stuff. And I mean, we do pretty sophisticated machine learning at Coma. Not sure how familiar you are with exactly what it is we do or how the system works. But yeah, I think we're we try to build like an end to end machine learning driving system, which means learning to drive by watching humans drive. So there's no hand coding, no rules, stuff like that. And so far, it's been pretty successful. I mean, what we've shipped is truly end to end and has learned how to drive from human behavior. So that's kind of a short run down, I guess.

Ryan Lazuka: Yeah, no, it obviously research your project before you come on. But it's also good to have you explain things like we don't know what it is just for the audience. But it's pretty incredible.

I told Hunter I was blown away by what you guys are doing. So from my understanding, the Coma is the company. And then the technology behind it is open pilot. Is that correct?

Harald Schäfer: Yeah, exactly. So I mean, as a company and we sell devices, like our most recent product is a comma 3x, which is a device you can install in your car. It runs open pilot and it will give you essentially test the autopilot like functionality on cars that would otherwise not have it like Toyota Corolla's or Honda Civic stuff like that.

And yeah, it gives you so I mean, no control gas break steering and can basically drive for you on the highway. I mean, it's level two. You have to pay attention. And we are trying to expand that similar to Tesla FSD. I would say our approach is quite different though. As of recently, Tesla has kind of pivoted towards more end to end. Not really clear to me yet what they mean by that. But it seems like they're going more in the direction that we've been aiming for for a long time. And it seems like they're making a lot of progress there.

Ryan Lazuka: Yeah, no, that's crazy. So just to be clear, someone can use the code on open pilot, for example, put it on their own computer and run this in their car right now if they wanted to.

Harald Schäfer: Yeah, I mean, in theory, it's not quite that simple. Like OK, you set up to support it and then you have an interface to actually connect to the car and you do like a camera that looks similar to the cameras we use. But I mean, basically, yes. And I mean, it's licensed so that you can do that. But there's some like practical implementations on running it on arbitrary hardware.

Ryan Lazuka: OK, so does Kama offer a product that the end user could buy to make it simpler for them?

Harald Schäfer: Yeah, so that's the comma 3x. So that's just out of the box. You just install open pilot. Everything works. That's that's that's what we sell.

Ryan Lazuka: So you're saying in the US right now, or I don't know if it is it available in the US or is it available?

Harald Schäfer: I mean, it's actually quite interesting that you know what open pilot is and you don't know. I mean, this has been a recurring theme, I think. You know what open pilot is, but don't know that we sell a product that runs it. And I really wonder why that is, I guess.

Ryan Lazuka: I don't know. Well, it's because you I guess, from my perspective, is like I hear I hear things about self driving cars with Tesla, other big car companies. But to hear that you can buy a product and then put it in your car to self drive when Tesla doesn't even have that perfected yet.

And I know that I'm sure you guys are it's always a work in progress. But I was like, how is this not a bigger story than than I've heard about before?

Harald Schäfer: Because it came out of nowhere. But I mean, to be clear, it's not a self driving car, right? It's it's it's Lane. It's a level 2 8S. OK, that's a autopilot. But just to give you some information like so out of our user base, I mean, I'm just looking at our own website now and so yeah, 56% of miles driven are driven by open pilot with users that have the system installed. So it drives a lot of the time. Humans have to pay attention and take over when it makes mistakes. But it is clearly very actively used and useful.

Ryan Lazuka: I mean, is it true that like self driving cars, nobody's has that right now, right? Like it might not happen for a while.

Harald Schäfer: Oh, yeah, I mean, that's my perspective. It depends what you mean by a self driving car like crews and waymo or driving cars with no actual driver in the driver's seat.

OK, I'm just but there's a long list of caveats. Whether you call that a self driving car is kind of depends on what you define. I would say no, because they're supervised remotely.

They operate under caveats that I think kind of deal breakers. But yeah, I think there is no legitimate self driving car and there's no expected to be any time soon is my perspective.

Ryan Lazuka: But if you got 56 percent of the time driving, that's pretty awesome.

Harald Schäfer: I mean, sure, but it's like 56 percent is great. And that's a very useful product in which is why I use it like using it. But, you know, even if you're at 99 percent, if you want a real self driving car, you need 99.9999, right? Sure. Humans don't get into accidents that often. And you need extremely high reliability to replace that.

Ryan Lazuka: I guess my point is just the fact that you can 56 percent is a pretty large number and for not a lot of people know about you guys. And I think that's a huge thing. Like someone could go out right now, buy your product and have their car be self driven by you.

I know I'm using the wrong terms probably. But assisted 56 percent of the time. That's pretty significant and pretty awesome. I mean, I don't think like I think the large majority of the public does not know that this product even exists.

Harald Schäfer: Yeah, no, I agree. Which more people would know about it. So how long have you been with the company? I've been with the company a little over six years. And do you know how long the company's been around? Yeah, I think it would be like maybe two years prior to me joining.

Hunter Kallay: And then you mentioned that you came to the United States. Where did you come from? Where are you from originally?

Harald Schäfer: So I'm from Belgium originally. So I went to university there and stuff. And I moved here for like my graduate studies on a scholarship and kind of went from there.

Hunter Kallay: So there's obviously two sides to this. Like there's this AI component behind the scenes component. And then there's some sort of hardware component. Could you just tell us about each of those components? What does this hardware look like? Like if somebody gets your product, the comma three, and what does it look like? How do they actually hook it up to their car to get this working? And then what is happening behind the scenes to get this work? Sure.

Harald Schäfer: Yeah, I can give a quick overview. At some context as well as like when, you know, comma I started that the big thing that we were standing behind, which is that this is predominantly a software issue, that the reason we don't have self driving cars is not because we don't have good actuators or good hardware or good computers. It's because we don't have the software to be a self driving car. So the focus at the beginning, especially, was to be driven by AI. And at that time, we were essentially using Android phones like one plus three T's and later the equal the pros, which we were repurposing, flashing with new OSes and running open pilot on them and selling that with some additional electronics.

Now we sell something that is purpose built and, you know, it's completely built in house and actually designed to like run in your car and stuff. But in essence, a lot of the parts are still there of it being essentially like similar to a mobile phone in that it has, you know, central computer. It has connectivity.

It runs, you know, some version of Linux. It has cameras. It has microphones.

It has speakers. And then that interfaces with the car in, I mean, something that we call the Panda, but it's essentially some kind of can interface that allows us to read messages from the cars can bus and send messages on the cars can bus. And the way it worked on most cars that we support is there is a camera mounted in the windshield that will send messages to the car to steer left, right and to break to use the car's own like a B system, a C system and L cast system. And so we can read those messages, reverse engineer them. And then when we run open pilot, we block their messages and send our own, essentially having full electronic control of the car while still maintaining all their APIs. So all the safety systems that they implemented for themselves are still completely active when we use it.

And that's a really important distinction. There are ways you can, you know, electronically control a car that you definitely wouldn't want to drive on the road. And we don't do that. This is the way, you know, these APIs are intended to be used.

Ryan Lazuka: I mean, there's a huge safety concern, right? Like, so are there any kind of regulations that you guys have to work with in order to put your product out on the road?

Harald Schäfer: So not so much regulations as like specs and guidelines, like there's an ISO specs. And yeah, I mean, they're extremely clear about what they recommend. And we follow all those recommendations. I think almost all of them are very sensible.

And I can give you examples of what those are. It's like, so we have a safety model that we, you know, abide by that we like came up with, and it says a few things, which is if you take over control of a vehicle, you should always have control and not be resisted. So control should always be given back to the user when the user wants it.

That makes total sense. The second is the car should never react so quickly that if the user notices it doing something wrong, it doesn't have time to take over. Like for example, you could imagine if you turn the wheel so quickly, even if the human was paying attention, they couldn't safely react.

So that's also, it shouldn't react so quickly. And then the last one is that a human must be paying attention at all times. And we also enforce that by monitoring the driver, the head position, their eyes open, closed, et cetera. And all of those things are in some way also worded in the ISO specs. They'll say things like the car cannot jerk laterally faster than this much over this amount of time. And we also enforce all those limits. Okay. There's a whole spec. There's a whole guidelines. And usually people respect them and we do too.

Ryan Lazuka: But there's no legal things that you have to go through to test the software through government regulations or anything like that. It's all just recommendations. Is that correct? Yeah, exactly.

Harald Schäfer: They're recommendations. They're sensible and generally people in the industry follow them. That's kind of how it works.

Ryan Lazuka: So for example, when you say if the car was about to get in an accident and the person's behind the wheel, but he doesn't have his hands on the wheel, it's from what you're saying is the program open pilot's not going to jam on the brakes super fast because it's going to be too fast for your threshold. So the person would have to sort of see what's going on as well and then slam on the brakes himself.

Harald Schäfer: There's some nuance there, which is there's some limits on how hard you can break in what's called like ACC mode systems, which is. is if it requires extremely hard breaking, what is expected is that you enter AEB mode, which is automatic emergency breaking, which most modern cars ship with and is active even when you don't engage any of the assistance systems. So even when open pilot is disengaged, for example, on the cars that we support, the systems AEB will still kick in. So that in the spec also, if you use that, you're allowed to use very high breaking. There are some more different caveats there, where you can never really be jerky under any conditions is with the steering.

Ryan Lazuka: And so what is the use case most of the time for the Kama system or for opening open pilot underneath it? Is it mostly for, is the best use case for driving on the freeway it sounds like?

Harald Schäfer: Yeah, yeah. So it's a convenience feature. If you drive long drives on the highway, I mean, you'll hear people say this is how I feel about it. It just makes long drives on the highway way, way more comfortable. I can drive 10 hours now, no problem. And sometimes I travel, I get a rental car, and I have this attitude of like, six, seven hours driving, no problem. And I always immediately notice like, oh, crap, this car doesn't have open pilot. This isn't as great as I remember. So I mean, that's the use case. It's just a really nice convenience on long highway drives.

Ryan Lazuka: And what does it look like? So you get on the freeway with it installed, and how does it work? Like, can you just, does it shift lanes a lot? How does it, does it go in a fast lane? Do you set like certain speed limits? Like, how does it feel when you're actually in the car with the working?

Harald Schäfer: It works very simple to like cruise control in that you set like a cruise control speed, and that's the upper limit. And in the default way that we ship, it just slows down for lead cars and respects the speed limit you set. We also have an end to end mode where it like, use the gas and brakes the way things are human would, in which case it will slow down for turns, it will drive, you know, what it thinks is an appropriate speed, stop at red light, stop at stop signs. That's, we call that experimental mode.

It's not really that polished, and only like 25% of our users use it because it's not that polished. But yeah, so those are kind of the two modes. And the way you use it, I mean, again, we override the buttons of the stock ACC of the car. So it's the same as engaging cruise control. It's just when you engage now cruise control, it's like, it will do, you know, the whole thing.

It will do the steering, it will brake for cars in front of you, that stuff. So switch lanes, everything, and then when? So it doesn't change lanes unprompted. So the way it works is, it will stay in your lane, and then if you want to make a lane change, you put on the blinker and you touch the steering wheel, and then it will execute the lane change smoothly. But so you prompt the lane changes.

Ryan Lazuka: Got it. So it just sees that you're touching via camera, the steering wheel.

Harald Schäfer: No, there's like a sensor in the steering wheel on all cars, basically.

Ryan Lazuka: Really? Well, that's true. Like I know my wife's, we just got a new Honda Odyssey minivan, and you know, if you take your hands off, it'll say keep steering. So it sounds like that's what you're talking about.

Harald Schäfer: Yeah, I mean, it's just a torque sensor. There's like a, I mean, every steering on, with electronic steering, you have a torque sensor that measures human torque on the steering wheel to amplify it with the electronic steering, and you can just use that to detect like if the human's touching, basically.

Hunter Kallay: So is that kind of why it can't be used on some cars? What's the limitation that determines it can be used on this car, but it can't be used on this type of car?

Harald Schäfer: I mean, there's various limitations. Generally, it's just we need like a certain set of things, which a lot of cars have and some don't. So we need to be able to intercept the camera, basically, so that we can sense steering gas and brake commands. Part of the thing that's necessary for our experience is we don't want people to have to touch the steering wheel to prove they're paying attention.

We want you looking outside forward and being awake to be enough. Some cars don't allow for that. They hard code you having to touch the steering wheel every six minutes, for example, in their ADAS system, just random limitations like that. Some cars, the install is very inconvenient. You have to like plug stuff in under the dash or in the boot that makes it kind of like impractical as a product. So there's all kinds of reasons, really, but the surprising amount of cars are supported.

Hunter Kallay: So it's not necessarily an old car versus a new car. It's kind of these random little things.

Harald Schäfer: Yeah, yeah, it's kind of random little things. Your car kind of has to be newer than like 2016, 2017, because that's when they started building all these APIs in the cars to start accepting these messages. So generally that's kind of the threshold.

Most likely if your car has some kind of lane keep assist and like cruise control ACC, then it will be supported if it's newer, unless there's any weird stuff like I just mentioned.

Hunter Kallay: Okay. So you said there's like 56% of the time your users are using it. You mentioned it sometimes not being polished on certain tasks. So what is the limitation there? Where's the current work being done to kind of improve that 56% to maybe like 70 or 99?

Harald Schäfer: Yeah, I can just want to add to that, that it's a 56% of miles. And given that most of the engagement is on the highway, that comes down to like 30% of time. But yeah, anyway, with that said, so we want to build an end to end system that basically looks at human driving and learns how to drive like that. We want to do this by writing a simulator and have the machine learning train in that simulator. That means that however good your final system will be is limited by the quality of your simulator. So one thing you could do is you could train in a classical simulator like for example, GTA 5, you know, GTA 5 is a pretty good simulator, you can drive around and you can kind of figure out how to drive in that simulator, but it's not realistic. The road widths aren't real. It only shows American environments, whereas you want our system to work everywhere.

So that's a no go. I mean, you can build dedicated simulators which is what some companies do, but it's kind of this endless project like, okay, now you've got to add the India texture pack and all that stuff. So we want to build a simulator that's based off real data so that if we could just take in more new data, our system will become capable of driving in that kind of data. The way our simulator works now is essentially you take a video of driving and you can simulate like movements forward, back, left and right by estimating the depth of the scene and reprojecting it. And this allows you to deviate from where the human drove and then we can teach a model to recover from those deviations. Does that make sense? It does.

Ryan Lazuka: Is it, is your philosophy for lack of a better word to go about emulating human driving compared to, it sounds like your system is a lot different than other self-driving systems or self-assisted systems?

Harald Schäfer: Yeah, I mean, I would say the other ones that call them like rule-based classical in that they have some set of optimizations which is like stay between lane lines, stay behind stop lines, don't hit pedestrians. We don't really believe that kind of stuff scales. We think the best way to do this is to mimic human behavior and then maybe if you did that well, you can do some tweaks at the end, but I mean, we're nowhere close to that and neither is anyone else. So yeah, that is a pretty big distinction.

Ryan Lazuka: And when you say like simulate human behavior, that's based off the simulation that you put in the machine that you're talking about?

Harald Schäfer: Yeah, so there's more to elaborate there, which is that, so like I was saying, we simulate in the simulator where we reproject the scene, knowing something about the depth of the scene, but this is limited because the quality of that simulation is limited, first of all, by how good you can reproject and how much reprojection in general makes sense.

And both those things are flawed. We are not very good at depth estimation and you can't reproject the entire scene. Like you can't reproject reflections, you can't reproject the illumination of headlights and other lights.

So that's a big limitation there. And what you also can't reproject is counterfactuals. If you move towards a bus in the lane next to you, in the real world, that bus will react, but in our simulation, because we're just reprojecting it won't. So that's a limitation of our simulator.

And the biggest one is just in general, it doesn't look that good because reprojection isn't that great. And so when you ask, okay, why isn't it engaged more? That's the main reason. The simulation is limited in quality and we want to move to machine learning simulation, which is essentially just a machine learning model that just predicts video. And that's what we're working on actively. Other people are working on this too. And once that works, we expect the performance of our system to just skyrocket.

Ryan Lazuka: It's a really exciting time. Like it sounds like, yeah, once machine learning, it's better than machine learning, your product is gonna be way better, exponentially.

Harald Schäfer: So I can send a video if you just want to see it of basically what we're working on. That would be great. It's difficult to get machine learning models to predict video, right? But that is the goal. And once you have that, it should predict the entire distribution of like human driving that it's trained on. And so that's extremely powerful idea because then you don't need any engineers to hand code any rules to hand code any types of road, any of that stuff. So it's super scalable here.

Ryan Lazuka: So are you essentially writing your own like large language model for self-driving? Yeah, exactly.

Harald Schäfer: I just posted a Twitter link of a video we posted, which is just a video basically that's completely imagined by this model that we trained. And that's kind of what we imagined this future of simulation to be. And you can see in the video, it's clearly not perfect. So, you know, but it's pretty good. You can see the basics of what it's doing. And that's kind of the future that we're working towards.

Ryan Lazuka: And what's the backend tech look like? I know you're the CTO, so you have an extensive background in this area.

Harald Schäfer: Yeah, I mean, so what do you mean with the backend tech? Like how we train these models?

Ryan Lazuka: Like what's your code? What code do you use? What's the, I mean, it's all open source, but like what's the tech stack behind everything?

Harald Schäfer: Yeah, I mean, it's so open source. We mostly use, I mean, the training stack is not open source, by the way, but we just use like kite. I mean, it's all pretty standard stuff. We mostly use Python. Some of the time critical and runtime stuff is written in C.

Ryan Lazuka: Is there any like, is there any calls to the cloud or anything like that? Or is it all locally on the computer, like in the car?

Harald Schäfer: No, there's no calls to the cloud. I mean, so everything on the device runs on the device, not the cloud. And then even our training, we have a local data center that, you know, we built internally that runs all the training. We don't train in the cloud, it's very expensive.

And I mean, so that's, there's some of the things that we actually, you know, are strong believers in is, and where we deviate from the norm is some, most companies train in the cloud. We think it's a really bad idea. First of all, because it's expensive.

Cloud can be up to an order of magnitude more expensive if you have 100% utilization of training GPUs over a year. But it also just incentivizes the wrong things. It makes, let's say you want to do a random experiment. You think some engineer thinks, oh, if we change this to the models, they'll perform better. Now, if you're training in the cloud, that's a pretty significant cost to do that experiment. And it makes experiments harder to justify. Whereas if you buy an entire data center and it's sitting downstairs and it's idle, now all of a sudden you're incentivized to do experiments because the alternative is that your expensive GPUs are not doing anything. And so I think it incentivizes the right kind of behavior you want in machine learning research, which is that somebody has some ideas, experiments are the answer, not like arguing about it and figuring out if it's worth the cost.

Ryan Lazuka: Yeah, I think you hit the nail on the head when you said, you know, what are those GPUs going on wasted? So it's like self-motivating right there to do something. That's cool.

Hunter Kallay: Yeah, exactly. Yeah. So something we ask a lot of people who we talk to is, a more general question about AI, apart from this project in particular, what is your general opinion of the direction of AI? And when I mean that, I mean, you can answer it as open-ended as you'd like, but some people have said, you know, we're going towards a... a time where AI is going to take these mundane tasks out of your life so you can focus on other things. Other people say it has more of a relational component of people having more relationships with AI, AI friends, AI companions, things like that. Where do you see the direction of AI more generally headed?

Harald Schäfer: I'm not sure of where I expected it to go, but I know that what I'm excited for is to have household clinging robots. Even a future where fallout, you've got your own little nanny robot roaming around the house, keep stuff clean, things up after you when you cook, cooks for you when you ask it to.

That's what I want. And that doesn't seem impossible, especially with recent innovations. This seems like, hopefully, we'll all die with robots taking care of the house. And that just seems so incredibly exciting. I've got one of those new Roborock vacuums that has all the AI and LiDAR and whatever.

It's cool. It's not actual AI by any modern standards, and it can be way, way, way better. But even that makes me happy. So if that's the future, I think it's possible someone's going to build it. I don't know how big of a part of future AI innovations it will be, but that's the part I want to work towards. And that's what I'm excited about.

Ryan Lazuka: Yeah. I've got three little kids of my own at home, and our house is a mess. So that sounds amazing to have someone clean up after everyone.

Harald Schäfer: Yeah, no, exactly. And I mean, you buy these clean robots now, and it just always feels like you're babysitting them. And less and less so over time. I mean, they used to be worse, but I expect, you know, with the innovations that we're happening now, in years, they will become incredibly reliable and hard to get them to mess up, which would just be incredible. I'd pay $20,000 for a household robot that cleans the house and doesn't have issues like at least once every few months. I mean, that sounds amazing.

Ryan Lazuka: Well, I mean, paying a cleaning lady once a week, it might equate to that much money. Anyways, yeah, and it comes with all these

Harald Schäfer: other issues too is, okay, you got to talk to them, you got to make sure they're happy. Yeah, yeah.

Ryan Lazuka: But you guys have a little, I saw you have a little robot too, right?

Harald Schäfer: So we have what we call self comma body. And so originally, when the company started, our mission was to solve self driving cars while shipping intermediaries. And when we say self driving cars, we mean, build some kind of AI solution that will, you know, drive itself in a car. And that's much bigger than like encoding the traffic rules, we've always imagined some kind of end to end system, like how do you learn driving from scratch in a sensible way, right? And in our case, that means, you know, start by looking at how humans drive and you start to imitate them.

That's how we learn how to drive to, you know, you drive with other people, they supervise you, all that kind of stuff. And so now, you know, we've made a lot of progress on that and the rest of the world has too. And we've expanded that long term scope in like solving robotics, while shipping intermediaries. Self driving is a subset of robotics. And in general, in both machine learning, anywhere, when you move to more general approaches, you tend to see better performance.

It's like more work and long term. But, you know, there's all these models previously like language models that would do stuff like, you know, they would do legal help, they would do medical help. And all these models and all these companies look ridiculous now, in comparison to chat GBT.

And chat GBT is just a general language predictor. And I expect the same will happen in robotics, which is that general purpose robotics AI solutions will solve self driving indoor navigation, household cleaning, all better than any specific solution ever has until that point. And so that's the line of thinking that we're long. And also why we're like randomly experimenting with like household robots, because it makes us think like, okay, what is the general purpose approach here that will solve all these problems better than anything specific ever has, which is what I expect will happen.

Ryan Lazuka: I mean, it's okay, because your your the technology for driving cars can be ported over to robots pretty easily in a way, like once you

Harald Schäfer: get at that point, I mean, the you know, the basics is it's going to be video based, you know, humans navigate the world most with your vision. So you're going to have some kind of machine learning model that takes in video has some general understanding of physics and the scene around it, and we'll make some decisions based on that. Obviously, it'll be somewhat different when it comes to indoor navigation versus navigating on roads. But there is more similar than there is different.

And generally speaking, when you generalize more in machine learning, you get better results. And so yeah, we built this robot. And we built it as kind of an experimental thing to work with that to think about robotics, to think about which solutions would solve would work for both self driving and robotics, and can they be, you know, better than what we were relying on so far. And so that's why we did it. And yeah, so this weekend, we're doing a hackathon with them, we've got like, for any of those little wheeled robots, they don't really do much by themselves, they basically run open pilot, and we'll get people to kind of tinker with them, we've got like a little, you know, route through the office, we want them to drive as kind of the challenge for this weekend.

Ryan Lazuka: Is there a link to that as well that we can post? Or that's this weekend, I think it's going to be too late.

Harald Schäfer: Yeah, that's this weekend. I mean, it's like a, it's like an invite event, invite only event. But we'll try to post some like videos of, you know, how people did and the prizes and presentations at the end for what people did, I think we'll post all of that on our Twitter. Awesome. Okay.

Ryan Lazuka: And for someone that wants to buy your product, like what does the install look like? Is it sort of like installing like a dash cam or something like that? Can people do it themselves? Do you guys recommend someone install it for you?

Harald Schäfer: How does that work? It's actually like a dash cam. It's pretty much the same. So you just like stick it to the windshield, and then there's like one cable you plug in by the camera, which just requires moving the trim. I mean, if you watch the instructional and you've got a normal car, it should take like 20 minutes. And then you route a cable down to the OBD2 port. It's really, it's definitely meant for people to do it themselves. It's not that hard.

Ryan Lazuka: All right. And can we ask you how much this costs as well? Yeah.

Harald Schäfer: So let me just go to the website. So I don't know. But yeah, it's $1250 now. And then do you need like a special cable for each car? And that's like $200. So all up, it's $1450. And then it should be the car.

Ryan Lazuka: I mean, for especially if you're driving long distance on the freeway, a lot of people I think could put good use to that.

Harald Schäfer: So yeah, we have a 30 day no questions, ask return policy. If you want to try it, please, we can even try it on rental car if you're just curious. All right.

Ryan Lazuka: And how does it work in terms of like firmware update? Well, it's not, yeah, firmware updates and software updates. Are those done over the over the wire? Or like, how do those get updated?

Harald Schäfer: Yeah. So I mean, just connect it to Wi-Fi and it'll update. It doesn't, it doesn't modify the software in your car or anything like that.

Hunter Kallay: I had a question about, you mentioned before and then you mentioned again, how there's kind of two way, there's a way you're doing things differently. And that's by trying to replicate human behavior rather than giving them a set of rule, like giving the AI some sort of set of rules, like stay between these two lines or something like that. And he said that that scales better. Could you just kind of reiterate or go into a little bit? Why do you think that that scales better than maybe a rules type system? Andy, are there any safety concerns with mimicking human type driving rather than some rule type driving?

Harald Schäfer: Yeah, I'll answer both of those. So first of all, I can give you an example, right, which is you want to drive through an intersection. Your lane is clear. The light is green. There's a 45 mile an hour speed limit. Okay, no problem, right? You keep driving on your lane.

No issues. What if there's a pedestrian, you know, waiting to cross? Now, if that person's wearing a suit, right, and they look calm and they're just standing still, you know, three feet away from the road, you're not going to bat an eye.

That's normal. He's waiting to cross. But depending on the demeanor of that person, any normal human driver is going to react. If that person is like dancing around and looks like they're on drugs, which I mean, you know, I don't know if you live in California or not, but that is the thing that happens here.

Around me here. So I get it. I mean, you're going to react differently. Or if that's a child, you know, that's very young and looks like they're looking at some ball or something they drop on the highway, you're going to react to that and you might consider changing lanes or driving slower or just slightly erring away from the sidewalk.

And that's stuff that really is relevant to driving and is important. And it's just a random example. There are many examples like this, that you cannot really imagine which kind of rule set would encode that.

So that's the answer to why we don't think that scales. Like, if you have a rule based system and you can see when people work on this, they just keep adding conditions and they keep adding things. And of course, they never got so far as to differentiate the kids from the adults.

But if they kept working on it at some point, they would and they deal with a lot of nuances now, they're much less complicated than that, right? Which is, you know, you have to follow lane lines, what I usually do, but sometimes there's constructions, sometimes there's an exit and the lane lines diverge, sometimes there are wrong lane lines, like there's all sorts of stuff.

Ryan Lazuka: So are most the technologies that your competitors out there or the self-driving or self-assisted cars that exist right now, are they all rules based? Are you guys one of the only companies that are simulating human behavior?

Harald Schäfer: Oh, yeah, we're definitely the only one that's shipping to any consumers that's simulating human behavior. Tesla recently is claiming to do the same thing. They're doing it somewhat differently. I don't fully understand. They haven't released full technical details and they haven't shipped it to users yet, but it does look like they're pushing the same direction, which I think makes sense. I think they're at a good point to do that. And then there are a few other companies that are like us and they say fully end to end all the way, no alternatives, but they don't have any products yet or really anything you can interact with except like demos. Awesome.

Ryan Lazuka: So you're one of the only players out there, which is great.

Harald Schäfer: I want to answer that second question too, safety, which is, comes up a lot, one of the reason that, I would say the older conservative people in this space don't like this approach. Is there like always a black box? How can you guarantee it works?

And I think that's kind of a bad argument, because that's not how humans work either. How can I guarantee that you're going to drive well when you're on the road? Well, I can't. We have some tests that we do on people and some expectations and by how they perform on those tests and how they perform in our evaluations, we make assumptions about how they will drive most of the time. And you can make more assumptions about humans and how their brain works than you can probably do about AI's. But with some statistical tests, you can totally come up with a pretty confident statement about how the system will perform.

Right? You can simulate it. You can simulate it in all kinds of driving conditions that you're worried about. And you can do that on enough data so that you can have the confidence, okay, look, it's not made mistakes for this amount of time. And in all these adversarial cases, I'm confident that it's reliable.

Ryan Lazuka: Because you could statistically analyze comma, if you just say comma is a general driver and then compare them to any other driver out there on the road, you could say by statistically, From stat, you could say, oh, they're better than 99.9% of all drivers on the road, something like that. Yeah, exactly.

Harald Schäfer: I mean, we can't now, but if you have a system that's a complete black box, but happens to be a perfect driver, I am very confident the issue won't be proving that it's good.

Ryan Lazuka: And like back to what you talked about earlier, like driver assisted cars, or I don't know the exact technical terms, but like for example, my wife had a Honda CR-V, and in it you could put on cruise control, right? Sorry, you put on cruise control and it would sort of keep its distance from the car in front of you. So if you put on cruise control at 80 miles per hour, I mean, everybody knows this, and the car in front of you slows down to 70, we're gonna slow down to 72, because it's gonna keep its distance. It doesn't change lanes, but it will stay in its own lane. What's the difference between what comma offers, because you guys don't switch lanes, and what's already out there, like built into say a Honda CR-V right now?

Harald Schäfer: Yeah, I mean, so they're basically a same feature set. I think there's just a massive difference in quality that it's hard to put on paper, but I think the numbers should speak for themselves.

Those people that have those CR-Vs, they may use those systems sometimes, they definitely don't use them, 50 to 60% of the miles that they drive. And I think that should prove that, there's some pretty significant difference there. But yeah, it's really just reliability. On the highway, our system is incredibly reliable and incredibly comfortable. It's not to the level of reliable that you don't have to pay attention, but it is to the level of reliable that you can, sit there pretty comfortably, have the system drive, while you just look out for any anomalies.

And that's a pretty big upgrade, right? If you, I don't know if you've driven much with these other systems, but generally speaking, it won't take long before you end up in a situation where it slams on the brake, because the car, lead car is going a little slower, but you would have hit the brakes later, for example, because it's way more comfortable, stuff like that. Like it's just like this minor stuff, where you just get annoyed very quickly when the system's not that good and not that polished.

Ryan Lazuka: Gotcha. And can you take your hands off the steering wheel with your system? Yeah, yeah. The problem is you're... There's a huge difference to what's built in the car, is that right?

Harald Schäfer: Yeah, I mean, it's hands-free for hours, as long as you're paying attention and look like you're ready to take over.

Ryan Lazuka: All right. And if something does happen, is like, what's the, is there an interface on this product, or is it just a black box, or do you put in your car?

Harald Schäfer: Like is there a screen or anything like that? There's a screen, yeah. I mean, if you, I don't know if you can open a new tab, but you can just open our website, and it should be, you should just see a picture immediately of how it works. Awesome.

Ryan Lazuka: And like, so there is a screen, if there is any kind of danger on the road, is that flash for users or?

Harald Schäfer: Yeah, if there's like a reason to take over, so like we have four collision warnings, or if there's an issue with the system or whatever, it will flash red and like beep at you. But yeah, the interface that you, you know, manually have with the cars is the same, as it normally would be. Like if you turn the steering wheel, you, it will let go of the torque and you'll have control. If you press the brake, the system will disengage, stuff like that.

Ryan Lazuka: And so if you, if I take over the steering wheel, then I let go of the steering wheel, it just automatically re-goes back to.

Harald Schäfer: Yeah, there's some smooth blending that it does. It will slowly release torque when your hands are on it, it will slowly re-engage it when you release. Yeah. Cool.

Ryan Lazuka: I mean, I, instead of asking you questions, I'll just have to buy one of these, test it out, and see how it goes.

Harald Schäfer: If you're excited about AI and stuff like that, it's, and you want to have some fun on the weekend, you know, rent a, support a car if you don't have one, you know, try it out. If you don't like it, you send it back. All right. Yeah, sounds awesome.

Hunter Kallay: Definitely exciting. And your website is comma.ai, right? Yes. Make driving channel. Yeah, it seems like there's a lot of stuff on the website that takes you right through like all the different cars, the videos of the cars, kind of good breakdowns or everything.

Harald Schäfer: Easy to buy. And we're struggling with this issue of people knowing about OpenPilot, not knowing that you can buy the device, which is funny because it happened to you guys too. I mean, I don't know what we can do about it. Cause as you can see, we've put effort to changes in the website, but I don't know.

Ryan Lazuka: Well, I think I found you guys on GitHub, like the, one of the most trending AI projects out there. So that's how I found you. I found you in the reverse way, you know?

Harald Schäfer: I see. I think that is more common way. I mean, we've got kind of more clout in that community, I guess, than in the consumer electronics community.

Ryan Lazuka: Once your name gets out there, I'm sure that will change. What about like, aren't, I'd imagine a lot of car companies are reaching out to you guys to sort of want to buy you out or have you partner with them. Is that true or no?

Harald Schäfer: I mean, car companies reach out to us sometimes, generally not for reasons that excite us or sound like good ideas.

Ryan Lazuka: Well, they got to get their act together. I think that would be a good partnership.

Hunter Kallay: So you're more concerned, you're more concerned about selling straight to the consumer. You're not going to planning on, at least as a company, planning on working with any companies in the future or manufacturers or anything like that.

Harald Schäfer: Not saying it should never happen, but we're not really planning on it right now. I mean, there's so many advantages to direct to a consumer. One of the bigger ones is, if we were to ship Popon Pilot on a car in a similar price range that we mostly, I mean, that we're mostly selling to now, we would have to make so many sacrifices in the quality of the experience. And, you know, we would have to make sure how to, we've had to figure out how to train smaller models that can run in the compute limitations of the models that these cars would have. All these compromises that kind of take away from our mission of trying to iterate faster on some really good product, or some really good AI product that people enjoy using.

And those categories are hard to get around. The other problem is their release cycles are just so tragically slow. Like if we work with them today, it would take like four years before that's on the road. And I mean, that means you'd perpetually have a four-year-old system, like that kind of doesn't really work that well for us. And I mean, Tesla, for example, has a completely different approach to this. They have, you know, OTA updates and new hardware and retrofits and all that stuff.

So they also don't like this. So yeah, that's kind of why we, and also our software is MIT free and open source. We think core companies should use it and they should try it and they should put it themselves.

It's free, it's legal, it makes complete sense. Companies like Lucid come out, you know, they advertise this great ADAS, they ship, the ADAS sucks, they have the compute to run OpenPilot and somehow they're wasting their time developing an in-house system. Whereas even if they thought they could do it better than us, at least start with what we have. Don't start from scratch and like, not ship anything. I don't know, it confuses me.

Ryan Lazuka: It's like more of an ego thing than anything. It sounds like.

Harald Schäfer: Yeah, I think it's something like that. It's hard for people to not want to do it in-house or something like that. Yeah. It totally makes sense for them to try our software. It's free, it's licensed MIT.

You know, try it. You don't like it then sure, but I think it makes sense as a car company, especially in a situation like Lucid where you have the compute and no software to start with something open source that works.

Ryan Lazuka: Okay, maybe one day. Maybe. What about like, so the software is free obviously, we went over that a lot already. What hardware, if you don't mind sharing, do you use for the system?

Harald Schäfer: Yeah, so I mean, like I said, it's pretty similar to a mobile phone. It has like a Snapdragon 845. It has like a nice mobile phone display. It has like GPS and connectivity chip. It has, you know, better GPS than a normal phone. It has like a high quality antenna. And then it has like three really good cameras that are like HDR for automotive use, very high quality, much better than a phone.

Ryan Lazuka: Is this a hardware you guys built custom or had built custom or is it something that was already on the market?

Harald Schäfer: No, no, it's completely custom and built-in house as well. We have like a circuit board assembly line and everything.

Ryan Lazuka: Okay, so it's not like you just went out and bought a cell phone and stuck it in here. This is something that's very proprietary on that aspect at least in you guys built-in house, which is really cool.

Harald Schäfer: Yes, so we used to repurpose a mobile phone with several modifications. That was like our product from years ago. And then, you know, with what we learned from that, we started designing one from scratch.

Ryan Lazuka: One more question, we're getting close to the end here, but are you guys VC backed? How did you get funding for this? Can you go into little details about that?

Harald Schäfer: Yeah, I mean, there's been some funding through the years. I forget what the total amount now is. I think it's like 18 million total, but I mean, we've had more revenue than we've had investment. And, you know, obviously we've built up a lot of valuable things like the circuit board manufacturing line, the data center. You know, we want to eventually have made more profit than we got investment. You know, our goal is to be self-sufficient and profitable. We're not really looking at raising more investment. We kind of want to be bootstrapped.

Ryan Lazuka: Well, you're definitely different than most of the startups out there. I mean, you've been around for a long time, but... That's the idea.

Hunter Kallay: Yeah, that's great. Was there anything else you wanted to talk to us or share with us or the audience or anything like that about the project we didn't talk about?

Harald Schäfer: Just reiterate, I mean, the reason that I'm really excited about is that, you know, this is a pretty modern machine learning solution. It learns from human driving. You know, there's no hand-coded rules. And yet we made a product that, you know, it's not super polished yet and we're working on that, but it'll stop at red lights.

It will stop at stop signs without us ever coding that it has to stop at red lights, ever coding what a stop line is. And you can buy that today with a free refund if you don't like it. So, you know, if you're into the kind of seeing what the state of the art ML products are like, I think it's pretty exciting. Absolutely.

Hunter Kallay: You can try out a self-driving car literally this week if you want.

Harald Schäfer: It's a new consumer product. I think it's exciting, you know, try it out if you don't like it.

Ryan Lazuka: In fact, we study projects every day because we have to write tools and stuff for the newsletter or newsletter. And your project really stands out. It's really cool. And I love people know about it. And I think once they do, it's you guys will be very successful. It's really cool what you're doing. Cool. Thank you.

Hunter Kallay: Yeah, there's a few times when we were researching different projects and either Ryan and I would get really excited about a project. And Ryan's the one that found this project. And he was he was really excited about it. I mean, Ryan doesn't get that excited about projects that often. This one he was he was really pumped up about.

Ryan Lazuka: So yeah, it's really fascinating what you're doing. So I mean, we're excited about it too. I mean, most of us are here, you know, a lot of times the office where all this kind of having fun and keep working on stuff. I mean, it's exciting to see this stuff come together, right? It's like building a consumer product in-house or doing these cool ML stuff. We've got a data center in the basement, you know, it's also cool to see the whole trajectory, right? It's like we started in a house and then, you know, you get a small office, you get a bigger office. Cool to see the whole evolution.

Ryan Lazuka: Yeah, you're growing slowly, like not compared to most of the VCBAC companies out there. So it's more power to you guys. I guess now's the time to promote or anything that you want to promote. The website is comma dot a I correct. I don't want to make sure I'm not screwing that up. Anything other any other kind of links or anything you want right now? Go ahead, feel free to promote what you would like.

Harald Schäfer: No, I mean, yeah, if you're interested in the project, I'd say just check out the website. You know, if you're interested in the product, you can check that out. Otherwise, you've got a GitHub, you know, we've got a lot of community contributors that either contribute a lot to our project, which is great. A lot of people learn a lot from it, which is also great. It's always growing. So if you're interested in this sort of stuff, just check out our project. Check out the website. I would say.

Hunter Kallay: All right. So yeah, check out the website comma, C O M M A dot a I try it out. Very cool project. And check out Ryan and I's newsletter, a newsletter fry hyphen a I dot com. We do weekday, a I news Monday through Friday, cool tools, community and latest stories. And a fun and easy to read, easy to understand way. And then we also provide some links and different things for you to look at. And then on Sundays, we do our deep dive articles into very cool AI developments, cutting edge stuff, cutting edge developers.