Subscribe
Share
Share
Embed
Deep conversations with the founders, investors, and operators building real-world AI - robotics, automation, industrial systems & AI infrastructure. Past the headlines, into how these technologies are really built, deployed, and scaled. Hosted by Bogdan Cristei, venture partner and former systems engineer.
THE OPTIM UPDATE
Determinism Is King: A Game Engine Built for Robot Training Data | Devrim Yasar, Lucky Robots
Guest: Devrim Yasar, Founder & CEO, Lucky Robots
Host: Bogdan Cristei
---
Bogdan Cristei (00:00)
This is The OPTIM Update. I'm Bogdan Cristei, and today I'm talking with Devrim Yasar, Founder and CEO of Lucky Robots. Lucky Robots built a game engine from scratch for one job: generating training data for robot AI. You describe a robot, a task, and a room in plain English. The engine builds the scene and runs the episode on real physics while recording every joint and every camera frame, so a policy can fail a million times in simulation before it ever touches a real robot. Devrim is a repeat founder. He built Koding.com to over a million developers, then built Superpeer and sold it to Skillshare. He started Lucky Robots after building robots with his kids and hitting the wall every robotics team eventually hits: there's nowhere near enough data. His CTO is Yan Chernikov, better known to 1 million YouTube subscribers as The Cherno, a former Frostbite engineer at EA, whose Hazel engine is the foundation Lucky Engine is built on. Without further ado, here's my conversation with Devrim.
Bogdan Cristei (01:00)
Hey Devrim, welcome in, man. So excited to talk to you today.
Devrim Yasar (01:02)
Hey, welcome. Same, same.
Bogdan Cristei (01:05)
All right, you ready to talk about simulation, training data, robotics, all of the above?
Devrim Yasar (01:11)
Yeah, of course. Always.
Bogdan Cristei (01:13)
Awesome. Well, before anything else: why choose the name Lucky Robots?
Devrim Yasar (01:18)
I started a couple of years ago, so it's been two years and some. I first started playing with robots when my last company was in the M&A process. It was being acquired, and when you are getting acquired you can't do anything. You can't add a new feature, lawyers are asking questions every day. It was that time, my son was two years old, and our home was a chaos.
Why Lucky Robots? At the time we were getting these Boston Dynamics videos or Unitree videos where they were just kicking the robots with sticks. And I'm like, don't do that, man. When they start kicking you back, it's not going to be nice. They're going to remember. And since I started doing simulation, in the end I thought these robots are going to be lucky, because we never need to kick them with a stick or hit them with a stick. So that's where it came from. And also I was in Austin, and when I was thinking through these things I was actually having sushi at my favorite restaurant in Austin, Lucky Robot. I think it all came together.
Bogdan Cristei (02:24)
Very good. So you're not new to startups, you've built a couple before. You built Koding to over a million developers and you sold Superpeer to Skillshare. Then you started building robots with your kids, like you mentioned. What happened in the garage that made you start a simulation company instead of a robot company?
Devrim Yasar (02:43)
I wanted to make a robot. That was my impetus. When the company was going through that M&A process, I knew that there was one more startup in me, but first and foremost I wanted to just play with the newest things. ChatGPT was just out, Sora, DALL-E, all of that. And when I saw those image models, where you can ask a question and the AI model would tell you there are three cats and one person, and the person is five meters behind, I'm like, okay, this is the time for general robotics. This is the moment. Of course the ChatGPT moment for robotics is far ahead, but at the time I'm like, the technology that we needed to get there has finally shown its face.
Then I decided to make a robot, and I thought, what's a hello world robot? I saw the chaos that my boy created at home and I'm like, okay, I wish I had a Roomba with a robotic arm to pick up these Legos and put them in a box. And I thought I could just do that. I'm a good software engineer. I thought it would take me a few weekends. Probably there are a bunch of tutorials and hello worlds online. That's how it started. And then I realized that there is nothing out there. I just couldn't get that simplest hello world out of project mode to an actual robot doing that in my home, because I realized that there is no simulation. You just need to do that in real life.
Bogdan Cristei (04:19)
Gotcha. Okay, so for people who don't build robots: a language model gets the whole internet to learn from. A robot gets maybe a few hundred hours of humans steering it inside a VR headset, most of the time. What does robot training data look like when you open the file, and why is there so little of it?
Devrim Yasar (04:39)
I don't want to offend anybody, but I have some strong opinions here.
Bogdan Cristei (04:44)
Feel free. We love hot takes.
Devrim Yasar (04:47)
We do teleop because there's nothing else we can do. Literally, I hired my first roboticist and I was playing with it. There was no Lucky Robots, there was no company. I just wanted to get that robot to do something. And that roboticist was actually working for a respectable robotics company. The year was 2024, and he told me, you need to buy a robot. Not a hundred-dollar robot from Amazon, but a fifty-thousand-dollar Franka Panda, because that's reliable. All right. And that robot is as tall as me. It's tall, it's heavy, and it's dangerous. All right, fine, let me buy this forty, fifty-thousand-dollar robot for no reason other than picking up toys after my boy. And it's not going to attach itself to a Roomba, it needs a proper base. All right, how much is the base? Fifteen thousand dollars. All right, sixty-five thousand dollars, cool. Then what? And then he said, you're going to sprinkle some toys around it in the living room, take pictures of those toys, upload them to Roboflow, and train your dataset, train your policy. And I'm thinking - I thought the R word, but I will skip that - is this stupid? Am I stupid? What's going on? Who trains an AI model with five toys? Ten, or maybe twenty-five. And am I spending sixty thousand dollars to have a clumsy robot that only works in my living room?
In that moment a couple of things clicked in my head. I come from game engine development. When I was at university in the Netherlands, we were making architectural visualizations, so that we could build a skyscraper and then you could take the elevator, go to the kitchen, change the floors, change the kitchen configuration. We were using game engines to do that. I was familiar with it, and I'm like, okay, these people need a 3D engine to simulate real life. Because otherwise, yeah, I can strap that VR headset on and sprinkle five toys, but then what? Then I can hire a teleop company and they can sprinkle a hundred more toys, or a hundred and fifty more toys. I think the reason we don't have general models today is because we haven't built the Matrix for robots. You need so many more living rooms. You need thousands, millions of toys to generalize these things. You need to simulate the lights. You need to set the building on fire, if that's what you're training towards. Without a simulation - forget about Lucky Robots - if someone doesn't do this, it's just not going to get done.
Bogdan Cristei (07:25)
Gotcha. So in plain terms: a customer types "a Unitree G1 in the living room, pick up the clutter, the toys, and put them in this bin." What then happens between that sentence and a dataset landing on their disk?
Devrim Yasar (07:41)
LeRobot, if you're familiar - Hugging Face has a project called LeRobot, and I think they've done an excellent job of standardizing that. Before they did that, everybody was doing their own thing. The dataset is actually pretty plain and simple. Imagine you have a spreadsheet and it has rows. The first row is the first frame of the picture: you are a robot, you're looking down, so you can actually see right there. And there are pliers on this table, and the robot is supposed to pick them up. So what is happening there? The robot is going to move towards it, and every time it moves it's going to take a picture, and it's going to have to record the actuators. This actuator is going to be at the zero point, the position. Then it's going to turn a little bit, and a little bit, and a little bit, all of those six degrees of freedom. And you need to record that exactly, right? It cannot drift. Let's say that every joint is at position one on the fifth frame, and this is what I'm seeing. That picture that I'm seeing and those states cannot drift. If it drifts, then your AI policy is just going to be bad.
What people are doing with teleop, when you put on that VR headset and start joysticking this robot around, that's the data that's coming in. Teleop data is also not perfect. They skip frames and this and that, but largely it's correct, above ninety percent. In the simulation, what we're trying to do is make it a hundred percent. Even if you start recording data for millions of hours, Lucky Engine is never going to drift.
Bogdan Cristei (09:23)
Got it. Okay, so let's talk about simulation. You tried Unity and Unreal first, right? What breaks when you use a game engine to make training data? And at what point did you decide the only way was to write your own?
Devrim Yasar (09:37)
We were doing it in Unreal for the first seven, eight months of Lucky Robots, and I'm pretty familiar with how Unreal worked and works. The main problem with game engines is that they are not built for this purpose. That spreadsheet I'm talking about: say you have six actuators per robot. If you have four robots, now you have twenty-four. And if every robot has two cameras, now you need to capture that frame and turn it into a video. So you need to write all these states down, and then save the lidar data, the camera data, and everything. Game engines are not made for this. Game engines are made for you to play a game. If you're playing a multiplayer game and 50,000 people are connected, and you shoot at someone and your CPU spikes, maybe you haven't seen that guy's shot. But when the computer comes back, you continue from where you left off, and all those frames are dropped. And this little piece is very critical. If you do it on Unity or Unreal or any other game engine, they are optimized to skip these frames and continue with the gameplay.
Unreal sends you a tick function, literally called tick, and in that tick function, one sixtieth of a second, you need to save all the data. And Unreal is not made to save data. It's made to display graphics on your monitor. In our case, we don't care. Nobody's watching these things, we're just saving them. We are saving terabytes of data and then delivering it to the client. So I tried to decouple Unreal from that, and their physics engine is not MuJoCo. It's not real. It's a game engine. Game engine physics is like, you put your hand near the object and the object teleports to your hand, because game engines don't care about contact physics. Game engines don't care about any of this stuff. So to be able to use Unreal without the tick function, or to say, hey, tick function, wait for me to correctly record all of these states - that's another two hours of podcast. But 3.2 gigabytes of source code. We couldn't make MuJoCo a first-class citizen there. We couldn't really make Unreal wait for us to save all that data. Even saving the data is so foreign to Unreal, you have to jump backwards to make that happen. So then I was like, my God, this is not going to give us five terabytes.
And you have to imagine, robotics training, policy training - you've done a couple of SO-100s yourself - requires 10,000, 20,000, 50,000, up to a million episodes per task. So if you don't know, or if you can't determine, that your game engine is not going to drift... We are making that spreadsheet, and we are on the four millionth row, and if it drifted ten seconds, then you're not going to get anything out of this data. So we need to make sure that everything we record is a thousand hertz, ten thousand hertz, ten-thousandth-of-a-second precise. And we needed to do something other than Unreal or Unity.
Bogdan Cristei (12:53)
Gotcha. So you can't really have error propagation, because that's just going to destroy your data, more or less.
Devrim Yasar (12:59)
You don't even know when you start to have errors. That is the problem. The tool is not made for that. It's like with your car, right? Do you know when you're going 68.3 miles an hour? It doesn't say the point part. If you can never make your car say that to you, and you need that data, then you're shooting blanks and hoping that things work. And when things don't work, people go back to teleop. With Lucky Engine, if you've got five terabytes of data and you train a policy, you want to see 92, 92, 98. And if you can't see that, then you're like, yeah. So that's the short history of how everything came to be.
Bogdan Cristei (13:45)
Awesome. So you tried a bunch of things in Unreal, it wasn't working, and then you said, okay, I need to get myself an amazing game engineer. Somehow you convinced Yan Chernikov, who's an amazing game engineer, to come on board, and he brought some Hazel contributors with him. So explain maybe a little bit what Hazel is, and how you convinced him to write something for you that you couldn't have written from scratch in a year.
Devrim Yasar (14:11)
I'm a software engineer. I wouldn't be the top ten Stack Overflow guy, but I can come up with an Uber app. I will get you from A to B. Those are the types of apps I've done. With Koding we've done some advanced stuff, but we had some advanced people on the team. So I wouldn't put myself in that bucket, but I did write the first Koding. I could have written Superpeer, et cetera. But the level of software engineering that Yan and the gang brought - sometimes I check the PRs and I'm like, my God. I thought I knew software engineering, and then I met Yan. Game engine development is just a whole other level. These guys are looking at the screen pixel by pixel. What we did in our careers, over ten, fifteen years, we are just writing things to the database, basically. We're doing distributed systems, message queues, and we were so proud: my God, we just scaled the server to fifty thousand servers on AWS. Those are the types of software engineering I'm used to. But game engine development, my God. I think that's peak software engineering. I don't know if there's anything more than that.
So how did that come to be? I'm spending my nights and days with Unreal, trying to deploy the plugin we are writing, and I have millions of errors. I hadn't seen C++ for over fifteen years. Not my favorite programming language, to say the least. And I found a C++ tutorial on YouTube. I did not know Yan Chernikov. He has this hundred-episode series, and I was around the fifty-fourth. I wrote to my AI researcher team, like, guys, I'm going through this course and this dude is amazing. And they're like, The Cherno? Are you kidding me? We all learned from him. That's where we learned C++. So I kept watching his videos, and one of his videos was about why everybody uses Unreal - give these small game engines a chance. And I was right in that mode of, okay, shit, Unreal is never going to give me five terabytes of data. Whatever we are building here on Unreal is only going to be an MVP, a "this might work in the future" kind of thing. I wrote an email to Yan and I said, would you want to build a game engine for robotics?
Obviously he didn't respond, he was busy. Then I talked to Jim, his main contributor, and we had a couple of emails back and forth. He's like, what do you mean? And I explained the situation. Finally Yan picks up the phone and we met. Amazing dude. At Superpeer I was dealing with a lot of creators and influencers, and I was like, this dude is going to come around and want to charge me twenty thousand dollars an hour and tell me, okay, I can consult your team. And we hit it off. We liked each other a lot. It was a meeting of two minds, I guess. I invited him over to Austin. We did a company offsite. I brought the entire company and Yan flew over. He was so awesome. He never mentioned money. He never said, this is going to be my daily rate, or anything like that. He just liked what we were doing. It was so obvious.
In Austin he literally took what we built in Unreal - okay, here are the actuator states, this is how you record data - and he brought Hazel, included MuJoCo as a first-class citizen in a couple of days, then brought the first parts of Lucky Robots into Hazel. And then we had the prototype. What we had been working on for eight months showed itself throughout that offsite. And at that offsite he told me that he was looking for something other than games. I think that's what made the connection. It's like, I don't want to go back to Electronic Arts. I don't want to make games. I didn't want to be in entertainment anymore. Using my game engine towards this new goal, this new vision, is an amazing opportunity for me as well. So that's how we got together. We liked each other a lot. He immediately started as our CTO of simulation, and now he's my co-founder, and I can't really think of this company without him at this point.
Bogdan Cristei (18:47)
That's amazing. And then remind everyone what Hazel is.
Devrim Yasar (18:50)
Hazel is the game engine that he started building seven or eight years ago at this point. I think it was geared towards indie games. An open-source game engine that was built from the ground up by him. All the modules were super pluggable, and thus it worked for robotics super well. And dude, every time we did something in Unreal, if I changed one line of code it took about fifteen minutes to compile again so that I could see it. Fifteen minutes. Imagine, with web development it's instant. You just refresh your page, your Node servers are up, and okay, this is my website. With Unreal, it was fifteen minutes of compile everything if you're testing for deployment. It was just awful. Once we switched to Hazel, 3.2 gigabytes of Unreal code became - I think Hazel is about twenty megabytes. With the packages, two hundred forty. You click, and you just see what you just made. It was fantastic. I think it's the best decision and the best thing that happened to us.
Bogdan Cristei (19:57)
That's awesome. All right, so then let's talk a little bit about physics. You have MuJoCo for physics, a Vulkan renderer, C# in the editor, gRPC and Python outside. Where does the physics still lie, and how do you find that out?
Devrim Yasar (20:15)
Physics is just regular physics in a game engine. Lucky Engine is nothing different from Unity and Unreal. It's just that the purpose of it is different. The building blocks are the same. When you're making a game, what you want is: if something falls, or if you shoot at a target, you want whatever you're shooting to move in the air. Or if you're looking at a bird flying, you want to see the wings flapping. In a normal game engine - Hazel had Jolt physics - it's game physics. It's not accurate. It's accurate to your eyes. It's an optical illusion. When you are watching or playing a game, things look and feel like they are physics-based, but they are actually not. If you're slamming a door, it doesn't slam. MuJoCo makes this physically accurate, real physics as we understand it in the world.
In a normal Unity or Unreal, you can't just go to Unreal and say, hey, I want you to forget about the physics engine that you have, change it to MuJoCo. Even if you did that, Unreal never had contact physics. Unreal never programmed something towards, okay, if you hold something, if you push this with two newtons, what happens then? They have none of these things. Not that they couldn't have done it, but that was never the requirement, because that's a waste of CPU and GPU. Why would you do that in a game? Also, hot-swapping those things was not designed in. When you are building that software, imagine you are Tim Sweeney, and someone says, hey, let's make this future-robotics compliant and enable a MuJoCo type of physics engine to come around. Obviously the answer is, don't be ridiculous, we'll never need that.
So starting from scratch gives us the ability to say: your first-class-citizen physics engine is MuJoCo, and we will always make sure that the friction values are correct, that the contact points are correct to the millimeter. This is what we want you to do. Our second-class citizen is Jolt. Jolt is: if the bird is flying in the air, you don't really care if that's physically accurate. It's just the visual. So if the robot is moving in the room, we draw a diameter around it, and only the things that it can reach are added to the MuJoCo scene, so that we don't waste any CPU or GPU on that. Those are the types of things we need to do. That was your question, right?
Bogdan Cristei (22:46)
Yes, yes.
Devrim Yasar (22:48)
And Vulkan and all that other stuff, they are exactly the same. You said gRPC. gRPC is when you export your spreadsheet with your video data. Now you have this LeRobot 3.0 compatible dataset that you can train a policy on. Say you just pick ACT and you train an ACT policy. Now you have an ONNX file. This is an AI policy. Now you can import that AI policy back to your robot and prompt it. You can say, go bring me a cup of coffee, if that's what you trained the robot on. And this ONNX file is going to control the robot in the simulation through gRPC, and you can test if what you trained, if your policy, is working. gRPC is basically our joystick into the environment. It's our VR headset. If you connect to Lucky Engine with gRPC, you can have a Python program running, and then you can move the robot, make it walk, make it crouch, all of that. And you are not using the engine's internals for it. You are just externally controlling your robot.
C# is the internal version of the same thing. The reason it's C# and not Python is that C# compiles to binary, and it has access to everything the engine has access to. So you write your C# script. This is for your training data. If you want your robot to go bring you a cup of coffee without the teleop, without someone joysticking that robot into the kitchen, you are going to write it in C#. And if you have a policy on Hugging Face that knows how to control that coffee machine - someone has already trained it - you can drag and drop that. You say, hey, when you're going to interact with this coffee machine through C#, use this ONNX policy to do that. One thing I like a lot about Lucky Engine is that it's not about just one policy from scratch. You can bring eight policies - T-shirt folding, coffee machine, sink, water, fridge, all of those policies - load them onto your robot, and right now write the C# script to orchestrate those policies. Out comes your robot that does everything in your house.
Bogdan Cristei (25:14)
Fascinating. I would imagine there are some folks today using your engine who never touched a simulator or a game engine before, right? So who are these people that are coming in and getting excited about using this?
Devrim Yasar (25:30)
I don't blame them, because simulations - remove Lucky Engine and one or two others, and I can't even say. MuJoCo has a built-in simulator, but it's so basic, it doesn't even compare. With Lucky Engine, your limit is Cyberpunk. You could literally create a scene, add so many robots, make things produce steam or smoke, things like that. The MuJoCo simulator is basically, MuJoCo made MuJoCo and then they're like, all right, we need a visualizer. That's not a game engine at all, you can't program it. MuJoCo exports one frame, you take it to your Python, do whatever you want to do, and go back to it. Or you use JAX. So these people, unfortunately, when you tell them, hey, we have an amazing simulator now, a AAA game engine that you can export robotics data from, they have a healthy skepticism for it, and I understand that. But we have built something phenomenal. Once they see it, they're like, my God. This is actually a quote from one of our customers: "I am so glad software engineering has come to our field." Because everybody is just hooking up a bunch of models to policies to Python, and trying to gRPC into Isaac Sim and get things out. Everything is so chaotic. It's almost like everybody's shooting blanks and seeing what sticks. With Lucky Engine, we are hoping that we will fix all that. One consistent thing: you do your training data export here, test back in here, or test it elsewhere, and then finally it comes to the table behind me and we do the sim-to-real tests. Hopefully next year we're going to be the de facto tool. That's my hope and dream, and we are getting there.
Bogdan Cristei (27:29)
Awesome. So you mentioned Isaac Sim. Isaac Sim has the GPUs, the distribution, the budget. What's the deeper difference between the two, and what happens if, let's say, Nvidia re-architects Isaac for episode generation?
Devrim Yasar (27:45)
In a parallel universe, in a universe where anything is possible, that's a possibility. But it's about as likely as me becoming president of the United Kingdom or something. I don't see that coming anytime soon. And the reason Lucky Robots exists is that Isaac Sim literally wasted three months of my life. It's not a game engine. It's not done for the new AI architecture, the transformer architecture. It was done in 2016 and they're trying to mold it towards the new stuff. I hope they do it. I don't mind competition. We're in a public forum, but I'm just going to say it: their documentation was not updated. That's the level of what we are talking about here. It took me three months. I had to buy a 4090 for it to just render me a couple of frames. And I decided to do Lucky Robots because I'm like, this is unacceptable. I even went to GTC, talked to their developers - amazing people. But I'm like, could you just please update your documentation? We get stuck on the third step. So I'm not worried about Isaac Sim. Even if they fix their shit and become prominent - they are prominent now, which I think is going to change. If you're asking which competitor I would be more worried about: if someone in a garage with five amazing people starts doing what we are doing, that would be more worrisome to me than this enormous corporation moving in this direction and fixing their things. So I'm not worried on that front. If Isaac Sim is BlackBerry, I think we built the iPhone. That's what I would say.
Bogdan Cristei (29:36)
Well said. Okay, so today there are three ways to get robot data, right? You have your teleop fleets, like Tesla - we just talked about teleop in general. You have world models that generate video. And you have physics simulation. What's the case for simulation against the other two? And is there a situation where the other two are better suited?
Devrim Yasar (29:59)
You said world models and teleop?
Bogdan Cristei (30:01)
Yes.
Devrim Yasar (30:02)
I'm not a big believer in world models either. I think determinism is king. I don't think I need what world models are good for in robotics. If I make a fridge and create five million episodes on that fridge, then randomize the fridge and create another five million episodes, and I've covered five hundred fridges, I think I'm okay. I don't need all the variations of all the fridges in the world for the model to generalize. World models all started as "we're going to do games," and two years ago, I guess, everybody moved to robotics. I don't know; your guess is as good as mine. But I have never seen a world model that created a deterministic dataset. And if you're not deterministic, then you're not training a policy properly. Maybe I'm horribly wrong on this, that's fine. They could do the thing as well, but I'm yet to see it. And even if they do the thing, it's going to be super expensive. Why am I spending that many tokens, when I could just create a scene in Lucky Engine and create five million episodes for almost free? I'll give that to you in a week. Ask that of an AI model and it will have to compute every single frame. That might be the second thing against it.
Teleop. People compare teleop to sim, and I think that's not apples to apples. Teleop sounds cool: I can collect data in teleop and in sim. No, there's no such thing. For example, if this bench behind me is done on teleop, that means you need a real bench. Some real people need to put this together. Now you want the bench to be five centimeters wider. Am I asking those people to come back in? What if I want the bench in a different color, in a rainbow of colors? How are these things happening in real life? They just don't. If you want to set the building on fire, it's not happening. Teleop is never going to compare to simulation.
Another example. Let's go with the cup of coffee. I bring it to you, and say I pour it on you by mistake. How am I going to debug that? The teleop company has already given you all the data, and now you need to call them back: hey, I want this exact situation replicated, and I want to understand how it's failing. Say the pouring happens on the policy, not on the teleop. You can't revert to that exact frame and start fine-tuning on it. That is a logical impossibility. You can't do it with teleop. That's why you have a bunch of controlled-environment demos and not a lot of robots doing things in real life. I can easily show you a proof of concept. But I can't say, hey, now this robot is serving coffee in a diner. That's a whole new step, because now you have so many edge cases, millions of things that can go wrong, and teleop didn't cover any of that. What teleop does is show you it is possible, if I had a million more hours of this. But we don't have the million hours, and we can't have it with teleop. There's no way we can do it in a manner where you can divert, breakpoint, debug what's going wrong. And needless to say, you call the teleop company again and they're like, we are fully booked, we can do this in six months, it will take another three months, I need to allocate a thousand people, I need your new robots, new specifications, new places. We need to build the Matrix to really fix the problem.
Bogdan Cristei (34:03)
Yeah. And then cross-embodiment is also not solved, so the minute you change the hardware you've got to do it all over again, right?
Devrim Yasar (34:09)
Yeah. And people throw AI at these things. They go, okay, we're going to train from the video, there will be good AI models, Astra is going to figure out the differences between these two robots. Maybe true. Okay. I can become the king one day. It's not happening today. But we can do it today. That's the difference. If I have a game engine that does these things, why am I waiting for AI models to catch up? It is possible today. Let's do it.
Bogdan Cristei (34:38)
Got it. Okay, so let's talk about sim-to-real. That's the question everyone asks. What's the strongest evidence you have today that a policy trained on Lucky episodes holds on a real arm? And is there a gap?
Devrim Yasar (34:51)
There's always a gap. It's impossible to get it a hundred percent. There will be a gap on teleop data as well. There's no one-to-one transfer, so AI models, policies, need to compensate for it. But if Lucky Engine drifts more than ten, twenty, thirty percent, then obviously it's just going to get harder and harder. In fact, Lucky Engine might be creating too much perfection, so you need to add some noise, just like you would have in teleop. What evidence do we have? We are training models and testing on these robots and picking up the things. We were heads down creating the engine for the past one and a half years. We released it; the first version got out in early August. Now the second version is coming, and it's unbelievably good. It's a hundred percent agentic. I hate the word, but it is true. You don't need to touch it. You just open your Codex or your Claude or your Hermes and say, use Lucky Engine to do that, and it just goes and creates the scene. You can upload a picture and it creates the exact replica. I think I shared it on your group chat today. This exact replica of the plant is done by Lucky Engine, and then it just moves the thing and picks up the pliers. I think I'm a little bit DHH-pilled on this one. I don't think people want to use tools anymore. They just want to use their AI agent, and the AI agent uses the tool to get you what you want. Now you will tell Astra, hey, I want five million episodes of this task. It will go to Lucky Engine, create the scene, add the robots, and out comes your dataset.
Bogdan Cristei (36:32)
So your customers today are labs building robot foundation models. Is that true, or who else is a customer?
Devrim Yasar (36:39)
We want some diversity there. We actually have one customer that has no AI personnel. It's just a company that wants a robot to do something. Basically the customers of a model company, like Skild or Physical Intelligence, would go to that company and say, what do you need? And they will say, I want this robot to do X. So we would welcome that type too, but right now it's easier for us to work with model companies, because they have their AI researchers, they have their teams, they know how to use Lucky Engine, they know how to train their own policies. Also, if you want to go to BP or Exxon, for example, to do a robot for an oil rig, imagine that sales cycle and the deployment of those robots. We would like that job to be somebody else's. We're just going to stay on the data layer.
Bogdan Cristei (37:40)
Got it. So especially when we talk about the labs: once the approach becomes proven, I wonder if they will think internally about whether they should buy instead of build. How do you think about that?
Devrim Yasar (37:56)
One of the frontier labs told us in a meeting that this is exactly what they wanted to build in-house. That was an amazing moment for our team to hear, because at the time world models were not out, and we didn't know if something magical was cooking behind the scenes that we didn't know about. I think no architectural firm ever wanted to build Autodesk. No engineering company said, I want my own AutoCAD. It's not their job to build the thing that creates the data. Those model companies are also not saying to themselves - at least I don't know any - "we need to own our teleop company." They contract these things out, because the teleop company is the most boring type of sweatshop you could have: ten thousand people, really hard to manage. You don't know if they're doing the right thing or not. They could send you twenty hours of junk data where nothing was picked or placed in the right spot. All of these things are just so hard. So why would they start building a game engine? If I'm OpenAI, if I'm Anthropic, if I'm any company of that size, I would just use Lucky Engine and be done with it.
Bogdan Cristei (39:17)
It's interesting that you say you don't know what you're buying with the teleop data. I don't know if you saw one of my posts recently, but it was an interesting story. Somebody had bought data from India, and the policy was working for whatever task they were doing, but once in a while the robot would make this motion as if it was trying to smoke. It just kept lifting its arm, getting it close to where its mouth would be, because the robot had no mouth. All of a sudden it stops what it's doing, takes a little smoke break, and continues the task. I thought that was pretty interesting.
Devrim Yasar (39:56)
What do you think that was? Was it someone teleoperating the robot who was smoking?
Bogdan Cristei (40:00)
Yeah, it's probably somebody - in India it's very common that people smoke. So it's probably one of these organizations that do a lot of teleop recording, and everybody's probably smoking. Then you train a policy on the data and you get that result, right?
Devrim Yasar (40:16)
Yeah. I could tell you more that's problematic with teleop. You basically have raw footage of the thing you asked for, but someone needs to label it, right? Someone needs to say, this worked, this didn't work. If that's the teleop company, they're going to label it "yep, it worked," because they just want to get paid and go to the next client. Then your guys in your company, or another company you contract, need to tag this stuff and say, all right, you have five million successful episodes. And you never know exactly what you're getting, even if that second company did the tagging well. With a game engine, we know exactly where everything is in the scene. We don't need to guess if you picked that thing, because I have the pixels, I have the objects, I have the contact points. All of that comes for free in Lucky Engine. Compared to teleop, I can get you five billion episodes and I can guarantee you five percent of it is failed. If you want failed episodes, I can deliver that to you. If you don't, we can trash them. And obviously you're using the engine yourself as well. You have access to everything in the scene: how many robots, how many Kelvin the light is, what's the hue, how much light was there when it failed. All of that is in the scene, so you can always go back and see your parameters.
Bogdan Cristei (41:42)
Interesting. Let's talk a little bit about the business side of things. You charge per episode generated, I think, today. What does a customer order look like? And how does the price of an episode change as compute gets cheaper over time?
Devrim Yasar (41:59)
Since it's a game engine, the compute cost is not that high. Most robotics policies do not require HD images either. If you're doing something simple, you could create, I don't know, a hundred episodes a second. Our upcoming release - not next week's, but the December one - will also contain a reinforcement learning stack, the Isaac Lab part of the stack, and we are aiming to get to 400,000 frames per second, that level of mastery, if I may say.
What does it look like today? I'll give you some reference points. If you want teleop data, you are probably getting quoted from twenty to fifty dollars, up to two hundred dollars. You need to talk to them. It's like before AWS, that's how you would buy servers. You would go to some sales guy from Dell, and they would ask you all about the servers, the racks, the cables, where, here, there, all of that stuff, and it would take three months to finalize your server order. With AWS, you now click. It just changed. You no longer know what server you're using. Similarly with teleop: where is it, how is it, is it a kitchen, what kind of kitchen, what kind of countertop, what's the robot, what's this, what's that. It will take you three months to actually finalize the teleop order. Let's say best case two weeks. I don't want to paint it too dark, but it's difficult. They are creating a real life for you and employing real people for you. That takes time. Lucky Engine doesn't need that. You just create a level of a game, basically, and that becomes the teleop scenario for you. If it's simple it will take you a few hours. If it's super complicated it will take you two weeks. But once you are ready, you can drag and drop, you can domain-randomize it, at no cost at all.
So the current cost is, say, twenty, fifty bucks and upward. One of these big, big robotics companies, that everybody knows of, told us they only employ in-house teleop operators. He said, our robots are not toys, unlike some of these Chinese companies. One of them costs five hundred thousand dollars, and our teleoperators are two hundred thousand dollars a year. He did an over-the-napkin calculation and said it's probably upwards of eight hundred bucks an hour. And one VR headset operator can only work two hours a day, because they get dizzy. And they are literally on the factory floor, looking down, using those robots. So the reference is a hundred to eight hundred bucks per hour, probably. Could be cheaper, could be more expensive, depending on what you want. Are we doing a coffee? Are we cracking eggs? Are we spreading peanut butter? What are we doing, really? When someone asks me, what's the sim-to-real gap, I don't know. What are we simming? What are we realizing? That's a very, very big spectrum.
With Lucky we charge per successful - well, if MuJoCo fails, and sometimes robots fail in the simulation too - every hour that you export from the simulator is one hour that we deliver to you, and depending on how many hours they want, we price it based on that. But it will be not even a tenth of what teleop would cost you in time and money. It won't take you three months to set anything up. It won't cost you six months to collect the data. It will be weeks. And once you are happy, the next one, and the next one, is going to be a lot cheaper and faster.
Bogdan Cristei (45:51)
Got it. So if I remember correctly, there were a number of things on your roadmap: drones, fridges, cabinets, liquids, deformables. What comes first and why? And what is the one thing a customer asked for recently that you said no to?
Devrim Yasar (46:07)
I think the second part didn't happen, because all the robotics companies right now are trying to solve very basic but difficult problems. You're not seeing a robot trying to parachute itself from a helicopter today. We're not saving humans from a fire. Those are very advanced. Right now the best robotics demos from the frontier labs are, we can flip a burger. I think it was Generalist that had the state-of-the-art demo, where they could change tools while in motion. That was pretty impressive. But if you look at it, it's basic. It's not something you and I couldn't do. When you use a computer, the computer can do a lot of things you and I cannot, and we're like, okay, this is amazing. When you look at robots, there are very few things where you go, I couldn't do that, that's very, very hard.
Bogdan Cristei (47:01)
We're talking about humanoids here. There are some robots that you see that work very well.
Devrim Yasar (47:04)
Yeah, on the manufacturing floor. But they are not using policies. They are programmed.
Bogdan Cristei (47:10)
Yes. So the ones with policies, yes.
Devrim Yasar (47:12)
I can't recall off the top of my head a policy that I saw where I went, my God, okay, that's something else. I think we're going to start seeing them maybe at the end of next year. If Lucky Engine is successful, and if we can give people those millions of hours of missing data, I would hope that we will start seeing that.
Bogdan Cristei (47:34)
Very cool. So you moved to San Francisco this year. You have a team spread across the Bay Area, Melbourne as well, some remote folks. What do the next twelve months of Lucky Robots look like from here?
Devrim Yasar (47:48)
Another contrarian bet I have is that game developers will play a significant role in solving robotics, not just AI researchers. AI researchers, unfortunately, don't come from the simulation world. I'm talking to a few of them, and I don't want to offend anyone, but most AI researchers are coming from Jupyter notebooks. They're not really writing software while training the AI models. Game developers, on the other hand, know the most about software engineering, and I think that if we create the Matrix, and if that Matrix can be programmed, it will be programmed by game developers. If someone is able to build an amazing game, they are going to be able to build an amazing training level in Lucky Engine, and that will export the data the frontier AI labs need for that robot to do the thing. So I think next year it's going to be game engineers. You know how you have the Fab store on the Unreal side, with levels and backgrounds? I think we'll have a similar situation with Lucky Engine. Lots of game engineers are going to come and start building for robotics companies, for their data needs. They are going to create the tasks, levels, backgrounds, domain randomizations. That's what's coming next.
And I'm going to have a few game engineers here at this office, mostly working as forward-deployed engineers. If we have a client, the game engineer is going to create the level. That's how we work with customers right now. We say, tell us what you need. You don't need to touch Lucky Engine. We will make the scene for you and deliver it, so you can start recording and training your policies. But I think next year, if Lucky Engine becomes the number one thing in the world, robotics companies will start hiring game engineers who can use Lucky Engine.
Bogdan Cristei (49:49)
Yeah, full circle. Nvidia used the GPU to break open the AI world, and gaming folks will come in and solve robotics.
Devrim Yasar (49:59)
I think so.
Bogdan Cristei (50:00)
Demis from DeepMind started as a game developer, a game engineer.
Devrim Yasar (50:06)
What an impressive life story he has.
Bogdan Cristei (50:11)
What a crazy life, yeah. Okay, so what do you need in the next thirty, sixty, ninety days? Customers, hires, introductions. What can people help with?
Devrim Yasar (50:20)
Customers are always welcome. We want to work with customers that will give us great success stories. Some customers, as you'd suspect, have very unrealistic expectations of robots. There are things we can do and things we cannot do. But most pick-and-place, data center automation, RJ45 insertion, things like that, are super easy for Lucky Engine. And we can obviously put humans and pets in the simulation. We can train robots to be safe around living things. If customers want to train their robots, they just need to give us a shot. Writing the engine and at the same time getting these robots to do something in real life was hard. We're a small startup, we're thirteen people now. We couldn't say, hey, half of you, we want sim-to-real demos. But we started now. Yan, our most senior developer, and one more are now fully focused on getting the sim-to-real demos out of Lucky Engine. When we release them, people will be like, my God, okay, this is possible now. That is what we are working on. We are not going to focus so much on adding more features to the engine, because I think it's good now. We are going to focus on getting those demos out, and they are going to be very, very impressive.
What we need: if game engineers want to join, that's always amazing. Great customers, model companies: please give us a shot. We really built something very cool, and it really creates the missing layer of robotics. And if you don't want to give us a shot, wait a month or two, and we will impress you so that you have to come around.
Bogdan Cristei (52:09)
Very good. So I think you have your fellowship, right? You're paying senior roboticists to break your engine. What do you hope to find there, and what was the first cohort like?
Devrim Yasar (52:21)
Everybody invents their own thing in academia, and we wanted to create a cohort of people who want to create the scene in Lucky Engine and make it work from there. It's not a ten-people, twenty-people cohort. It's one or two amazing people who are working on things. That's our way of giving back, and also of telling people that this is a legitimate platform to train your robots on. If an AI researcher in academia writes a paper on Lucky Robots, we love that. That's why the fellowship is there. Please apply, anyone listening. Let's make cool things together. They have full access to our team. They can ask any questions. If they get stuck, we help them out.
Bogdan Cristei (53:11)
Gotcha. Okay, so I have maybe two or three more hot-take-type questions. First one: there's a humanoid that folds laundry in a home with no one in a headset. What year is it?
Devrim Yasar (53:25)
I really can't tell, because without deploying that thing, it's really hard to tell where you'll need to fine-tune the model. Is it bumping into doors? Is it not able to climb the stairs reliably? This thing is very dangerous.
Bogdan Cristei (53:46)
What do you have there? Is that a G1?
Devrim Yasar (53:47)
Yeah, that's the G1 up there. When you deploy a new policy - we're going to build a cage here, because when you deploy a policy it flails around, and it's very, very powerful. It almost cut my finger off one day. And you don't want that thing to fall backwards while climbing the stairs at your home. So you don't know exactly when we are going to hit that time. But if all goes well, probably a year, a year and a half, I would say. If the Astras and Claudes of this world continue to advance as fast as they do, I think we'll be able to get there a lot faster. The latest Astra update, for example, solved a big, big problem of ours: creating the scenes. Now we can upload a picture and it uses Lucky Engine's primitives to create the scene. You don't need a technical artist to come in and make your scene and add the robots. Astra does that now. You can just start collecting the data and training your policy. If it goes like this, maybe you train the foundation model and the fine-tuning becomes a lot easier, and generalizing to the next adjacent task becomes easier. We are living in such a timeline right now that it's really hard to predict what happens four weeks from now. So I'm not going to say one and a half years, or two, or five. I don't even know what the world looks like in five years, to be honest with you. But I hope that at the end of 2027 we'll have robots doing things near us.
Bogdan Cristei (55:20)
I'm still waiting for the flying cars, but okay. Let's see what happens.
Devrim Yasar (55:24)
I never bought into that. Who needs a flying car? When do you need a flying car?
Bogdan Cristei (55:29)
All right, next question. What's the most wrong thing people believe about simulation?
Devrim Yasar (55:35)
I think it's not wrong - they're skeptical, because the simulations today are just bad. Isaac Sim is not good. I have not met one person who likes to use it. In that sense they are not wrong to say, we don't believe in simulations, they've never worked for us. All of that is true. But if you let your cousin fix your computer and the computer doesn't work, is it the computer or is it the cousin? There's something there. This simulator needed to be built by Yan Chernikov. It cannot exist otherwise. I couldn't have done it myself. I don't think three AI researchers with some cycles to spend can go online, pull some renderer, and do it that way. It's a really, really hard problem to solve. So that's what I would say. Give Lucky Engine a shot.
Bogdan Cristei (56:33)
Okay, last question. Five years from now, what share of robot training data comes from simulation?
Devrim Yasar (56:40)
Ninety-nine percent. We all believe we live in a simulation too, right? This all is a simulation. There's no way you can get robots from where we are today to, say, you go out on the street and there are five robots doing things. That world is not going to come with teleop. Teleop is giving us good POCs, good prototypes, good "if I had more data, if I had more X, Y and Z, this would work" kinds of ideas. But simulation needs to come in and play a significant role. And I don't only mean Lucky Robots. Maybe we will be good at five verticals, and some other company will be good at another five. There's going to be a huge surface area here. And hopefully Isaac Sim fixes itself and becomes good at fifteen things. It's all going to be amazing. But this problem requires simulation. There's no other way.
Bogdan Cristei (57:45)
Okay, that was the last one. All right, so with that, Devrim: where can people learn more about Lucky Robots? Website, LinkedIn, where should they go?
Devrim Yasar (57:54)
Yeah, luckyrobots.com. We have our Discord, and we will start live-streaming every day starting next week. We have hired our first game engineer to be tasked with just that. So if you want to learn how you can put things together in Lucky - and largely it's going to be just prompting your AI agent to go do something for you - they can do it from there. LinkedIn is also Lucky Robots. My name is Devrim Yasar, if they want to connect with me on LinkedIn or elsewhere, or on X. I don't post a lot, but I welcome people to connect.
Bogdan Cristei (58:30)
Awesome. So good to have you on, man. Thank you so much for your time today. It was a really awesome conversation.
Devrim Yasar (58:35)
Yeah, thank you, Bogdan. Of course.