Subscribe
Share
Share
Embed
CRAFTED. is a show about great products and the people who make them. Top technologists reveal how they build game-changing products — and how you can, too. Honored twice by The Webby Awards as a top tech podcast, CRAFTED. is hosted by Dan Blumberg, an entrepreneur, product leader, and former public radio host. Listen to CRAFTED. to find out what it really takes to build great products and companies.
[00:00:00] Mario Rodriguez: I just get so excited about these micro experiences that we could use to teach, educate, and then get something done. Not everything has to be a productivity app. There's a lot of stuff at the very beginning of software development that got done that was super creative that we lost over the last 50 years that I'm excited to get it back again.
[00:00:23] Dan Blumberg: That's Mario Rodriguez and he's the head of product at GitHub, and he believes that copilot and other AI advances are gonna unleash a wave of creativity and enable a billion people to be software developers.
[00:00:34] Mario Rodriguez: And I think it's gonna feel a lot more like how kids play to the degree. It's like you create something you play with and you're like, Nope.
[00:00:41] And then you instruct it again, change that specific thing, and then you do it again. And so it's gonna be more like uh, what I call real time development.
[00:00:48] Dan Blumberg: Today on CRAFTED., we're exploring the power of AI to change the types of apps we build and the way we build them. Plus, Mario has tips on prompting and how to build experiences that delight developers.
[00:01:00] Mario Rodriguez: You might ask me, Hey, why are you telling me that? And you're not gonna go and be like, shut up, Mario. So you have to now think about, okay, how do you want to introduce AI in a way that builds trust with the user?
[00:01:17] Dan Blumberg: Welcome to CRAFTED., a show about great products and the people who make them. I'm Dan Blumberg. I'm a product and growth leader, and on CRAFTED. I'm here to bring you stories of founders, makers, and innovators that reveal how they build game changing products and how you can too. Crafted is brought to in partnership with Docker, which helps developers build, share, run, and verify applications anywhere without environment confirmation or management.
[00:01:43] More than 20 million developers worldwide use Docker’s suite of development, tools, services, and automations to accelerate the delivery of secure applications. Learn more at docker.com
And CRAFTED. is produced by Modern Product Minds, where my team and I can help you take a new product from zero to one and beyond. We specialize in early stage product discovery, growth, and experimentation. Learn more and sign up for the CRAFTED. Newsletter at modernproductminds.com.
[00:02:17] So Mario, I've thought of lots of ways to start this conversation. We could start with how you foster developer happiness, or how you and the team are building, you know, the new versions of copilot and what comes next. And we'll get into all that, but maybe let's just start this way. What are you most excited about right now?
[00:02:32] Mario Rodriguez: Wow. AI is changing so fast that probably you asked me that again next week and maybe I have a different answer, so maybe like an evergreen answer to that question. In my opinion, it's the ability to democratize software development overall and. Today a lot of software is being written by professional developers, and we have been talking at GitHub about what if we could actually have 1 billion developers in the world, how, how much potential can we unlock overall?
[00:03:09] So I, I think the pursuit of that and all of the changes that will need to happen in order to get there is. Really what gets me out of bed, like if, imagine by 2030 we're 10 billion, right? Like 10% of that would be 1 billion people doing some type of software at, at that moment though, you're gonna have to redefine them what a software developer is because it's not only those professional people, it's now, it's gonna be.
[00:03:33] My daughter who is nine years old, or you know, I have two daughters, 1, 9, 9, 1 7, or even the 7-year-old one. Yeah, it could be my mom. Um, as an example. It could be, you know, someone in high school that really has a specific need on something and wants to actually develop something really quick. Jonathan Carter, he's the head of next at GitHub, and I think he says this a lot more eloquent than I am, but I think a lot of the applications are gonna go back into this very simple.
[00:04:02] Root and like if you think about Unix and the Unix commands and how you could compose them. Mm-hmm. But they're extremely simple. They do one thing and one thing very well. So if you think about these applications becoming these micro apps and then us returning to this era where a lot of these micro apps are very simple.
[00:04:22] They're compostable. And then you could actually even tweak them to what exactly you want out of them. And I think that future is, is really. Interesting. So like I said, I have, uh, two daughters and I have noticed with them, at least at the age that they are right now, is that they love to learn, but they love to learn through play.
[00:04:42] Like if I actually go and sit down with them with a notebook and I make them do some type of. You know, math or anything like that, uh, then becomes, you know, yes, they do it, but it's not something that dad Exactly. It's not going like that. And, you know, I wanted to teach, um, the oldest one a little bit more about probability.
[00:05:02] Mm-Hmm. And chances and to a degree, you know, transparency, luck. And I was like, well, maybe we should just play a little bit of poker. And I wanted to just create. Like this little micro app that every time we will have a hand, you know, you could think about with today's models, all my daughter has to do is kind of point the camera to that, take a picture on it, and then ask a question.
[00:05:25] Be like, okay, what's the probability of this hand winning or not? And then she could figure it out if she wants to bet, you know, the fake money or not. Yeah. But. That is not a professional app that someone is gonna have out there. Maybe it does exist for the people that actually do tournaments and things like that, but not actually made for a 90-year-old, let me just tell you that.
[00:05:43] Yeah. Well
[00:05:43] Dan Blumberg: also the, the, the point of I think you doing it with her was for her to build it and understand the lesson while building it. It's not to use an app that someone's already made of. Had a place. Yeah, exactly. So that's what I think
[00:05:53] Mario Rodriguez: the future is. Like, Hey, you just having a moment, you need something ephemeral.
[00:05:57] And I just, I just get so excited about this. Micro experiences that we could use to teach educate. Not everything has to be a productivity app. And I, I think there's a lot of stuff at the very beginning of software development that got done that was super creative that we lost a little bit, um, over the last 50 years that I'm excited to get it back again.
[00:06:21] Dan Blumberg: That's very cool. I'll tell you, it's funny you, you brought up Poker and Probability. The, the very first app that I ever built, and I'm not a software programmer and this story will emphasize that, but on my ti 83 calculator in 10th, there you go. I remember this 10th grade. Yeah. I, I thought you might, I learned, you know that there's a random number generator in there and I think I was learning statistics or something.
[00:06:40] We, we've been taught by our teacher how to program the quadratic formula into the. The thing, but I was like, I think I could build blackjack. And so I, I built blackjack into it and the part about me being a terrible programmer is I could never figure out how to make an ace a one or an 11. So my, my game was terrible for that reason.
[00:06:57] But like, it worked. Unless you got an ace. That's this. There you go. I hadn't thought of that in a long time, so, exactly.
[00:07:02] Mario Rodriguez: So that brings some memory. So, and you probably had fun. I did when you were trying to program that and eventually you hit that bug. If you had copilot, then it'll probably assist you in, yeah, we should actually try that and try to see if copilot can understand how to program a T six.
[00:07:16] Oh, I bet it could six or an 90 something or whatever the latest one at the, at
[00:07:19] Dan Blumberg: the most.
[00:07:23] Paint a picture for us of what software development may look like in the future. I mean, right now it is largely in a text editor. It's largely, you know, writing things in sort of a linear fashion. Yes, but I don't think that's the right thing to picture in the future.
[00:07:35] Mario Rodriguez: Correct. And this is why I am so excited about it as we, I think, have.
[00:07:40] At this moment for the first time, a medium where we could actually challenge what the status quo is. Um, not again for, you know, professional programmers or anything like that, but just to do unlock this 1 billion. And, you know, there's, I think two things that are really intriguing for me. One of them is, you know, for us at least at GitHub, is the human is always at the center.
[00:08:02] So you always have to keep that in mind. One thing that humans have that I struggle thinking AI systems will have is this concept of creativity. Mm-Hmm. Like we could literally create something that did not exist before. And so like if you put the human at the center, the reason why I mentioned that, and I I think it's important, is you put the human at the center and you allow that creativity to shine.
[00:08:26] Then you could get a lot of things done that, in my opinion, advances, you know, humanity. And one of the key changes I think that are gonna happen is, you know what, like natural language is gonna take center stage. Mm-Hmm. Because just like part of humanity and it has a lot of things, a lot of meaning that is so hard for us sometimes to actually put in some machine format.
[00:08:48] The second thing that I think also it's gonna happen that doesn't happen today is that you should be able to create a program then. On your phone. You know, look, we're very privileged to have computers in some countries in the world that, that do not have the technology that we have. But they all have phones, right?
[00:09:06] So you could imagine that a kid, a person in high school, even a person in a university will have a phone that now through natural language, they could actually express the intent that they want out of that software and get that back. And then the third thing that I think is gonna start changing dramatically.
[00:09:23] Is how we experience this loop of creating this application. So for the professionals, they're gonna have this AI native editor getting that loop for the non-professionals. I think there's gonna be a lot of innovation on how you're gonna go through that loop, and I think it's gonna feel a lot more like.
[00:09:42] How kids play to a degree. It's like you create something you play with and you're like, Nope. And then you instruct it again, change that specific thing, and then you do it again. Yeah. And so it's gonna be more like, uh, what I call real time development, and then it's gonna have two worlds. One for that professional, and then one for these people that are just creating something for them that they needed to get something done.
[00:10:02] Uh, either for fun or for work or uh, for education.
[00:10:07] Dan Blumberg: I love that future because what you're talking about is, is increasing the iteration speed, which I, you're, you're the head of product and like the quicker you iterate, the better the product is. And you're talking about like this rapid, rapid feedback that's gonna be even more possible with these new tools.
[00:10:20] A hundred percent. I'd love to hear how you see the craft of product management and the way that we build test, measure new products is changing with the rise of ai.
[00:10:30] Mario Rodriguez: I don't think this is a hot take, but maybe it is not, uh, an opinion expressed or shared by many, but I think doing. AI products is incredibly hard and completely different than doing features that did not have ai as an example.
[00:10:47] So look, I've been doing product for 20 plus years in traditionally, I. Whatever it fails at is considered a bug. AI is completely different. If you were to think, even if in the lab or in your offline evaluation you might get 90% right and like getting 90%, it's like you're doing something good into like you really like exercising the right test in offline evaluation.
[00:11:12] You're not lying to yourself, et cetera, et cetera. That. Means at the end that 10% of the time that your user tries that it's not gonna actually get that same answer back or it didn't do what it was supposed to do and they're gonna be disappointed about it. Yeah.
[00:11:27] Dan Blumberg: Or it gives them dangerous information. I mean, you can go even darker than that.
[00:11:30] Exactly. So you gotta
[00:11:30] Mario Rodriguez: be so careful and you gotta be so mindful, you gotta. You gotta really understand quality. What does quality really mean? You know, like you have to pay attention to all of these things and apply science people. And there people actually do an amazing job on that as well. But you have to really from a product perspective, think how to protect against that non-determinism.
[00:11:53] And look, even me to my team, like I was testing a feature the other day and I asked it a question. The first, you know, I responded back beautifully and I'm like, okay, I'm gonna ask it five times and see what happens. The fifth answer that it gave me was not only completely wrong, I thought it was like, wow.
[00:12:10] Like we need to figure out how to actually protect, not from an our like responsible AI perspective, but mainly from a trust of the customer perspective against that. And, you know, and there's ways that you could do it, like. People could say also whatever they want about prompt engineering. But if you know how to do prompts very, very well, I will hire you tomorrow.
[00:12:29] Like for us, prompt engineering is a real thing, and you could actually create these prompts to protect you from that non-determinism, uh, from the AI and actually create a better user experience for the customer. So the first thing is that, is in my opinion, is yeah. Like, Hey, before we knew how you could behave, now you had to really think about, well, these edge cases where you're not gonna get where it's gonna hall.
[00:12:54] You know, like say something that is not really true. How are you gonna protect against that? So you could still have customer trust. And then I'll tell you another one that is, we started to talk more and more about in the team. And it has to do more about psychology, in my opinion. But I'll give you a real life example.
[00:13:14] Like pull requests. And a pull request is what is, um, or a code review. A pull request is actually an artifact in GitHub Mm-Hmm. That you make in order to actually get your code review by other people in, in this code review process, multiple humans, uh, that are in a team participate. And I like imagine you did the, the pull request and I'm reviewing your code.
[00:13:36] 10. And you have this beautiful code of blackjack in I can give you, I can give you, I could give you, yeah. Subjective feedback. For the most part, if we're in a team and you know, we're that subjective, feedback is respectful. Mm-Hmm. And all of those things, you are gonna be like, oh, Mario, that's okay. This is why I did it this way.
[00:13:57] I'm not gonna take part of that feedback, but this is the other thing that I'm thinking on it. Yeah. And you're probably not gonna flip a table and stop talking to me. Right. Like, you're gonna come back and you're gonna take that subjective feedback as part of like, this is how we work as humans. It's like, okay, I put something in in there.
[00:14:13] Guess what? We have been seen when copilot gives extremely subjective feedback.
[00:14:19] Dan Blumberg: Mm-Hmm.
[00:14:20] Mario Rodriguez: Us humans really do not want copilot to ever give us feedback again. As an example. Yeah. We're like, Nope, you're completely wrong. It is completely subjective. I know what I'm doing. You are a machine. You don't know what you're saying.
[00:14:32] I am not gonna ask you to ever review my code again. Yeah. Like we literally. Yeah. See that psychology at times play out.
[00:14:40] Dan Blumberg: Yeah. I have the same reaction when I'm driving and Google Maps tells me something that I know is stupid and I'm like, no, I'm not gonna turn here so I can save 17 seconds and make 17 more turns.
[00:14:51] I'll just stay on the highway. Thank you. I.
[00:14:52] Mario Rodriguez: Correct. So I have that
[00:14:54] Dan Blumberg: same visceral, like shut up reaction.
[00:14:56] Mario Rodriguez: Yeah. And you know, and maybe if we are next to each other and I take, oh, turn here, you might ask me, Hey, why are you telling me that? And you're not gonna go and be like, shut up Mario. You know, so, yeah.
[00:15:04] Like, but like that's, that's reality what you said right now, it's, it's reality. So you have to now, from a product perspective, think about. Okay. How do you wanna introduce AI in a way that builds trust with the user? So I, I think human psychology is gonna play a significant bigger role in the life of product managers going forward in as we utilize more or as we infuse.
[00:15:31] Uh, AI into more and more of the applications that we use every day for this to be successful.
[00:15:36] Dan Blumberg: Yeah. GI give a few more practical tips on how you test a product that is using AI at its core. You talked about you got this prompt and you did it once and it was great, and you did it four more times and you the fifth one was terrible.
[00:15:47] So that's like one specific thing that people can do is like, don't just trust the prompt the first time because it is by definition, non-deterministic. You will not get the same answer every time. Like, what, what are a few other ways that you can build successfully with ai?
[00:16:00] Mario Rodriguez: So we strongly believe that in, in fact, we have this, um, internal product, um, that we named, it's not very clever.
[00:16:08] It's called copilot offline eval evaluation. But the abbreviation is actually pretty cool. It's coffee like C-O-F-F-E. So what we try to do with coffee is to figure out really what are the tests. That we should be having and sometimes, and what are the responses and like a human has to be part of that too.
[00:16:28] Yeah. Later, after you get the human. Like the right test cases, you could actually use synthetic tests, and you probably have heard that in your podcast. Mm-Hmm. Or in other ones too, which is like, Hey, we're gonna go and create all these synthetics and, and then we're gonna actually even have the AI rate itself on the answers that it gives.
[00:16:45] What we sometimes do too is like, you know, we ask this model. A question and we could ask this other model to be the rater on that. We then are very careful about what are the tests that we wanna run and being true to that. So you do wanna try a significant amount of variation on those tests to get better and better signal that your, what I call your system prompt together with the other prompts that kind of add more context into what you're feeding for inference, you know, ends up behaving the way you want it.
[00:17:17] So what I would say is like. If you're doing product and you're doing an AI product, please Overin invests in offline evaluation. If not, I think you're gonna pay it at the end. So that's kind of like number one. Number two, it's we have this concept of staff shipping and where we just give it to all of the GitHub developers or all the, you know, all the GitHub staff, and then you have people play with it.
[00:17:40] And then that's your second thing where you're like, okay, we have some offline evaluation. Now, let's see what actually happens in the real world, but with some friends, right? I think you need that human loop evaluation too. Um mm-Hmm. And then the third thing that we do, and this is the most, I think there's a lot more innovation that needs to happen in the market on this one, by the way, but it's online evaluation and then paying attention to online evaluation at the end to be able to make the right decisions.
[00:18:10] So we were working on this thing where. We wanted to. Increase the acceptance rate of copilot and we also wanted to make sure that the latency of getting that suggestion by copilot is, is small. Mm-Hmm. Overall, because we think like you don't wanna be waiting too long to get something suggested while you are, you don't wanna disturb your flow is really the main thing.
[00:18:37] We put it over there in the real world and we started to see that even though. We were actually given faster suggestions. The acceptance rate was decreasing even though, like even on offline. Mm-Hmm. Like we did some things to better even the quality of those results. And we're like, how? How is this possible?
[00:19:01] And we started digging deeper and deeper and deeper. And what we saw was that, hey, we had regress. The ability for copilot to give multi-line suggestions '
[00:19:13] Dan Blumberg: cause you wanted them faster.
[00:19:14] Mario Rodriguez: Because it's faster. It was just giving you less, less and less. But guess what? The developers love multi-line suggestions because it will save them a lot more time.
[00:19:23] For them, it was a lot more so we saw Nsat, but in go down we saw acceptance rates on certain things increase. But on these other things of lines of code acceptance rates, we saw a decrease significantly. Yeah. So like we were like, what is happening? So there's this trick now, like, yes, they wanna fast. And yes, they want high quality, but if it's just like three lines, they just don't like that, you know?
[00:19:49] Yeah. They want to actually add a value as well, so. So I think that's a place where, look, we're still learning as a company, and I think there's a lot more innovation out there that we need to do, and I say we, in this case, I mean the entire industry on online evaluation on how to actually tease up our things a little bit better there.
[00:20:08] To be able to create the great product. Yeah. Because it's a really hard problem at times. Yeah.
[00:20:11] Dan Blumberg: Yeah. The other thing in that story just now is, is it's an example of you get what you measure, right? So if you were, if you had a team and their whole objective was like, make acceptance of, you know, code suggestions go up, the, the prompts or their response would've just gotten shorter and shorter and shorter and shorter and shorter.
[00:20:26] And developers would've been less happy. But the, it would've been like 99% acceptance. Yep.
[00:20:30] Mario Rodriguez: Correct. Exactly. You get what you meant. Yeah, definitely. And by the way, this is true about all of us, like. Yeah, if you have a metric, you wanna move humans because of their creativity, they could get very good about moving that metric.
[00:20:41] So yeah, be multi metric in the way that you evaluate your, your AI products. You cannot just rely on one or two. I think you need to rely on three, four, sometimes even as high as five, in my opinion. I've seen that five gets a little bit problematic at times, but like I would say if you are only using one, you're failing, but you wanna be probably two or three.
[00:21:00] And maybe as far as five.
[00:21:01] Dan Blumberg: Yeah. Well this, this. Kind of gets to product being, I've heard you say this before, like more art than science. Like machines are really great at optimizing for one and only one objective. Once you have more than one, you can set up, you know, guardrails, different things. But, but you're basically saying there's like, for lack of a better word, like product sense and you know, you have to apply here to get this
[00:21:20] Mario Rodriguez: right?
[00:21:20] Yes. And look, the, the name of the podcast crafted, right? Like. Craft becomes really, really, really important overall. And as part of that craft, I think you need to have product sense. And that product sense comes from a, you know, experience. Uh, definitely a set of things that you have in your DNA that give it to you too.
[00:21:40] And, but also appreciating that we are now not necessarily just in the technology field, like you really are in that intersection. Be, you know, between science and art for sure. Yeah.
[00:21:54] Dan Blumberg: Are there one or two things that you've noticed about, you know, for, for developers who are using, let's say, copilot, uh, for them to get the responses that they want, are, are there a few patterns you've observed of ways to prompt it to get great responses without, you know, having to ask it five different ways?
[00:22:10] I.
[00:22:11] Mario Rodriguez: Yes. The less ambiguous you are in that prompt, the better it is. So, in my opinion, if you use shorter sentences and you're more directive, you get a better answer. Like if you try to tell a story. To sometimes you're not gonna get the answer that you get back, but if you're very much like, you know, you try to get the ambiguity out, yeah.
[00:22:32] Then you're gonna get significantly better answers. So I even have seen people hack copilot and not hack it in the way that you think, but um, on the sense of they go in, they copy and paste this thing and they put it as comments at the top of the file. Just so they has all of that context, because right now what Copal does is it on the open file, it tries to actually see what is before the code completion and also also what is after the code completion.
[00:23:01] And then we, we do a couple of other magic things. But we kind of go and use that and pass it to the model in order to generate the right code suggestion. So it's not, and we do adjacent files and a couple other things too, but think about it that way. So sometimes if it's not in the adjacent files and they know the context, they copy and paste it into this file.
[00:23:20] So Copala has all the context before, and then they just go and say through a comment, oh, do this thing, and Coval does an amazing job. And then they go back and delete all that context in from the file because it really was. To give it to Kapil. So they were still in the early days of what should be an AI native experience because that's kind of, you know, not ideal for me to copy a bunch of things for Kao to do a better thing.
[00:23:43] And you know, the more descriptive you are, the more time it takes. And if you are only gonna get five lines of course suggestion, you don't want to spend that time either, so, so I think there's a lot more innovation that will happen in here, but be direct. Even say please, I have seen that work as well.
[00:23:58] Yeah. And you know, use shorter sentences on exactly where you want and be descriptive in the places that you wanna be descriptive.
[00:24:04] Dan Blumberg: Yeah. Can you share a bit about the development of copilot and some of the things that you tried that didn't work along the way and, and how you landed on today's experience?
[00:24:13] Mario Rodriguez: Yes, yes. So. I was in, in the first founding team of Copilot. I I, I do know there was a paper that was called Automated Per Programmer Fact or Fiction. I think it was done around end of August of 2020. This was before SHA GBT by the way. The model that they were using was called Codex by OpenAI, and this paper, uh, was by uga, Alex and Albert, and the paper made the rounds, um, clearly, you know, to a select set of leaders at GitHub.
[00:24:47] And, but that was kind of like the first time that I think in, I think Brian Loving and a couple other people maybe did a video to, to kind of, so you could see kind of the power of it as well. And that was the first time in my opinion, that I saw what this could be in the way that we, that that paper was structured was saying, look, we tried to give Codex.
[00:25:11] All of these questions, like, uh, there were a set of engineers that ended up doing programming, um, questions or, or problems. Mm-Hmm. And, uh, and then they gave it to Codex and said, oh, solve this. And Codex was able to do an extremely good job at that. And then the other thing that they did, which I think that that's what blew, blew me away at times, was they had this function.
[00:25:33] There was comments on that function. And then there were tests on that function, and then essentially what they did is have, you know, copilot be able to then have to recreate the function just based on the comments and then see if it passed the test, and then try to figure out how to actually then change what the function was.
[00:25:54] Took them pass the test and never saw the function. Only saw the comments before. And the reason why I'm telling you that, that was the thing that changed my mind completely. Yes, there were before ways for you to evaluate ML models on things like that. Like, Hey, I'm gonna give you a problem to solve. Go and see how many shots it take you to solve it.
[00:26:12] But a lot of these things never actually took natural language and try to recreate the function, in my opinion. And. Just by reading the comments and the function name and kind of what the arguments were, this thing was able to try to recreate it and then eventually write enough to pass it. You know, like that synthesis was like, it was amazing.
[00:26:34] I, I think Alex had the first, um, commit on that repo, that synthesized repo. And I, I, I think that was beautiful. So like, that was the beginning of it. That that's when you could actually see the. Holy. Like we have a technology that we haven't seen ever before, but then like anything, you have a tech. How are you gonna get that into a product?
[00:26:55] I. That's a really hard thing to do as well. Many good techs have died because they never made it to a product. So the team started exploring various things, and I was speaking to Edan on this and one of the things that, um, that that team tried is it's not what you see today, by the way, that happened later.
[00:27:16] What it was is you were going to the editor and you had this thing that you invoke and it's a moral window and you say, generate and. You would ask it to generate and then it would do something, and you could expect how we gave this to internal developers. And they hated it. Like, you know, they could see the power, but they're like, I wanna not use this.
[00:27:35] Like, cool. Yeah. Like cool tech, you know, like, hey, this is cool tech. But that's how, that's not how I develop. I don't pause all of a sudden and be like, oh, you know, model window generate. So they started iterating, iterating, iterating until they got to, you know, IntelliSense had already existed for a long time.
[00:27:53] In tele sense, you pause, you know, you maybe do that, and then you wait for, you know, a millisecond or two or 50 milliseconds, and then you get something back and you could accept it by just hitting tab as an example. And we're like, oh my God, what happened? If we could get the latency of copilot to be that good.
[00:28:12] Mm-Hmm. And then you just type in. If you don't like what co compiler is telling, and by the way, we see this also in, in many of the word processors too, like sometimes or email products, you're typing something and it gives you something in this great text and you're like, oh, whatever. Like, I don't want that.
[00:28:27] You know, you could type in and then sometimes you're like, no, that's exactly what I wanted you to say. You'll save me five, you know, 10 or five, um, strokes on this. So, so on that end, like. We started getting to that latency, latency, latency, latency until they got that ghost text to be fast enough and to actually have the right quality and that.
[00:28:47] Is how copilot began.
[00:28:49] Dan Blumberg: That's cool. And, and I, I love also that it's, you know, the behavior of hitting tab is that's, that's a user experience insight. The real, real work is the latency had to be low enough that that actually would work in the first place. That would work. Exactly. Yeah. I love that. So you, you spend a lot of your time thinking about developer happiness, developer productivity.
[00:29:06] How do you measure these things?
[00:29:08] Mario Rodriguez: That's a very interesting question because I think it's, again, art and subjective. Totally. For us, what developer happiness means at the end, it's, Hey, are we keeping that developer in the flow and that flow can, it's not only while programming, if you think about. Software development, you spend a significant amount of time also doing sense making.
[00:29:32] So you have to look at the code and figure it out and say, okay, we wanna accomplish this in. Your brain needs to go and say, you know what that means you're gonna have to edit this file. This, this other file too. And this other file too. Yeah. Maybe we don't have, like, we need to bring in this dependency.
[00:29:49] Oh. That means that we are gonna touch this code and this other code is, is actually calling us. And then after I make more all of that sense, yeah, you might touch only one line of code. By the way, that's, you might just go to one place and say, you know what? This is the way we're gonna do it. Boom. So you have to give them tools for that.
[00:30:05] lio allows you to actually explain files. It needs to grow in actually making you sense, make about the entirety of the application. That's a really hard problem to do today, by the way. Then you need to actually keep them in the flow while programming. We do a pretty darn good job about that today. Then you have to keep them in the flow at build time and at test time as well.
[00:30:24] And then you try to abstract, in my opinion, as much as you kind of weigh, you know, platform engineering is a hot topic because of that from like the deploy and all of the automation there's, and then deploy. And if you do those things like yeah, your productivity as a developer goes through the roof.
[00:30:39] Your happiness as a developer goes. Through the roof. And guess what? Probably those things happen. The company and what they're doing, unless they're spending a lot of time on politics or independencies, their outcomes for that business also goes through that roof. So what we ask them to actually try to measure is that is how much time are you spending on sense making?
[00:30:58] How much time are you spending developing? How much time are you spending actually waiting for bills or waiting for compute? So like bills to pass, checks to pass. All of those things. How much time are you spending in the. Deployments, and then how much time are you spending in meetings and waiting for other humans?
[00:31:12] And like those things, if you end up optimizing them and having colo help you in many of them, then you're gonna get the outcome that you want as a, you know, developer happiness and then also as a business metric for the user.
[00:31:25] Dan Blumberg: Yeah. I, I love this and I love how passionate you are about this. How did you come to this passion about making engineers happy?
[00:31:31] Mario Rodriguez: That's a good question, but maybe it goes back. So look, uh, so I was born in Cuba. My mom and dad are educators and I, we got to United States. I didn't speak the language to start, so that was a little bit tough when you are in high school and you don't speak the language, um, wow. High school people are not very kind at times on that end.
[00:31:51] So, but that had this, um, ended funding an electrical company. And I went to school. I was supposed to kind of take that business over, right? I went to school mainly for electrical engineering. I was trying to do a c uh, computer engineer and electrical engineer at the same time at the University of Miami.
[00:32:07] And, but my passion really was like circuit design and like, you know, RF frequencies, like, you know, radio frequencies and try to see, at that moment, wireless was not what it is today. And I ended up taking an internship at Loosened Technologies and. During that internship, I was supposed to be doing what I, Ian technology had this, I guess they were servers, but they were racks at the bottom of the cell phone tower.
[00:32:33] And I remember they had this stack that was beautiful that if you are on the highway on a cell phone, you could go from tower to tower and you, they did not drop the call.
[00:32:43] Dan Blumberg: Mm-hmm. And
[00:32:43] Mario Rodriguez: incredible feat what they were doing in those towers to do that. Because at that moment. And I was trying to design a circuit that will go into one of those servers or those racks in that cell phone tower and they're like, okay, Mario, go and do it.
[00:32:56] And I started doing it and they're like, well, well write some tests. They had me programming per please, you don't have to learn pearl. Uh, but like I ended up programming 10,000 lines of code. And in that internship it's like 10 weeks, 12 weeks. And I was like. What am I doing in my life? Like I stress about dissipation, about all these advancements that need to come into hardware.
[00:33:18] If, if you're a hardware engineer, you know what I'm talking about, it's a hard life to do that the right way in, in and to do innovation in that space. But in software, you just type and that's it. Like there's this computer that does this thing and I'm like, I'm in the wrong career. So I went back and told that, that I'm not gonna take over your business, I'm gonna just go back into software and I'm gonna try to spend my time.
[00:33:38] Try to make software development amazing. I think because it's just something, it, it, it just captivated me, you know, transparency. Then the education part of it that, the reason why I love making tools for software developers too is like, there's a lot of education that goes into there too, and, um, my family has always been big on that.
[00:33:56] Um, so I probably got the bug from both of those aspects.
[00:34:02] Dan Blumberg: Tell me more about where your love of learning and teaching came from. I know that your family was a, a big influence for you there. Uh,
[00:34:08] Mario Rodriguez: yeah. My dad was a physics professor. Mom was a, a university professor. Then my dad eventually then went into electrical engineering.
[00:34:14] Yeah. And then after that he became an engineer as well. Uh, but he, I. I think he was one of the youngest professors of physics, um, in his university back in Cuba. So I remember when I was nine years old, mom will come in and give me all of these papers to grade because she don't wanna, she, she wanted help in grading them.
[00:34:33] So I, she gave me a red marker and it's like, okay, go and figure it out if anyone had any misspellings or things like that. And I would do that. And then that sometimes would also give me things are, are, are very similar but from a different angle. So I mean, even so much right now that. I helped found a school here in North Carolina in small, in the rural parts.
[00:34:55] It's not in the city, so it's probably an hour and a half from where I live. And just to try to figure it out, how can we actually increase the test scores of. You know, the students that are, do not live in the city and that maybe don't have access to the same teachers or don't have access to the same resources that people that are in big cities, uh, have, and you want everyone with an amazing education and you wanna innovate in giving them that amazing education.
[00:35:23] Dan Blumberg: Yeah. You're, you've, you're saying this so. Almost casually like, oh, I launched a school. I talked to a lot of founders on the show. I have not talked to someone who's founded a school. Like, could you tell us what was it that caused you to found a school?
[00:35:35] Mario Rodriguez: I mean, I think, I think we all agree that humanity can benefit from, I.
[00:35:42] Better education anywhere in the world. Right. And since we have a privilege sometimes in these big cities, that privilege doesn't exist at times in the more rural areas, um, of this country and many other countries as well. Yeah. So I. Making sure that everyone has access to amazing education. I think it's part of who I am and also why I think it's what we should all be doing is trying to figure out how to, how to provide better and better education to, to all humans.
[00:36:14] And look, education is hard. It's not easy. There's many ways kids learn at different rates. Like I think humanity knows how to do agriculture very, very well. We figured that thing out like to a. Degree. We do not know how to educate kids. Many places, uh, actually are just babysitting your kids, in my opinion.
[00:36:34] I love that a lot of companies and a lot of institutions now are making their stuff available for free online. I. 'cause then that allows these students, by the way, so we give them one laptop to each one of those child. But what that allows us to do is, let's say we have one child that really gets physics very well and he wants to just advance and we don't have that teacher available at the moment.
[00:36:58] Then that that student can just go ahead. And if you let that space happen, which is what I do, it's like, Hey, that kid is advanced. Let the space so they could just go to a computer and learn more about physics. And that's okay than the teachers mainly facilitating many, uh, sometimes that,
[00:37:14] Dan Blumberg: yeah,
[00:37:14] Mario Rodriguez: then you get a better education at the end.
[00:37:17] Um, and I think we just, we just are not innovating enough into how to. Try different things to increase the reading level, the proficiency level in math, um, even things like programming and, and what software development is in many of these schools.
[00:37:36] Dan Blumberg: And, and as, as we think about, you know, all the things we've just talked about about AI in the future and what, what are some of the skills that you hope these kids, that your kids are learning today they think are gonna be really essential going forward?
[00:37:49] Mario Rodriguez: Systems thinking is, so one of the things that I. Being, thinking more about, it's like, okay, how do you teach system thinking? It's a hard thing to do. Mm-Hmm. I, I think 'cause you have to, in system thinking, you have to look at the world. Um, extremely analytical from one side and, and like actually try to put some things together.
[00:38:12] Yeah. Um, that are not intuitive at times, but I, I, I think system thinks this is one of the things that is good because in system thinking, then you could start expressing these things in natural language. Then you could express then the outcomes that you want depending on these inputs. And then you could let software be the middle glue of those things.
[00:38:32] So I think in the future, system thinking is gonna be really powerful. Um, overall, some people tell you too, like thinking through first principles would actually accomplish the same thing. Well. Thinking through first principles. Easier said than done the majority of the time as humans. Psychology is another one.
[00:38:50] We don't teach psychology or really some these other softer uh, skills in both education. In, in both, you know, I would say middle school, high school or university in my opinion. And how to then end up designing and interacting with software and ai, I mean, clearly math and. Anything that has to do with, with software is always something that we deeply encourage.
[00:39:14] And then specific on that, I was in Wai and I was really impressed. Uh, we were also there, um, visiting some kids. We enable 126 kids to be able to, you know, do their first get command, understand what copilot is, and they were teaching them loops. I think like teaching that from a very early age, I think is.
[00:39:34] Really powerful and, and we probably need more innovation and try to see if that creates what we want at the end or not in the kid. Meaning like the, the, the teaching itself actually allows them to understand things a little bit better when it comes to programming those type of things, which is like, hey, teach things are first principle, and then from there you could understand a lot more.
[00:39:53] Like I always, I was taught that F ISS two ma, like if you really understand F ISS two ma. You understand a lot of physics. Yeah. And you can solve a lot of equations. Yeah. And I don't think the education system that I went through really encouraged me to look at things from the base of it, from the foundation, and from the first principals.
[00:40:14] And that's what I think we need to continue to do for this kids and for the next set of programmers in the world.
[00:40:21] Dan Blumberg: Amazing. Mario, thank you so much.
Thank you, Dan.
That's Mario Rodriguez. I'm Dan Blumberg, and this is CRAFTED. CRAFTED. is brought to you in partnership with Docker, which helps developers build, share, run, and verify applications anywhere without environment confirmation or management. More than 20 million developers worldwide use Docker suite of development tools, services, and automations to accelerate the delivery of secure applications. Learn more at Docker.com.
Special thanks to Artium, where I launched, CRAFTED. to see how Artium can help you build your future at Artium.ai
And CRAFTED. is produced by Modern Product Minds, where my team and I can help you take a new product from zero to one and beyond. We specialize in early stage product discovery, growth, and experimentation. Learn more and sign up for the CRAFTED. Newsletter at modernproductminds.com.
Please share CRAFTED. with a friend or have your copilot do it for you and see what your friends say.
[00:41:19] Mario Rodriguez: Nope, you're completely wrong. It is completely subjective. I know what I'm doing. You are a machine.