Subscribe
Share
Share
Embed
Free Form AI explores the landscape of machine learning and artificial intelligence with topics ranging from cutting-edge implementations to the philosophies of product development. Whether you're an engineer, researcher, or enthusiast, we cover career growth, real-world applications, and the evolving AI ecosystem in an open, free-flowing format. Join us for practical takeaways to navigate the ever-changing world of AI.
Michael Berk (00:00)
Welcome back to another episode of Freeform AI. name is Michael Burke and I do data engineering and machine learning at Databricks. I'm joined by my cohost. His name is...
Ben (00:08)
Ben Wilson, I attempt to avoid complexity at Databricks.
Michael Berk (00:12)
Wow. Very on topic. Ben, what does that even mean? What is complex? Well, actually let's, let's take a step back. So today we're going to be talking about minimizing complexity in your developments. Complexity might need a definition. Ben, when you think complexity, what does that mean?
Ben (00:28)
It means for code, right? ⁓ it means if I'm reading through something and like my, my unskilled brain has to like really work hard to understand what the heck is going on and why, not so much why it was built the way it was, but more of just how does this work? If I'm looking at
at a module of code and assuming that I understand where, you know, utilities are defined or interactions with, with other parts of the code base. If I have all of that up in front of me and I'm looking at it and I'm like, I don't get why this needs to be this hard to use or this hard to, to extend. That's when I start saying like, okay, this is complex.
And it's not something that's just intuitive looking at it. That's a sliding scale. I got to admit. So if I were to look at code bases that I contribute to now, if I compared what my capabilities of determining whether something is simple or complex from say five years ago, it would be dramatically different. I would have found more things like very complex than I do now, but there's still things that I look at.
at least once or twice a week that I'm like, huh, this is hard. And it's not like this is empirically hard. It's hard for me.
Michael Berk (01:50)
Yeah, exactly. So complexity can take a lot of forms in the AI space. It's not just code complexity. It can also be ⁓ dependency complexity. So the amount of packages that you're using, it can be infrastructure complexity. So if you have a bunch of model serving endpoints or vector search indexes or whatever it might be, the more of those often the harder it is to manage. And there's a variety of ways that complexity can be introduced. But before we get into why complexity is
probably bad? Ben, I'm curious, when you think about a hard problem, do you think that complexity is the most challenging aspect to you, or do think there are different definitions of hard that are more challenging?
Ben (02:31)
That's an excellent abstract question.
It's not so much like coming up with a solution to something is hard. ⁓ It's almost like if I have a new feature that I need to build, I can pretty much solve most things now with just brute force attempt of trying to make it work. I can build prototypes of things.
The hard part is how do I make this so that it's as easy as possible for a user to use? And how do I remove as much of that user interface complexity as possible? Just like, what is the minimum API signature that is required here? And how can I, if complexity is needed to solve this problem.
How do I hide that from the user so they don't have to see how the sausage is made? And then how that sausage is made, how do I make that and write that implementation in such a way that other people on the team can look at it be like, yep, makes sense. Totally understandable why this is built this way. And also future me or future people on the team when they need to change that code or add new features to it or.
Michael Berk (03:27)
Interesting.
Ben (03:46)
maybe rewrite the entire thing from scratch, it's way easier for them to do that.
Michael Berk (03:50)
Got it. All right, well, this is soon becoming a rabbit hole, so I won't go too deep. But it sounds like the challenge that you typically face these days isn't the amount of moving parts. It's the amount of different stakeholders and your role to cater to all those different stakeholders, so the user and your team.
like balancing priorities.
Ben (04:10)
Yeah. Like if,
yeah. So you think about your customers, which are your users of your product and you want to make it something that
It solves their problem in a way that they don't feel stress using it. They're like, yeah, this solves this need that I have, but this API just like sucks. in order to use this API, I have to like build in all of these other APIs to interface with this. So if you're pushing all of this cognitive load onto your users of like, hey, you got to learn all this shit that we built. Hey, we built it for you. Just like figure it out.
Michael Berk (04:26)
Mm-hmm.
Ben (04:49)
Like either nobody's going to use it and they're going to dunk on it. They're to be like, this sucks. Like it's so bad. Or they'll have to use it because it's the only option out there. And they're just going to feel miserable about it because they had to waste hours of their day while they're, they've, they're already stuck in shitloads of meetings. They've got other development tasks they need to do, or they're just trying to work on this project to get it shipped.
And now they have to go and read your fucking docs to understand how to use this API. That sucks. Like you shouldn't do that. Right. And we try to avoid that as much as possible. We're not perfect at it. I've made plenty of, of like bad decisions that have been shipped into prod and you've you'd end up feeling bad about it. Sometimes you're like, this sucks so bad. Like I gotta go and fix this before that our next release or
before the end of the year, like I want to redo this because it's so terrible. And sometimes we get to do that, which is awesome. Like deprecate APIs, create a new one and be like, Hey, this is, we thought a little bit harder about this and this is kind of where we want to go here. But then that other aspect of like people that are dependent on you, like the other people that maintain that code base, like you got to think about them and you got to get their approval to merge anything. So if something's a true abomination, which I
Michael Berk (05:53)
out it.
Ben (06:05)
done it many times, you'll get tons of PR feedback of like, hey, maybe we don't want to do this or maybe we want to think about this a different way. hopefully design phases can help to avoid stuff like that. Sometimes not. But yeah, you got to get the approval of the tribe. And they have to think, yeah, this is good way of doing it.
Michael Berk (06:14)
Got it.
Yeah, that's actually a really great segue. So let's take an example where you're building out a feature and you have to cater to different users. It's both your internal team and the users of that feature. And you want to minimize complexity. ⁓ The way that I think about this is you basically have a big high dimensional space of potential solutions where I can go this way or this way or this way. And it's theoretical and abstract.
And then what your job is, is to search through that space with increasing complexity and stop basically once the criteria of good enough are met. That's how I think about it. Do you think about it in a similar way?
Ben (07:08)
Not really, man. I think we think about it the same way. ⁓ Mine's probably, instead of thinking of all of the possible permutations, because they are literally infinite, it's incremental, but it's the same sort of thing. What's the dumbest, simplest thing I can do to maybe make this work? And I'll go and write that and then test it and be like, okay, that met two of the seven requirements.
I need to do this other thing as well. Okay. Let me build just like that in and see how that feels. And you iteratively introduce additional features and potential complexity. I wouldn't say I like write code that way, but that's my mental model of it. I've like only introduced what is absolutely needed to solve this problem and don't get fancy. Can I do fancy? You're goddamn right. can do fancy.
Michael Berk (07:41)
Mm-hmm.
Yeah.
Ben (08:02)
⁓ and I've done it in the past, where I write something sometimes like for a fun prototype and just go hog wild with like, I can use this like super cool thing that's in like AST or in the inspect module in Python. And like, I can, you know, at runtime, inspect the state of this object and do conditional logic. When you look at the code that you've written like that, it's like,
Yeah, that's cool. ⁓ Is this ever going to make it into an actual PR? Hell no. Because it's so fragile. You're introducing things that you don't understand, like all the side effects, or I don't understand all the side effects. So it's just super dangerous.
Michael Berk (08:41)
Yeah.
Okay, cool. Before we get into how you actually go through this process, let's just take a step back. Why is complexity bad?
Ben (08:51)
Complexity is bad for a number of reasons. ⁓ One is maintainability. It's it's so much more mentally taxing to understand what the hell is going on when a bug happens. Number two, it's harder to test.
Michael Berk (09:10)
Well, hold on, hold up. Do you have a horror story?
Like I feel like people who haven't maybe been on the hook for on-call or like maintaining production code, they might not know what it's like. Can you just in a couple words explain what that's like?
Ben (09:22)
I mean, you get a support ticket coming in that, worst case scenario is you get a ticket at, 9 AM. You're like, oh, jeez, this user is reporting this weird bug. OK, I'm going to go look at that. And then at 9.40, you get a second independent report from another user that's the same bug, effectively, but in a completely different mechanism.
There might be like a reproducer associated with it. Like, here's some example code that shows it. Or it could just be like, Hey, like this is totally broken. And you start working through combining these two things, kind of getting a mental model of like, what the heck is broken? And then 15 minutes later, third, and you're like, we have a serious incident here. I need to get this fixed. Like well understood and fixed. then test coverage added.
as soon as possible. And yeah, it's stressful as people are pinging you and asking like, what's the update? my customer's workflow is totally broken. Yeah, it's.
Michael Berk (10:22)
If there's
a sev zero at Databricks, what happens?
Ben (10:25)
For our team, ⁓ I yanked the package. It's usually something like that happens right after release. Somebody's got an unguarded install of MLflow. And it's happened twice in the last four years, where we released a package that was fundamentally broken. One was it broke Databricks model serving, ⁓ which is not good.
⁓ so we yanked that package, I think less than 45 minutes after it was released, just yanked it from bye bye and then applied the fix and re-released later on that day. ⁓ and the other one was. ⁓
the details of it, but it was just, it was self-contained within MLflow. Like tracking wouldn't work. We just fundamentally broke something. ⁓ I think it was something with the dependency that just broke. So we had to yank the package. So that's SevZero for that. SevZero for Databricks is usually because of how complex the platform is and how many different engineering teams involved in it. You have something that
affects like a large number of customers and it's usually impeding their ability to use the platform. A lot of things happen like war rooms get called. You have to dial into a meeting live while you're doing like figuring out what is the mitigation here? How do we roll back? How do we fix this? And there's, there's run books on all of this stuff. Like you open up docs that teams go through checklists and
It's very formal and very involved.
Michael Berk (11:56)
and you don't sleep until it's done.
Ben (11:57)
Unless you can hand it off to a secondary on-call first. If it's something that's so complex that nobody can really figure out what the mechanism is, those are the worst. Like, hey, customers are reporting this issue. I can remote into their session with their approval. And we're looking at this failure mode on multiple different customers, but we can't reproduce it on our side. Those are the worst.
Michael Berk (12:21)
Yeah.
Ben (12:21)
where
you have to start pulling in multiple engineering teams, people that are specialists in security and networking and file store, like IO type stuff and compute. And yeah, it gets crazy.
Michael Berk (12:34)
Okay, so higher complexity leads to a higher probability of those very scary bugs with war rooms and many teams. So maintainability, that makes sense. Why else is complexity bad?
Ben (12:44)
Yeah.
The other point that makes it really bad is it's really hard to test. So if you're doing an implementation where you have just lots of low level code that interfaces with, you know, the kernel in a way that it's really hard to follow reading the code, what is actually going on, or you're dealing with
like low level base programming that's interacting with the language itself. Like those things I would classify as fairly complex. ⁓
Generally, only really do that if you really need to and you know what you're doing. And there's got to be like a real justifiable cause there. And the trick with that is how do you break it up into discrete pieces that can be testable so that you can be more confident that, this is what I'm trying to do. And in order to guard against or to aid in debugging.
either during development or later on once this thing is shipped, if there's a bug that is discovered in it, it'll help you diagnose it faster and potentially come up with a fix that minimizes the blast radius of a change.
Michael Berk (13:50)
That makes sense.
Ben (13:51)
Yeah, like testability and extensibility is another one. Like, hey, what if what we built wasn't exactly what people wanted? And you now need to add this other key critical feature that you just weren't aware of when you're building it. If you're digging into super complex code and you have to add additional functionality to it, the more complex it is, the harder it's going to be to do that without introducing bugs.
Michael Berk (14:15)
Yep. Then I'll add a third one, which is it just takes more time. Like your job is to solve problems. It's not to build cool shit. Usually the cool shit is hopefully a nice bonus. And so if you're going to spend 10 hours solving a problem versus one hour solving a problem, just choose one hour. ⁓ and at least for me, sometimes I get a little carried away with the cool stuff and, ⁓ over engineer, but
Ben (14:20)
Yeah.
Michael Berk (14:41)
Getting burned from the maintainability perspective will quickly have you change your tune on that. I now almost never do that. But sometimes it's too hard to resist. ⁓ yeah, if you can have a one line change versus a hundred line change, always use the simple
Cool. So that's why. Now, now that we sort of have like a motive to direct us towards a specific style of solution. I think the next step in this process is understanding how you build out the solution space. So the way that I think about this at least is I think about it like hyper parameter optimization as a metaphor. So let's say we're building a random forest model, a linear model, whatever it might be.
You need to tune things. need to change hyper parameters, like for random forest, the number of trees that we're going to be building, the sampling rate, the density of density is not a parameter, but like all of these different parameters that can change how random forest model will be fit. Um, there's a bunch of different values they can take. So max depth can be one to a hundred and all these others can be true false, whatever it might be. And as you think about all the different combinations that creates hyperspace.
where the number of dimensions is the number of parameters you're messing with. And then each value, whether it be discrete or ⁓ continuous, those will then be different points in your hyperspace. So that's sort of the solution space, but let's scale this out metaphorically to be all the possible solutions for building feature X. Let's say we are predicting hot dog sales and we're using a
Ben (15:59)
He
Michael Berk (16:14)
let's say AWS we're building a online model that has some time series aspects and it's going to predict basically where the highest amount of hotdog sales are going to be in New York city, by hotdog cart. So there's a lot of different ways to do that. Like on one end of the solution space, can go clap five times and then roll a dice. And then whatever that dice says, that's the number of hotdog sales at that location.
Ben (16:36)
you
Michael Berk (16:39)
Conversely, you can train a big neural network on all of your historical data. That's another solution. You can do a time series model. So metaphorically, we're now building out the solution space with different levels of complexity and different levels of efficacy.
Ben, I have a question for you. Let's say I am even a senior engineer. I've built some of these things in the past. How would you go about mapping out this by definition infinite solution space?
Ben (17:06)
So I do it the same way that I would with like framework code, which is what's like, try the simplest thing first. Like what is the, the, the roll of the dice. Like let's fake this. That's bad. But how do you meet the actual requirements of what the product description is? Like, Hey, we need to, we need to forecast out sales on this thing. Well, can I do that with.
a heuristics based model. Like is the data.
Michael Berk (17:33)
Well, how do you know what the simplest
is? taking a step back, like how do you know the solution space? Let's say you've built some of these things in the past, but how do you, well, what I'm trying to get at is how do you know what you don't know?
Ben (17:44)
so we're assuming like we're fresh out of school, first job. We're trying to figure out like, do I predict hot dog sales? Go ask somebody. Like ask somebody who's done this before. And by asking somebody, that doesn't mean go find the most senior person on your team and ask them a bunch of silly questions. It's very senior people have written books about these topics. Go read them. Like get acquainted with
Michael Berk (17:51)
Well, even if you have built s- Okay.
Ben (18:11)
the way that they have approached this, learn from their failures. The best books that are out there talk about like, hey, here's ways of doing something right, but here's ways of not doing something right. And you can kind of get a perspective on how you should mentally approach things. And you get exposed to these topics and you can go and read more in depth on these things. So you need a base level of knowledge of understanding
what's out there, it's potential ways to solve something. If you don't have that and there's no book to refer to, then whoever's giving you that task should know whether you're comfortable with doing it. And if not, they're going to like pair you up with somebody who's more senior, who has solved problems like this in the past. And then they should be your sounding board for ideas and questions. And maybe they do most of the work and you're there to watch.
So you get experience with like, okay, I contribute like 20 % to this, or maybe I did 80%, they did 20%, whatever that mix is, at least you're getting experience from that person and they're reviewing everything that you're doing. So mentorship is super big in this space, whether it's Gen.ai, ML, AI, or even software engineering, it's super critical.
Michael Berk (19:25)
How do you know when you know enough to start building the simplest solution?
Ben (19:30)
You don't, to be honest. Speaking from personal experience, there is that ridiculous level of self-confidence that comes once you know just enough that you think that you can figure this out and you'll go and do it. And you could be somewhat successful. You could make something that's pretty good. A couple of people have some tips on how to make it better.
But you're going to get from that when you actually ship that this confidence level in your own abilities that it does not align to reality. Some people never escape that, right? They never have peers to challenge them or they never have self-reflection to like think back on, I do like a really good job here or do I need to get better at doing this? But
Assuming that you're in an environment where there are people who can challenge you, you're going to go through almost that trough of disillusionment. it's, it's, I mean, what we're talking about here is the Dunning-Kruger effect, right? Which is when you don't know very much, you think you know a ton. And then there's this period of realization that hits you of like, I don't know as much as I think I did.
and you almost feel like you're completely useless and don't know anything. And then as you learn more, your skill actually goes way, way up provided that you follow this process of self-discovery. But the more that you actually learn and the better you actually are at it, the more your skill versus your own self-assessment diverges in the opposite way. So the people who are best at things
who are really, really excellent don't have a high opinion of their own skills. They kind of think like, not, like, I don't know that much about this thing and I don't know about how to do that, or I'm an idiot about this thing, or I'm really bad at this. Whereas to outside observers or even equal peers, they're like, no, this person's really good at this. Like, they just think they're not that good. And that's how you get that sort of humble approach to everything's kind of scary at a certain point.
Like every time you're working on something new, you're like, I'm probably going to screw this up and I need to really think through this and test this out because I don't trust myself sort of thing.
Michael Berk (21:48)
Yeah, exactly. So the feeling to you is you start off with not even knowing what you don't know. From there, you get sort of this false sense of confidence, be like, this probably isn't that hard. And then the next step is a lay of the land and actually understanding the amount of complexity involved and knowing that there's so many areas that you just have no idea how they work or how to operate within.
Ben (22:00)
Mm-hmm.
Michael Berk (22:13)
Is that when you stop mapping? When you start knowing what you don't know?
Ben (22:14)
Yeah.
No, like that. once you get to that point where you, you're aware of the vast majority of what you don't know is stuff that you'll never know, or you'll never be like super deep knowledge of this thing. When you're operating in that mode, you can, you can now approach things like selective complexity in an intelligent way, which is because you don't know how dangerous something is.
Like in software, like, I'm going to use this package because it seems really cool. And it does all these things. When you really know what you don't know, you know what to look for in that package and how to evaluate it and do a fairly quick assessment of like, do I understand everything that's going on in here? Like as I'm reading through the docs or like reading through the source code, is this very straightforward to me?
Like it says everything just click. If you have that reaction, you understand everything in it without having to really look at anything up. And you're just like, yep, that's exactly what I'm looking for. Sweet. I'm going to use this. Great. Awesome. Then it's no longer complex, right? It's like a pretty good solution for what you're looking for. But when you go and just pick something random that, you know, people have said,
solves this problem or it's super fast or like, this is like the coolest thing out there. You go and start reading through the docs and the source code and you're like, no idea how this works or like, what is the mechanism that this does what it does. Now you're playing with fire and you're soaked in gasoline. Like you're going to get burned in that situation unless you spend the time to grok what the heck you're looking at.
Michael Berk (23:51)
Yeah.
Got it. Okay, cool. So that generally concludes us mapping out the solution space. So I know I can do X, I can do Y, I can do Z. Each of them have different pros and cons and different levels of complexity. How do you iterate through this, again, infinite solution space, starting at X or starting at Y or starting at Z? What would you pick to do first?
Ben (24:18)
I mean, I always try to pick the easiest thing. like whatever, is there something that's out there that makes the number of lines of code I have to write the smallest possible, the tests super easy to write. And it just generally works. Like I don't have to mess around with a bunch of configurations. I don't have to set environment variables. I just use the API. The API is a couple of arguments.
It does what I need it to do. And then I'll go and write that and then write a crapload of tests. And then I'll also start working on integration stuff where as I'm iteratively developing this new big feature, I'll have this script that's garbage code that's just running things in a real system up until the point of what I've built. Like,
I did the SQL store backend, like the database storage layer. Okay. And I use this, this approach to solving this problem. Well, I have this script that will start up a database and create the table needed and it'll run through using the APIs. And then I'm doing stuff like make sure that data is in the database or just, okay, I wrote the thing that writes to the database and then I need to write.
the thing that reads from the database. And then I have the object that I went into the database and then the object that came out. Make sure those are equal. Are they equal? Okay. Cool. Got that work. And my interface supports like the input of I can be a string or I can be a JSON string. Okay. Now I need to write a bunch of like
Python dictionaries and a bunch of different types and make sure that JSON serialization works properly between these things. And that's what that testing is doing is as I'm adding new functionality, whatever the scope of this thing is, I'm testing it as I go. And every time I add a new component to it, I'm running that script with the additions to it to make sure is this all working. But while I'm doing that, I'm writing discrete unit tests for each of these core components.
because I don't trust myself.
I don't trust that I'll do that without making loads of errors. So test as I go.
Michael Berk (26:24)
Okay, that makes sense. But going back to the original question, how do you, like, what's the intuition for you feel like that this is gonna be the starting place? It's like, you generally know what it would take to build it and it seems pretty easy relative to the other solutions. You've built it before and you're confident it'll solve the problem. Like, what are the things you think about?
Ben (26:45)
If it's something that I've built a version of before, I already have kind of a mental map of what's needed. Like, I need a new REST API that does this thing. Well, I need an interface to talk to the backend. Then I need the backend implementation. Then I need the storage layer and everything going the opposite direction. I need to build that whole stack. I have done that so many times now that I just kind of intuitively know what's involved in all of that.
When you need to deviate from that, that's when it gets a little tricky, but you just write your thoughts down on paper and leave them for a day. Come back and read them and see if that still makes sense. Give yourself a bit of time to kind of think through the problem. And then with a fresh set of eyes the next day, it could be like, yeah, that sounds reasonable or...
Wow, that sounds way too complex. Why did I think that that was the best way to do this? Let's try this simpler way. That's my process. It's like trying to pursue simplicity as much as I can.
Michael Berk (27:46)
Yeah, I'm, think an interesting case because I'm really bad at learning things without doing them, or I'm good at learning things by doing them, which is the inverse. So often it takes a good amount of time for me to map out all these solutions. I'm still pretty quick with it, but if you're like me in that space, I really find it useful to just create a document.
Cause that gives you the feeling of having done something and basically outline every single solution. So conceptually it can be a design doc where you have pros and cons of each solution type, as well as sort of first principles or tenants of what the problem is looking to like, or leverages. And also having success criteria of like must have, could have, should have. Those are really great ways to anchor and structure. ⁓ if you potentially like to be very hands-on and just want to be building stuff.
⁓ Ben, for you, I know you have a very good brain for understanding complexity. ⁓ do you write stuff down typically? What does this mapping process look like? Or do you just read a textbook and then you now know what to do.
Ben (28:53)
read a textbook. That's funny. Not that. ⁓
I've learned all this stuff like the most painful way that you can, which is similar to what you're saying. I learned it by building abominations and learning how bad of an idea that was, or working on projects in isolation, which I don't do that anymore because I'm on a team that, you know, we have professional software engineering development going on.
But stuff that I had done years and years ago when I was like a data scientist or an MLE at previous companies, I go off and build something and I had no context of, I building something that is the worst thing ever or?
Like, does this even meet the needs that we have? I would usually write down on some sort of document, not as structured as what I do now. Where what you mentioned, we do that, but on steroids, with like design documents and product requirement documentation and Hilemiers and all this stuff that's like formal engineering approaches to anything that you build of
sufficient complexity. ⁓
Yeah, like when you're doing it on your own and you don't have a mentor figure checking what you're doing, you learn a lot by just building garbage. Like, I want this to go as fast as possible. I'm going to use this obscure internal thing in this language and I'm just going to, I'm going to make it go. And you find out six months later after working on this project and you're like,
Man, it is so painful to add new features to this thing because of this backend that I built that is so insanely complicated. And now I've developed myself into a corner where in order to keep going with this backend, I can't actually build these new features. It doesn't allow me to. Oops.
So then you go and rewrite it all. like, okay, now I need to actually get this thing working. I'm going to rewrite it from scratch and make it simpler. And then you look at that and you're like, okay, now I can add this feature and these other seven features that I wanted to add. This is so much easier. So I learned it by just doing it like that. It was not a quick and easy process.
Michael Berk (31:03)
Mm-hmm.
same. I just relentlessly build shit and it'll break a lot of the time, but it's kind of fun. And it's also something that's pretty self-sustaining once you get into the rhythm. I feel like it's hard to enter and exit those modes, but if you're in a flow state and like just coding for four hours, you can try a lot of stuff very quickly. So food for thought.
Ben (31:28)
Yeah, and one of the things that did help me out a lot with understanding why things should be built in a certain way, particularly when you're talking about like...
What triggered this entire episode? Like before we started recording, we were talking about asynchronous calls. Like how do you get maximum performance interfacing with like a REST API? And if you had asked me that question five, six years ago, I would have had a completely different answer than what I gave you. Like completely different. I'd be like, in order to get maximum performance, we should be thinking about, you know, running threads inside processes.
or let's kick off a bunch of sub-processes and we'll run thread pools in each of those. I wouldn't have known that that's a bad idea because I didn't know, like, how does the CPU actually execute code? What is the concept of, of like hardware isolation of things? And what are the limitations of hardware? And how does this language that we're writing this thing in, when it does something like creates a new process, what is it actually doing?
in order to do that needs to have like a memory, like an isolated memory. Like a new process is like basically starting up Python again from scratch. There are some aspects that are shared, but effectively every time that you create a new one of these, you're creating multiple versions of the same application running. And then threading is a little bit more, a little bit more interesting where you start talking about.
How do I get dedicated hardware resources for this particular thing that's executing right now so that it doesn't have contention with other running threads? There's limits, there's physical hardware limits on the silicon about how many of these things you can start and how many processes that's like your memory limited. So when I started to understand how the hardware to software interface worked, I started to understand a little bit more about like,
training my brain to think about what is actually happening in this thing, this feature that I'm building. Is this thing doing a bunch of computation on the CPU or is it talking to a remote computer? Is it talking to a database? Is something else doing the work? And then you can start thinking about, okay, since all the work is being done here, do I want parallelism here? Do I need to limit it in some way?
And it starts getting a better, it starts giving you a better understanding of what these tools can do and why they're used for certain things and not for others.
Michael Berk (33:46)
Exactly. Yeah. And do feel like you know what you don't know in that space? I would hope so at this point.
Ben (33:53)
I'd say I can identify when I'm out of my depth fairly well. When I can look at something and be like, I don't know ⁓ what the solution is here. So, but I do know that I need to block off three days of time to go and read a bunch of stuff, test a bunch of things and understand in greater detail what the hell is going on here before I write any sort of that
Michael Berk (33:56)
Okay.
Ben (34:19)
would be going down this path. But usually now it's more, I need to prove to myself that I can't make the simpler thing just work. I'd say 95 % of the time now, on most things that I work with, the simpler solution just didn't come to me because I just didn't think of it yet. But giving it a little bit more time adapting a very simple way to solve something is successful 95 % of the time.
Michael Berk (34:35)
Mm-hmm.
Ben (34:43)
And I have that 5 % of, oh shit, like I actually need to use this like low level construct. This is going to suck. And it's going to take me 10 times longer to do this. And I need to write probably way too many tests to verify like what I'm doing is not completely stupid.
Michael Berk (35:02)
Right. Okay, cool. Final question. In the age of Gen. do you have tips, tricks, advice on minimizing complexity?
Ben (35:11)
Yeah, treat coding agents like they're the most ambitious person two years out of college. So they can write code super fast, but they don't know without you expressly telling them to how to minimize complexity. If you let one go off, off the rails and be like, Hey, I need to build this thing that
you might think is a good set of instructions to give like, I want it to be able to, you know, query these rest API endpoints the fastest as possible. It takes that at literal face value. And if it doesn't have any context of like reference code or like what the hell you're trying to do and what it is that you're looking for the system to do, it could just go like, okay, you want the fastest way to do this. Here you go. I'm going to write this. And
I haven't seen it do without prompting. I haven't seen a cloud code go really off the rails in this, in this case. ⁓ but I did prompted it a couple of them, weeks ago. I was like, I want the absolute fastest way to do this thing. Like no holds barred, like just build me the fastest way to do this. And it started writing like Scython and I'm like, cool. ⁓ we're not going to ship this, but thanks for that.
That's fascinating. ⁓ And of course, my response to it was like, do you know how much memory to allocate for some of these objects? It's like, you have an excellent point. You're absolutely right. We need to have protection against that. Let me write 30,000 lines of code in here. like, okay, stop. So, but when you're interacting with it and it's operating on a repository that you maintain, there's something that you're very familiar with.
Michael Berk (36:37)
Yeah.
Ben (36:53)
You need to tell it, like, hey, check to see how this is done in this other location in this code base. And here's the feature that we need, this new functionality, and see what you can do. Make sure you write tests and stuff and things look the way that the rest of everything else looks. It can replicate that style, but check the code that it comes up with. Like, really check it. And then start challenging it.
saying like, Hey, I don't think that this is the simplest way to do this. Can you look at this again and give it evidence of like why you think it's not the optimal way to do it. It'll go and fix it. It'll be like, you're absolutely right. Let me go and like simplify this. So a lot of my time working with my buddy's sonnet four or five, ⁓ is
is a lot of that. It's like it goes and writes like some part of a module and then a bunch of unit tests and it runs them and make sure that everything's passing. And then I go and look at the tests. like, can you stop mocking everything? Like we have a fixture for this. So test this for real. And I also don't think that this is correct the way that you did this. And we should probably break this function up into two separate utilities because that functionality is mirrored in this other function.
So how about you go and do that? And it doesn't understand how to do all of that because of the way it's trained. It's trained to make self-contained things. It'll put like local imports everywhere because that's how its context window operates. It doesn't understand unless you're, you're prompting it continuously. Even with system prompts, these things aren't perfect. ⁓ but getting it to build.
Clean code is not what I think a lot of people assume, like how easy it is. It's not just like fire off like, go build this for me robot. And it just produces like exactly what you're looking for. It's not like that at all. It's it builds something. give it feedback over a period of like 20 back and forth session, you know, statements, and then I'll go in.
edit the code manually and then tell it that I just edited it and say, okay, now that I've cleaned that up for you, can you now build this next part of it? Thinking of like, look at what I just did and how I just did that. Now you replicate that for this next thing. And at that point when it has that in its context window, it's pretty good. It can kind of learn within that session, like what I'm looking for.
Michael Berk (39:01)
right.
Right. So hope it's still complexity. Got it.
Ben (39:23)
Yeah, because otherwise it'll in that question that you asked before we started recording, it would just go and build that. You'd like, you want an example of multiprocessing with multithreading ⁓ inside each of the processes. It'll go and write that for you. It won't tell you like, Hey, this is a bad idea. Like this could blow up in so many different ways and it's going to be so insanely complex. Not to mention there's no guarantee that this will run on differing types of hardware properly. So.
Michael Berk (39:38)
Mm.
Ben (39:50)
that massive, you know, VM in the cloud that you've been running, that you're going to be running this thing on. It could have a completely different, you know, it's going to have a completely different hardware specification than what's running on your laptop where your unit tests are running. So what could work for what you're trying to do might blow up in production. So you need to test all of that out.
So the more complexity you add into those systems, the worse it's going to be for you. And then you just start to to bolt, like bolting on more and more fixes to the super complex thing. And you're just playing whack-a-mole with like stability things. So yeah, screw that dude. Like I hate the idea of that actually like puts a pit in the bottom of my stomach. Just thinking about that.
Michael Berk (40:15)
Exactly.
Yeah.
Ben (40:37)
Like what would that process be like for me to get this stupid thing running? No, thanks
Michael Berk (40:43)
Yeah. And also if you're, if you, listener don't have that pit in your stomach, I suggest go and learn it the hard way. Like I used to not have that pit in my stomach and then I learned it the hard way and now I have that pit and I hate complexity. it's, it's really like a useful lesson to learn and learn it quick. ⁓ so yeah, anyway, I will summarize. We mentioned why complexity is bad. Maintainability suffers when complexity is high. It's also hard to test and it also takes extra time to build.
So if you can not do those things, don't do them. ⁓ How can you learn about solutions is the hard part, I think, which is like mapping the solution space and figuring out where you should start and how you should iterate through that solution space until you meet the criteria. ⁓ Step one is just like look around, read books, Google, use LLMs, talk to people, try to get a lay of the land of what is possible and what are the areas that could actually solve your use case.
mentorship is really important. If you can collaborate with someone who knows it's a lot easier. ⁓ and you can generally stop searching around the solution space when you know what you don't know. So you have areas where you're like, I generally know what this is like, but I'm not going to go further for now. ⁓ once you have that space mapped out, start small and incrementally increasing complexity until you have met the criteria and with gen AI prompt the crap out of it. Don't trust it.
Anything else?
Ben (42:05)
Just don't trust yourself either.
Not just Gen. AI.
Michael Berk (42:08)
Delaborate.
Ben (42:09)
tests, the amount of code that I've seen people push to even staging where there's no testing has broken my brain where I've seen people just like, Hey, it works in dev. And what they're doing is a full end to end integration tests, happy path test. And then they pushed to prod and things work for like a week.
and then they stop working. That process of trying to diagnose what the hell just went wrong. If you don't have tests to isolate the complexity of your large application that you're deploying, good luck. Like you're going to waste so much time debugging that without, you know.
Just applying basic dev principles of like, hey, use functions instead of like a massive script. Use some level of abstraction to understand how to discreetly test different parts of your project. Tester key.
Michael Berk (43:04)
Well, until next time it's been Michael Burke and my co-host and have a good day everyone.
Ben (43:09)
Ben Wilson.
We'll catch you next time.