Ian Nowland was at AWS in its early days, and is now Head of Hardware at Baseten. Camille Fournier built infrastructure at Goldman Sachs before the cloud took over, and now is a VP of Engineering at CoreWeave. Together they wrote the book on Platform Engi
Ian Nowland was at AWS in its early days, and is now Head of Hardware at Baseten. Camille Fournier built infrastructure at Goldman Sachs before the cloud took over, and now is a VP of Engineering at CoreWeave. Together they wrote the book on Platform Engineering.
Now agents write code faster than anyone can read it, and the problems Ian and Camille spent their careers on are back: who reviews (or doesn't review) code, how you test it, where the servers come from, and who gets paged at 2am when it breaks.
Presented by Railway, your agent’s favorite cloud: https://go.technically.dev/railway
Original music by The Jazz Boys.
Show notes and the full transcript: https://technically.dev/podcast/all-about-platform-engineering
Corrections
Thanks to AI, everything we knew about software is stale: how it's built, and what it even is.
The best people to bring us up to speed are the ones building it. (Honestly, they're figuring it out too.)
Each Thursday, host Megan Reynolds asks one high-profile technical leader to explain how their work works, and where it's headed.
Season 1 features founders and leaders from WorkOS, Databricks, Datadog, Railway and more.
Original music by The Jazz Boys.
Megan Reynolds: you were CTO Rent the Runway you led platform engineering at Two Sigma and an 800 person engineering organization at JPMorgan and Goldman before them and now you are VP engineering at CoreWeave Ian you were at AWS in the early days and you built some of the cloud services most popular cloud services that we use today you were also at Two Sigma with Camille and then you led platform engineering at Datadog and you're now leading hardware at Baseten. so you seem like pretty good oh and you also wrote the literal book on platform engineering so you seem like good people to ask some simple questions about platform engineering so let's start off I mean Camille you were running infrastructure engineering at Goldman
Camille Fournier: well I was I was an infrastructure engineer at Goldman let's not let's not over inflate my my influence at the time
Megan Reynolds: building infrastructure at Goldman before the cloud was dominant talk us through before we had the cloud what did it mean to put an application into production at that point
Camille Fournier: well first you had to make sure you had a server to put that application on and so you know for a long time the you know there was just a very long procurement process that was you know run by another part of engineering but you would you know you you needed to figure out okay like where am I going to run run this thing? You know, how big of a machine do I need to run it? Do I need, you know, multiple machines? Do I need, you know, also a database?
Camille Fournier: And I may not even run that database. I may need to talk to the database's team, right? And they may need to then get their own hardware, Do we have the networking set up? You know, do I need to run it in multiple data centers? And what would that look like depending on the resilience characteristics of the application? So, you know, know, VMware actually, you know, came along and created virtual machines and some amount of the kind of like OpenStack cloud, which some of the audience may remember, that made it somewhat easier for you to get virtual machines out of your physical infrastructure.
Camille Fournier: And so that that made it a little bit faster to get.
Megan Reynolds: Why were the computers coming from otherwise?
Camille Fournier: I mean, otherwise, you know, they had a they had a they had a long procurement process that would that would happen where they would you know plan however many servers they were going to buy in a year and you know maybe sometimes they would turn off old you know old software turn on new software right a lot of times applications would live for a really long time because you would get your hardware and then you would just deploy to the same hardware for a long time and maybe occasionally you needed to grow a little bit but you would just kind of make use use of the hardware that you you had as much as you could because that lag of months you know weeks to months to
Megan Reynolds: who knows. Wow, really? That long?
Camille Fournier: yeah it was it was a long it was you know it was getting physical stuff you know set up and installed and you know networked and all of that even when it was good it still was you know complicated right and so yeah it was a it was a big you know heavyweight process
Megan Reynolds: that and so what like how does that work with like maybe walk through a specific application something that you might have felt at the time what would your process be if you know that it's going to be months for this to get set up like how how would you start that planning
Camille Fournier: yeah I mean you know there were usually kind of annual planning cycles and so you really were thinking very far ahead what am I what am I going to do in the next year you know what what is growing so I one of the things I worked on was a very like like, large in -memory caching system that actually required, like, servers that were at the very edge of capacity. So, like, the biggest servers we could get so we could fit everything we needed in memory.
Camille Fournier: And that was actually really, that was really challenging because, of course, you know, our systems were starting to grow faster than the available memory in a single machine was growing. So we did various different tricks to try to squeeze that, eventually actually turning it into a distributed system you know which which was the right thing to do at the time but
Megan Reynolds: distributed system maybe clarify that
Camille Fournier: yeah wait so we we would we took something that used to run as kind of a single a single machine you would run the entire stack on the single machine and we said okay well we can't fit it all on one machine because we just need too much data and memory and we can't put that much memory in a server without going to like a supercomputer. So we figured out how to shard the data that we needed to have in memory and basically run it across multiple servers and make the requests over the network to each of the shards to get the data aggregated and then sent back to be sort of finally aggregated on the main machine.
Camille Fournier: Very kind of common MapReduce -ish type of model. But yeah, you would have to think very very far ahead and work closely with your, you know, partners in the central, you know, in the central team that were actually doing the procuring and, you know, make sure you were getting this stuff planned and budgeted for. And it was, it was just, it was just quite a process.
Megan Reynolds: Especially if you then need multiple servers. And so you're kind of starting to distribute out and yeah, it sounds like a lot of planning. Sounds challenging. Sounds like a lot of time time and planning and not very fast process. So Ian, you were earlier at AWS. Where does the cloud come into this? Like what did you guys at AWS see as a problem here that you wanted to solve?
Ian Nowland: So I joined AWS in 2006, where actually Amazon, sorry, I joined Amazon in 2006. I didn't join AWS until two years later. Amazon was basically the server procurement staff was all totally the same. I'd say Amazon was probably earlier in their journey towards a service architecture. There's a fairly famous Bezos memo that gets mentioned where it's like, if you don't move to services, you are fired. Have a nice day.
Megan Reynolds: And what does it mean to move to services?
Ian Nowland: So basically what had happened is, sort of what Camille sort of Goldman is, you'd end up with like 30 engineers from like five different teams trying to collaborate on what you now call a monolith and just end up with like rampant political fights. And I talked to people who were there at the time and it was on stuff as banal as your Berkeley database is taking too much DRAM, who owns cleaning this up? And so it became, it was very much a collective action problem between product teams that Bezos just decided, I don't want to hear any of these arguments ever again.
Ian Nowland: You're moving to services. I know it's not, doesn't solve all the problem. And we see this in the microservices versus monolith argument today. But this is what I'm going to do. I'm the dictator. And in hindsight, I think he made a really good call, particularly for the type of business that Amazon.com was running, which was a retail website. And so they'd invested a lot in what you now call fairly decent CI/CD techniques that were absolutely fantastic for deploying applications rapidly. But actually, the server -side of the stuff was all exactly as Camille said.
Ian Nowland: And again, I think Bezos saw this in 2004, and that's when he kicked off AWS effectively. It's like, okay, somehow this has to become a lot more virtual and a lot less physical. so yeah when I joined like it's funny in hindsight I sort of took for granted what was probably probably bleeding edge thought that sort of had been pushed from above just purely based on the politics that were coming out of Amazon just trying to move really fast.
Megan Reynolds: Wow interesting that it stemmed from actually more like I mean I guess timelines and taking a long time but also like political political challenges and people being upset by yeah different servers not being split between between them okay and so that was the insight behind AWS and the insight was like how do we split this out make it more virtual make it easier to consume and so you worked on a few dominant pretty popular cloud services at AWS what was the thinking behind that what did you want to cover that wasn't already covered
Ian Nowland: so that the first one I worked on they now call EMR but used to be called Elastic MapReduce which was taking this fairly dominant at the time open -source called Hadoop which these days no one who is sane still uses they have moved to Spark but at the time sort of
Megan Reynolds: and what's that what's Hadoop what's Hadoop Spark doing
Ian Nowland: Hadoop basically allowed what you'd now call a data engineer but at the time wasn't a data engineer to sort of apply imperative programming techniques towards manipulating data so before that right it was you shove it in your big Oracle database and you essentially are limited by SQL or maybe T-SQL or PL/SQL these different stored procedure languages but really the what Hadoop meant outside Google, what MapReduce meant outside Google was much more, I can program against it, I can unit test against it, I can actually build software against it.
Megan Reynolds: For what kind of applications? So you're talking about data?
Ian Nowland: Data engineering. So typically anything you think about manipulating large amounts of data, which at the time was nothing compared to what we had today. But it was, I want to say, terabytes, if not petabytes at the time, stuff that would take an hour to run through. I remember an early customer was a dating site that I think went out of business. But doing their matchmaking, say was a one -hour job that had to run every day on time for people to get new matches okay and so it was those types of use cases and so EMR though sort of came out of okay this is difficult distributed open -source called Hadoop EC2 itself at the time just had a lot of paper cuts and probably EC2 is the elastic compute cloud which is the virtualization layer that you know most people one way or another their workload is running on in AWS today so there's
Megan Reynolds: Wow, okay, so that's pretty serious scale.
Ian Nowland: Yeah, exactly. There's definitely higher level primitives today. Lots of people never think about anything other than their container running on Kubernetes. But underneath it all is just, hey, we need to take a physical machine and divide it up into multiple virtual machines.
Megan Reynolds: Okay, and that's EC2, making
Ian Nowland: that very simple. Exactly, that's EC2 making it simple enough. Making it more simple. And that was part of a product pitch for EMR is it was still too difficult for individual product teams at customers to plumb together Hadoop and EC2. So, you know, at the time my manager was like, we're removing the muck. We're just making the data engineers, just thinking about the data engineering part of their job, because the combination of this open source, distributed open source and EC2 at the time, was just too hard to combine and run reliably every day in production.
Megan Reynolds: For the purpose of processing a large amount of data much more quickly. Exactly. Okay, so they would have to kind of string that together themselves. It would take a long time. and so you combined the kind of compute and the storage to
Ian Nowland: make it easier. Again, there's this other primitive S3, which I think everyone knows at this point, it's the big object store, but you can sort of see EMR combine the compute in EC2, the storage in S3 with the processing engine in Hadoop and just made it easy.
Megan Reynolds: That whole end -to -end process
Ian Nowland: was
Megan Reynolds: what made that easier to work with large -scale data. And also, so EC2 was like the, make it easier to have your compute scale.
Ian Nowland: Yeah, if you come back to Camille's earlier thing, right, where you don't want to have to procure things years in advance what you want is service to be smaller and you can just sort of buy dynamically about what you want so that's where you call it a virtual machine rather than machine and that's where like the claim to fame of EC2 was you know credit card call an API two minutes later you have what you need now it was a little bit of a lie because we didn't have infinite capacity in our data centers but we had enough behind most demand that for a lot of people it was magical but just suddenly this wasn't the four four months
Megan Reynolds: versus four months to two minutes there must have felt like I mean do you remember the
Camille Fournier: first time
Megan Reynolds: you guys used that at Goldman
Camille Fournier: I was not at Goldman when they started using oh because that was
Ian Nowland: in Goldman had virtualized right
Camille Fournier: but we had virtualization and it definitely was a you know definitely was a positive change it was still I think the two minutes or the the minutes thing there was more bureaucracy I think in most internal internal internal offerings of companies even at that time but it was you know definitely like Amazon doing EC2 I think opened everyone's eyes to we shouldn't have to put up with these really long slow procurement timelines you know ourselves like we we should make it so that developers at companies don't need to plan for that months and months and months in advance that still has to be planned for by the way Amazon still has to plan for it right like you know someone has to plan in for it at the company or at Amazon or whatever.
Camille Fournier: But get it away from the people at the edges who really don't have any business having to think that deep about it far in advance.
Megan Reynolds: And why didn't Goldman use EC2 early on when it happened?
Camille Fournier: Oh, well, EC2 took quite a long time to stabilize and make enterprise ready, particularly for the rigor that a financial services institution expects. And that's very normal. right it is very very particularly
Ian Nowland: security that really wasn't till 2014 2015 where the product they call VPC virtual private cloud and then the more infrastructure feature they called IAM which is identity and access management really became something that an enterprise could use to safely manage the cloud so it was actually almost 10 years in before for a Fortune 500 enterprise you wanted to use it at any scale
Megan Reynolds: and what was the risk there like what was the reason that
Camille Fournier: I mean, you know, you're still holding sensitive financial data, you know, for for the firm and you almost everything that is built, even if it's not directly touching financial data is, you know, is important to be relied on by some other financial system. right so so essentially you you want to make sure there's no ways to get into the internal systems through this you know through this you don't want to leak any data you don't want to expose any information about your business you don't want bad internal actors external actors there's many many threat models that financial institutions put to bear and I think the other thing to point out is that you know at a big company like Goldman they also knew how to procure hardware and run data centers they didn't they It wasn't like a startup where they didn't want to have to learn that or maybe go to a Rackspace or some other kind of, you know, bare bones provider.
Camille Fournier: Like, you know, they didn't need that. And so there was also, you know, there's also political pressure to be like, this thing has to be actually really good for us to put any of our workloads on. Otherwise, we've already got all these people that kind of know how to do it. They could do it maybe faster, maybe better. But like, you know, we don't need this expertise as desperately as the startups of the, you know, 2000s, early 2010s did.
Megan Reynolds: But still, 10 years is a good chunk of battle -hardening Amazon services.
Ian Nowland: The thing I'd add to it on the technical side is when you run your own data center, you know, the thing that really protects you from the Internet and the outside world used to be these big Cisco firewalls that you hired, grumpy network engineers. and they were another source of friction that sort of oh
Camille Fournier: my god but at
Ian Nowland: the same time it was a for all of it they were grumpy and would take three
Megan Reynolds: months and why are they grumpy because
Ian Nowland: they're constantly being pushed by the organization to open things up so that they can move faster okay yet they personally are taking all the accountability for the security posture of the business
Megan Reynolds: oh wow so they're the responsible point
Ian Nowland: exactly and so the thing with the cloud is it does away with that and suddenly you get the IAM APIs the and like you know the S3 APIs the EC2 APIs and some of your security posture is just spread everywhere and like even like you know a company like Wiz say their whole business is the fact that it's still very hard to secure this you know 100 different APIs versus what one of these painful firewall devices used to be so it's so this is like you know all the problems we're sort of seeing with like AI attacks on infrastructure structure today in real if we were back in 2005 we would have other problems of course but things would be more secure like that would move a lot slower but you could reason about the security posture of your company a lot more easily
Megan Reynolds: so what's the shape of a company today that would maybe benefit from less cloud services I
Ian Nowland: think people buying a lot of GPUs so
Ian Nowland: when Camille and I work together Two Sigma had been very much on -prem and we wanted to move to the the cloud. Once you did the math, even if you were doing the best three years of instances where you're reserving hardware for three years and committing to Amazon for five years, on -prem was still two to three times cheaper. But everything Camille sort of opened with about the frustrations that were true at Goldman in 2005, I guess, were just as true at Two Sigma in 2017.
Ian Nowland: So it was very obvious for unlocking business agility that we should move towards the cloud. The interesting thing with GPUs at the moment though is they're in such short supply that actually you aren't getting much dynamicism out of any of the big providers that everyone is in shortage everyone looks
Camille Fournier: to be
Ian Nowland: in shortage for another 12 months so so if if it's like paying Amazon a bunch of money to rent servers GPU service for three years or doing it on -prem or buying through CoreWeave or whatever yeah it actually looks a lot more it's a much more sensible trade -off than actually when we were at Two Sigma together the people who were like ah on -prem is better they really weren't seeing that for agility it was worse but for GPU you need the GPUs but
Megan Reynolds: where does that cross over I guess you're saying now there's shortages and so it takes a long time to get things so you may as well kind of take on all of the other complex things in that process because you kind of have the time to do it like at what point do you think that flips like where does renting versus buying change over I
Camille Fournier: mean it's it's hard to say because because
Megan Reynolds: at the same time well
Camille Fournier: because at the same At the same time, so GPU workloads themselves can also be very elastic and dynamic. And so the cloud is also, there's a reason that the early things from Amazon were all called elastic. And the elasticity was actually very useful because when you have sort of unpredictable workload demand, it's nice to be able to get compute that you need, get access to resources when you need them, pay for it when you need it, and don't pay for it when you're not using it, right?
Camille Fournier: That's actually very nice, right? Now, if you have a large enough company with a bunch, enough different types of workloads, you yourself can sort of smooth the elasticity and, you know, reschedule things into idle places, theoretically, anyway. It's actually quite hard to do, but, you know, it is possible. But, you know, so with, you know, with the GPUs now on the the training on the training side that can be a very like elastic demand I mean researchers will will get as much GPU whenever that they possibly can
Megan Reynolds: elastic upwards
Camille Fournier: they will they will yes like quota management of GPUs for for labs and other companies running training workloads I'm sure is a is a never -ending challenge and battle for them but you know if you are a if you're a small company that isn't you know that is running some big batch training jobs and then running ramping them down and running some and then ramping them down you may not over time as the price price curves get better you may not over time want to have to have a contract on all of those GPUs all the time the flip side of course is that inference serving once you have a product that people want to buy and they want to use inference serving is much more much more kind of like your typical web serving well you know ish and you might have some some some thoughts on that but but it's a much more like steadier pretty steady or demand maybe not predictable
Ian Nowland: yeah you just think of any coding agent right oh maybe this is changing a little bit with the nighttime coding agents but historically when engineers are in the seats you're gonna see way more usage than you are during the nights and then you want to go sell that to someone else and that's where I guess both our companies trying to build products that find a way to do it but it's still you know the I would say we're early in the elasticity story for GPUs compared to where we got with the CPUs yes yes
Megan Reynolds: and so who would you recommend to be buying their own GPUs at this point
Camille Fournier: well can you get them that's that's actually I mean that is
Megan Reynolds: about I mean you can't get them I guess that's a question like who is able to get GPUs right now I
Ian Nowland: mean you can get GPUs can you get a powered data center with liquid cooling at this quantity you need in the US so
Megan Reynolds: really okay
Ian Nowland: no I do think if you're Fortune 500 and you have high technical bar and for whatever reason you know you do a cost but like I do think it's legitimate that at that size if you can hire good people to do it but you do need really good people I think that's like one of the ways both our companies succeed is that in the end the infrastructure experts GPUs are just way more complicated than CPUs so what
Megan Reynolds: makes they're more complicated
Ian Nowland: you know
Camille Fournier: they've been more complicated I feel like our entire lives they're they're just it's just trickier hardware
Ian Nowland: yeah I think number one like the reason why you offload something from a CPU to a GPU is you're getting much higher throughput you're getting much higher throughput because the programming model is much different you're not running your operating system on there you're running it across a bus today for inference right you're not just using one you're using them together transferring a tremendous amount of data over the network work so it's just a lot of operational pain to to use those effectively if you have a lot of money where outlaying a billion dollars to buy 20 megawatts is fine with you which is a lot of money of course and you can hire a great team of probably 15 engineers to operate them for you but often you find the Fortune 500 have the first but not the second and then startups have the second but not the first and
Megan Reynolds: they don't have the talent they have the capacity
Ian Nowland: and they can
Megan Reynolds: get the data centers but they don't have the talent
Ian Nowland: they can't yeah they don't take
Megan Reynolds: the pain
Ian Nowland: without casting aspersions they aren't paying the same amount as Anthropic is paying in in San Francisco at the moment and so you just don't get people who are willing to bang their heads against the keyboard in quite the same way they just make it work and so so yeah so what you find at the moment like there's definitely companies who can do it but it is definitely they're very very expensive and you need to have very good people and so it's it's probably a lot of CFOs are very curious about on -prem at the moment definitely with data sovereignty as well I think in practice a lot of them will come back to both the cloud and companies like ours for the exact same reason they moved to the cloud in the first place like it make that someone else's problem
Megan Reynolds: okay that's helpful and so maybe like grounding us back to platform and so you know say you're building like a simpler app like a like a ticketing app for example you know you have AWS for for example, so you can rent your compute, you can rent storage, what else do you need? I think one of the things you talk about in your platform engineering book is that cloud is not a platform, and so we have the cloud, what else do we need around it?
Camille Fournier: Well, so the cloud tends to provide all the pieces you need from the platform, right? And it sort of depends on what you want. You usually, you'll want some kind of databases often, right, so you need some compute, you need some storage, you need some databases. databases. You may, pardon?
Megan Reynolds: Why plural databases?
Camille Fournier: Oh, you need a database. I don't, I'm not, I'm don't, don't take anything from me saying databases, right? You need a database or multiple databases depending on the architecture of your system. And so, you know, and then you may also need messaging buses or, you know, you may need, what else am I, what am I not thinking of?
Ian Nowland: I don't know, queue.
Megan Reynolds: Yeah, maybe work through a ticketing app example. So, you know, you won't have a lot lot of people buying tickets at once for example like what are the things that you would need that would go into you know what would you start with
Ian Nowland: I mean honestly like if you were just trying to do a SeatGeek competitor you'd you'd take Postgres you'd take you'd have a bunch of blobs through an S3 you'd choose some type of framework in which you're going to create the web the web the direct thing you want some type of asynchronous workflows I hear you often hear Temporal is the dominant one today but there's been plenty over the years you could do a pretty good job building something at small scale with a tiger team with just those components where platforms sort of come in is okay that's been done in an organization over 10 years you've got five different semi -implementations of that and that new thing you want to do needs to talk to all of them and that's really where a lot of the you know say it you know Camille sort of dates herself with message bus but when you look at where message buses
Ian Nowland: today you just say Kafka today you just say freaking Kafka but but if you go back to 2000 it was a message bus and the idea was okay how do we decouple these different parts oh sorry these different parts of the the team to work independently but share information at the right thing and it's yeah it's a message bus I don't know Moms Moms was IBM's message bus
Camille Fournier: I don't remember that one.
Megan Reynolds: I mean, people would say they would ideally not use Kafka today, so I don't know what people are evolving to.
Ian Nowland: People always hate the... They are complicated technologies, but they're trying to decouple teams. And so people end up hating the technology, but the actual problem with the technology is just inherent, that it is actually hard for teams to agree on stable APIs that allow both to move independently. And so they all get hated on over time. Yeah, there's no... Like, the things that engineers tend to love, like take S3 well you're just usually using it S3 within your own team so you tend to love it but any of its protocol buffers anything that tends to cross teams just becomes a hated technology because it's a collaboration with some other team who doesn't agree with you over some detail and has a totally different road map about when they want to do things and so so it's it's a place of where social disagreement will be pointed at the technology you know
Camille Fournier: yeah and
Megan Reynolds: so where does is platform engineering coming to this? I guess we've reached a level of scale with our ticketing app where we've got a lot of these different services running. At what point, like Camille, at what point do we need platform engineering specifically? And what does that mean?
Camille Fournier: Yeah, I mean, I think, I forget what the... The
Ian Nowland: magic number in the book was 200. 200,
Camille Fournier: But that's, you know, who knows? Nowadays, the companies are so different, it's hard to say. But so the magic, I think the point in platform engineering is when you are paying sort of pain the same kind of tax in a lot of teams that would be better done centralized so you know not just you know how do these how do these people all you know kind of communicate and work with each other but also like if is everybody having to learn Terraform and like manage their own Kubernetes or manage their own cloud stuff and are they all doing it in different ways.
Camille Fournier: They probably all need similar kind of IAM primitives. And maybe you do that in libraries, but maybe you want to, you know, actually kind of run some systems to support that and just auto hook into this whole model. So, you know, I think that the point is when you have when you've grown to a point where people can't just kind of voluntarily support the shared things that everyone needs and you actually need people who are going to be focused on supporting those shared, you know, constructs, those shared systems, those shared, you know, those shared platforms, right?
Camille Fournier: That, you know, you can't just rely on the goodwill of the team to kind of keep it going and say, oh, we have the shared code base and everybody's just gonna kind of chip in and they'll fix it when it's broken. So
Megan Reynolds: you might have like different app teams within this ticketing
Camille Fournier: app,
Megan Reynolds: for example, and so people running Kubernetes, they're they're responsible for their own networking and orchestration, Terraform. So people are having to do that on their own and you're saying that you would actually bring that and make it centralized.
Camille Fournier: Yeah, I mean, some of it may, like it always depends, right? But I think the idea is like, do you really want everyone to have to know how to do Kubernetes networking or what have you?
Ian Nowland: I mean, your rant in the book was always Terraform,
Camille Fournier: Oh my God, although AI has made Terraform.
Megan Reynolds: I was gonna say, why is that so painful and how does AI tend to change that?
Camille Fournier: I mean, it definitely changes it. Like, it's just, it's, you know. I think
Megan Reynolds: maybe explain Terraform. Oh,
Camille Fournier: God, I don't know. I don't know if I...
Megan Reynolds: Terraform is essentially
Ian Nowland: trying to take all your cloud APIs and make them declarative in a way that it's easy just like, oh, I want this new thing. I just need to go at it.
Megan Reynolds: What do you mean by cloud API? So, like, a database?
Ian Nowland: We'll come back to EC2. Like, you just want to provision 10 instances in five different regions across the world. Rather than calling the API directly and having to have a human wait, you declare it as Terraform and then it's sitting there in a reconciliation loop where what is caught in that configuration file would just become magically appear in the cloud okay
Megan Reynolds: so it's kind of building like automations for provision
Ian Nowland: exactly that's exactly but it's
Megan Reynolds: like its own language okay so
Ian Nowland: part of the problem is always it's its own language that people who I don't know if anyone loved the language ever but some people would at least used to defend the language but it wasn't that like your software engineers were not programming in that language every day and Camille's right AI makes this better because now you can program very quickly in languages you barely understand but
Ian Nowland: we've all done it but yeah but that was a so a lot of a problem was you sort of get the software engineers who are the people in these application teams going Terraform we just need a Terraform wrangling team but then the Terraform wrangling team every every all of the product teams want different things and so they end up needing to work out you know what we call a lot in the book the glue like so the shitty way of doing this from a you know engineering is here to bring leverage through our thing but when engineering is done poorly you operationalize it and there's no leverage like so basically you need to just keep hiring this team and nothing if it becomes more coherent it just becomes more complicated and so where platforms come in is you finally just say okay this mess has grown too big we're going to hire four people and we're going to make them happy to just wrangle Terraform all day which I think if you have a good leadership team you can do it if you have a bad leadership team they're the most miserable people at the company
Megan Reynolds: why is that important
Ian Nowland: because they need well
Camille Fournier: no no I mean well I mean why is it what like because if you if you have a good leadership team you can at least recognize them and make them feel good and give them political backup when others are yelling at them because they're not moving fast enough or whatever like you can make their jobs feel tolerable
Ian Nowland: because
Megan Reynolds: they become this kind of
Camille Fournier: because
Megan Reynolds: office of no like
Camille Fournier: well yeah exactly I I mean, I do think like there's always going to be those in companies, but like, you know, the fewer of them you can have, the better. But a good leadership team does make a difference in how those people approach their job and think about their work. But if you do a really good job, of course, then you say, do we need to do this as just kind of like an operationalization of ticket requests and whatever is coming in from the teams? can we actually look for common patterns and common needs across this organization and build software abstractions that will manage this that that then we just sort of operate the software abstractions rather than doing the ticket per per team
Ian Nowland: request I completely agree and the thing why I think a good leadership team and sort of the book is trying to teach what good leadership looks like is generally in that when you move from a tactical team to that more trying to build a platform team for six months or maybe longer stuff gets worse actually you aren't quite you know you're taking your best people and asking to build the the next thing and not focusing quite as much it's just churning through each day so you really need to get organizational buy -in into the vision of why this will be better for everyone and I look at the book in a lot of ways it's just here's all the things you got to do like everyone everyone is going to critique all your ideas over time and so you've got to be playing across multiple fronts to get their buy -in to allow you to do that
Megan Reynolds: so it sounds like there's almost this kind of like optimal point of people engineers and engineering teams are owning their own platform for a while until you hit that exactly that point where it starts to break do you see that getting longer with AI like you know people don't have to learn Terraform necessarily now they can understand things like do you think that changes the point I'm
Camille Fournier: not sure it's getting longer I think it might actually be getting shorter but But in a weird way. And I'm curious Ian's thoughts on this. So a few things, right? People just produce so much, so much faster. And it all still needs to work together. And I actually think that, you know, even though AI is powerful and increasingly powerful, it still works better when it's working against a more, like, a more constrained sort of set a more a smaller set of constraints right so okay you know if you're basically like look like you here's here's a bunch of you know a bunch of pre -existing tools that you can use that we know work well and we support here's here's a platform that you can just build against okay you know a human can build faster against a known platform I think an AI can build fast it can build way faster against a known platform and you have I think less surface surface area that the AI is going to, you know, just sort of make its own decisions and possibly create weirdness and drift across your teams if you can kind of point it to a somewhat more standardized interface to build against.
Camille Fournier: And so, and I do think we're hearing, I've had a few conversations with folks and, you know, in various companies where they are saying like, oh my god, like platforms are becoming so much more important. I was like, well, that's kind of nice. I feel like like we wrote our book, then AI immediately happened and sucked all the air out of the room so that no one was interested in platform engineering at all. And now I think, you know, a couple years later, everyone's like, oh, actually, wait, what was that about?
Camille Fournier: And it might look different a little bit with AI, but it's not the concepts, the baseline concepts of you need to agree on things. You need to, frankly, you've got to build consensus among humans because humans still exist in companies, at least for now, right? About what you're going to do and what you're going to work on what you're going to support and not support all of those things still remain true even if you have much more AI you know making it faster to move forward
Megan Reynolds: I guess I guess part of the premise there is that you want the constraint more constrained environment because at the moment at least like the more decisions that the agent has to make like the more likely they might make bad decisions and it will drift off like do you think we will hit a point where that becomes less true
Megan Reynolds: what
Camille Fournier: do you think I'm curious what Ian thinks
Megan Reynolds: on this
Camille Fournier: one I
Ian Nowland: think it's it's really hard to predict like there's this term Gell-Mann amnesia which was originally talked about the newspaper where you you know we all read the newspaper every day and we sort of trust what it says then it writes an insider article on something you know and you know exactly how ridiculous it is and you because you're an expert in the particular thing you sort of know this journalist is saying the wrong things and a bunch of people are going to read this and get wrong impression so so I mean I use AI of course every day and it's still funny that like when I know better than it like I'm constantly correcting it like my sessions before I get something that I'm happy with they're going forever but then like everyone else sometimes it's just like well I'm gonna I'm gonna vibe it's for me it's more like I'm gonna vibe write an email because I don't really and I think a lot like if I think what platforms are trying to do we're trying to bring technical experts in what's below to build software for people of what's above and I think a little bit of people are fooling themselves at the moment is like just you know Claude is going to understand exactly how to do this because it's in the training set or maybe it's getting really really good at reasoning whereas I don't think we've seen much evidence yet that these things are that great I mean you can maybe say the math, the stuff that's doing math at the moment are different but but are that great a sort of reasoning about deep complex systems which is what platform engineering is supposed to to help with so yeah I don't think I think it's too early I think we have to see advancements in what these models are doing compared to what they're doing today, in my mind.
Camille Fournier: Yeah, and I think, look, I think ultimately companies
Camille Fournier: are powerful because they are large -scale collaborations to some extent, right? They share something in common that kind of makes them bigger than the sum of their parts. And I think that, at least right now, like, you know, AI and vibe coding is really just like a splinter in many, many different directions. It's so easy. You can get something that works in a million different ways. And if you don't give it that many constraints, you're going to get something that works in a million different ways throughout your company.
Camille Fournier: I think a lot of companies are seeing this right now where it's like everybody has an idea. They run off. They do their thing. They run off. They do their thing. And those ideas individually may be great. but how do you make them better than the sum of their parts how do you make them work together how do you make them kind of able to be evolved as as kind of a shared set of things and I think if you're gonna have individual humans driving individual AIs where do they get the common context where do they get the common you know where do they where did that where does the AI get the common set to keep it from just really splintering into a million different pieces I think that's that that's platform among other things right but I think you know yes you could say it's like oh we have a knowledge graph of the company or or what have
Megan Reynolds: you right and teach us all at any time
Camille Fournier: you know that
Megan Reynolds: will frame it perfectly for you
Camille Fournier: and I actually think like knowledge graphs of the systems in the company is probably part of the modern platform you know but but But these, you know, someone needs to think about and maintain these constructs. And again, until we have the AI that really does that all for us and then coordinates all the other AIs. And then I don't really know where humans are at any point in this.
Megan Reynolds: I guess part of the one of the biggest tensions in that flow is like who is responsible when something goes wrong. And you can have all the primitives of the world. But in the end, we're layering all these abstractions. and at some point you know you vibe coded a few things Ian and like you're like oh I don't actually know how this works so like how how are you how do you think about that challenge of who is kind of becomes responsible when you've you're I mean even with platforms you're already a few layers of abstraction and even with a platform now who has that responsibility and maintains it
Ian Nowland: yeah I don't think the industry really has answers yet so what a decade at Amazon teaches you by having to wear the pager we used to be a pager is now a phone call is who is going to actually debug this to no longer having a customer impact at two o'clock in the morning and if you look at like Amazon sort of strategy as a company sort of you know what has led its growth it's like a team will sort of form around a bunch of software where a five to eight person pager rotation and so so a lot of what Amazon engineers tend to think oh I'm here to build new software really they're there to start the pager rotation and and honestly I think about platform engineering I mean anyone at Amazon pretty late to time sort of realizes that's
Megan Reynolds: true but
Ian Nowland: and so a little bit with platform teams is a little bit like who are the four experts in Postgres at two o'clock in the morning and I think there's this hope that you know AI SRE has got a lot better like I've watched some of our engineers on call like quickly mitigate you know in 20 minutes what should have been a two -hour outage but you still need the human there and you still want the human to validate oh am I going to press the button to do a quick rollout to production and so so I think maybe platform teams have a greater amount of software they can be accountable for because these five people who have to you know once once every five weeks are on that rotation but it doesn't get rid of them like the human accountability aspect just seems to still there where I think we're well away from from serious platforms which is like you know anyone can sort of fly by night at a sort of small product focused startup but at serious companies you want like part of what you're paying your engineers for is that and you need them to be experts so yeah I I think it's I think AI SRE is still very early in you know now I'm sure you know if you're an investor in one of those companies or like what they're saying is well we're learning a lot from our training data and we're going to do reinforcement learning on top of it and maybe that doesn't I guess my sense is always every business is very different and so it's very very hard for an AI to do well
Megan Reynolds: and what would it I mean what would it take for you to trust something like that I guess like if you're higher level and you have less context on the technology, like what would you need to see from something like
Camille Fournier: that? I mean, at the end of the day, it's got to be like, you've got to have the data on how often is it right? And do we know, you know, I think, so I think so many of these systems, right? So, you know, Ian and I actually sort of intentionally shied away from making our platform, talking about platform engineering as really about the software development lifecycle and the developer experience. But that is a big part of platform engineering at most companies is thinking about like, how do you do the developer experience?
Camille Fournier: And, of course, now a lot of people are talking about, well, we just want agents to be able to take tickets and write code and, you know, review each other and basically, like, submit things, right? And how do we know when we can do that or not? I think it's a similar, although probably in some ways easier problem than AI SRE. and to be honest like everything I've heard from anyone who's done that even moderately successfully is you actually need to have like and start to have nuanced notions of is this ticket simple or is this complicated is this fix that the AI like how how risky do we think the fix that this thing has written or the the code that this thing is written actually is do we think we need human eyes on it or not how risky is it across security across blast radius across this that and the other so you need to start having these more detailed you know kind of classifications of the different steps where you can know when it is very likely to be safe to move forward versus when you need to page in a human to act and I think that's the discipline and that is not a discipline in my experience most software companies have most companies have at all about almost anything
Megan Reynolds: to be able to grade
Camille Fournier: right to be able to really think about like you know know what what is this just you know analyze this and find the signals in it and analyze it statistically and you know make decisions to move forward
Megan Reynolds: because what would you is there something you would trust it to do today like what's kind of the sure something simple but
Camille Fournier: yeah I mean I think there's like if you think about if you think about like code changes right like I've got a like a minor you know CSS issue in the like about us page on my website you're right you know I mean it's a very simple example but like
Megan Reynolds: I mean platform like infrastructure wise is there something I mean that like the kind of classic example and I was digging into this space and made an investment in this space was that people would speak about auto scaling like you're already doing auto scaling and so if there was something going wrong with auto scaling you would sort of be comfortable with there being an automation that would roll back your auto scaling that felt like that
Camille Fournier: yeah I'm not I'm not sure I feel like rollback is actually one of the one of the hardest
Ian Nowland: cases I was going to say rollback in general but rollback auto scaling like at Datadog you know incredible scale of streaming auto scaling up is easy to think about auto scaling down with these semi -stateful systems right where you're not quite sure how things like that was very slow
Megan Reynolds: okay it's amazing not so not even that example you know that was that was at least something maybe they meant like triggering auto scaling
Ian Nowland: yeah what what serious platform in the old school they're talking about like hysteresis in these systems and there's There's a term that I'm completely, but it's like, it's actually very, very difficult to reason about certain types of changes. And so scale down is one of those classic ones. Scale up, you usually, oh, I'm going to spread my connections. Scale down, it's like, okay, what is actually going to bear the increased load? Is it the thing that I'm scaling down or is it a dependency on it?
Ian Nowland: - And so Sort of
Megan Reynolds: all that. Yes. So what would be the kind of most complex platform thing you would let an agent be responsible for today? day
Ian Nowland: it depends on how important it is to the customer. so both of us work for infrastructure businesses where if we go down they go down and that's very very different say even to Datadog which was an observability platform so I would imagine when I talk to like data can be pretty gung -ho in their front -end code about pushing changes particularly behind feature flags I don't know how much they've sort of incorporated that knowing that at the end of the day this is the observability tool if it goes down we're going to get yelled at but the customer isn't actually down now the stuff under the platforms that I used to run was treated by Datadog as much you know much higher complicated and so I had a much higher standard and so so that was you know at Datadog you could see this classic bifurcation between product engineering and platform engineering I feel like product engineering again if you're not I mean it's always so tough right like I think about like Harvey AI they're down for two hours a bunch of lawyers can't get any jobs right like so it's very easy just through your success to find yourself in a place where it's very hard to trust you're not going to break someone's workflow and you know the way it used to feel at AWS is like every time the CEO had to fly down to a customer to apologize we were going to come back with more restrictive policies and you see that I think in sort of Amazon statements today what was it like two days ago right you know it's not about token maxing it's about customer obsession maxing or something oh god but it's the like Amazon for its fault like its greatest strength is operational discipline and people really knowing like the operational culture of Amazon is people getting really upset when their package is one day late and it just informs everything that they're great at in a way and so for both of the companies we work at I think we're at a higher bar than maybe a lot of product first startups who can you know they can get a few black eyes before it really becomes oh we're going to lose a bunch of our customers because we just have too many outages
Megan Reynolds: and you mentioned the core constraints are important for for using agents more freely? Do you think we need different primitives now than we have previously? Like if you were like building an AWS from scratch today, for example, like how would you build it differently?
Camille Fournier: I mean, I feel like agent identity is the big, is the big like elephant in the room that I know many people are thinking about, must be thinking about, because I can think that it's an important thing, then, you know, this is not my, I'm not an early adopter of thoughts in this space, I'll put it that way. agent identity and like how do you know if an agent you know how do you know what an agent is allowed to do and how do you differentiate an agent acting on behalf of a human from an independent agent and when is it where you know even though they're acting on behalf of a human are they really allowed to do that still I think there's a huge amount of space around agent identities that is I think it's just a I don't think we have a solution to it yet as an industry but like that's going to be really important for future platforms
Ian Nowland: yeah I think I totally agree we don't have a solution and I don't like I know people are getting excited about stuff but it's still hard to see what's there
Megan Reynolds: like what are some examples of things that people are I
Ian Nowland: mean this is all tied to the MCP really changing as a protocol right which I'm not that close with but I know there's a company I invest in called Keycard and they're like very excited that there's notions of delegation in there actually we both worked at Two Sigma they took delegated identity very very seriously and it was super painful to administer because it's a lot of okay I trust you as a human but do I trust your agent or your intermediary to have the same amount of access you do and so I think all of these attempts to standardize delegated identity just struggle more or less that it's actually administrative complex humans don't want to define it but maybe you can have AI check it very safely.
Ian Nowland: But yeah, like what we all know at the moment is everyone is running agents with too much access.
Megan Reynolds: Yeah, very dangerously skip permissions.
Camille Fournier: Yes, and I think to your point, Ian I think most companies have not thought about delegated auth very deeply. Finance actually probably has an advantage. Financial services companies have thought about it. It's just because they are so paranoid about who has access to what data who can do what actions and it's very important regulatory there's regulatory you know considerations there's just you know big money on the table that they think very carefully about it but I think a lot of companies don't just don't think that much about like should Ian really be able to do that should an agent acting on behalf of Ian you know Because like Ian, we can say, well, if Ian does the wrong thing, we see that Ian did the wrong thing, and we're going to go yell at Ian or we're going to fire Ian or whatever, right?
Camille Fournier: If Ian has an agent that has access to whatever Ian has access to and doesn't think about asking him, should I actually do this or not, and then goes off and drops the production database, you know, right? Like that's, well, can I yell at Ian? I don't know that it's Ian's fault, though. Like sometimes it is and sometimes it isn't. like that's a that's a that's a tricky it's actually like a it's a tricky thing to reason about and I think we've traditionally in a lot of companies been very loose with internal access because it's like you're in the system we know who you are worst comes to worst our you know internal threat people can like find you and
Ian Nowland: also it's just humans don't want to go to jail and they they know very much they don't want to go to jail the agent may or may not and so yeah there was just a lot of trust that everyone had excessive access but not everyone but I'd say you know the vast majority of people were very careful in what they did and didn't do because they knew what was important to their company socially in terms of what they should and shouldn't do again finance was ahead of this because they were heavily regulated but yeah like the other one you worry about a lot today right isn't even my agent doesn't do the right thing like my laptop gets gets exploited by North Korea, right?
Ian Nowland: And they can just run whatever agent they want at that point in time. It comes up, I'm a Slack admin at Baseten and I need that for a little bit of my job. And I know I'm trustable, but it doesn't take too much. Click a bad phishing email, right? So yeah, I think delegate identity in general, the industry's behind and we're about to go through about a year or two of pain trying to catch up.
Megan Reynolds: Yeah, I don't know the exact details of the hack, but the Binance hack recently, it was like through a DevOps engineer, and maybe that was access that they had and should have had, and you can't prevent that, but yeah, the blast radius there was large.
Megan Reynolds: Okay, so identity is important. If you were building AWS from scratch today, what are the primitives you'd put? So identity is one. we
Ian Nowland: definitely inference I think the other one we all sort of see is sandboxes but like sandboxes sort of just look like a better Lambdas and what
Megan Reynolds: does that mean
Ian Nowland: so Lambda was AWS's about third attempt at taking pure application code and having it run completely independently without even having to think about even the Kubernetes layer now came with a lot of restrictions it was like maybe five minutes, the total runtime it could run but what it meant is like you You know, the great thing about Lambda is you could scale to zero when you didn't have any load. You could quickly scale to 1,000. And so for certain use cases, it really took off. [Correction: AWS Lambda's limit has been 15 minutes since 2018 (5 before that)]
Megan Reynolds: What's an example?
Ian Nowland: What is an example? I've used a lot of AWS. I've actually never used Lambda.
Camille Fournier: It was used a lot for things like, I mean, again, things like I'm processing things off a message bus and I get bursts of, I keep saying message bus,
Ian Nowland: which ages
Megan Reynolds: me apparently. but it's
Camille Fournier: not it's not called Kafka at it's what is it called SNS or or whatever at anyway I'm processing I'm processing stuff as it comes in and you know there's a lot of ways to do that but like all right you know I'm expecting kind of a burst of of users and what they're going to do is like kind of this light these lightweight things that doesn't require a huge amount of state to be you know instantiated in order for them to to act accordingly and so it's It's very useful for activities like that.
Ian Nowland: Yeah, talk to people at Capital One. I think they just said, like, this is a fairly open posture, but, like, they try to use Lambda every way that they can for financial stuff where, again, 2 o'clock in the morning, maybe there's a whole bunch of books that need to be, you know, just make that a Lambda job. It scales up, scales down. And so, yeah, there's just a lot
Megan Reynolds: of... Small jobs that
Ian Nowland: may have an unknown
Camille Fournier: amount of computing.
Ian Nowland: And I think where you're seeing a lot of excitement about sandboxes, even though it's not really substantially different to Lambda is the agent is the thing running in the sandbox and of course you want to have a swarm of agents and the swarms of agents are calling into you know inference which is calling back with tool use maybe that's like even more sandboxes
Camille Fournier: that you
Ian Nowland: want okay
Megan Reynolds: this is kind of the perfect like it's a specific function of a job the agent's acting on something spins up spins down so like the the premise of Lambda is kind of fine but
Ian Nowland: exactly but it's normally longer lived than five like Lambda's again I don't know why they didn't just expand this but this limitation of five minutes really allowed a bunch of startups to sort of come up and I think part of it is just capacity but I think part of it is they just you know if your agent wants two hours it's just going to run for two hours [Correction: AWS Lambda's limit has been 15 minutes since 2018 (5 before that)]
Megan Reynolds: you mean because you've
Camille Fournier: only
Megan Reynolds: got the five minute limit that's why you've got all these sandbox companies
Ian Nowland: exactly well again part of it part of it anywhere in the market today is just access to capacity exactly but I do think the other part of it is they found a better product surface for agentic workloads and yeah AWS Lambda at least as it was two years ago
Megan Reynolds: I mean touching on we've spoken a bit about the GPU shortage but you know one of the things that I'm hearing from all the people in SF particularly is the cloud is full and that's GPUs that's also CPU capacity is constrained explain what does that what does that mean what's actually happening right now
Camille Fournier: well we are definitely in like a you know we we hit this capacity constraints on on memory was just like a huge huge hit about a year ago or give or take
Megan Reynolds: and what does that mean in practice we hit
Camille Fournier: it means it means that you just couldn't get it so like that and that's why your iPhone is so expensive and that's why you know so like all like you know an SSD that used to cost a hundred dollars may cost $250 now, or
Ian Nowland: seven hundred dollars I
Camille Fournier: mean it's crazy you know so that meant that what's
Megan Reynolds: the shortage what was the short of this hardware shortage so yeah
Ian Nowland: the silicon like SK Hynix and the other one who I always forget but basically spinning up a thing to print memory is just a massive investment that takes years and a big leap of faith from them that the demand is going to be there over time that they've been burnt by in the past so they're not the most reactive companies so it's just a massive I don't know maybe trillions of investment needed from them to open up new lines yeah and so three years later you know three years from now the DRAM will be cheap again yeah but it's it's while the demand is just 10xing in a lot of use cases you know they're just printing as fast as they can it's not okay so we
Megan Reynolds: crunched with memory a year ago
Camille Fournier: so because of because we have this memory shortage that affects all piece like all kinds of parts of what you need for a cloud your storage your servers you know it your networking equipment even right like everything needs needs memory and so all these different pieces now they have longer procurement timelines or they're more expensive and companies don't want to like overbuy again when the price is really high or they just underplanned you know for for their own for their own access or like you know they thought they had a contract and they didn't have a contract because the vendor was like actually sorry this is three times as much you got to pay us three times as much you're not getting it which you know whatever that that that happens right so I think that a lot of it is there's like a spike in demand as we just have you know despite you know despite every all the all the news and contrary I think we're actually engineering employment is still growing so people are still hiring software I mean that's what I believe what I've seen in the news and you know in the job in the sort of like job reports is that actually we're still hiring engineers they're still producing they're actually producing way more stuff we have of course a huge startup boom in the AI space they all want to be be putting their things online where, you know, the cloud is full.
Camille Fournier: I suspect also we're coming up to Black Friday, Cyber Monday, which is big shopping in the U.S. which means all of the big retail, you know, e -commerce companies, they procure a lot of excess capacity in this time. So they all need backup stuff because they're expecting to get slammed by buyers in the holiday season. And all of that, I think, is contributing to the very current cloud is full. I think it's It's like a, right now it's worse.
Megan Reynolds: What does that mean practically?
Camille Fournier: It means practically is that you need, you are like, if you need anything, any major amount of capacity, even if it's just like CPU -based servers or, you know, databases, storage, whatever, you just, it's very hard to get it from your cloud provider.
Ian Nowland: Yeah, if you come back to the earlier, you know, the two minutes, call an API, you get that. So what EC2 would call on -demand capacity, that just is in way much shortage. And a lot of times then you're compromising on, okay, I've got to go talk to Google for the first time to find it and like you know I've got to go talk to CoreWeave and so so a lot of it then is you're just searching for the capacity in the shape that you wanted what
Megan Reynolds: happens if there's no capacity like I guess we're saying the cloud is full which gives this impression of like you know there's there's no more to be had but
Ian Nowland: everything's available
Megan Reynolds: scaling we
Ian Nowland: leave it in the market economy everything's available at the right price it's just yeah how much are you willing to pay the
Megan Reynolds: economics and stuff are really like there is you will find it somewhere
Camille Fournier: well and And the economics and also, like, you know, I think it wouldn't be the worst thing for the tech industry for us to remember how to do, like, performance engineering a little bit and optimizing our usage of things. And, you know, now you can rewrite Postgres in Rust in a month or whatever with your AI agents, right? And, you know, maybe that's more memory efficient than it was. I don't know if that's true or not, right? But like, you know, so I do think also that people are going to have to scramble to be like, okay, just like back in the old days, when it would, you know, you had five different teams sharing a server and fighting over scraps of scraps of memory allocation being like, okay, who can turn off that Berkeley DB got to clean it up, got to, you know, got to delete those logs
Megan Reynolds: somewhere. It's just kind of,
Camille Fournier: yeah, but it's work, you know, it's sort of work that, you know, the economic, there's an economic price either way, either you're paying more money, or you're you're paying your people to find efficiencies.
Megan Reynolds: Cause I mean, is there a world where these cloud providers will be saying no to customers? Like, is that, was that what's happening? Is that something you'd have to deal with?
Ian Nowland: So if you look at how AWS does things right, they either have reserved instances or on demand. I think definitely by just failing the API, they're saying no to on demand. I would imagine at, you know, I know Jassy sort of came out in the last week and was like, oh, we're all out. And then -
Megan Reynolds: Yeah, what does that mean?
Ian Nowland: Well, you back door and it's like, no, no, I mean, supply is getting short of certain things and please come talk to us. It feels a bit like, honestly, like, you know, a CEO playing bring people, bring in demand because it isn't quite true. But I don't think it's totally false either. I imagine for certain types of stuff, they just can't give you a three -year reservation full confidently where you want it at the moment. It might be, oh, you can't get it in us-east-1. You know, work out how to make this work in, I don't know what battery, I don't want to say, they say Europe's pretty good like bad latency I was thinking of like south america but yeah just just further away and then you know engineering team make that work so yeah my sense is it's still there it's just the people who have really especially startups right who have just got really used to when I want it it's going to be there I'm not really going to come in in advance because I don't I don't want to do that they're going to struggle the most I think and a lot of it then is paying more capital you know paying more than they want to get what they want
Megan Reynolds: anything you want to speak more on that we haven't spoken about yet I
Camille Fournier: I think that is the big thing that everybody is most obsessed with right now is like how do I get rid of code review how do I scale get
Ian Nowland: this by the way it's total victory for me and Camille so you know we're Gen X and we grew up where code review was a nice-to-have, not a must-have
Camille Fournier: yes and
Ian Nowland: we ended up managing a lot of millennials who socially just love the culture around code review the collaborate like it's always like what
Megan Reynolds: like why no I
Ian Nowland: don't know I don't know
Camille Fournier: I think I mean I think it was I think it was a combination I think it was a combination of a number of things I think it was a lot of like the Google Microsoft culture where you did have mandatory code review and everyone was like well if Google does it that way plus the move to Git and GitHub that was very open source influenced and open source of course you need code review because it's randos on the the internet so you want somebody who you could trust reviewing it so between I think the GitHub thing everybody uses GitHub Google Microsoft kind of setting the tone for tech culture and then you had Sarbanes-Oxley which actually does basically require code review for any code that touches financials and a lot of companies being very lazy and just being like fine I'm just gonna make everybody code review I think it it's like a but then people create just-so stories about about how it's not a thing that's being imposed on us or that we're accidentally carbon copying from open source or big tech companies.
Camille Fournier: It's a thing that's good and we really want it. And look, I get in so much trouble from the youth of America every time I say this, but it is true that Ian and I both grew up and early in our careers, code review was maybe reserved for very sensitive areas of the code base or it was, you know, it was optional.
Ian Nowland: It was trust your judgment. If you've got something very complicated, especially in a system maybe you don't understand, then you absolutely did it. But it wasn't like, if this was the code base that you wrote, it wasn't go ask someone just to check a box and, you know, maybe find a couple of typos. And, you know, I would estimate that 75% of the code I wrote in my first 11 years of my career was not code reviewed. Wow, interesting. So would you
Megan Reynolds: recommend not code reviewing now?
Ian Nowland: Well, so now the AI is the problem.
Camille Fournier: I mean like the thing
Megan Reynolds: is better than no code review I would assume yeah
Camille Fournier: I mean the world didn't the world didn't end when we didn't have code review I think there are more I mean I think there are legitimately more places where you have to do it like like if you are touching financials and you are in a publicly traded company you better freaking be doing okay some kind of human code review regulators and auditors have no sense of humor I say this to people all the time they do not have a sense of humor don't play with them like maybe somebody will figure out how to crack that but like until someone else does I am definitely personally not going to sign up to be to be the first there but I do think that again I think it goes back to you know okay people aren't reading code right now I mean even places I think with required code review I think most people are just like looks good to me looks good to me looks good to me because it's just you cannot keep on top of it it's too much even if people aren't using AI excessively you're either getting really long changes which are just very painfully hard to reason through or you're just swimming in you know lots and lots of small changes which just takes a long time and so at the end of the day I think right now code review is is dead whether we admit it or not you know and we need to be and I think we would all be better suited to say okay where do we absolutely need human code review and it should be a combination of regulatory requirements compliance compliance requirements plus where do we think the code is high risk and how do we risk rate this code beyond you know you know it's not just about the AI agents code reviewing one another I do think there is some value
Megan Reynolds: and where do you draw that line today
Camille Fournier: I
Ian Nowland: mean my gut is always this goes down customer goes down okay and so trying to and but that's like financial being you know this goes down money gets funny we would probably apply to that as well but because I've just typically been in platform teams it's like this goes down the customer goes down A lot of it, honestly, could be junior engineers versus senior engineers and how much you trust them to prompt it, how much it's within the team that sort of has shared knowledge about what this does, how much it's an outside person because it's a, oh, you've got a monorepo and anyone can change anything in it.
Ian Nowland: So it'd be a bunch of heuristics essentially just trying to work out risk to production.
Camille Fournier: And I think heuristics are ultimately going to be implemented by a different AI I think you're going to have some kind of like AI scoring and you're going to have to see whether the scoring we think the scoring is good or not to turn tune it up until we're like okay like you know does what is the security scoring of this change do we think this might have security implications and therefore a human should take another look at it what is the production you know surface area scoring what is the you know I don't know whatever like we haven't touched this code in a very long time do we really think we know what we're doing you know kind of scoring and then there's the other half of it of course which is testing and wow like you know we really are back to the the back to the stone age we're back to the ages of like you know what full integration end -to -end testing is kind of the gold standard again which you know I think for a long time unit testing was was great I think it taught developers how to write better code in some ways again I'm not like an excessive test-driven development everything all the time but I do think that like forcing developers to think about writing testable code write testable code unit tester code unfortunately when AI is doing that it can certainly do that it can do it very well it's not clear how well it proves anything what
Megan Reynolds: do you mean by that
Camille Fournier: I mean I think that AI is going to test what AI thinks it should test and
Megan Reynolds: if it stories of like it just like deleting the tests
Camille Fournier: you know yes I mean like you know I think that's that's part of the challenge is like you know AIs are like satisficers and they are they are trying to like you know achieve the goal you have given them and if the goal is write this code and like make it pass the tests they may do weird things like
Megan Reynolds: an argument for more code review like I guess like
Camille Fournier: and so but you can put other checks obviously you can put other checks in right does
Megan Reynolds: it
Camille Fournier: you know do we have code coverage do we have this that and the other right but I actually do think that that testing and in particular like the heavier weight old school testing end-to-end, bring up an environment and actually run through a bunch of flows I think that's going to be more important which you know is expensive it's expensive work it's hard work it tends
Megan Reynolds: to be in terms of
Camille Fournier: intensive yeah in terms of compute in terms of like you know you're just you know a unit test is a small little bit of code you just run it can run really really fast the bigger changes the bigger integration tests they both are slower they require a lot more context to be spun up to run with any kind of fidelity and then they often are flaky and break for you know for stupid reasons and we're gonna spend a lot of time I think chasing down all of that man that's that's that's where I see this going
Ian Nowland: I mean
Megan Reynolds: I'm already hearing from people that for a lot of companies like CI compute is almost becoming parallel to production
Camille Fournier: yes yes
Megan Reynolds: which is
Camille Fournier: it's crazy it was always expensive it was always more expensive than anybody wanted
Megan Reynolds: to
Camille Fournier: admit but yes now it's getting worse and worse yeah
Megan Reynolds: painful and so where does the bottleneck move what's the bottleneck now is it CI compute cost like is that but does that become the new constraint or what becomes the strain constraint in software development I
Ian Nowland: think it's CI can like you know I I've attended some of them whereas someone at at Anthropic, and this was maybe nine months ago, and the nice thing about Anthropic, right, is tokens for developers on the latest models are more or less free, so they're often the earliest to discover, and it really was CI compute, but also, you know, Camille and I both, from when I worked together, the test flakiness, and like, you know, who turns off the flags, and everyone.
Megan Reynolds: What does that mean, test flakiness?
Ian Nowland: So, test flakiness is when you've written a test that is usually an integration, you can get test flakiness in the unit test, but usually it's an integration, where it actually is only succeeding because of some softer contract maybe it's timing or we you know this is running on this CPU
Megan Reynolds: the test is kind of faulty
Ian Nowland: the test is faulty and it's usually someone else's change that causes the flake to actually happen and then the problem is right it stalls the entire pipeline and then so a lot of people were doing was like sort of building flaky test detectors and then but then the problem is you end up just turning off and never turning turning back on and you can sort of imagine like you have AI sort of like constantly trying to work to keep the you know number of turned off flakes below a certain level but yeah what you end up with the big problem with centralized CI is you just end up with a massive bottleneck for all your software gets written or gets pushed through what is essentially a single a single pipe and so so yeah so I think it's both the pure compute behind that but also yeah can you maybe it's 20 different environments yeah something just to keep you know to keep the agent flow happening without it being a single bottleneck.
Megan Reynolds: yeah last question what would what would convince you that a team can ship software end -to -end safely without understanding the code
Camille Fournier: do you have to understand the code or do you have to understand the system okay so I would be comfortable saying if they understand the system okay even if they haven't looked at the code and excessive I think we can get there I don't think most non-engineers but not all but many non-engineers truly understand systems I mean plenty of engineers don't understand systems either it's a little bit of the problem here right but like I think like ultimately what you're talking about is like okay we're gonna have a lot of even if you ignore the code the direct implementation of the system you have a lot of rules and a lot of things that that have to happen in state that needs to be true in various ways for anything interesting to happen and then to be able to change a rule and make sure that the system itself sort of stays the way you want it to or improves that is a particular you know like systems thinking or what what have you right that's It's a particular, you know, logical mindset that I think that non-engineers, you know, of certain backgrounds place and many engineers do not have.
Camille Fournier: And I think that if we, you know, I think if you have a people that is very, a group of people that's very good at systems thinking and AI has gotten to a point where you can really pretty much rock solidly, trust the code. And therefore, the people building these things can understand the constraints and optionalities of the components of the system. and it's not probably doing anything that's like extremely performance sensitive or like I do think there will always be edges of engineering that are very hard to get away from if you don't understand like then you know if you can't peel back the surface but there's plenty of business types of applications that could do real important things that don't necessarily ever hit those edges of engineering if you understand systems but that's a big if
Ian Nowland: yeah particularly like if you go back you know the engineer product manager relationship right like one really understands what the customer wants the other understands what and so sort of why that partnership always worked together is the the engineer would bring the system knowledge to push back against the things that the customer wanted that doesn't make sense this is sort of the frustration of any engineer dealing with a regular human who you know why can't you just code that right and and the feeling at the moment is you can just code that but I think coming to Camille for a simple enough our system but as that you know as this is a ticketing app that is you know if it's got to do the Taylor Swift concert and there's going to be you know a peak load of a million whatever they hit on those those crazy concerts humans had to have thought about that and again I think I think it's got to tell you right like you need that depth attention to detail around how the system's going to do this to make sure that the AI was correctly prompted to think about that in advance maybe eventually all of this ends up in the training set and the reinforcement learning gets a lot better
Camille Fournier: I feel like there's a point at which you have to slip over to and then we have the AI overlords because then what are the humans even doing in any part of it at that point but
Megan Reynolds: I guess the baseline for both of your answers is that that assumes that the code quality has become so good that you can depend depend on what it's what the AI is telling you it's doing it is going to do and execute it's more about how does the system function around that
Camille Fournier: yeah I think the code quality and I think you probably still also need the AIs themselves to have increasingly better understanding of the system themselves because I think there is like a system there are obviously at the join points in the code system details that matter and so the AI would have to be able to know at least enough for the system join points to write that code effectively where the humans would not don't understand the code and they don't need to review it because they can trust that the AI can get the join points as long as they get the overall you know workflow right
Megan Reynolds: how far do you think we are from that I
Camille Fournier: don't know man I'm not a futurist I'm sure you know I think we're getting I mean I think it's getting better at doing at doing that for increasingly complex things all the time and the question is is it going to stop getting better at doing like we're not there yet for most I think genuinely interesting things but you know we're getting better so it's I don't know I'm very I'm very bad at predicting the future I've been
Ian Nowland: consistently wrong about AI especially about how quickly they fix hallucinations not that hallucinations don't happen which is that they happen far less regularly because they got much better I guess you you know, just the, what is it, the thinking model, the reasoning models they brought in, you know, but I'm still not San Francisco, you know, singularity -brained at all. So I remain a skeptic that, you know, I think it is the human and the AI are doing great things today, and I can't see that changing too much in the next three to four years, I'll say, you know.
Megan Reynolds: We have a timeline.
Ian Nowland: My p(doom) is close to zero.
Camille Fournier: Well, that I agree with you.