Subscribe
Share
Share
Embed
Simon Elisha is the Head of Technology and Transformation for the Australia and New Zealand public sector at AWS. He’s also the host of the AWS Podcast and an advisor for Bugwolf, the curated marketplace for bug bounty hunters. Previously, Simon served as the CTO and director of field engineering for Australia and New Zealand at Pivotal, Inc., a principal solution architect at AWS, and a data center enterprise architect for Cisco, among other positions.
Join Corey and Simon as they discuss why it’s different to sell services to the public sector versus the private sector, why Simon decided to launch the AWS Podcast, what the most rewarding thing about hosting the podcast is, how increasing concurrent EC2 instances can actually help lower total spend, how tagging is a critical tool in getting a handle on your AWS bill, how the cloud has lowered the barrier of entry to tech considerably, career advice from Simon for those starting out, how the word “resilient” doesn’t have the same definition as it used to, and more.
Screaming in the Cloud with Corey Quinn features conversations with domain experts in the world of Cloud Computing. Topics discussed include AWS, GCP, Azure, Oracle Cloud, and the "why" behind how businesses are coming to think about the Cloud.
Transcript
Announcer: Hello, and welcome to Screaming in the Cloud with your host, Cloud Economist Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.
Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I'm joined this week by Simon Elisha, head of technology and transformation, Australia and New Zealand public sector for a little company called AWS. Simon, welcome to the show.
Simon: Hey Corey, long time, first time.
Corey: Exactly. Which I imagine is one of your colloquial expressions, which means it's great to talk to me. And you're right: it is. So, what is it that you do at AWS because I understand that you run the AWS podcast, which is a subject near and dear to my heart. Feels a lot like this one, except you probably spend less time trying to figure out who should sponsor it.
Simon: [laughs]. Probably true because that was one of the considerations I didn't have to worry about. So, I wear a few hats here at AWS. So, firstly, I lead a team across Australia and New Zealand who work with our public sector customers on building, deploying, and getting the benefit from really cool technology for citizens. So, kind of nice and very, very rewarding.
But also, as you mentioned, I do host the AWS podcast as well, which was a crazy idea I had back in 2012 when I actually went to search for podcasts about AWS, going, “I'd like to listen to one,” and there wasn't one. So, I did that crazy thing of saying, “I'll make one. How can it be?” And now I learned that having a podcast is like having a puppy. It's for life, not just for Christmas. And I really enjoy making it.
Corey: Oh yes, having spent a little bit of time, myself, around AWS folks, I know that if you play the popular drinking game of taking a drink every time someone in an AWS meeting uses the word customers, you will die. So, you just mentioned that working in the public sector, you work with citizens. How often do you wind up accidentally using one term instead of the other and having to self-correct? Or is that something you've been able to train yourself out of doing?
Simon: It's more the case that it’s a different mindset when you think about citizens versus customers. And our customers are the departments and the agencies that we work for, but their customers are the citizens. And the distinction I like to make is a citizen can't choose the service they choose. So, you have your tax department, and that's the one that you'll be providing your tax details to. You have your immigration department, your defense department, your health and human services department. You don't get to choose.
A customer gets to choose, a citizen get a service by their government. And so my goal is always to help out agencies provide the best possible citizen service. So, it's an interesting nuance, but it's an important one. And the nice thing is a lot of governments are speaking about citizen-centric services, which ties very much our own customer-centric thinking.
Corey: I like that quite a bit. One thing that I find interesting is in my own mapping—mentally—of big companies, my impression has always been that the bigger a company gets, the more inherently narrowly defined various employee roles become. So, it's interesting to me that you're not the, for example, head of the AWS podcast on the following four days out of the week. Instead, you have another full-time job that isn't speaking into a microphone. How did the podcast come to be, and why you, I guess, is probably the rude version of that question?
Simon: [laughs]. It's because I have an absolutely beautiful head for podcasting is what I would describe it as.
Corey: Yes, a face for radio, is what I have on this end.
Simon: [laughs]. But look, I think it's interesting is that one thing that we do when we hire folks at Amazon, and AWS in particular, is we want to hire builders and let them build. And so what that means is we hire, looking for people who the leadership principles really resonate with them. You know them just as well as anyone else, so I won't list them out here, but what those leadership principles do is keep us these broad running rails and decision making filters to go do stuff that is really, really good on behalf of our customers. And so what that means for me is when I sort of saw a need, which was, “Hey, there's no podcast. Maybe people would be interested in this.” I could go build it.
And yeah, I had to talk to some of the right folks and say, “Hey, this is something I'm thinking of doing. Can I do it?” And I [unintelligible] trust with those people. They said, “Yeah, it sounds like a great idea. Give it a go.” See what it is, it's what we would call a two-way door. So, it's a decision we can unmake later on if we want to. And history tells the title, that a lot of people found it useful. And probably one of my favorite things is especially people that I meet at conferences, saying, “Hey, I really love the podcast.” That’s really gratifying. But more importantly, is when people say, “Hey, I got a job and your podcast helped me prepare.” That’s, like, awesome. Very happy with that.
Corey: Of all of the feedback that I get around the different conversations I have with folks from this podcast, and newsletter, my being obnoxious on Twitter, the most common is, “What's wrong with you?” And other various forms of insult. But the ones I like the most are where I've helped someone do something next in their career, where it's getting someone from where they are to someplace else. Maybe it's career based, maybe it's solving a pernicious problem. But that's always meant a lot more to me than the slings and arrows I get, which, let's face it, I definitely invite that criticism onto myself with basically everything I say and do. But there is something to be said for having been in a position to impact people's lives in a positive way.
Simon: It is humbling, and not a sort of twee way, but if I think about it—like, I've been in IT for 30 years now, I'm an old person, now, relatively speaking, but I think back to those folks who helped me when I was a young whippersnapper and gave me guidance, gave me an opportunity, took a punt on me and said, “Hey, this person can do something. Let's let him do it,” type of thing. And you’ve got to pay it back. There's so many amazing folks coming into the industry for many different aspects. One thing I'm really conscious of is not just saying, hey, he's an IT graduate, that'll be perfect.
Well, what about someone who's had a completely different career trajectory, but he's really interested in this domain? How do we get them into that? Giving that assistance is huge. And the weird thing is it has a really outsized effect compared to the input you put in. So, much like yourself, you put work into the podcast, you do it, but you kind of get it done. It's done; it's out there. And what you don't think about is there’s people listening, sometimes years later, saying, “God, this is really useful to me. This is inspiring me, getting me to that next level.” Can’t ask more than that.
Corey: A realization I was somewhat late to was the, I guess, dawning awareness that a lot of what AWS was releasing—things like SageMaker, or the DeepLens, or the DeepRacer, or the DeepComposer—also known as Dr. Matt Wood’s piano recital at re:Invent last year—is, sure, on some level this stuff is fun and goofy, and an easy target to mock, but on the other, what what you're fundamentally doing with these things is making new fields available in a fun and engaging way to people who might otherwise never go in that direction. And that's a very powerful thing.
Simon: It is. And that's one of the things, particularly DeepRacer as an example is something I've seen so many people grab that and use it to learn. And they've sort of done it because, yeah, writing remote control cars with computers, who wouldn’t want to do that? But afterwards, they're like, “I learned so much about how I can apply this, and importantly, where I can't apply it as well.” So, just because you have a tool doesn't mean you should use a tool.
So, one of the things about DeepComposer and DeepRacer and DeepLens is to learn where's the best fit. In my toolkit of things what should I use, and how should I use them? And by making them more available and fun—fun being a relative term—it means that people can get their hands on these technologies, and that's the big shift. If I reflect on, sort of, what's changed a lot over the last 30-odd years, 30 years ago, if you want to use a new technology, you had to get on a project that used that new technology, and that was hard. Whereas today, it's like, “Oh, I’ll just spin up my account on AWS and I'll just get a DeepComposer, or I’ll do DeepRacer. I’ll spin up some SageMaker, or whatever.” The barrier to entry is much lower, and that's really exciting.
Corey: That really is the differentiator here. And I'd argue that is the real transformational power of Cloud, where I know I'm a little late to the observation here by about 15 years, but the fact that I can have an idea, something I want to experiment with, and more or less run a single command, and, these days, within seconds—once upon a time, within many minutes because, hey, the provisioning plane needed to evolve from time to time—I could have an environment set up and ready to go. And when I was done, I could then, in theory, just turn it off and never be billed again. In practice, I would then spend 22 cents a month for the rest of my life for ancillary resources that wound up being spun up that I could never fully track down. Which brings us to, I think, a topic that is near and dear to both of our hearts. Specifically, the idea of—not necessarily reducing costs because that topic gets done to death, but more about understanding and attributing it to different aspects of the environment. Tell me what you're seeing.
Simon: Yeah, I think you raise a good point, is that really the high-value discussion, the discussion you want to have with your CFO, or COO, or whoever pays the bills is, “Is what we're doing valuable to our organization as a function? Is it worth spending the money? And is it worth spending this amount of money?” And the beautiful part of Cloud and AWS is that you can track down to the cent, what you're spending on a transaction, on the system, on an operation, on a development process, et cetera. And that puts you in the absolute box seat to make decisions about whether it's worth even doing.
And then secondly, you can think about, from an architectural standpoint, what are called economic architectures. And this is where you sit down as an architect or as a developer, and you make informed choices about design patterns you apply that effect in price is a consideration. Now, this is really different to how the model used to be in my day. In my day, we’d specify all the hardware, software, storage, network, et cetera, that we thought we needed upfront before we even built the software, and hoped that we got it right, which we, spoiler alert—
Corey: Oh, yeah, we pushed the purchase request uphill both ways.
Simon: Yeah, absolutely. Never got it right. So, we either had too much or too little. And really, what's important here is to think about, am I using the best possible tool? Is it the most efficient? Does it deliver my functional and non-functional requirements? So, am I getting availability? Am I getting security et cetera? Am I avoiding some operational cost in the future?
Well, honestly, my projects at the moment, I [audio break] think I've spun up an EC2 instance in months because I just do everything serverless now because I just don’t like patching systems, and I don't have to. So, that has an ongoing benefit. And so looking at holistically, really, is the difference here. And one of the trends that's happened is DevOps and DevSecOps—or call that domain whatever you will—but the better understanding of developers in terms of how these systems operate in the long term, and the operations teams in terms of some of the decisions that get made early on in the development lifecycle that have long term repercussions is enhancing and improving substantially. And the organizations that start to get that right tend to be much happier places, and they deliver systems that have better value and can show back that they're running as they should run.
Corey: That's an increasing challenge. And I find that there are two reasons that that's challenging, at least in my experience. One is the obvious of there's an awful lot of moving parts, and not everything is easily attributed back to a single cost center—shared services make that a challenge—but the other is that when you're having this weird cultural dynamic where there isn't a culture of cost attribution, of understanding how to transform legacy processes into something that is more dynamic, it seems like a win when once upon a time, it took us six weeks in a good day to wind up getting a server provision, and now with the Cloud and the company's processes, it only takes four. But it feels like there are opportunities to optimize that. And that, in turn, leads to other weird anti-patterns, such as if it takes that much work to get something spun up, you'll never give it up because you might need it again, and who has that kind of time to kill?
Simon: Yep, yep. Let me, maybe, tell a story that relates to that, and answer that question because I think it's a good point. And this happened a few years back with a customer that was all-in on AWS. And they had a big developer team, like hundreds of developers. And they had set a EC2 limit of about, it was about 500 concurrent EC2 instances, and they were always at that limit. And when they dived deep on it, they discovered that the reason was exactly what you just said: the developers would not release the systems because they were so close to the limit that they were worried that they wouldn't get one back.
So, counter-intuitively they upped the limit to 750, and the actual usage dropped to 400. So, they saved money by increasing the limit, and it was a really interesting study of human factors of that supply and demand, and concept of scarcity. If there’s scarcity, human beings tend to hold onto things, it's just kind of how we’re wired. If there's abundance, then we're not worried about it. And so to your original point of how do we take the time to understand cost? Who cares? Et cetera.
At some point, someone is paying the bill, and what I find works really well is getting as good a handle as you can into your environment—and tagging is a big part of that. And there are some great tagging strategies you can use to get a view of cost at various levels—but the other thing is having the right conversation with the right people in the organization. And this is where—you know, IT and finance don't often talk in detail. They tend to talk at the high-level numbers, the CIO goes and submits a budget, gets some budget, applies it, et cetera. What I've found is that when we bring the finance department into the conversation and show them how much granularity we can see around our spend compared to our utilization, we're suddenly speaking their language, and they're like, “Well, I'm invested in this. I want to know more. What can you show me? How can I understand more about what we're doing?”
And then, once I understand the visibilities that I have, that then informs reducing or lightening the governance frameworks around it. Because once I can see something, and I can see it in near real-time, I don't have to put these heavy gates in front of me. So, like you said, I don't have to have a four-week delay to get an EC2 instance if I know—if you spin it up and it doesn't heed the policy, it'll just get turned off automatically within five minutes. I'm good with that. That changes that mental model, and having those high-level conversations back with data is really changing some of these organizational structures and governance processes.
Corey: This is a difficult thing to achieve in most corporate environments. I can't imagine how much more difficult it is when you have a lot of these practices enshrined in the law, or process that is so documented and required by so many other aspects, it may as well be law. How do you begin to pivot the culture around, I guess, a transformative opportunity that was never even considered when a lot of these laws and processes were written?
Simon: I think the thing to remember is that there are many stakeholders for whom change is their goal. They don’t come to work going, “You know, I want to keep it exactly the same as always been, and it shall always be ever thus.”. I’m not saying everyone comes to work not thinking that, but there's a big component of people who like, “Hey, how can I make these better? How can I improve this? I don't want to sit in an architectural review board meeting every single week for six hours to approve something I know is going to be ok.”
So, tying into that cultural change, and again, finding those right stakeholders, giving them the information, and having the right conversation is really important. And let me give you a for instance, or an example of this. I remember many years ago, many years ago, early on in the days of Cloud, I met with a significant bank. And I had a workshop with the security team. This is 15 of their heavy-hitting security architects that basically got the opportunity to say, “Hey, Simon’s going to come in, you got an hour and a half with him. You can ask him any question, throw anything at him, have fun.” And they put me through the wringer for an hour and a half, and at the end of that session, they said to me, “Everything's going to be on Cloud in the future. This is fantastic. What can we do to help?” And I was like, “Okay. That's an interesting lesson learned, which is that there are people who—they are there to protect and apply governance and requirements, but also to evolve thinking based upon new data. And one of the biggest challenges I find is people aren't always aware of what is possible today. You work around AWS and Cloud all the time, as do I, so we just assume, “Oh, spin up an instance, create a VPC, or spin up a database or—”
Corey: Or talk about new service, and then have AWS employees look at me and wonder if I'm making the service up because who can possibly keep track anymore.
Simon: [laughs]. But that's a mental model that we're very comfortable with because we live it all the time, but for a lot of people, they've never worked, or lived, or really been exposed to that kind of velocity, or opportunity, and so they get delighted when they can do that. So, there's a lot more latent change opportunities than you may think.
Corey: And that’s always a challenge because looking at, I guess, the trajectory of companies as they go through various ‘digital transformations’—which I despise the term, but don't have a better one, so I begrudgingly use it but always with a cynical tone of voice—and you see that they go from this idea of data centers, where everything is, effectively, very fixed in terms of cost, and then as they move through the, I guess, spectrum of how Cloud-y or not something is, it becomes a lot less expensive—in theory—and it winds up instead billing more upon usage. And at some point, you wind up hitting the pure serverless ideal of transaction-based billing, where it’s, oh, I can now trace, as Simon Wardley says, the flow of capital throughout my organization. I haven't really seen that yet because almost nobody is full-on serverless to that degree, and the few shops that are—for example, A Cloud Guru is famous for this—but during the Serverlessconf, they get up and show their AWS bill and it's something like 500 bucks a month. It doesn't actually mean enough money to matter. So, what does transaction-based pricing start to look like?
Simon: So, one of the things with that is, firstly, it's often really, really low. And that's scary, as you said because we're not used to dealing with those types of numbers. But what it looks like is an evolution in certain systems within organizations versus the whole thing. So, again, this is a continuum. You're right, for a lot of organizations, this is hard to change; there's an inertia that builds up over time.
But much like planting a fruit tree, the best time to plant is 10 years ago, the next best time is today, people need to start. And so we have lots of customers who have saved huge amounts of money, like millions and billions of dollars just by changing their operational model. Now, they haven't necessarily had to say we're going to go all-in on Cloud, or we're going to move everything across. They've picked their nastiest, most expensive, problematic, non-scalable system, what have you, the one that's the most opaque, and chosen to attack that and deliver that much less. Now to your story, sometimes it's so much cheaper that people don't necessarily want to talk about it.
I was working with a customer once, a little while back, who was replacing a major system, this system probably cost them—just thinking—it was about $10 million a year to run this system traditionally. And we said, “Okay, we can lift and shift, make a few changes, cloudify it a bit, run it serverless, and it'll cost you $1 million dollars a year.” And they said, “We can't go back to our leadership and tell them that.” I said, “Why not?” They said because it's too cheap, and we're embarrassed that we spent $10 billion over the last few years on the same system. And I said, “I hear you, but that's not actually the problem here. This is a big win. It's not about the decisions you made in the past with the best knowledge you had at the time. It's about the decisions you make now, knowing what you know.” And that mental model shift is really important. And doing it on a case-by-case basis is really important.
In what you might be forgiven for mistaking for a blast from the past, today, I want to talk about New Relic. They seem to be a relatively legacy monitoring company, and I would have agreed with that assessment up until relatively recently, but they did something a little out there: they reworked everything. They went open source, they made it so you can monitor your whole stack in one place. And most notably from my perspective, they simplified their pricing into something that is much more affordable for almost everyone. There's even a free tier with one user and a hundred gigs per month, totally free.
Check it out at newrelic.com.
Corey: And that's something that I found is that one of the hardest parts. The technology is relatively straightforward. The getting people to a point where they can have that pivotal moment, that shift in how they view these things, is always the hard part. And even now, going from the idea of—for me—running on a bunch of virtual instances that are running everywhere. Great; now moving to containers, heaven forbid, or serverless is still a whole ‘nother sea change.
But backing up, how do you even, in today's world, go from a time of understanding the world through a lens of data centers in computing to moving to Cloud? Because to be blunt, when I did this, there were a lot fewer services in Cloud that I had to worry about. I looked at the AWS Console, and, oh my god, I'm never going to be able to learn about all of these services. And there were 12. Now, there's significantly more than that, and I don't know where to even begin in a modern era. Where do you stand?
Simon: Yeah, so let me tackle that from a few places. So, firstly, one thing that I think is really important in a long term IT career is continual learning. And I think many of us start our careers off loving to learn new stuff, and then we kind of stagnate after a while because life gets in the way, but if we're not relearning all the time, we're not going to be able to deliver the best for our stakeholders, for our customers, for our sales, what have you, take advantage of the latest and greatest. Now, I'm not saying always use the cutting edge, the bleeding edge, et cetera. I'm saying use the things that make sense given the best of what you know.
The reason why I mention that is the way I look at all the services we have for customers is, it's kind of like a painting palette. You might not use every color in the palette, you just use the ones you need at the time. So, the mental model I use is to say, “Think about what you're trying to achieve, and then pick the services you're using.” So, for example, you just mentioned, “Hey, when I host my website, how would I do that?” It’s like, okay. Hosting a website; that sounds like some content: lives on S3. Sounds like I need to give it to some people: CloudFront probably fits the bill there. Probably need to have some security, HTTPS: that's Certificate Manager, strap that in there. And then maybe I'll do some logging, some CloudFront logging, maybe I'll report on it using Athena, and I might call it good. If I want more functionality, I can add more functionality, but I can just stop there. And I don't need to go super deep in each of those services, I just need to know they’re kind of there. And I can use them. Because I want to tie into your drinking game, Corey, so I can have a drink. Is, because our roadmap is 90 to 95 percent driven by customer requirements and feedback. They're telling us what they would like us to build, and take off there [unintelligible].
Corey: That 5 percent gap, the things you release that no customer asked for, is amazing.
Simon: [laughs]. Well, you know, the thing is, is that we don't always know what we want. So, sometimes we need to see things that we didn't think of. But in terms of your question, “We've got this wall of services, what do I do?” It's about thinking about what you're trying to solve for today. And assuming that, well, I'm sure other customers have asked for this, too. Let me go check if Amazon has tried to solve this on our behalf. And usually, the answer is yes.
Corey: One of the problems I have is that invariably AWS has this incredible knack for releasing a service that solves those global pain points directly after I've implemented something badly to work around it. I think one of the terms that Forrest Brazeal likes to use is ‘spackle punched’ where, “Oh, I built this ridiculous janky thing. And now AWS has spackle punched me, and solved this problem globally.” Which is awesome, but it would have saved you a couple weeks worth of effort, if it had just come out a little bit sooner. Is that just me with my terrible timing, or is that one of those universal moments?
Simon: I think it's just a feeling that you get with all technologies in a way. But I think if you think about it, tying into it say, “Well, If I'm having this problem, and I'm spackle-filling, as you say, then there's probably literally thousands of people at the same time all around the world doing that.” And then, as you say, it becomes a timing issue. So, whereas you may have already built it, and now you get to replace it, there will be many, many, many more people who will never have to build it in the first place because it's already there. Now, related to that, I might make an interesting counterpoint here is that experience is really important, and understanding how things work versus how things are, gives you a relative concept.
And you talked about people not necessarily understanding how hard it is to get a server racked up, et cetera. Now, I've a got large cohort of solution architects who work in my team, and many of them are much younger than me, and they've never seen a data center; they've never had to order a server after a PO process, et cetera. It's always been on demand. So, we actually created a little learning series to say, “Here's how we used to do it. Just so you understand what was involved.”
And that was fascinating. When you sit there describing connecting to a storage area network using HBAs, you watch people's eye roll in the head and go, “Well, that's not a career I would have chosen,” compared to now just going, attach EBS volume, get on with my day. So, understanding what came before helps you understand how much better things are now, but it's no excuse not to continue to make them better. And so, we will continue to feel that spackle wherever we can.
Corey: And again, I'm not sitting here saying that we should stop the march of progress, or well, it might upset someone who's already built something janky, so you should never release a service. But there is that moment of, at some point, I take a step back, and I realize that, huh, I'm spending an inordinate amount of time solving what feels an awful lot like a global problem. Maybe that's not the best path forward. Last time I did one of these things in earnest was about trying to get replication working with encryption on RDS between AWS regions. Today that's, click a button, but at the time, it was not.
And that was painful and challenging, and I'm looking at that, and it felt like I am probably not the only company in the world that has this problem. Maybe there's a better way forward? I can't shake the feeling that by going down this path of either cloud agnosticism with going multi-cloud or building everything in your own data center yourself with an eye towards I want to avoid lock-in at all costs, you're effectively having to build those solutions that can be done for you by an organization that focuses on solving those global problems for you, like AWS, case in point. I feel like it's so easy to wind up getting wrapped around your own axle of, “So, what are you doing right now that adds business value?” It’s, reimplementing a load balancer doesn't really seem like the right answer, unless your business is load balancers.
Simon: Yeah, yeah. I think the concept of undifferentiated heavy lifting—[take] a drink—is really important because, as IT people we know, we've built stuff that’s really hard to do and you're kind of proud of yourself. Hey, I made this work, but my goodness, that was hard, and no one really cares that I did it. And really what it's about is doing as little as possible to get the outcome you need. And I remember early on in my career, I studied software development, and I got to work with some really, way smarter than me developers who really knew their stuff, and one of them took me aside one day and said, “You need to learn to be a lazy developer.” And I'm like, “What do you mean? Lazy? That's bad.” “No, you need to learn to do as little as possible to get the outcome you're trying to get for the software. Write as little code, be as efficient as you can, use as few services as you can, reduce the complexity, reduce the time it takes to get done.” I was like, “Aha. That's a really interesting insight.”
And that was 30 years ago, when I spin forward to today, what it means to me is, when I'm building a system, I'm going to work really closely with the cloud provider of my choice, I'm going to write as closely to their APIs as I can because I know if I want to make a change, I'm just going to change a few APIs, talk to a different service provider, use a different service, drop-in replace it, whatever, but that's going to get me to my outcome quicker, which means I get my solution in front of my customer quicker, which means they can tell me whether they like it or not, and I can either make a change, or double down on it. And so, it's that mental model of building as little as possible as quickly as possible that, really, has helped in this current environment.
Corey: So, a question I have for you that is a common refrain on this show—it’s, sort of, our themes go—if you were starting today and you're at the beginning of your career—because let's not kid ourselves, you and I have been doing this for decades at this point—where would you start? And at some point, are you done? I mean, is there a place where, “And now I have learned the cloud. Box checked.”
Simon: [laughs]. Check. It's a good point. And to give you some context, I mean, when I started, I started on mainframes. So, I learned CICS, COBOL, DB2. I can talk about JES queues, and ISPF till the cows come home. I still miss it. But then I had to learn client-server, which was the brand new hotness at the time, and then the web came out, et cetera. Really what the lesson is, is that we're always learning, so never get too fixed in your mental models around technologies.
Now that said, if you're starting today, there are some really good foundations to build upon; so lucky. And so, one of the things I point to customers do all the time, now that's available is something called the Amazon Builders’ Library, which is really a set of blog posts and explanations about how we build and operate software at scale. And this is not saying this is the only way to do it and this is how you should do it, but this is saying, well, this is what we do at global scale, and it seems to work pretty well. You might want to learn some things about this. So, now one of things that I really like talking to customers about is how to deploy software and rollback software.
Super hard problems to solve, but we do it all the time, many, many times a day as you can imagine. How do we do it? What does testing look like in that environment? What is validating that's going to work? How do you roll features forward, roll them back, maintain compatibility? All this stuff, it’s already there. There's a lot that exists to learn upon. Even simple things like you mentioned, “I want to get started in machine learning. What do I do?” Well, even if you jump onto the AWS Console, there are pre-built videos, labs, instructions on how to get up and running. So, you can at least learn and stand on the shoulders of giants to get going. What is existing now, we'll look back on in 5, 10 years ago, and say, “Well, how cute. You're doing this, you're doing that. Look how far we've come.” But you can do that at any point in time in computing.
Corey: Now, it's six lines of YAML.
Simon: Yeah, exactly. You can do it at any time in your career. I mean, now, I joined AWS back in 2011. I think about some of the stuff I built back then that, like you said, I can build now with literally a few lines of code, but back then I was going, “Wow, look. I didn't have to spend six months doing this. I did it on my kitchen table.” Just maintaining that mental flexibility is super important. And you’re right, you’re never done, and I think that's really hard for all of us as technologists because we tend to be quite mathematical in our mindset, which means we like complete proofs, and finality, and a solution, and it's solved, and it’s done. And I can, like you say, I can tick that box.
Corey: I've gotten to the end of AWS. The final boss was super hard.
Simon: Exactly. Whew. All done. And you just can’t. And it's hard doing that. And I can tell you that that's something Amazonians struggle with all the time because we'd love to know everything. And I'll share with you a personal story, there was a brief shining moment where I could, hand on heart, say I knew AWS. And, like you said, it was early on. We didn't have that many services. I knew them upside down, two ways from Sunday, inside out. I knew them all the way. But you can't now. There’s, like, over 175 services. [laughs].
Corey: Meanwhile, one of the early engineers who's there and still there, and is now Distinguished Engineer or something, is just sitting there mumbling, “No, you didn’t,” Because there's always another level to go down to.
Simon: It does depend what level you go down to. But as far as I was concerned, I had done it. I was on top of my game, I knew everything I need to know. And then we released another five services, and another ten services, and just [unintelligible].
Corey: Programming, I got it. I have both languages: JSON and YAML. Yeah, so one of the challenges we see, too, is that at some point, you can't go by experience anymore. When I was starting out in my career, and I was trying to sound like I was someone a company would want to hire, there was a point where I wanted to add as many years as I could to experience.
Now, on some levels, I kind of want to shave some off because your skill set is what it is, and how long it took you to get there is in some ways, an interesting metric. I guess the depressing end of the spectrum is I've met people who've been working in tech for 30 years, but they don't have 30 years of experience. They have one year of experience repeated 30 times, and that always really depressed me because at some point, the tide rises, the thing that you do winds up getting washed away, and there aren't very many opportunities to continue doing that one thing. It feels like tech is one of those areas where you have to reinvent yourself the entire time that you have a career.
Simon: Yeah, I think what we say at Amazon is we say yeah, we want to be stubborn on the vision, but flexible on the details. So, being stubborn on the vision is, “Hey, I want to be a technologist, I want to build systems, I want to solve problems, that's what I want to do.” But whether it's COBOL, or C, or Python, or PowerBuilder, or Delphi, or Visual Basic, I don't really care. Like, I care at the time, but I'm not going to be bound to it and say, “Well, there is only one true language from here on out.”
Corey: Oh, I was so angry about things like that back then. Oh, god, picking fights about programming languages, and systems, and architecture. That was one of my favorite things. It turns out, I was a terrible person. Some of us evolved past that, hopefully.
Simon: You've recovered. You've recovered. But it's true, but that's the thing is, we tend to get into these, kind of, almost philosophical arguments about this chipset versus that chipset, or this operating system versus that. It just doesn't matter. What matters is the outcome you're getting, how easy it is to run, easy to manage, easy to learn, et cetera.
And when the time comes to replace it—because we're all building the legacy systems of the future—how easy is it to replace as well? That's what you want to think about. And that'll set you up for a long satisfying and invigorated career, versus just fighting these battles that you’re just going to lose eventually. You can’t win that conversation.
Corey: And that's part of the challenge is, it's even hard to talk about because it’s, on some level, you definitely don't want to be coming across as saying evolve or die, dinosaur, but at some point that, that's kind of what you do. I mean entire jobs that were things when I started my career—like firewall engineer was a six-figure salary, if you had that skill set—now, more or less—I was going to say that, wow, any basic network engineer should have that skill set, but even beyond that, today, it's kind of pretty much—do you know how security groups work? Spoiler. No one knows how security groups work but roll with me here. And that gets you where you need to be, the baseline level of experience is necessary. How do you find that the fundamentals—the things that I guess we had to learn at one point because there was no other option—manifest today? Are they still necessary?
Simon: Well, I think the detail is not necessary, but I think the fundamentals are. And the fundamentals don't change. Now I need to build a system that's resilient. Well, what does resilient mean? Well, resilient means a lot different now than it did 30 years ago. I need to build something that's user friendly. Well, when I was at study at university, I remember clearly my lecturer telling me, “For a good user experience, when they hit enter on the console, it should take no more than four seconds for it to come back. That's the baseline of good user experience.”
Corey: At this point, it can go to the moon and back.
Simon: I know. It changes a lot. But understanding that you need to care deeply about user experience will get you a long way. Understanding that people don't really care about the details of technology—depending on their perspective—so as IT professionals we care deeply we're all about it all the time. That's what we love, we're passionate, we're enthusiastic. See, most organizations doesn't care. They just want to know, does it work? Is it fair value for money? Does it put me ahead of the competition? That's it. That's it.
Now, whether it's all singing, all dancing this, or an all singing, all dancing that, they don't really mind; they just want to get it up and running and done. So, learning how to speak to non-technical stakeholders in an accessible and meaningful way is a super valuable skill. And let's face it, Corey, you’ve made a career out of it. And I'm sure it wasn't a course that you did, or an instructional video you followed. It was realizing, “Hey, I need to talk to these people that don't look like me from a technology perspective, but need to hear what I'm saying.”
Corey: One last area I want to talk about with you before we call it a show was announced last year at re:Invent—back when we would all gather in the same place, and not worry about our lives—was the Amazon Builders’ Library. And that is ‘Builders’ Library’ not ‘Billers’ Library’ which presumably is a compendium of all the different API calls that you can make that will cost you money in embarrassing ways, obvious only in hindsight. Talk to me about that because I love it personally, but I want to get your take on it.
Simon: Yeah, it's one of those great things that we have a lot of really experienced engineers building software here at Amazon. And Andy Jassy often says there's no compression algorithm for experience, and he’s right. And the other way I like to look at it is, I'd much rather learn from other people's mistakes than my own because then I don't have to suffer from the pain of that. And this catalog, the Builders' Library, really showcases what we've learned. How do you build a system that is distributed, that can recover from an outage without the thundering herd of new transactions coming in, thus storing it again and creating a repeatable loop of [unintelligible]? How do you build a Continuous Integration/Continuous Deployment Pipeline that genuinely works at scale and you can easily roll back from? How do you deploy technologies like Shuffle Sharding, or leader election, or all these other interesting things that are highly difficult problems, but have been solved and solved effectively at scale?
And what this library does is gives you all that information for free. Like, if I was a graduate student today, I would just sit down and read that for a day, and understand it deeply, and go, “Wow. I've just saved myself 20 years [laughs] to learn all that.” And so I'm really excited that that's publicly available and continues to grow and have new content placed on it because it is genuinely—now, this is not about Amazon. It's not about AWS, it's just about building software at as high quality as you can.
The lessons you learn, even simple stuff like introducing jitter in the way you handle workloads because that improves your ability to handle it at scale. Like, just stuff that you wouldn't necessarily think about in your day to day work that can inform your own design decisions and the way you choose to build software and may give you ideas to improve the way we build software in the future. So, it's a big one, to go to the Builders' Library.
Corey: Do you think that it has the potential to cause problems for folks, in the sense of architecture as imagined by Hacker News where someone is building out something to work internally at their company, and if they follow every tenant laid down in the Builders' Library, if assuming such a thing was even possible, they would be building this world scanning system that more or less would run payroll once a week for 200 people. It feels like it could lead to scenarios of stupendous overkill, and more or less writing code for the joy of it, rather than to solve a business problem. Do you think that that's a realistic concern, or am I not giving people enough credit?
Simon: I think it comes down, again, to that mentality of building only what you need and evolving the architecture over time. And that's where this information is baked into that. And there's actually a talk we do at re:Invent—I did it many years ago, my colleagues do it now, we evolve every year, which is the scaling to 10 million users—or it could be 100 million users now, but really talks about moving from, I've got one EC2 machine, now I’ve made it highly available, now I'm scaling out, et cetera, those thought processes to go through. You're not building the end state. You’re building the start state and evolving. And I think a lot of the lessons in this can help inform when you might need to reach for that technique in your tool belt based upon the scale you're at. It's like, “Ah, now I've got this problem. I understand what they are talking about. I know how to solve it,” versus, “I didn't know this would be a problem. Now it's a problem, and I don't know what to do.” So, it positions you better to tackle the future.
Corey: If you were to dive into the Builders' Library today, what would you start with? Because it turns out that there's an awful lot—I wouldn't say an awful lot. Not by Amazon service terms—but there are enough documents in there that it could be challenging to pick which one to start with, and on some level, they get very deep very quickly. Is there something that you would say is the most accessible way to get started?
Simon: If you had to read one, it would be the one that's called Ensuring rollback safety during deployments. That's the one. Because that's all about risk management. It's about saying, “Hey, how can I deploy software frequently and safely?” And it solves a lot of problems. What if I moved from XML to JSON? What if I change my protocols? What if I change a database? How do I test that I can roll back? What does that really mean? And there's a comment in that particular article that I had a chuckle about because it says, “At Amazon, one of our leadership principles is frugality, but we don't believe in frugality when it comes to testing.” And I thought, “Yes, [laughs] that is correct. That is the way to think about it.” So, that article, I think, is an absolute go-to. If you read one article, that's the one to read.
Corey: Excellent. We'll throw a link to that in the show notes. Simon, thank you so much for taking the time to speak with me today. If people care more about what you have to say, for some ungodly reason, where can they find you?
Simon: Well, they can indeed find me at the AWS Podcast. It's available where all good podcasts are caught. So, the podcast catcher of your choice, and if you search up AWS Podcast, it will be the first it webpage-wise as well.
Corey: One day I will beat you on that listing.
Simon: Challenge accepted. [laughs].
Corey: [laughs]. Thanks so much for taking the time to speak with me today and suffer my egregious slings and arrows.
Simon: Always a pleasure, Corey.
Corey: Simon Elisha, head of technology and transformation, Australia and New Zealand public sector. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on Apple Podcasts. And if you've hated this podcast, please leave a five-star review on Apple Podcasts, along with a detailed comment filling out exactly why you felt the need to dislike it, in triplicate.
Announcer: This has been this week’s episode of Screaming in the Cloud. You can also find more Corey at ScreamingintheCloud.com, or wherever fine snark is sold.
This has been a HumblePod production. Stay humble.