Subscribe
Share
Share
Embed
Heidi Waterhouse is the principal developer advocate at LaunchDarkly. Prior to moving into a career in advocacy, Heidi worked as a technical writer, documentation consultant, and content manager for 17 years — for companies like Pluralsight, BlueTalon, Dell, Intel, Amdocs, and Microsoft. She was also the very first guest on a podcast you might have heard of called Screaming in the Cloud.
Join Corey and Heidi as they talk about feature flags, the difference between temporary feature flags and long-lived, permanent feature flags, how everyone tests in production but not everyone admits it, best practices to getting started with feature flags, Heidi's vision of Flag Markup Language and what the future of personalization looks like, why the transition to virtual conferences has a lot of hidden benefits, the rise of the digital librarian, how features flags are all about feeling safe about software, and more.
Screaming in the Cloud with Corey Quinn features conversations with domain experts in the world of Cloud Computing. Topics discussed include AWS, GCP, Azure, Oracle Cloud, and the "why" behind how businesses are coming to think about the Cloud.
Transcript
Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.
Corey: This episode is sponsored in part byLaunchDarkly. Take a look at what it takes to get your code into production. I’m going to just guess that it’s awful because it’s always awful. No one loves their deployment process. What if launching new features didn’t require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren’t what you expect? LaunchDarkly does exactly this. To learn more, visitlaunchdarkly.com and tell them Corey sent you, and watch for the wince.
Corey: If your mean time to WTF for a security alert is more than a minute, it's time to look at Lacework. Lacework will help you get your security act together for everything from compliance service configurations to container app relationships, all without the need for PhDs in AWS to write the rules. If you're building a secure business on AWS with compliance requirements, you don't really have time to choose between antivirus or firewall companies to help you secure your stack. That's why Lacework is built from the ground up for the Cloud: low effort, high visibility and detection. To learn more, visit lacework.com.
Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. So, this promoted episode is honestly one I’ve been looking forward to for a while. Three years ago, almost exactly from the time of this recording, I started this ridiculous podcast, and we’re now a couple 100 episodes in or so. But my first guest was Heidi Waterhouse, who is now back for more. Heidi, thank you so much for joining me. You are still at LaunchDarkly; you’re a transformation advocate. First, thank you for getting this whole ridiculous thing started. The world may never forgive you.
Heidi: It’s always good to be blamed accurately.
Corey: Exactly. It’s as long as you spell my name right, there’s really no such thing as terrible publicity, now is there.
Heidi: [laugh]. I feel really sorry for the other Corey Quinn on Twitter.
Corey: Me too because he worked in marketing; he had the name longer than I did. And he gets tagged every so often in ridiculous nonsense that I’m in, and at some point, I feel like he’s going to pivot to marketing in the cloud space just because of inertia. You can’t fight the tide forever.
Heidi: Go with what’s working.
Corey: So, LaunchDarkly. There are a few things interesting about this to me. The first is that—thanks, you folks are sponsoring stuff now, like this podcast. So, thank you for that. Secondly, you’re still there, which is not a ding on the company itself, I want to be very clear, but it feels like the average shelf life of an employee in tech these days is somewhere between 12 to 18 months. You’re over twice that.
Heidi: I am, and I am employee 20 or something. It’s kind of amazing, but it turns out that you can retain high-level employees if you treat them well and allow them to keep growing. So, I think that’s the thing people might consider taking under advisement.
Corey: That feels almost like it’s one of those ancient business practices that everyone likes to frown upon, like it’s something from our grandparents’ era, like make more money than it costs you to deliver your goods and services.
Heidi: That seems fake.
Corey: Exactly. One of those old-timey business models? All right, so we talked three years ago about feature flags. For those who have not taken the time to go through every single episode we’ve ever done, what are feature flags?
Heidi: Feature flags are a way to control your code, after it’s live in the world. At its most basic level, actually, that’s what LaunchDarkly does. Feature flags don’t actually have to work live in order to work effectively. A lot of people create feature flags as database queries or environment variables or runtime changes or settings. It’s a pattern that most teams end up needing, and they either recreate it or they buy it.
Corey: A line that I heard recently that reframed the entire topic for me—and perhaps you covered this on the first episode; I don’t actually know, I was way too nervous to pay attention to what you were actually saying back then—is it separates out feature toggles and rolling out features from your code deployment. And that one resonated because there are two kinds of shops out there, really: those who have terrible disturbingly bad code deployment processes, and those who lie about having the exact same thing. No one has a terrific code deployment process to go safely and repeatedly from developer laptops into production. And the only people are going to argue with that are the people who sell something that claims to do that. But it’s always scary. It’s always frightening. And having to do that to change what your application is doing feels like it’s a little bit—how do we put this—overwrought. Is that a fair characterization?
Heidi: Yeah. And the way I think of it is in sort of a Vegas thing. So, I come from an era where we had something called a gold master. We burned things on CD, and that was what you got, so we were really cautious about what we put into a release. And now we live in an era where you can YOLO stuff into production 100 times a day and not have a problem with it.
But that doesn’t mean that people want their website that they’re using to change 100 times a day. That’s bad for the business. We’re not just moving their cheese, it’s just popping around like a video game character. But on the other hand, if we do a release every hour, it gets a lot less scary because we have to have streamlined it. Like, if you can’t run your test suite in enough time to do a code push in under an hour, then you optimize your test suite so it gets better.
So, once we separate this idea of like, “How do we get things on to the server?” From, “How do we deliver the things to people?” We realize those are actually two different roles, and why did we conflate those?
Corey: I first learned about feature flags a while back at a presentation at a meet-up from some big tech company. And it felt, “Oh, that sounds like an awesome idea that you would run at a big tech company, but here in the real world, no one’s actually going to do it until you’re at that scale.” And talking to you the first time, it was clear that that is not strictly accurate. That is the whole point of LaunchDarkly. Three years later, has the industry changed? Is adoption of this pattern becoming more widespread? Are we learning new things as we go? And if the answer to that question is no, I’ve really just painted myself into one heck of a corner, but we’re going to try it anyway.
Heidi: [laugh]. This is honestly something I’m very proud of professionally. We have done, as a company and as a movement, so much to get the idea of feature flags in front of people and teach them that it is not just about how your frontend behaves and it’s not just about A/B testing, which is what most people think it’s for, but it’s actually about being able to mitigate your risk to be more cloud-native, to think in a way that says, “I don’t actually have a deterministic way to say everybody is getting the same thing at the same time unless I’m using a tool to make sure.”
Corey: So, is the idea of feature flags strictly a frontend concept? Is it something that winds up mapping to backend as well? And, God forbid, is there a story about using feature flags for infrastructure?
Heidi: Oh, absolutely. So, I think the most compelling story that I keep running across when I talk to people in Ops, is the idea of a permanent kill switch. So, most people, when they talk about feature flags, they’re talking about deployment, and rollout, and testing. Those are what we call temporary feature flags; you’re going to pull them out after they’ve done their job. But there are also long-lived or permanent feature flags that you put on bits of your infrastructure so that you can control things when shit goes sideways.
So, for instance, imagine you have an inbound API that’s writing to your database. This is a normal thing that happens: you sanitize the data, it’s a known sender, you’re accepting all this data, and you have a monitor on it. And all of a sudden—Datadog or whatever—your monitor goes off and says, “I’m getting 100x traffic. I don’t know what’s happening. Beep, beep, beep, I’m not happy.”
And the first thing that you want is to start shunting that traffic off because it’s probably a DDoS, or some other kind of bad data. So, rather than have to wake somebody up, figure out what the problem is, figure out where to shunt it, you can set up a permanent feature flag that says, hey, if this alarm goes off, I want you to shut all that data to this overflow database. And I will wake up and look at it, but first, maybe I will ingest some coffee or at least, you know, wash my face before I try and stare at a screen. That gives people so much more time to react in a smart way and uses our automation to sort of delay the need for the human in the loop. The human has to be there, but they don’t have to be there instantly.
Corey: The traditional idea of feature flags seemed like it was something that you would use to roll out experiments, on some level, to 1 out of 100 users of your site, and then you could start validating: Is this feature working? Is it breaking? Et cetera. Facebook, I seem to recall, had something vaguely similar. This is misremembering many moons ago, back when I gave the slightest crap what anyone from Facebook had to say to me about anything, just based upon ethical reasons. But they started off with, I think, seven concentric circles, or six concentric circles that spanned from a single developer’s account all the way out to the entire world. That feels like the feature flag story to some extent, isn’t it?
Heidi: It is. And Microsoft uses it, and Lyft uses it, and they all have teams that are doing that. And the story is, put it into production, but nobody can see it. And this works especially well at Facebook and Google because they’re using trunk-based development. So, everything is always live in their codebase, it’s just hidden behind different feature flags.
So, they put it out into production—they’ve deployed it—they turn it on for themselves, they see if it works; they turn it on for their team, they see if it works; they start turning it on for beta users. Microsoft calls this ‘ring deployment.’ And that really works for getting stuff out and making sure that it’s not going to be overwhelming in a weird way. And also, it turns out that even though most of our test engineers are amazing geniuses, and we should take them more seriously, you can’t really test how a distributed cloud environment is going to respond, except in that cloud environment.
Corey: So, the question I have, at the idea of expanding out from effectively just the developer’s test account, all the way out to the entire world regardless of who you are, is there an ethical concern here? I’m not trying to wind up putting you on the spot, but the idea of, I want to roll out a test to my paying customers, in many cases. The idea of chaos engineering and running experiments, and breaking production intentionally came out of Netflix, among other places, but at Netflix, the failure mode was on some level, “Okay, someone has to restart their stream in the event that something goes sideways.” That isn’t really the end of the world. But there is still a question, okay, you’re actually slightly degrading a paying customer’s experience. Given that feature flags are seeing adoption significantly outside of the entertainment space where the stakes are almost invariably going to be higher, okay do you stand on the ethics side?
Heidi: So, I feel like most of the life-critical and financial clients that we have, do manage to work with feature flags in a regulated environment because they already have rules about how to do that. It’s not like I’m saying because you can test on anybody, you can test on everybody. So, if you can test on yourself, that’s fine, but you still need an approval to test on any customers. There’s still an approval process. And in fact, we built in a new set of features that allow you to do approvals and say, “Okay, developers can try this out for themselves, and internal people, but it has to go past the approvals board if it’s going to hit any customer effect.” And actually, the thing that I say about this is that we are all testing in production, it’s just that some of us admit it.
Corey: That’s fair. Do you think that there needs to be additional scaffolding put in place before you can do this in an effective way?
Heidi: So, LaunchDarkly provides you a lot of that scaffolding, if you already have some concept of having to be regulated. I think that if you are a new financial startup, I hope that you are consulting best practices on how to set that up. I think that the thing about feature flags is that they are not a pattern, but a tool that can have many patterns. And that gives you the ability to say okay, the way we’re going to implement feature flags is with this extreme change management control, staging environment, soak time, like, we’re going to have all of these safeguards, or you can be somebody who doesn’t have to be that careful and you can move fast and YOLO things into production and see how it works.
Corey: So, on some level, what you’re saying is that the folks that are going to need to build additional scaffolding to do this responsibly, more or less already need to have built that scaffolding already and they’re already behind the curve to some extent.
Heidi: Right, exactly. Because how else are you going to say, “I can auditably and verifiably say that this person got this exact variant of the deployment, of the release?”
Corey: One of the problems I have with the idea of feature flags—that’s right, I brought you on the show on a promoted episode to basically tell you, “You know what your problem is”—and basically berate you for the way your entire product works because that’s how we roll here. But I have to ask, I wonder how I even begin getting started with something like this. I want to go ahead and test it, sure. It feels like I may have to wind up doing two, three sprints worth of work just to get into a position where I can even test something out. And at that point, it almost doesn’t matter what you’re going to charge me for a product or service. The sheer engineering time investment makes it a relative non-starter.
Heidi: Right. So, one of the things that we found is we are so frequently replacing some homegrown solution. So, people have the concept of being able to control how their software operates, they’ve just set it up in some way, and that some way is not scaling. One of the patterns that I’ve seen when people get started is they get started with something that’s not mission-critical because it’s easier to learn that. So, we have a customer called Xero who does a ton of payroll stuff in Australia and New Zealand.
And the way that we got in was, they wanted to use it for their financial transaction stuff but they didn’t want to do a ton of risky messing around with that. So, what they did was they’re like, “Okay, we’re going to buy a small license, and we’re going to use this to control our website. And once we’ve internalized how it works, then we’re going to use it for financial stuff.” And so these small non-mission-critical projects are learning labs for the team and then they can go on and share that knowledge with a more core business value team.
Corey: When we first spoke a few years back, my initial takeaway was, it sounds awesome. I can see the value. But it also felt like you were more or less running into the wind in that first you had to teach the market about the thing that you solved then immediately afterwards had to go ahead and sell them something. That always felt like a very heavy lift. But looking around now, I can see broad consensus in the customers I talked to about the understanding of the value of feature flags, and, “Oh yeah, it’s a reasonable thing that we should be doing.” It seems like in that respect, the heavy lifting has already been done, and on some level, it, “Oh, it just sort of happened organically. That’s just a natural evolution of the market.”
Heidi: [laugh].
Corey: It feels like that may have partially been what you're doing.
Heidi: Thank you. I have worked really hard. It turns out that category creation is enormously fun. I mean, you know, founder of Duckbill, a consulting group that comes in and says, “You’re doing it wrong and here’s how to fix it. And we’re not going to charge you more for being dumb.”
It was actually kind of a risky stance to take. And in the same way, LaunchDarkly came in and said, “Okay. We see this need and we’re going to explain to you why your life will be better after this.” And fortunately for me, CI/CD really took off, I think there’s been a ton of great work from the IT Revolutions people, putting out Accelerate putting out Project to Product putting out Sooner Faster Happier. This ethos, this zeitgeist of being able to move faster, and safely, is exactly the group movement that I needed to catch on to.
Corey: It’s a really neat thing to see the natural evolution of a product in a space, going from, “What the hell is this thing?” To, “Oh yeah, it’s a best practice and if you don’t do it, you’re probably doing something that is at least marginally dangerous.” It’s really something to behold and I have a hard time identifying other major players in the space that aren’t you folks. Not that I’m asking you to, because, “Yes, now let’s talk about your competitors,” is never a great look on an episode. But as you mentioned before, I strongly suspect your strongest competitor, the one that we all fight against commonly is, “I’m just going to build this internally myself.”
Talk to me about that. What does that usually look like because very often when I see people building things internally themselves, they don’t quite contextualize it in the context of the thing they can go out and buy; they view it as something different. As I look around the shattered remnants of my build system for the crappy software I build and deploy myself internally, what parts of that are going to look like, hey, that’s a feature flag option but I don’t think of it that way.
Heidi: Right. I think that this is actually a super exciting place for tools vendors to look at because it turns out that not only do people build it themselves in large organizations, they continue to build new things themselves. By my count, Google has at least ten different feature flagging systems.
Corey: Wow that’s almost half as many as they have messaging options that they release and then deprecate.
Heidi: Right?
Corey: I don’t know if we’ve seen the google messaging application for 2021 yet, but I’m sure it’s coming.
Heidi: I’m sure it’s coming. And then we will get attached to it, and then they will kill it off. I’m still salty about Reader so, you know.
Corey: As am I. They are never going to live that one down.
Heidi: Never.
Corey: Nor should they.
Heidi: It was rude. It was very rude.
Corey: It absolutely was. It showed a flagrant disregard for an entire ecosystem.
Heidi: The thing that I find when we’re competing with homegrown is that people are solving the problem in front of them, and it is absolutely true that it will take your engineers less than a week to code up some kind of feature flagging system, but it is also true that we have invested a ton of money in all this infrastructure so that we can serve flags at the edge in under 200 milliseconds around the world; that we have done a bunch of integrations, we have like 23 SDKs now; we have all of these abilities to hook into Salesforce and ServiceNow, so that you can have this seamless throughput and so that people don’t have to leave their native tools in order to use feature flags. And your developers aren’t going to replicate that and they’re not dedicated to researching where we need to go. So, when I’m competing with a homegrown solution I’m always like, “Yes you can build it, but it’s free as in puppies. You’re going to have to maintain it.” And that’s the expensive part of any software.
Corey: Any engineers who build this kind of thing just please skip ahead 15 seconds on your podcast players. Go ahead; do that now. Great. Managers, yeah, do you really want this sort of thing being built by the exact same people who built whatever horrifying monstrosity you’re using to deploy your existing software into production? Really stop and think about that for a minute.
Okay, now let’s talk a little bit about the future. Now, I’m not asking for roadmap information because that’s always in flux and no one likes to pre-announce things, but what do you think the future of feature flags is? Now, that it’s broadly accepted, okay’s it going from here?
Heidi: Individualization. The future is personal. And I think that the thing that we want to be able to do is let people set their own experience of their phone, and their web, and their smart home devices, and say, in all of the ways that we sort of have control now, we get more control. So, I have all of these things that I’m like, “I can change the settings on it, but not as much as I want.” And also, the settings are pre-assuming a bunch of things about me.
So, if I had the ability to do some of the things that we can do in browsers—like you can set your browser text to be something more dyslexia-friendly, okay—But not all web pages respect that. If I could force that, it would be awesome. I’m not dyslexic, but I want that ability, I want the ability to say, this is exactly the experience that I have chosen for myself.
And I’d love it to be portable, I’d love it to be a markup language because I think it’s a real accessibility statement to be able to say, this is what my web experience is like. And I have separated the content from the container. It’s a really old tech-writing concept that the words and how they are presented are almost entirely separated from each other. And in the same way, I want somebody to say, “I’m giving you this web content or this application, and how you present it is up to you.” And breaking that linkage is going to be so empowering for so many people.
Corey: Tell me a little bit more about this. I was worried when you said personalization because, “Oh, good. More creepy tracking of people.” But that’s not at all what you’re talking about. You’re talking about something that winds up transcending devices and sticking with a person, but for, I guess, the power of good rather than for the purposes of, basically, spying on people.
Heidi: Right. So, this is my futurist hat and not necessarily LaunchDarkly, like, end goal, but what I want—I’m not allowed to call it ‘Flag Markup Language’—
Corey: For the obvious acronym purpose—
Heidi: Yeah.
Corey: —of course.
Heidi: But Flag Markup Language follows you around and says, “I never want to see day mode. I’m only a night mode person, and if something appears in day mode, I want you to override it.” That’s like the simplest explanation of it. But it would also follow you around and say, “I’ve reduced the screen width.” Or—here’s an important one—I’ve taken out everything that makes this page very heavy because you are accessing it on a very narrow pipe.
Like, my parents don’t have cell phone service, and their WiFi is, well their rural internet is not great. And so, every time I visit a page that’s not a problem when I’m in the city, it takes a minute to download because there’s all this stuff. And I’m like, what if people could still get their ads through, but they were simple text instead of, like, video animation, based on the size of the pipe that is trying to go through.
Corey: I love the idea, but it feels, on some level, like that also requires broad-based acceptance across the board from every site that they visit, wouldn’t it?
Heidi: It would, or you’d have blockers. What it actually requires is broad-based acceptance by the browsers.
Corey: Got it. That feels like it is simultaneously easier and far more difficult all at once.
Heidi: Well, I don’t think it could be a solo project, but I think it would be a fascinating step forward in accessibility to have Lighthouse run and say, “Okay, but your page is not only inaccessible, it’s also too heavy for people who are on this bandwidth. Do you want a reduced fidelity version?”
Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn’t translate well to cloud or multi-cloud environments, and that’s not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.
Corey: So, one more question before we wind up calling it a show. You were one of the people that basically got me on board the train of giving conference talks from an iPad. It was transformative. It worked super well. I was really in the swing of it, and then the pandemic hit and I’m not traveling at all anymore and my iPad is mostly gathering dust here.
What’s your experience been on that? Are you still using iPads for any of your digital presentation works, or are you basically putting that on hiatus until it’s no longer taking your life into your hands more than it normally is to go on stage in front of a roomful of people?
Heidi: I think the earliest I will be at a physical gathering is possibly November. And honestly, conference organizers, you’re going to have a hell of a time doing anything this fall. And it had better be single-nation. We are not going to be able to cross borders.
Corey: Oh, absolutely. And beyond that, the first few conferences that rush to come back in person, you’ll be able to take a look at who attends those things and realize whose company considers them expendable.
Heidi: Right?
Corey: It’s a harsh thing to say; it is also accurate.
Heidi: Yeah. It’s really interesting to see how this is going to work out. But as far as my iPad, what I’m actually using it for now is I watch talks, I don’t live-tweet them as much because I’m watching on the iPad, and its multi-screen capability is okay-ish, but not great.
Corey: Adorable. I would classify it as adorable.
Heidi: Yeah, like, points for effort.
Corey: There was a solid attempt.
Heidi: But I will watch talks because it turns out that a lot of what makes me work is getting to listen to engineers, and developers, and ops people. And if I don’t have that input, I don’t have any grist in my mill. I figured out this is why I was having such trouble writing blog posts is because I wasn’t talking to anybody out in the world who was having problems. And I guess that whole developer advocate title wasn’t just hot air because what I really care about is making sure that those voices are getting represented.
Corey: There really is something to be said for what has happened to conferences in the past year. Suddenly, attending a conference no longer requires the ability to travel places, take time off, pay for your accommodations while you’re there, and in many cases, pay the not small fee for the conference. Suddenly, there’s a wealth of content that is available online, universally. And sure, the experience is relatively crappy, but it’s universally crappy. There’s not a better experience for some subset of people who are able to spend more, and the folks who can’t cover that expense are sort of forced into a substandard, degraded mode. This is what it’s like for everyone. And I have a hard time seeing how that is going to continue once the world opens back up. It’s one of the vanishingly few items on the list of things I’m going to miss, post-pandemic.
Heidi: Yeah. I was speaking at a conference based out of Russia. And there was a guy logged in from Kinshasa, D.R. Congo. Now, the thing that nobody really knows about me is I grew up, until I was five, in D.R. Congo. And it had never occurred to me that I was going to get to talk to a technologist from Central Africa because they can’t get visas; money is an entirely different thing.
And now in this new time, all you need is an internet connection to be able to attend. And honestly, I kind of choked up because there are all of these technologists all over the world who have been shut out for various reasons: because they can’t get childcare, because their company won’t pay for it, because they’re not like working for either a very large company or a cool Silicon Valley company. It makes me painfully aware of what we haven’t been providing all along, and I hope that conference speakers and me—this is a thing that I’m working on—will start doing more fixed time, like, here’s a webcast, here’s a podcast, here’s a conversation that enables everybody to access it.
Corey: On the other side of it—again, not from a engineering insight and knowledge perspective—but what I think is a slow-dawning awareness among a few people that I’m super excited about, is they’re watching me, effectively, livetweet slash aggressively shitpost various big company keynotes. And they’re finally realizing something that, wait a minute. Corey is not doing this because he had early access to what’s coming out and is ready to go with it. He doesn’t have special front row seats; he hasn’t been behind the stage talking to people before it goes out. He’s just watching this thing in one window, screen-capturing it as he goes, pasting it into his Twitter client, which is just the Twitter web app, adding some stupid commentary, and whacking send.
That’s the entirety of what I do, start to finish. And I apologize for just using the word stupid; let’s say ‘nonsensical’ instead. Let’s be a little less ablest. That is all it is. And I think that there’s now a slow creeping awareness that, wait a minute, it doesn’t require stupendous amounts of money, or access, or privilege beyond the normal level of privilege that I have in this space.
It’s just the ability to do it. And of course, the tremendous privilege I have of not being able to be fired for the things I say on Twitter, which is a separate problem entirely. But it isn’t because I have this magic ability to reach behind the scenes and grab things. It’s just, I’m doing it because no one’s stopping me from doing it. And I glad to see that other people are starting to take notice of that.
Heidi: Yeah. I don’t think you need to be an insider to be a good reporter. And I think that one of the things I’d love to see is for companies to hire some people that they haven’t been thinking about lately. My current campaign at LaunchDarkly is I really want us to hire a librarian. Honeycomb did it and I think it’s frickin genius.
Because we have all of this content, and we don’t have a good information architecture system to find it. And Confluence’s search is… not great. [laugh]. But I want us to hire librarians and I want us to hire investigative journalists, and say, “Look, we are developing so much stuff, but we need somebody to make the connections to make it accessible and usable, to make it something that you can use.” Like, you can absolutely teach yourself programming from YouTube, but where do you start?
Corey: It’s a terrific question. I think it starts with doing it. I wish I had a better answer, just because it’s, “Oh, I just go out and I do the thing that I do.” One thing I admire about you and I’ve always admired about you is you view a primary component of your job as being teaching people to do things. The problem I have is that so much of what I do is an outgrowth of things that have worked for me that it’s not easy for me to teach it because it’s just oh, just be yourself.
Well, most people aren’t themselves, they’re, you know]actually pleasant people. And for me, it’s always been hard to get to that point of being able to articulate what I do in a useful, constructive way. And I am extremely conscious of the danger of, “Oh, just do this thing because it’s what works for me.” That is a perfect recipe to unintentionally create a masterclass in how to be a white guy in tech who is swimming in privilege. Because I am, whether I want to admit that or not. And the things that work for me will not work for someone who is not themselves overrepresented. So, I am very torn on, how do I teach things? What do I teach? What can I effectively teach? And how do I not turn into a monster while doing it?
Heidi: Right. I think actually, the live-tweeting big keynotes is an interesting take because you can be an asshole and nobody is going to fire you. And also, the internet is unlikely to fall on your head because you’re a dude.
Corey: Exactly. My failure mode is a board seat and a book deal. I’ve been using that joke for a long time because it’s not really a joke.
Heidi: It’s not wrong. And I think it’s important to note that every once in a while I pull my punches because I don’t want the internet to fall on my head. And I’m a nice white lady in tech and have axis of privilege along that. So, I think it’s interesting when we’re talking to people who are under-indexed, and we need to be really careful when we listen to that feels unsafe. And one of the things I like about when you’re talking about how you’re funny online is, you talk a lot about punching up and not punching down.
And by sheer odds and percentages every once in a while you get it wrong, and then you apologize and try not to do it again. And I think that’s certainly a model I would like to see a lot more people adopt, where I don’t want people to necessarily be permanently canceled, but I do want them to say, “I caused harm and that was wrong, and here’s what I’m doing to fix it.”
Corey: And sometimes I feel like all I can do is try and lead by good example. And on some level, that’s really what the story about feature flags—and now, that’s a hell of a reach—is. It’s a—[laugh] it’s about setting examples and giving good demos. And that’s what you did. That’s why it became normalized.
Let’s stop beating around that particular bush. The reason that it became normalized is because people like you got up on stage, from an iPad, threw up some random website that you’d just thrown up, and said, “Here’s how feature flags are going to work. And here’s what it took to instrument it.”
And it turns out, not that much. You can do it in a live demo.
And then there was an interactive approach. And seeing that again, and again, went, “Wow, if you can use this for upscale shitposting mid-conference talk, done live, then what’s my excuse for not being able to do this thing in production?” And the answer is, “Oh, right. I’m bad at things.” But for most people, that’s not the answer.
It worked for them. And that, I want to be very clear, is no small thing. I’m not blowing sunshine up your butt when I say that you have fundamentally changed the way the entire industry thinks about feature flags. It’s true. It really is. One of the unique things about—you mentioned at the beginning—is that you have been at the same place for as long as you have, consistently on message but also dragging the rest of the industry forward, politely. You aren’t doing it with my brash way of getting up there and picking fights. You’re doing it by making people feel good by listening to you.
Heidi: Yeah.
Corey: If people take nothing else from this entire episode—forget the feature flags; forget the how to make your software better—if the only thing they take away is just watching you and using that as a life lesson in how to become a better person than they were, then this podcast, every episode has achieved more than I dream to when it’s set up.
Heidi: Oh. All right, but it’s a sponsored podcast. So, I’m going to say my thing.
Corey: Excellent. Please, by all means, take it away.
Heidi: I want you to feel safe about your software. And I want you to be able to do that while your software is operating in production. And the way you can do that is by being able to control it live in production. And if you can’t control it live in production, and if you can’t commit broken code, you’re not doing CI/CD. You’re doing some kind of hideous mini-waterfall. And so when you’re thinking about all of the things that keep you up at night about your deployment and your production, remember that there’s a better way to do it and it involves finer-grained control.
Corey: And again the whole point of this podcast is, I have a bunch of sponsors who say different things at different times. There is a bar: we don’t have sponsors on this show if I am not convinced that there are people for whom their product or service is the right thing. We have rejected sponsors on those grounds before. But to be clear we also don’t ringingly endorse the companies either. With what you do, and what I’ve seen, I do endorse it. I want to be very clear; and that is not something a company can buy, though some have tried.
Heidi: Well, you know, the sacks of cash that VC gives us are ours to spend either responsibly or irresponsibly, and since John, our CTO, is not getting his kombucha tap anytime soon, I think that what we’re going to do with it is do as much as we can to help people sleep better at night. And also—oh, this is a cool thing that just happened at our annual meeting. I just found out that LaunchDarkly is completely carbon neutral. We’re offsetting our AWS, we’re offsetting our travel, we’re offsetting our office space.
Corey: That is no small thing.
Heidi: And we commit to continue doing it. That’s just part of our budget now.
Corey: Well, I guess that really does sort of throw a wrench into the pivot option of starting to do one of those new NFT cryptocurrency things, now doesn't it?
Heidi: Man, I am so angry about that. I am so angry.
Corey: I want to own a representation of a jpeg, but I also want to burn down a forest, boil the oceans, and wind up effectively using more energy to do it than my house uses in 18 months. What have you got for me?
Heidi: I just don’t understand why this—I don’t. I just don’t understand anything that has to do with, I ran my car on idle for two years so that I could do a sudoku that is somehow fungible. The economics of it don’t make sense to me.
Corey: That is a whole separate podcast that I will get to one of these days. Heidi, thank you so much for taking the time to tolerate my slings and arrows and occasionally ridiculous compliments. If people want to learn more, where can they find you?
Heidi: So, find us at launchdarkly.com. We would love to give you a trial or a demo. And you can find me at heidiwaterhouse.com. And every once in a while, I update my blog but not on any regular cadence.
Corey: Excellent. And we will of course throw links to those into the [show notes 00:37:54].
Heidi: Oh, and the place that I really am is Twitter. So that’s—
Corey: As are we all.
Heidi: Right. That’s @wiredferret.
Corey: All one word, of course. Heidi, thank you so much once again. It is always a pleasure to talk with you.
Heidi: I had a great time. Thanks, Corey.
Corey: As did I. Heidi Waterhouse, transformation advocate at LaunchDarkly. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud if you’ve enjoyed this podcast please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast please leave a five-star review on your podcast platform of choice, along with an insulting comment with an embedded feature toggle so that you can wind up changing it to a glowing comment after it’s already been published.
Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
This has been a HumblePod production. Stay humble.