Screaming in the Cloud

This week on Screaming in the Cloud, Corey Quinn is joined by Kat Cosgrove, Lead Open Source Advocate for Dell Technologies. Kat catches Corey up to speed on the newest version of Kubernetes that Kat was the release lead for. The two discuss its unconventional name: Uwubernetes, what goes into creating and implementing a new version of the world’s second biggest open-source project, and which of Kat’s changes will be her legacy to Kubernetes. Kat also shares how she handles running a team that essentially works for free and what her Kubernetes role will be moving forward.



Show Highlights:
00:00 - Introduction and Welcome
00:28 - Meet Kat Cosgrove
01:46 - Kubernetes Release Management Insights
02:43 - Naming the Kubernetes Release: Uwubernetes
06:19 - Roles and Responsibilities in Kubernetes Releases
11:18 - Enhancements and Deadlines in Kubernetes Releases
14:22 - Kubernetes Incentive to Upgrade & Support Policies
18:26 - Running Old Versions of Kubernetes
20:17 - Challenges with Using Outdated Software Versions
22:15 - Best Practices for Version Releases
24:36 - Release Team Cycles
26:00 - Kat’s Release Legacy
31:58 - Kat’s Responsibilities Post-Release
33:04 - Future Plans and Contact Information


About Kat Cosgrove
Kat is a Lead Open Source Advocate at Dell focused on the growth and nurturing of open source through authentic contribution. In particular, her specialties are approachable 101-level content and deep dives on the history of technology, with a focus on DevOps and cloud native. She was the Kubernetes Release Lead for 1.30 Uwubernetes, and currently serves as both a Release Team subproject owner and SIG Docs tech lead.

When she’s not at a conference, she spends her time playing video games, watching horror movies, or reading science fiction, but her current hyperfixation is film photography. She lives in Scotland with her cat, Espresso, who is the real brains behind the operation and actually ghostwriting all of her tweets.


Links Referenced
Kubernetes: https://kubernetes.io/
Kat Cosgrove on Twitter: https://x.com/Dixie3Flatline
Kat Cosgrove on LinkedIn: https://www.linkedin.com/in/katcosgrove/
Email Kat: kat.cosgrove@gmail.com 

* Sponsor 
Prowler: https://prowler.com

What is Screaming in the Cloud?

Screaming in the Cloud with Corey Quinn features conversations with domain experts in the world of Cloud Computing. Topics discussed include AWS, GCP, Azure, Oracle Cloud, and the "why" behind how businesses are coming to think about the Cloud.

Kat Cosgrove: If you are willing to spend that much extra money to run on an old ass version of Kubernetes. Get it together.

Corey Quinn: Welcome to Screaming in the Cloud, I'm Corey Quinn. Very often I like to have conversations with people who are buttoned down and straight laced, and say the business talking things you would expect them to say, but that gets old and I have no patience. So my guest today is Kat Cosgrove, who is so many things.

Kat, thank you for joining me. Professionally, you are a lead open source advocate at Dell Technologies, which, last I heard, makes computers.

Kat Cosgrove: We do, in fact, make computers. We also make monitors.

Corey Quinn: Oh, that's right.

Kat Cosgrove: I think we also do keyboards and mice now.

Corey Quinn: Prowler works wherever you do. It's an open source powerhouse for AWS, Azure, GCP, and Kubernetes.

From security assessments to compliance audits, Prowler delivers with no hidden corners, just transparent, customizable security. Trusted by engineers and loved by developers, Prowler lets you start securing your cloud with just a few clicks with beautiful, customizable dashboards and visualizations. No unnecessary forms, no fuss, because honestly, who has time for that?

Visit Prowler. com to get your first security scan in minutes. And historically, very challenging to work with out of line management cards called DRAC. I have fears of those. Very nice quick rails, though, in server environments. But that's enough of me dating myself. More germane to, I guess, I don't know if this is really as germane to your day job, but it's definitely more germane to a lot of other people's.

You were the release manager for the most recent release of Kubernetes. Uh, what, did you lose a bet?

Kat Cosgrove: I begged for it and I fought for it because I like to make myself so excellent that people who don't like me can't ignore me.

Corey Quinn: Oh, does that ever resonate?

Kat Cosgrove: It was spite driven career development.

Corey Quinn: So let's be clear.

Your Kubernetes has become extremely corporate and ridiculously enterprise. You work at Dell, which let's be clear is many things, but to be direct, hip and with it is not something that has ever been of accusation lobbied at them. They, I, for example, they had the remarkably corny, but remarkably well done, uh, Dude, You're Getting a Dell series of commercials until one day they discovered that the actor who portrayed Steve, the Dell guy, once had fun at a party, so we can't have that.

And let's get rid of him immediately. That is kind of the brand. That is the enterprise world that Kubernetes inhabits. Why the hell did you name it uwubernetties? That is the formal name of the release for those who are less online than some of us.

Kat Cosgrove: Yeah, I thought it would be funny. So, three years ago, I made a joke on Twitter.

I just simply tweeted the word uwubernetties. And it went, like, minor corner of tech Twitter viral, but I threatened at that time, I was not even a member of the release team back then, that if I was ever allowed to lead a release, I was going to name it Umu Brunetti's. And I didn't believe anyone would actually ever let me lead a release, because I generally am a little bit of a loose cannon, right?

I didn't, at the time, have a reputation for somebody who gets shit done. At the time, it was an empty threat, but then after years of hard work, I did get the role and I thought, Man, it sure would be funny to pander to furries online and actually name it UwU Burnetties. Uh, so I did and it pissed off a lot of people on Reddit, which was very funny and very gratifying for me.

Corey Quinn: The children of Reddit are very fickle and temperamental, and I've learned not to cater to their whims.

Kat Cosgrove: Correct. They were deeply upset that, uh, an enterprise solution like Kubernetes would be named something so ridiculous that the project would have somebody so unprofessional lead a release But the reality is that most Kubernetes releases have goofy names.

The serious ones are few and far between. There has been more than one named after the release lead's cats. There was one named after Olive Garden breadsticks. Goofy ones is what we do. The one a couple releases ago was a sloth wearing sunglasses. I

Corey Quinn: like that.

Kat Cosgrove: It's a little bit of flavor that's fun for the release lead as a small reward for three and a half months, four months of hard work that realistically nobody remembers outside of the maintainers and the people who were on that specific release team.

A year from now, nobody's going to remember that it was Uber or Nobody's except for me and my release team. But it brought us joy and the furries on the internet love it.

Corey Quinn: How hard did you have to push to get that run through?

Kat Cosgrove: Not at all. It was my decision and my decision alone. The only people that have to approve the release theme are SIG Release.

So the SIG Release chairs have to give you a thumbs up. So I couldn't go and name it something like actually foul, right?

Corey Quinn: I would have come up with something that was complete non starter, so it looks good by comparison. Like, um, like, so what's, what's the counter proposal name? Uh, Less Shitty OpenStack.

Yeah, that's not going to upset anyone. So, yeah, okay. I guess in Wooper Netties it is.

Kat Cosgrove: You might get away with that one, because that's funny.

Corey Quinn: I'm sure there's a trademark issue somewhere, and people don't generally like, uh, getting cease and desisted.

Kat Cosgrove: I'd probably take a swing at it, though. You know, at least to get the reaction of SIG Release leads out of it.

Clearly they weren't disappointed with my choice or my performance as the release lead because after the release was done they gave me a fancy job and a title.

Corey Quinn: I have to imagine everyone in Seattle at AWS is still reeling because they didn't realize that you could give something a name that wasn't horrible.

They're, they've struggled their entire corporate existence with giving things good names.

Kat Cosgrove: If they want to pay me a few grand and fly me out, I'm happy to teach a class on how to not name things stupid.

Corey Quinn: That's why it's a non starter. The most they'll go is three shiny buttons, and the Amazon travel policy involves stuffing you into a dog crate and throwing you in cargo.

Kat Cosgrove: Oh no, that's brutal. I have standards. I fly for work. It's business class across oceans or nothing.

Corey Quinn: Exactly. That's always been my philosophy, which is why I'm not a culture fit at a lot of these companies. So other than giving it a name, what does a release manager actually do?

Kat Cosgrove: There's two different titles that I think are being conflated slightly.

There is a release manager and a release lead. And a release manager is responsible for the technical aspects of building a version of Kubernetes and managing the release branch. My release manager was Mark Rossetti, and he was very good at his job. He's excellent. He's one of the technical leads for SIG Windows.

A release lead is the boss of the release, the chief cat herder, functionally. So if you think of it in terms of like a company's organization, a release lead is like the president of a particular org. Right? And beneath me, I had a number of sub team leads, five of them, for comms, docs, enhancements, release notes, and release signal.

They each had a number of shadows who were there to help them with the technical and interpersonal aspects of making those job functions work. It's like having nine direct reports, because I also had shadows myself, and then 30, 35 indirect reports, depending on how many shadows each team has. And it is a lot of cat hurting.

It is being the person ultimately responsible for a release working or not, things happening on deadlines, things like code freeze or enhancements freeze or docs freeze actually happening on time. Um, anytime something needs to get escalated to a degree where it has to get a leader of another SIG involved or a rule has to be bent or broken, that was my problem to resolve.

Um, also a little bit of HR, so anytime there was an interpersonal issue between shadows or leads or, uh, any member of the release team and somebody else in the project, that was my problem to handle. Which is not fun. It is a full time management job for three and a half, four months.

Corey Quinn: My skin is already crawling.

I don't do well in those types of scenarios. You used the term a few times now that I feel like there has, there's a special meaning. I'm not grokking. Specifically, shadow. What does that mean?

Kat Cosgrove: Yeah, so shadows are people who are ultimately being trained to lead a sub team. I will use the Docs sub team for example.

Because I've led that sub team, but the docs sub team will have a sub team lead, and then four or five shadows who are helping with all of the aspects of running the docs team, so making sure that a particular enhancement, if it needs docs, it has them, helping to review the technical documentation, making sure that documentation is merged on time.

And the release sub team lead, the docs lead, will do some of that work. Most of it is done by the shadows, And those shadows are being trained to replace the Docs lead in the following release. We don't have people exist perpetually in a role across releases because it is a burnout factory. So you would never, like, lead the Docs sub team two releases in a row.

You lead it, at the same time you're training your shadows to do your job, and you nominate one of them to succeed you after that release. So It is a training program. It is not like a mentorship thing for students or early in career people. It's not an internship. It is, watch this person do this job so that four months from now you can do this job for them.

Corey Quinn: It's odd to see a community like this that focuses on succession planning and training. Usually most open source communities are, good luck, here catch. And everything always feels just a hair's breadth away from disaster, which people are like, well, it's just Kubernetes. You can't actually run an enterprise thing like this.

Yeah, cool. I have very unfortunate news for you from everything, starting with the Linux kernel and moving on up the stack for things you are critically dependent upon for your business to function. This is a, this is a beacon of an exception.

Kat Cosgrove: Um, we also have very excellent documentation because we have hard requirements about documentation for enhancements.

If an enhancement. makes a user facing change at all, it must, must, must be documented before the release is merged. If you don't have documentation for your enhancement before the release is done, we pull your enhancement. We're very serious about that. We're the largest Golang open source project in the world.

We don't feel like we have the luxury of doing less than this. It is a lot of work. It does require a lot of people to make it work. It requires a lot of companies to either allow people to do some of this on company time, or outright hire them exclusively to maintain Kubernetes. But if we weren't doing all of this, Kubernetes would be a lot worse.

Corey Quinn: We

Kat Cosgrove: covered

Corey Quinn: this already. OpenStack.

Kat Cosgrove: Yeah, it would be OpenStack. And nobody wants that. Clearly. Nobody wanted that, right?

Corey Quinn: Yeah, the vision was great. The execution was faltering, and that's really, I think, what doomed it on some level. Forgive the ignorance of this question here, but what sort of things change from release to release?

I first got my hands dirty with Kubernetes earlier this year. And I worry the stain will never wash off, but it was, it was an experience going through it. I have a, I have an ops background working on Linux systems, so believe me, I understand the value of keeping things patched and keeping current because there's nothing worse than having to basically sequentially upgrade through seven different releases of a thing because nobody bothered to look at this thing for five years.

But there's a But I understand it intuitively on a patching of computer systems approach. I'm curious as to the Kubernetes release, where it starts, where it stops, what sorts of enhancements wind up rolling out.

Kat Cosgrove: It varies from release to release, but we have a series of hard deadlines throughout the release that cut off when we can add stuff to a particular version.

So one of the first major deadlines within the release is enhancements freeze. And that is the point at which you can no longer choose to add your enhancement, your code, your feature, whatever, to that particular Kubernetes release. Your code doesn't have to be in by then. It's merely like you and your SIG lead saying in the PR, I would like to add this to the milestone for Kubernetes 130, right?

And once that's done, you're considered in for the release, and you don't have to do anything until the next deadline. But that could be anything from net new features to graduations to beta or stable bug fixes and patches and cleanup stuff like that that happens during the release. Sometimes get rolled into the release.

Sometimes they get released as a patch for the previous version. It just depends on what the patch and cherry pick schedule is right now. And we do publicize the patch and cherry pick schedule. It's very firm and we tend not to wiggle from it much. But, Could be anything. It's just that you're going to know very early on in the release cycle what it could possibly be included in the release.

Anybody can go look at this. It's all public. We work out of a GitHub Projects board. So right now, it's the beginning of the 1. 3. 1 release cycle, and enhancements are starting to be voted in to the release. So you can go look at the project board and see what has currently been added. What you see there is not necessarily all going to make it, because there is also a code freeze deadline later on, which is, your code must be here, and after that there is a docs freeze deadline, which is, your docs must be here.

The docs freeze deadline is new, previously that was not a hard deadline. That's something I changed with my release, because, Technically, it is a requirement for the cap to be considered complete. So I started threatening to pull enhancements if we didn't have docs by a certain date. So at those two points, an enhancement can get cut.

And we usually lose like around 50 ish percent of the enhancements that opt in at code freeze. So we had something like 98 at the beginning of my release, and we ended with 45 enhancements included.

Corey Quinn: Obviously, I think of this in the context of AWS primarily, when I think about cloud. They did something at the start of this year that has gotten mixed reviews.

Generally speaking, when they release a version of Kubernetes on EKS, their managed Kubernetes service, You get 12 months of support on that thing, at which point it's expected you will upgrade to the newer version. Should you choose not to do that, you get an additional 12 months of support, but the cost per cluster goes from 0.

10 an hour to 0. 60 per hour. And people were decrying this as a cash grab. I I am conflicted on this because they are the least sympathetic people in the world to wind up making the argument. But I appreciate the fact that given the negative externalities running unpatched software as causes on the rest of the internet, Yeah, there should be a way for you to have financial incentive to upgrade this, Because otherwise it's just technical debt that, eh, you'll get around to one of these days.

When, alright, your per cluster, uh, control axed, Maybe you'll pay more attention. Where do you stand on it?

Kat Cosgrove: My thing on that is, what constitutes support for that extra year? Is it literally just allowing you to run this old ass version of Kubernetes on EKS? Or are they providing security features for it?

Because if they are, we're not doing that. The Kubernetes project has like a hard and fast policy for end of life for a particular version. We don't wiggle on that at all. Not even for AWS, no matter how much they cry, we're not going to extend support for a version that we've already end of lifed. So if they're providing security updates for end of lifed versions of Kubernetes, presumably they're paying engineers to write That code, again, we're not doing it.

So that justifies the cost increase for me, because they have to be paying somebody to do that work because the Kubernetes project isn't doing it for them for free.

Corey Quinn: Presumably. So I also, I misspoke. Someone is going to yell at me for this because if there's one thing AWS PR loves, it's correcting minutiae.

Uh, when a Kubernetes version gets released on EKS, You're gonna get 14 months of support, not 12. Someone will wrap me with the PR stick when they emerge from their monastery where they've taken otherwise a vow of silence.

Kat Cosgrove: Yeah, so that's tied with our support timeline, because we support it for a year and two months.

Corey Quinn: Yeah, so it's unclear at the moment, looking through their blog post, exactly what that means as far as the, as what they're doing as far as patching goes. I mean, if they're running this and letting you run it, then on some level, they're going to have to wind up handling the security portion of it. That's their side of the shared responsibility model.

How they're doing that is unclear to me.

Kat Cosgrove: Yeah, I think if they're paying engineers to maintain that, then like, sure, take the extra money. And also, like, if you are willing to spend that much extra money to run on an old ass version of Kubernetes, get it together. First of all. Second of all, like, sure. I mean, there's like more fun ways to light money on fire, but if you could buy a boat or something, if you just love wasting money, but

Corey Quinn: it even has a nautical theme going for it.

Kat Cosgrove: Yeah. You know, sure. Sure. You know, but I guess if what gets you your thrills is outdated versions of critical infrastructure, I guess that's, that's cool, man. Couldn't be me though.

Corey Quinn: Tired of big black boxes when it comes to cloud security? I mean, I used to work at a big black rock and decided I was tired of having a traditional job, so now I do this instead.

But with Prowler, you're not just using a tool, you're joining a movement. A movement that stands for open, flexible, and transparent cloud security across AWS, Azure, GCP, and Kubernetes. Prowler is your go to everything from compliance frameworks like CIS and NIST to real time incident response and hardening.

It's security that scales with your needs. So if you're tired of opaque, complicated security solutions, it's time to try Prowler. No gatekeepers, just open security. Dive deeper at prowler. com. I gave a talk earlier this year about my exploration of Kubernetes. So I built a Kubernetes of my very own in the spare room, running on a bunch of Raspberries Pi.

It's a K3S, which presumably has some, I think they just self describe as a distribution of Kubernetes, but they do track upstream and they mostly punt on documentation to upstream, which great, awesome. So it's running a number of things that I find useful to have around, but are not themselves critical infrastructure.

Um, similar to the fact that I only go through and update the firmware on my UPS infrequently at best. Because, great, is it providing power to the computer? Terrific. Do I really care about enhancements? Not unless there's a critical bug. If I just treat this thing like an appliance and leave it running like this for years on I think 1.

29 at the moment, what's going to, what, what'll break in the fullness of time? Theoretically, nothing.

Kat Cosgrove: Theoretically, nothing. Yeah. Like if you just don't touch it, if you just leave it there, theoretically depends on where you're pulling your images from. Um, like that, that could break if the place you're pulling your images from changes or goes down, but that's going to be a problem no matter what.

So yeah, theoretically, Nothing, but there's also like a huge difference between your personal shit and your work shit.

Corey Quinn: Oh, absolutely. I, I don't treat this like it's a production environment. It's running, remember when I said spare room and raspberries pie and treat it like an appliance? Yeah, maybe don't do that for the thing that makes the money.

Kat Cosgrove: Right. Like at home, my electronics, I, I don't keep them up to date. I ran on Windows XP until Windows 7. 1 came out. You know, I truly. I'm not great about it with my personal devices, but for if your business and then your livelihood is relying on the thing, you should keep the thing up to date. You should not be running on a two year old version of Kubernetes.

Corey Quinn: I've seen this problem with languages before, where I wind up becoming the first non developer technical hire at a startup. And they're running an ancient version of Rails in some cases, and okay, great, how do we upgrade this to something reasonable? And the answer is, that's the neat part, you don't.

Python 2 is a terrific example of this as well, where you get to go and do a whole bunch of refactor work, and it's painful and annoying.

Kat Cosgrove: Yeah, there's no upgrade, there's no direct translation, you're just

Corey Quinn: Right. So you, you definitely want to keep current. It is easier. I'm at least my belief is that it's a lot easier to use the migration tools that go from Python 2 to Python 3 with earlier versions of Python 3, because at this point, when you're coming out with Python 3.

13 now, which is what they're working on, uh, You aren't really envisioning that the primary use case being people migrating off of Python 2. That people, they, I think they sort of think that people who have done that or are serious about it have already done so.

Kat Cosgrove: Yeah, especially since Python 2 has been end of life for like, what, six years at this point, right?

Python 2 is like dead, dead. Like it's been dead, their extended end of life was like six or eight years ago, I think. But you still see it, you still see it, we can't get away from it. But if you're scared of upgrading your cluster because you think that things are going to break.

Corey Quinn: There is a section called Breaking Changes, which is good because if you don't admit that there are breaking changes, you're lying to me because everything can be a breaking change in the wrong way.

Kat Cosgrove: There actually are no need to knows for the current version. And it's the first time that's happened in a very long time. But there genuinely are no breaking changes in this version that we knew of at release. There was a regression that was found afterwards that has been patched now. But of the actual enhancements included, none of them required any kind of like warning before upgrading a cluster.

Which is, again, unusual. Most releases do have, like, something that you need to need to need to be aware of, but in this case, there wasn't anything.

Corey Quinn: I am fairly conservative around certain things. File systems are one, databases are another, and production infrastructure, like Kubernetes, is another, in that You have released something new and shiny today.

Yeah, fine. I might go ahead and roll that out in my spare room, but I'm probably going to have some challenges with doing that or something like that in a production environment that does banking. So there's a, it feels like there is a sweet spot. You don't necessarily want to roll it out day of release, but you also don't want to be doing it just before it hits end of life either.

Where's the sweet spot?

Kat Cosgrove: Wait for the first patch. Yeah. Which will be like a month later. You know, so yeah, wait for the first patch. Ideally, if you want to be helpful to the release team and helpful to your, yourself in the future, um, you run the release candidates or the betas,

Corey Quinn: uh, and help us. In a pre-production environment.

Let's be clear. In a pre-prod.

Kat Cosgrove: Yes. Don't run it in prod. Uh, I mean if you do decide to run it in prod, in prod, like call me and let's talk about it. 'cause that's really funny and also wildly irresponsible.

Corey Quinn: Exactly. So what, what bank do you work for? I need to move some money out of it immediately. .

Kat Cosgrove: Yeah, please don't do that.

But in, in pre-prod or in something Prod shaped. Uh, run the release candidates. Run the betas.

Corey Quinn: The reason I have not gone down that path, especially when building my talk, is when you're just learning something new. This stuff is hard enough without trying to diagnose, okay, am I the moron here, or is this doing something unexpected?

Because usually when you don't know at all what you're doing, and you're following the tutorial, People have tread this path before. It is unlikely that it is, that you have discovered this amazing edge case bug. No, you almost certainly forgot a semicolon or something equally foolish.

Kat Cosgrove: Yeah, it's definitely not a stunt for newbies.

Like, if you're new to Kubernetes, please do not try learning running the release candidate. They are buggy as hell. But if you want to try. There's usually like a known issues going on in the GitHub branch for the current release and uh, you go look at the release signal or the release notes channels in the Kubernetes Slack.

You can see an ongoing discussion during the release about any issues we find and as they are solved. Which is, you know, fun and interesting to watch if you're new to how Kubernetes releases. I recognize that we are weird about it for an open source project. Can be a little bit alien for people.

Corey Quinn: Absolutely. So you mentioned that people don't do, they aren't the lead on the releases or other subteams to releases in a row. Do they cycle back through? In other words, is there a chance that we'll get another, uh, another release with a good name in a year or two?

Kat Cosgrove: I won't lead another release. Um, usually after you are the release lead, like the big kahuna, you don't do it again.

Um, and in fact, most people don't have anything to do with the release team at all. The cycle after their release, uh, SIG release likes you to take a cycle off. Because then they set you to

Corey Quinn: see on an ice floe and that's the end of it.

Kat Cosgrove: Yeah. You go to a farm in upstate New York, you know, that's usually how it goes.

You come back as the emeritus advisor usually, uh, two or three releases after you lead a cycle. Me being involved in the release immediately after my cycle is unusual and is a result of the fact that unfortunately I was very, very good at my job. And so they gave me more responsibilities than a fancy title.

So I have to continue to be involved. One of our previous release leads, uh, Xander, he is back on the release team now as a Docs shadow. You do show back up. It's also common for sub team leads to shadow on other sub teams, lead other sub teams. Uh, I led comms and docs myself before I was the release lead.

So we, we bounce around, but usually leading the whole release is your, your exit. You dip after that.

Corey Quinn: I would be somewhat surprised if having gone through the process of being the release lead, you had not made lasting changes to the release process as you went through it. Because let's face it, Kubernetes is not that mature.

It is a decade old that you have not ironed out every bug in this. And if you have, please share with the rest of the class. But so. What have you changed as you went through this that will, I guess, serve as your monument and a legacy?

Kat Cosgrove: I made two big changes. Uh, one was the instituting of a dox freeze. Uh, like I said earlier, the dox deadlines used to be very, very soft.

People would kind of ignore them and it was very stressful for both the dox sub team and, uh, SIG dox. It was a week and a half of tearing our hair out, Screaming at people, chasing down SIG leads to yell at their KEP owners to write the goddamn documentation. It's literally just a fucking feature flag.

It's two lines, please, for the love of God. And we hated that. And technically, in order for a KEP to be considered complete, documentation has to be there. So I instituted a DOCS freeze. It is a hard deadline, just like the enhancements deadlines. And if you miss that deadline, you have to file an exception request that I have to approve.

Or, I will pull your enhancement from the release. And people took it very seriously this time, and we did not have an issue with docs. So that was really nice. It improved things for the release team. It improved things for SIG docs. So, that I think will be remembered.

Corey Quinn: I can't stress how important that is though, just because when the docs are bad, especially as a newcomer, And docs are often bad.

It's a, you think that it's a problem on your fault, not, not on your part, not the fact that no, no, this just was basically never spoken to by someone about by someone who has not written it. So they assume everyone they're talking to also has that level of context. And this is really confusing and hard.

The docs were surprisingly good.

Kat Cosgrove: They are. I honestly attribute a lot of Kubernetes success to our good documentation.

Corey Quinn: It'd be impossible for us to see where it is now without that. It's, that, it's a critical path. But I'm sorry, you were saying there was a second change you made as well.

Kat Cosgrove: There was a second change that was more impactful, I think, and will be more impactful long term, but is unpleasant that it was required.

We instituted a way to remove inactive shadows and sub team leads from a release team mid cycle. Uh, historically, the release team has a problem with shadows, uh, applying for the release team. Getting the role, we get hundreds of applicants. It is genuinely very competitive. Uh, they apply for the role, they get it, they get the org membership, they get their name in the markdown file, and then they disappear.

Which creates a huge amount of extra work for the shadows that are remaining and the sub team lead who has to pick up all of that extra slack and they already have a bunch of other stuff to do. And that, that sucks. We've had it happen with sub team leads as well, twice in my tenure, at least. And that's, that's even harder to deal with because then another sub team lead or a release lead shadow has to pick up and it's, it's not good.

So, we finally gave up and documented a way to remove a shadow or a sub team lead against their will. I don't think that people necessarily took it seriously when, uh, Grace, my co owner of the sub project and I wrote that documentation. But then I used it in my release and I fired three shadows. I don't regret that at all.

I need people to understand that while I would love for the Kubernetes project's release team to serve primarily as a mentorship opportunity for people who are early in their career or for students, that is not what we're here to do. We are here to release the second largest open source project in the world.

We can't have that. people on the team who aren't doing the work. So, um, we can remove people mid cycle now. We have a way to replace them. We have Criteria for like, does this actually constitute inactivity? We make it very clear to everybody on the team what that criteria is and now they understand that we actually will use it.

I would hope that we have to use that less and less.

Corey Quinn: I have been removed from a number of open source projects myself, in many cases years after my contributions come to an end, just because it feels weird to send an email saying, yeah, take me off of this. Uh, but I also don't want to be there anymore. So for some folks, I have to imagine it's almost a sense of relief.

But for a short term thing like this, it feels like burnout hasn't had time to set in yet. And, and again, people have, people have changed in their lives. I get it. I'm not saying that, oh, if you're removed from being a shadow, you're clearly a terrible person who shouldn't have volunteered. And people also bite off more than they can chew and they don't know how to message that sometimes.

I get that.

Kat Cosgrove: We have a very open policy of like, if you have too much going on, like if, if stuff happens in your life or stuff happens at school or stuff happens at work.

Corey Quinn: Please drop, let this be the thing you drop because we, we can't have you not focused on it.

Kat Cosgrove: Yeah, just tell me, tell me, tell your sub team lead, tell your release lead, tell whoever, tell somebody in a position of power that you would like to step back.

for reasons. You don't have to tell us what the reasons are. Just say you would like to step back and we'll say, okay, no problem. And we will make that happen. We will replace you and we will remove you with no negative marks, right? Like we're not going to hold it against you. If you disappear without telling us and we have to remove you against your will, that is a different story.

You're not going to be welcomed back for the next release. And we're going to want to see you like succeed elsewhere in the project before we consider welcoming you back onto the team. But if you choose to step back and you're communicative about it, that's fine. We've had somebody who willingly stepped back from a sub team lead position, come back a couple releases later, and lead the whole release.

So, like, we really don't care if you communicate that that's what's going on. And if there is something going on in your real life, you should always, always choose to dump us first because we don't pay your bills. We're an open source project. I'm not writing you checks.

Corey Quinn: If not, there's another question out there somewhere for someone.

So, your, your task is done. You have done the release. Do you have to do anything else on the patch releases, the security updates to this, or is that handled separately?

Kat Cosgrove: Those are handled completely separately. Those are handled by the other half of SIGrelease, the release management subproject. So that is entirely on them.

I don't have to deal with it. All that's left for me at this point is stuff like this. Come talk about the release, handing out stickers at conferences. People think the release logo is very cute, so. Handing out stickers, talking to the press, answering the same questions over and over again, usually, although nobody else has asked me about the name.

How do you not? I don't know.

Corey Quinn: That's the first thing I saw is UbuBernetes. Terrific, great. What's wrong with you? And I bet it is a long name and it's expensive to fix. Tell me about it.

Kat Cosgrove: Yeah. It's cute and it's funny and I thought people would ask, but they don't. They just ignore it or they go rage post on Reddit, which is also very funny.

So, um, actually comparably little hate on Hacker News, which was disappointing because I do love upsetting them too.

Corey Quinn: The Horrible Orange website is, uh, Usually a reliable source of that, so I am disappointed. What's next for you in the context of Kubernetes? You're obviously doing a little bit of a stop and breathe again approach, but what are you going to dive back into?

Kat Cosgrove: I am now the release team subproject owner. That is a new role that didn't exist before, so I am now the release team's dad, pretty much.

Corey Quinn: You're effectively the release lead emerita, which means that presumably your word carries some weight to it.

Kat Cosgrove: It does. I have actual authority now, whereas when I was the release lead, I had authority that was on loan from the SIG release chairs, right?

But now I have real authority, and part of that was because there are some changes to the way we want to handle things like the shadow program, especially with respect to removing inactive people. And previously that was handled by the emeritus advisor, which is a temporary role that only exists for the release.

It doesn't exist outside of a release. My emeritus advisor felt like she didn't have the authority to actually fire someone because the emeritus role didn't feel like it conferred real authority. It's not a titled job. You're not part of SIG release leadership. Those will be my responsibility moving forward.

Unfortunately, also things like selecting shadows. are my responsibility for any given release, any changes that we make to the structure of the release team. I moved some responsibilities over from one sub team to another, and I'm looking at next release potentially eliminating an entire sub team and moving its responsibilities into two other teams.

Uh, so those like overarching changes are my problem now and then day to day operation of the release when the release lead runs into trouble. So my successor, if he has a problem, he comes and asks me now instead of having to go crying to the say, release leadership.

Corey Quinn: I look forward to seeing how that winds up playing out for you.

If people want to learn more, where should they find you?

Kat Cosgrove: Ooh, uh, you can find me on the Byrd app, uh, Twitter. at Dixie3flatline. Technically, I do have a BlueSky account as well, which I barely use, but it is there and it does send push notifications to my phone. And that is cat. lol. I have an Thank you.

Thank you. I have an email address. You can try emailing me at cat. cosgrove at gmail. com. I extremely don't recommend it.

Corey Quinn: That's where requests go to die is my email inbox.

Kat Cosgrove: Yep, I will look at your email, go, I will respond to that later. And then I will get 30 more emails and forget it was there. And you will never hear from me.

So if you do email me, Godspeed, I'll pray for you. Uh, probably not going to hear back from me. So Twitter, better chance. Um, LinkedIn also exists, but, uh, good luck there too.

Corey Quinn: We will absolutely. Links to all of that in the show notes. Thank you so much for taking the time to speak with me. I appreciate it.

Hey, thanks for having me, Corey. It's a pleasure to welcome you back, and don't worry, I'm sure we'll talk again in the somewhat near future.

Kat Cosgrove: Oh ho ho!

Corey Quinn: Kat Cosgrove, Lead Open Source Advocate at Dell, and so much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you enjoyed this podcast, please leave a five star review on your podcast platform of choice.

Whereas if you hated this podcast, please, Leave a five star review on your podcast platform of choice, along with an angry, insulting comment, probably on Reddit. And make sure you get that comment in before the documentation team's shitty comment cutoff date.