Cup o' Go

Visit cupogo.dev for show notes, Patreon link, Swag, and more!
★ Support this podcast on Patreon ★

Creators and Guests

Host
Jonathan Hall
Freelance Gopher, Continuous Delivery consultant, and host of the Boldly Go YouTube channel.
Host
Shay Nehmad
Engineering Enablement Architect @ Orca
Editor
Filippo Valvassori Bolgè
Sound Designer / Audio Editor based in Milan

What is Cup o' Go?

Stay up to date with the Go community in about 15 minutes per week

Jonathan Hall:

Show is supported by you, our listener. Stick around till after the news to hear some more about that. This is Cup o' Go for 06/2626. Keep up to date with the important happenings in the Go community in about sixteen minutes per week. This is Jonathan Hall.

Shay Nehmad:

And I'm Shay Nehmad.

Jonathan Hall:

Hi, Shay. It's good to Six see you

Shay Nehmad:

six. Number of the beast.

Jonathan Hall:

It has been, what, at least six days since I saw you.

Shay Nehmad:

Yes. The the release schedule has been kinda weird, but here's what happened. We should do a short start with a short recap of what happened in Canada. Right? Yeah.

Shay Nehmad:

Was very unique. So a few listeners of the show who work for a I will say what looks like a great company called Gena tek, not sponsored Yeah. Had a conference, like an internal company conference called Elevate Dev. Jonathan's actually wearing that their swag shirt right now. And they emailed me.

Shay Nehmad:

I think they emailed you as well.

Jonathan Hall:

I don't know.

Shay Nehmad:

Yeah. And we're like, hey, we're looking for speakers. And I guess the lesson is check your email because we went to speak there. We flew out to Montreal. We hung out at the hotel, we met a ton of interesting people, Brian Finster, Steve Yeage.

Jonathan Hall:

Bill Kennedy.

Shay Nehmad:

Bill Kennedy, of course. Just a ton of, very interesting people, and of course, from, Genentech themselves. They had a really cool conference. Do you wanna, like, try to explain how the conference, like, was like? It was very unique vibe.

Jonathan Hall:

Yeah. Oh, wow. It's hard to explain that. It was a great con like, so it was a large multitrack conference. Of course, they had the keynotes, the opening and closing keynotes that were, you know, on the central stage, and then they had a bunch of breakout rooms for workshops and and presentations, which I gave two breakout presentations.

Jonathan Hall:

You gave one on the main stage, but it was during the multi track section, right?

Shay Nehmad:

Yeah. It just had a of people signing up because it had an interesting title, I think. Yeah. Our talks, of course, were focused on Go. Not all talks were focused on Go.

Shay Nehmad:

Not all talks were technical. Some of them were like management, leadership, whatever. But I think it was mostly a dev conference. So a lot of the talks were were focused on development, a lot of them with really interesting speakers, like really talented people, people who wrote books about programming, people who with lots of experience, very high quality conference for sure. Definitely.

Shay Nehmad:

And it was in Montreal.

Jonathan Hall:

Yeah, it was. One of my highlights from the conference was a few weeks ago, spoke about on the show, I talked about Bill Kennedy will be presenting a workshop in Berlin at GopherCon Europe about agentic workflows in Go, and he gave the same a condensed version of same workshop there, and I got to sit in on that. So that a lot of fun. I'll put a

Shay Nehmad:

How was the workshop?

Jonathan Hall:

It was good. Unfortunately, I had a headache that afternoon. I didn't get to sit through the whole thing, But the part I did, it was quite enjoyable. Bill Bill's a great presenter. So it was

Shay Nehmad:

And such a chill guy. We hung out on a Uber after, like, I don't remember where we were going, but we were going somewhere. And we were both like, man, Bill is such a chill guy. We I we think of him as like a star in the Go world. Oh my god.

Shay Nehmad:

He's Bill Kennedy. He's the Go guy. But then you end up just hanging out with him and he's just like such a

Jonathan Hall:

cool dude. If you have a chance to see Bill in Berlin, go up and say hi. Or I think he'll be in Seattle for GopherCon in August as well.

Shay Nehmad:

Sounds a bit like a a mob threat. If you see Bill, tell him we send our hello. My highlight from the conference was definitely the the party, Genentech party they had. Mhmm. Because I got to hang out with the Genentech people and walk around there and it was a great event.

Shay Nehmad:

Was a lot of fun, but also a lot of people like talked to me and were like, oh, this is what I think about your session. I have some questions. I have some follow ups. They have a very unique company culture. They're all like very science y and and Mhmm.

Shay Nehmad:

Growth oriented. So it was a ton of fun. As a speaker, you're like uniquely positioned to sort of satisfy what they wanna do. So that was a lot of fun. And you said the main stage, I just wanna people to have the image in their head.

Shay Nehmad:

Like what happened on that main stage was it's a three it's like a full three sixty arena situation, you're standing in the middle, like production value is off the charts. They have a green room, they have someone who hooks you up to a microphone, They have video. It was really, really, really high quality. I I felt like super professional.

Jonathan Hall:

And you know, it it felt like I was at a conference center. It didn't feel like I was I was at an office building. And part of that's just the layout of their office building, but also, you know, just the way they they decorated it. And I had one of my Uber drivers ask me, what is this place? I'm like, well, I'm at a conference.

Jonathan Hall:

I I think this is their office, but I like was I was questioning myself as I said that because I felt like I was going into a lease conference center.

Shay Nehmad:

Yeah. So it was great. Again, shout out Genentech. Thanks a lot for inviting us. We had a ton of fun.

Shay Nehmad:

Jonathan and I hung out physically for the second time in our life. Yep. And we even got to travel a little bit. We walked around Montreal. We ate some food.

Shay Nehmad:

Makes

Jonathan Hall:

some popcorn.

Shay Nehmad:

Oh, lots of popcorn. That was good popcorn. Chicago style popcorn. And we forgot to buy jewelry for our significant others.

Jonathan Hall:

Right.

Shay Nehmad:

It did everything. So that was a lot of fun. That was also why the regular programming schedule is a bit was a bit wonky, but now we should be back to normal. Let's talk about normal Go stuff. Alright.

Shay Nehmad:

Let's do it. Kick us off.

Jonathan Hall:

First up, Go 1.27 r c one has been released. Yay. This means that there are no new features going into Go one twenty seven. It's been frozen. I guess it's been frozen for a while, actually.

Jonathan Hall:

But this is the first version that might theoretically be released to the public if they don't find issues. They always find issues. They always polish things a little bit.

Shay Nehmad:

You can help them polish it by moving your production load test, unit test, CI, whatever to run on this new version. And then any bugs that pop up, just create a new issue on the issue tracker. And also a good reminder, we talked about a lot when we it was like one of the main stories we covered when we started the podcast a few years ago, the GoTelemetry, opt in, opt out, blah blah blah. It landed on being opt in. So please consider opting in to GoTelemetry because it will help validate its release candidate.

Shay Nehmad:

They'll be able to know how many people are running it, etcetera, etcetera. Very simple. Just run GoTelemetry on. Honestly, zero downside. It's like very anonymized and whatnot.

Shay Nehmad:

I know it's hard to believe with with Google in the background, but I think the implementation is okay. And it's very easy to to download Go, just a Go install command. Cool. Following up on that, there's a new release for Delve, 01/20/1970, which makes sense. Released two days ago.

Shay Nehmad:

We won't go into all the issues because they are, like, in this release. There are, like, 30 or whatever. But the the I think the main thing this release has is it lines up stuff with one twenty seven, you can debug one twenty seven code. So, you know, it has a lot of changes that are like, oh, now you can step into the new RangeOver func. So you can like f 11 on the or whatever your shortcut is to do Range Rover funk stepping in the debugger.

Shay Nehmad:

And there's a reason why I'm talking about this now, Jonathan. You had a very one of my, talks was about Go tooling at the conference. And I was like, yeah, debugging is great and it's very easy. You should do a lot of debugging. And then you stepped up after and you were like, no, you're wrong.

Jonathan Hall:

You

Shay Nehmad:

That's care what to explain yourself?

Jonathan Hall:

That's not quite what I said. I said I very rarely use the debugger because I focus on writing lots of small independent tests. And I I feel like the only time I ever need to reach for the debugger is when I don't have good tests, which means it's on code that either I wrote three years ago or somebody else wrote.

Shay Nehmad:

Hot damn.

Jonathan Hall:

Or that's not that's not entirely true. I I do sometimes write code without good tests, but it's not it's unintentional, of course.

Shay Nehmad:

So I I do think I tend to write fewer tests than you do. I I think our coding styles are a bit different in that regard. But I also just like, you know, even if I have the test and the test is passing, sometimes it's if it's a complicated piece of code, just stepping through it. Think it's That just

Jonathan Hall:

would be valuable.

Shay Nehmad:

Yeah. It goes to show how you learned how to program. Because when I learned how to program, I started on a pen, like, pen and paper in school, and they had you write out, like, every single debug step on a big table. Don't know if you they had you do that and, like, when when if you ever done that?

Jonathan Hall:

No. I started programming on a Commodore 64 by myself with books from the public library. Did take some programming courses in university, but none of them were were like that. No.

Shay Nehmad:

So I had to, like, write out the every step and all the values, which is basically just being a human debugger. Maybe that's just, like, how I wired. Anyway, one twenty seven. The debugger's out, the new tooling all the tooling will line up pretty soon, and, it's a good time to start trying it out and giving feedback to the team. Alright.

Shay Nehmad:

You have a proposal you wanted to mention. Apparently, you hate dumb quotes or something

Jonathan Hall:

like that.

Shay Nehmad:

That's not very inclusive.

Jonathan Hall:

Yeah. Don't don't don't quote me on

Shay Nehmad:

that. Nice. That's a better segue. I should have thought

Jonathan Hall:

Alright. So the proposal is to don't rewrite into smart quotes. So this is a feature of GoFumped. I didn't know existed. Wait.

Jonathan Hall:

GoFumped or GoFumped? Well, both, actually.

Shay Nehmad:

Okay.

Jonathan Hall:

So the original GoFumped, Go FMT, apparently rewrites. If you do two backticks, it turns it into a smart quote, which might be kind of nice in some situations. But at least for Rob Pike's chosen font, the two the the double backtick and smart quote render identically. He says pixel identical. So he couldn't see that that his code had been changed under his nose, and he doesn't like that.

Jonathan Hall:

If if for no other reason than when you start to grep for your double back picks, it doesn't find anything because those bites have been changed.

Shay Nehmad:

Why would it change that, though? Like, why would it why would it do that?

Jonathan Hall:

That's the question. Why would it do that? Somebody thought it was a good idea once upon a time. The proposal is to not do that anymore. So this is in new status.

Jonathan Hall:

It hasn't been act hasn't been decided yet. So if you strongly want your magical smart quotes to stay, you need to go and comment on this issue and tell them not to accept it. If you agree with Shay's reaction of why would you do that in the first place, then maybe go upvote it.

Shay Nehmad:

I think it's I'm looking at the design. I, like, clicked on a few buttons here. It's like reformatting, doc comments specifically for, like, in in documentation, so the printer will look nice, for docstrings. Don't touch it. Don't touch.

Shay Nehmad:

Just don't touch it. It's better. It's fine. Don't rewrite these things. Think I am very much in support of this.

Shay Nehmad:

It's also very funny, I have to admit.

Jonathan Hall:

It is a little

Shay Nehmad:

bit funny. Like, I don't wanna be, that that guy who's like, it's it's a funny bug to to walk into, I should say. What would is this like a breaking change in your opinion? Does this require, like, backward compatibility, blah blah blah?

Jonathan Hall:

I don't think so since as far as I know, it's only in comments. Like, I don't think it would it would change that in like a quoted string literal or something like that. So it should only be a comment change. I don't think compatibility rules apply here.

Shay Nehmad:

Yeah. But GoFump I mean, GoFump only changes white space and stuff like that anyway. So any change can go into GoFump. Doesn't matter how you know, now we are using, spaces instead of tabs because you have to, like, rewrite all your Go code now. Theoretically, now you have to go run GoFump on all your Go code.

Jonathan Hall:

So I think in theory, they could change from tabs to spaces in GoFumped without breaking the backward compatibility promise. They wouldn't do that because that would be silly. But in theory, I I don't I I think that's orthogonal to the backward compatibility guarantee.

Shay Nehmad:

Got it. This is funny. Like, it doesn't look the same, at least in my browser. Yeah. My browser looks very different too.

Shay Nehmad:

I wonder what font he's using. I wanted to step away from proposals for a second to talk about a different type of a proposal, an acquisition proposal, in a very, I think, good news. Someone who was on the show, Matt Boyle, you may remember the episode we interviewed him on ONA HQ and all the work he's doing there. Filippo, you can now do the your thing with the

Jonathan Hall:

la la la la.

Speaker 4:

But, yeah, I'm I'm at I'm currently the head of engineering at a company called Ona where we are building an AI platform using Go. I also am very, very active to the best of my ability in Go community. I run a website called bitesizego.com, where I try and teach people, post beginner lessons on Go and spend a ton of time running a Go community on Twitter as well.

Shay Nehmad:

Ana is, you know, obviously they use Go and whatever, but it's like a platform for background AI agents. So you can run, like, your codecs in the cloud. And this is not really Go news, but we're just really psyched for Boyle, for Matt, sorry, because OpenAI are acquiring them. Have reached an agreement to acquire OnaHQ. Exactly.

Shay Nehmad:

I think specifically so co Codex can, like, run stuff in the cloud.

Jonathan Hall:

Got it.

Shay Nehmad:

Cool. Good good work, man. This is really, really cool. Congrats. I'm just I don't know what to what to add.

Shay Nehmad:

Like, if you're interested in running background agents, check out Ona, but and if you're interested in how to sell your company to OpenAI for presumably a lot a lot of money, just come on Cup o' Go and then wait for nine months.

Jonathan Hall:

That that was the direct line to success for them. No doubt.

Shay Nehmad:

Yes. Alright. We have two other interesting backlog items, but I don't think we have time. We have to move on. I will move them to next week.

Jonathan Hall:

Why don't we have time, Shay?

Shay Nehmad:

Because we're up on the twenty minutes. We have to get up to the ad break because we have a really cool interview to for you to listen to. Oh, that's right. A really, really interesting interview. You wanna give us a short preview where we're gonna whom we're gonna talk with?

Shay Nehmad:

Yeah. We had

Jonathan Hall:

a really interesting chat a few weeks ago with Francis Nickels who works at Riot Games and uses Go heavily there. So we're gonna be talking about Go at Riot. The interview was recorded a few weeks ago. We finally got it edited and ready to publish, so don't miss that interview.

Shay Nehmad:

Yeah. We're excited to let let you all hear it But a quick ad break. As Jonathan mentioned at the top of the show, the show is supported by you. The best way to support us is via Patreon. Just join and kick in a few bucks a month, and you can find the Patreon link, including many other links to our Slack channel, to our email, to our swag store, which now includes the new shirts that have finally arrived at cup o' go dot dev.

Shay Nehmad:

Other ways to support the show include spreading the word about it. We don't pay to advertise. All our growth is based on people telling other people about the show. And apparently, us going on a stage in front of, like, 300 people and talking about Go as well with a with a big Cup o' Go sticker. So, leave a review or write about it or whatever.

Shay Nehmad:

That also helps out a lot. Other than that, we do wanna mention we're thinking about a lot of changes to the show's format, And next week's episode will probably not be a news episode. It'll probably be just like a sort of stop and take stock episode. If you have any ideas or thoughts you'd like us, you know, just with this general prompts ahead of time, feel free to DM us on Slack or email us, but we'll talk about it more next week.

Jonathan Hall:

I'd add a little bit there. Like, if you have anything you would like to see added to the show or anything you don't like about the show, that would be really useful feedback. Yeah. Yep. So next week, a meta a meta show in the in the metaverse.

Shay Nehmad:

Yes. It might be recorded with video. We'll see. Oh, wouldn't that be fun? But it's not live.

Shay Nehmad:

We're done with that. I don't wanna see your your your face in the in meat space. Alright. Let's do a quick lightning round, then get to this interview already. Let's do it.

Shay Nehmad:

Lightning round. First

Jonathan Hall:

up, Go Please has a new experimental feature for those of you who who might care about AI and LLMs and whatnot. They have a new MCP protocol built into Go Please that you can run. It's experimental. I guess that means it might be broken or it might change. Probably mostly means it might change, but you can check that out.

Jonathan Hall:

It's a lightning round, so I'm not talking about it anymore. I'm done.

Shay Nehmad:

My lightning round item is a apparently, there's a Golang Brazil subreddit. Guess it goes really big there. We had the Go releaser, person. He was from Brazil. Right?

Shay Nehmad:

Oh, yeah. If I remember correctly.

Jonathan Hall:

I think so.

Shay Nehmad:

There is a project called Glaze, a seagull free desktop web view toolkit for Go, sort of released two days ago. It's a desktop WebView binding for Go, which is pure Go, so no C. It should work on macOS and Linux and Windows, and nothing native is bundled, which is pretty cool. And if you want to render some HTML in a in an app using Go, this is one way to do that. I wouldn't know why you would do that, but now you can.

Shay Nehmad:

I mean, I I do know. I'm obviously, I'm being facetious. But it's just an interesting project and something I would never, like, work on. So very cool. It's it's there, you know, similar to, like, whales and whatever.

Shay Nehmad:

If you're if you're having pain trying to build, a a little web like, desktop app, this is really good.

Jonathan Hall:

Very cool. Alright. Well, I'll see you at the interview. Do you hear that noise outside my window, Shay? Can you hear me at all?

Jonathan Hall:

There there's huge riot right outside of my front lawn. Oh, no. I wish I knew how to get rid of these, how to take care of riots. Hey, Francis. You might know something about riot.

Francis J.Nickels:

Yeah. I I know a little bit about riot. Been there for a few years.

Jonathan Hall:

Welcome to the show. Tell us tell us who you are, and well, I guess we know where you work now, but we could make it formal. Tell us who you are, what you do, and and why you're here.

Francis J.Nickels:

Yeah. My name is Francis Nickels. I'm a staff software engineer at Riot Games. I've been at Riot going on sixteen years. Prior to that, I did a short stint at Blizzard and have worked in a wide variety of other industries over the course of my career.

Francis J.Nickels:

But, yeah, I'm I'm currently a staff software engineer working on, mostly infrastructure and the operation side of things at Riot.

Jonathan Hall:

So is working at Riot fun? Is it a Riot?

Francis J.Nickels:

Well, I've never worked in one place as long as this.

Shay Nehmad:

So I

Francis J.Nickels:

I will say there there are definitely a lot of fun things about, working at Riot. And, yeah, as part of my job, I'm expected to play games. So how bad could that be?

Shay Nehmad:

That sounds nice. Just for people who may not, know, Riot are behind titles that probably you have heard of, even if you're not a gamer, like League of Legends, Valorant, Teamfight Tactics. I think League of Legends is the most one of the most well known video games that people have some opinions on and on the people who play. It's to the point where they're like stereotypes or whatever, but I think it's super cool just to put it out there. It's like a very cool game.

Shay Nehmad:

It's lot of fun. It requires some dedication that I don't have, but it's a cool product.

Francis J.Nickels:

Yeah. In in the early days, it had a a significant learning curve to get into it, but somehow millions of people around the globe have decided they really enjoy playing it and hence, caused me plenty of headaches over the years as how to solve problems at scale and, you know, keep the run game up and running twenty four seven globally.

Shay Nehmad:

So basically, every player who lost and rage quitted because of lag, they can blame you. Oh. That's what you're saying.

Francis J.Nickels:

Don't get me started on lag. Like, I I actually did a player facing tool, ten years ago called the lag report that we actually released to players. We recently mothballed it because it had sort of fallen out of disrepair. But I am very passionate about last mile connectivity and how that impacts or can improve the player's experience playing the game.

Shay Nehmad:

Interesting. How did you come to the interview? That's interesting. You're like a long time listener, first time caller. Right?

Francis J.Nickels:

Yeah. Well, I I think Jonathan made a call out a couple months back saying, yeah, I'd love to talk to somebody who uses Go at a game company, and I guess I'm you're second now. I'm sure there's plenty of gophers in a lot of different game companies. Matter of fact, I I know of some in other companies, so we exist. That's cool.

Jonathan Hall:

Yeah. Thanks for for responding to that. Let's just start with how do you use Go, or or maybe more broadly, how is Go used at Riot? Because I I imagine everybody has sort of concept. It's probably the boring stuff, but, you know, you're not you're not rendering three d graphics in Go, I'm imagining.

Jonathan Hall:

But let's tell tell us what you are doing in Go.

Francis J.Nickels:

Yeah. Like, the things we're not doing is, you know, our game clients that get deployed to players' machines and devices are not written in Go. They're they're usually written in c or c sharp or c plus plus. But, yeah, it's the back end stuff, which I don't find boring. I find it quite

Jonathan Hall:

I don't either. But, yeah, when think of games, it's not the sexy stuff that people generally tend to think of.

Francis J.Nickels:

So Our our security teams, use a lot of Go tooling. Matter of fact, we have one that's open sourced, called Key Conjurer. And so a lot of services I personally use it in building a lot of command line tools that other engineers within my sphere actively use. Mhmm.

Jonathan Hall:

Cool. So you mentioned I don't remember if it was before we started recording or or sensing because we we talked for a while. You're you're doing sort of operational type stuff. What what what kinda what are your responsibilities right now? Like, does your team have a name that that's meaningful for us?

Francis J.Nickels:

Oh no, not at all. The team does not have a meaningful name that would be worth sharing.

Jonathan Hall:

All right.

Francis J.Nickels:

Yeah, to give you a sense of like what I've been doing, like over the last five years prior to the current team I'm on, I worked on a team that was laser focused on building, maintaining, and supporting our Kubernetes clusters that the majority of our games all ride on top of. Being Kubernetes is a lot of use of Go around that, deploying services.

Shay Nehmad:

On one hand, this makes a lot of sense because of just the scale of Riot games. Like, these are games that have sometimes even like, League has millions of concurrent players. Right? Or We do. Yeah.

Shay Nehmad:

Tens of thousands. I know, like

Francis J.Nickels:

No. It's millions.

Shay Nehmad:

The number is millions. Right? I'm not I'm not kidding.

Francis J.Nickels:

There was a time I had the that, like, I would watch, you know, the CCU every day because whenever there was a dip, there was an issue, I needed to

Shay Nehmad:

jump So in and respond to it makes sense that the games are running on Kubernetes because it's also like, oh, a game is like a game of League is like thirty minutes or something. It's a server or whatever. But on the other hand, I'm thinking like, oh, pods, they die all the time, like, for, like, evictions or whatever. Doesn't that mean the game will crash? Isn't that, like, super frustrating?

Shay Nehmad:

I'd rather have, like, a dedicated server. When I'm imagining gaming servers, I'm imagining, like, you know, pizza, huge you know what I mean? Like, big servers, server racks.

Francis J.Nickels:

Yeah.

Shay Nehmad:

The game gets, an IP. How do you how do you run something that's, like, as customer facing? Because, you know, a background data job or whatever or a multi like an HTTP server, it makes sense to me to put on Kubernetes. But what's like the architecture that makes it work on Kubernetes without disrupting player games all the time?

Francis J.Nickels:

Well, we strive for very high reliability. You know, the men as many nines as we can get. And it's true. If your pod that's hosting your game props out, your game is disrupted. That fortunately does not happen very often.

Francis J.Nickels:

It's actually kind of a rare occurrence. And so, in the early days, yeah, we absolutely had the pizza boxes. I was in the server rooms, stacking and racking on, hooking up the cables. And for a long time, that's how we ran our game servers and all of our other services ran on more ephemeral stuff. But these days, we've gotten things running really nicely such that we can run game servers on pods that once we decide to drain that pod, we let the games finish out.

Francis J.Nickels:

And when the game's done and there's no more games running on that pod, we can kill that pod and start up a new pod to take fresh games. We use HBAs for handling auto scaling. So there are peaks and troughs in a day where there's lots of players playing and then there's periods of time when there aren't. And so we take advantage of auto scaling groups and HBAs to like spin up and spin down resources. Lots of fun stuff.

Shay Nehmad:

Super cool. So sorry I cut you off just because I got caught on that Kubernetes thing. But you said five years ago you were working on, like, on a team that's in charge of Kubernetes?

Francis J.Nickels:

Yeah. For actually, as recently as two years ago, but for a number of years, that's a team that I was on. More recently, I'm on a team that's dealing with all of our ingress traffic that is not game server traffic. So the majority of any TCP traffic that's coming into any of our services goes through various, you know, filters and WAFs and stuff like that. We're using caching a lot.

Jonathan Hall:

So is this stuff like websites, chat

Francis J.Nickels:

chat? Services, any yeah. All all that good stuff.

Jonathan Hall:

Good.

Francis J.Nickels:

And having been there sixteen years, some of our tech has gotten kind of old and just needed a refresh. So like one project I worked on recently was sort of bringing modernizing some of our ingress practices and tooling.

Shay Nehmad:

And the way you're both Kubernetes and, like, web servers and whatever, I think usually of Go it's like, oh, the underlying tool uses Go. Like, oh, I use Kubernetes, and that uses Go, so knowing Go gives me an advantage. Or, you know, I'm writing a a web I'm using a web firewall. It's written in Go. If I have to go into the implementation, I can read it.

Shay Nehmad:

Do y'all have like do you roll your own or do you add like plugins or whatever? Do you like, I'm trying to I'm trying to pinpoint, you know, what sort of Go development is going on just because such a big company and so much Slack and so much scale. It's you know, I I've never experienced something like that, so it's very interesting.

Francis J.Nickels:

Yeah. For my work, I am I am not actively writing a Go service that is massaging traffic or anything. There are other teams that might be good number of our service teams are written in Go, but we also use a variety of other languages. Python, I think there's Rust here and there. There's definitely C, like, no Go is the only language you're using.

Francis J.Nickels:

We're using a variety of languages. But for my specific work, I might write a tool that helps automate some process or a migration tool for migrating some systems from one way of doing things to another, and rather doing that manually because we have so many endpoints and so many services, we have to automate things. And so the two tools of normal trade is either Python or Go. So there's a fair amount of use of both tools, on the back end side of things for helping automate things.

Jonathan Hall:

So you you were talking a little bit ago about these these pods that manage your League of Legends games. I don't I don't imagine those are written in Go. I imagine they predate Go.

Francis J.Nickels:

I actually haven't worked closely with those teams in long time, so I don't even know what they're written in.

Jonathan Hall:

Alright.

Francis J.Nickels:

So I can't even read that.

Jonathan Hall:

That's fair. I mean, but just look like, looking at, know, League of Legends came out in 2009. Go was kind of barely a blip at that point. So I would unless it's been rewritten, I I doubt it's written in Go. But it sounds like you use Go mostly for for some orchestration, and you say some services might be written in Go.

Jonathan Hall:

Do do do have an example of a service that you know is written in Go or that that you worked on? Or are you working on mostly orchestration?

Francis J.Nickels:

Answer that question. Like, that is probably not a safe question for me.

Shay Nehmad:

He can tell you, but then he can this is just a League of Legends joke, but he can leave you in jungle to die. So Yeah. I actually wanted to sort of pick your brain a little bit. I know a lot of developers, me included, got into software development because of our love of video games. For me, it's a 100% what happened.

Shay Nehmad:

Like, I loved video games, and then I I loved the original Roller Coaster Tycoon, then someone told me, oh, it's all written in assembler. So I went really deep on learning assembler, and that was particularly useful for, you know, my sort of my first jobs, this is sort of how I stumbled into it. To me, there is like game development and there's working in a games company. You know what I mean? So clearly, if you love video games, working in a games company is better than working in whatever else company.

Shay Nehmad:

But I've also heard a lot of bad stuff about the, like, gaming industry for software developers. There's a lot of crunch before releases or anything like that. I'm wondering, does that extend to people who, like yourself, who, you know, developed like the backbone of this gaming industry, but you're not like a game developer yourself, like you're not sitting down and designing levels or or, you know, heroes for the game or whatever. Are these rumors about like the gaming industry for developers true that it's like very crunchy, very stressful, what it doesn't like, it's not actually fun like playing games, making games?

Francis J.Nickels:

Yeah. So that is a really interesting question.

Shay Nehmad:

By the way, just for before you answer that, for the context, you have experience in more than one gaming company. Yeah. I've like, you've worked at Blizzard, which is one of my, you know, childhood, like the core memories, the amount of hours I spent like in StarCraft and Warcraft and all that stuff. Man, if I just spend them on software development, I might've been good by now. Like it's really.

Francis J.Nickels:

Yeah. So why don't I just start by saying I actually swore off video games in college. Now, me, that was forty years ago and video games are not what they are were not what they are today back then, but they were still time sinks. My grades were suffering. I was spending way too many hours into the wee hours of the morning playing games instead of doing my schoolwork, and my grades were suffering.

Francis J.Nickels:

So I swore off video games in college and then ultimately graduated. I would not consider myself like a hardcore avid gamer. I sort of fell into it. Back in the .com days, I got involved in a startup doing interactive television and we were syncing game shows on TV to web experiences. So that was my first reintroduction to the vibe.

Francis J.Nickels:

Fast forward to getting into Blizzard. Prior to Blizzard, I was at a multinational holding company that had brands around the globe and I was running IT operations for that. And then two thousand and eight recession happened and knew some guys working at Blizzard who I worked with at a bank ten years earlier, and they brought me in to deal with a project that wasn't going well at Blizzard. And I came in and got that thing going. And then I as I was finishing up that project at Blizzard, I was looking for my next gig.

Francis J.Nickels:

Riot came along and offered me a position. At the same time, Blizzard turns around and offers me a full time position and I had to choose between the two. But to your question about crunch time at games, I think prior to probably Blizzard and definitely prior to Riot, most games were produced similar to how movies are produced. You pull together a bunch of people, you work really hard on this game, and then you publish the game on a CD or DVD, and you let everyone go because like the game is just it's done. Right?

Francis J.Nickels:

And then if they're gonna do a follow-up, that's a new crunch that you work on that. One of the things that the founders at Riot really keyed into, when they started Riot was this concept of games as a service. And, like, when I when I got to Riot, like, I thought I knew what agile was. I found the company being run more agile than I had ever been exposed to before. But really, the concept was we're we're making tweaks and adjustments to the game every two weeks.

Francis J.Nickels:

We'd have a release every two weeks. We'd introduce new champions into the game. And, yeah, the game you're playing today on League of Legends or Valorant, you know, is gonna be slightly different from what you're playing a month or two months from now or what you're playing six months ago. And so it keeps the content fresh, keeps it engaging. While they're like little crunches, because what's going on?

Francis J.Nickels:

It's just like how we do software development, right? With releases, What's in the release, what's not. Go is a great example of our release cadence. We're pushing this content at a regular interval and either the feature's in or it's not. Same could be said to how we operate most of our games in that way.

Francis J.Nickels:

Now granted, there are key things that we change in the game that have a little bit more fanfare, and there there's definitely target delivery dates for that.

Shay Nehmad:

Yeah. Like a new mode or something like that.

Francis J.Nickels:

But and and so there is a little bit of crunch, but I would say my experience has been, you know, we don't we don't crunch to the point where, like, you're totally exhausted and, you know, you need, you know, to disappear for three months just to recover from it. We we try to keep it a little bit more sane.

Shay Nehmad:

So it's interesting. Like, in the past, games were, like you said, like a big production. It had a release date, so people were, like, really working hard to get to that release date, and then it's out.

Francis J.Nickels:

And they let you go. Right?

Shay Nehmad:

Yeah. And then either it works or it's not, you know, it's a big success or it's not. And now it's more like normal software development where the game, you put it out, maybe, I don't know, I'm playing a game right now that's in early access, so maybe you even put it out while it's in development to get some feedback and you, like, improve it. As long as you have the core ready, like, improve it over time. And games as a service, like Valorant, whatever, League of Legends, they have even, like, seasons, that concept.

Francis J.Nickels:

Right.

Shay Nehmad:

And generally, on the flip side so the the most worrying thing I hear about working in a games company was actually the crunch. Is the flip side true also? Like, is it awesome to work in a gaming company because you play video games all day or you get like don't

Francis J.Nickels:

know. Wish I no. I I don't play video games all day, but, you know, there there is an expectation that you play so that you can appreciate the player's experience so that the technical decisions that we're making, like, you know, whether or not to add a caching layer onto something, like, what does that mean to the player's experience? Right? Like, being able to identify, how the changes you're making, technically might impact the player experience is something that we try to bring to the forefront.

Francis J.Nickels:

But there but to your point, like, there could be teams within Riot where, like, they have a hard deliverable and there might be some crunch on it, but I don't think it's, like, that crunch followed by the crunch followed by the crunch. Yes. There are companies that probably crunch harder. And so, you know, it may still be true in a number of gaming companies in the industry that, you know, the crunch aspect is real.

Jonathan Hall:

Are are all of Wright's games sort of online games, or do they I mean, I guess all games these days are at least to some extent, but, like, primarily, are they multiplayer? Joy.

Francis J.Nickels:

We just released the trading card game.

Jonathan Hall:

Oh.

Francis J.Nickels:

So we have our own trading card game. We've done

Shay Nehmad:

Wait. Like, card, like a physical card?

Jonathan Hall:

Yeah. That's funny. Okay. Oh, damn. Is that written in Go?

Francis J.Nickels:

Nope. It it's written it's written on the printing press.

Jonathan Hall:

Written in Postscript.

Francis J.Nickels:

Yeah. But we also have we have there's a board game we produced a couple years ago. We we experiment on a lot of stuff, and we've also done we've done an animated series on Netflix called Arcane.

Shay Nehmad:

Which is really good. It's like, I don't I don't need to promote it because I think it's like number one on Netflix Israel or something, but it's really good. If you Jonathan, if you haven't watched it, it's like a good watch. Not a good watch with kids. It it gets, like, it gets kinda dark, but it's I watched it with my wife, and it was like Alright.

Jonathan Hall:

That's the question. Can I convince my wife to watch it? So, anyway, as for video games, though, are they all online games with a multiplayer component, or are they are they more like the other traditional FPS type of thing where, you know, it's a single person playing a mission?

Francis J.Nickels:

Most are multiplayer online games. I think we've released some smaller games that are, like, PvE, player versus environment. I'm trying to remember, like, you know, the the the ones that I worry about day to day are all the online multiplayer ones. Right. Where, you know, we're dealing with, issues of scale and high reliability.

Jonathan Hall:

And and so if if the model is online multiplayer and you're releasing fresh content frequently, it doesn't sound like there's really the opportune I mean, unless it's perpetual crunch, you're not gonna have that opportunity like you do. Like, oh, we need to we need to hit the the the July 4 release deadline for this this whole new thing. You know? It's a different it's a different business model, and so probably a different, you know, just a different approach, different sets of of of constraints and and pressures.

Francis J.Nickels:

Yeah.

Jonathan Hall:

Yeah. What do you like most about working at Riot or in games, either or or both?

Francis J.Nickels:

What do I like most? Like, one of my favorite activities at Riot is our internal hackathons, which actually starts up next week. We've been doing this for probably fifteen years. I've only missed one and that was like the second one, like, you know, years ago. And like, literally we get The number of hours every time seems to change, but the last time was forty eight hours.

Francis J.Nickels:

I don't remember what is this hour, but it's two or three days you get to form your own team and work on whatever fascinates you. At the end of it, we do this big science fair thing where you walk around the room and you see what all the other teams have done. Some of the things that people work on actually make it to be real product that gets delivered to players. But the goal isn't to produce new product. The goal is to basically scratch whatever itch you want And maybe get to work with people that you don't normally work with on a day to day basis.

Francis J.Nickels:

So like some of the teams that form are with complete strangers, in the company our size, like, I don't know everybody. There was a point in time in the early days where I knew everyone at the company. Nowadays, when I do a hackathon, I might join a team that I don't even know half the people and you spend two or three days working on something intensely. You get to meet some new people and hopefully learn some new stuff too. So that that's one of things I enjoy most of the company.

Shay Nehmad:

Yeah. That's cool. That's interesting. So you mentioned scale. I assume I'm saying I assume because I think I'm actually wrong.

Shay Nehmad:

But, you know, when you're talking about scale in in in your industry, you mean suddenly, you know, now it's 6PM in America, suddenly whoosh, like oh, I don't know. Maybe 3PM, all the kids come back from school, and they boot up League of Legends all at the same time and jump on a game. And then as as the time zone you know, as 3PM knocks in throughout America, suddenly you have a lot more gamers trying to use your game at the same time. But even though League of Legends is a totally global thing, I assume you have peak times and, like, you know, some peak times during the day. I don't know if they're America centric or not, but I assume there's like peak times where there are most players and then there are very low hours when most of the world is like asleep.

Shay Nehmad:

Or is it very uniform just because it's a global thing?

Francis J.Nickels:

Like, one of the early jokes around Riot was like, hey, Rito, buy another server. Like, all the players don't play it on a single instance. Right? Right? We we actually break things up into regions.

Francis J.Nickels:

So, you know, if in your North America, we have a collection of North American services.

Shay Nehmad:

Oh, of course.

Francis J.Nickels:

You have

Shay Nehmad:

to have the service geographically close to the players as well.

Francis J.Nickels:

Exactly. Because we worry about lag. So we physically try to put our servers as close to players as possible. And then with Valorant, we even go beyond that. We stand up multiple points of presence throughout our region to provide players the best experience they can have.

Francis J.Nickels:

We've also stood up our own private backbone. So we have this service that we call Riot Direct, and we try to peer with as many last mile providers as possible. And we try to onboard your ingress traffic and egress traffic as close to the player as possible. And then we take ownership of the transit of the player's packets from that point of presence back to wherever the server's located. Seeing a scale like that in itself is probably I I I haven't looked at recent ranking, but it's probably a top 30 or top 50 network global network compared to large ISPs.

Francis J.Nickels:

The nice thing about players is they don't all log in at the same time. Having worked in television broadcast where the broadcast starts exactly at a specific point in time, you have a lot of spiky traffic. Ours gradually come on and then gradually decay. So we don't generally deal with spike traffic, but we definitely get DDoS fairly regularly. And so when DDoSes show up, those show up as spikes and we have to, like, absorb those and make decisions about is this legitimate traffic or not and try to to bat the the bad traffic, the vicious traffic away and protect the player's traffic.

Shay Nehmad:

Talking about bad traffic, now only now I'm thinking about it. Like bots and cheating and all that is a real thing in, you know, if I'm developing a Go backend service in some b to b SaaS company, I have to do like a penetration test once a year just to have my certifications. But honestly, a 100% of my requests are legitimate requests most of the time. For a gaming company, there's like incentives for the players themselves. Well, ideally the players themselves are logging on to have fun and whatever, but they're, like, botting to to level up their characters or they're cheating to get, like, an unfair advantage in games.

Shay Nehmad:

Is that something that the firewall is aware of, or is that the application layer thing, or is it, like, defense in-depth where multiple layers are trying to deal with it?

Francis J.Nickels:

Definitely defense in-depth. We have a dedicated team called the anti cheat. You can probably find some posts online about some of their things that they've had to do to, like, detect cheers and thwart cheers from playing. We try to make sure The game is only as good as the experience you have, right? And so if cheating is rampant and, you know, you're playing against someone who's cheating, you know, you're not having a fun time.

Francis J.Nickels:

You're not having an enjoyable experience because it's like like, how can I win? This person's just trouncing all over me and there's nothing I can do, right? That's the effect of cheating. So we take anti cheat very seriously. We take there are concepts like gold farming, where there are these places where people will try to automate and generate currency in the game to then sell to players who basically don't want to put in the time and experience on the game.

Francis J.Nickels:

There are services that do all sorts of things like leveling up accounts. Like we try to identify those and shut those down and making it really difficult for them to have a viable business in our ecosystem. And then

Shay Nehmad:

there's Because, again, because they ruin like the experience for legitimate players who are logging I'm ashamed to say you said it, and suddenly I remember like one of my first jobs when I was like 11 leveling up my friend's RuneScape characters, just logging on, but I didn't automate it in any way. I just it was literally, I logged into their accounts and clicked on trees and whatever. So I'm ashamed to say I was part of that industry once, but I've seen since improved my ways.

Francis J.Nickels:

Yeah. So it can be big business. When you have an ecosystem where you have millions of players, you know, there's opportunity for someone to figure out how to capitalize on that. And to that end, like in the early days, we used to have problems where bots were scraping our services both for illegitimate and legitimate purposes, and crashing our servers. So we actually stood up our own dev portal where you can get an API credentials into our dataset, and as long as you're using it for non malicious purposes, we'll give you data and metrics about all sorts of things in the game that aren't even fully exposed in the game clients.

Francis J.Nickels:

So there's a strong third party ecosystem wrapped around our services where you can explore, you know, building your own third party service that may provide information about your game history, you know, or give you suggestions on, you know, which characters you should be playing or which characters, you're weaker on and you need more practice on. There's a whole ecosystem tied to that, and we support that with our dev portal and API. And so that's something you could write in Go, right? You wanna make a League of Legends or Valvant related tool, you could access our API using Go and make your own little third party service. Cool.

Francis J.Nickels:

Unique use of our data.

Shay Nehmad:

And just so, you know, asking for a friend, what are latest anti cheat mechanisms? No. I'm just kidding.

Francis J.Nickels:

Well, used sit right next to that team, and so I would see and hear a whole bunch of stuff that they were doing.

Shay Nehmad:

Honestly, that sounds like such a fun team to work in because you're it's just like cat and mouse against people who are very motivated.

Francis J.Nickels:

But the same is true in DDoS. Right? Like, DDoS isn't necessarily cheating. It's people trying to make the service unavailable. And game companies aren't the only targets.

Francis J.Nickels:

Everyone gets DDoS. But the bigger you become, the more of a target you are. And I used to spend my Christmas break sitting in our NOC because that's when all the script kiddies would be off school and they would start like, hey, let's take League of Legends down. And we didn't have the tooling back then and you could, you know, with not too sophisticated methods, you know, cause real impact. So like my first two or three Christmases at Riot, I literally was in our knock probably, you know, more hours than I cared and not at home.

Francis J.Nickels:

That doesn't happen anymore. That's good. We've got things walking

Jonathan Hall:

these But

Francis J.Nickels:

it's still cat and mouse.

Jonathan Hall:

Yeah. I would like to ask how you use AI because I bet you have some insight because everybody's using AI these days in some form or fashion. What does it look like for you on a daily basis?

Francis J.Nickels:

Well, for me, I caught the bug probably two years ago. Like, you know, prior to that, I was pretty much an AI pessimist. It's like, okay, it's not going to be that good. It's a waste of your time. Don't do it.

Francis J.Nickels:

And I forget, I I I think I was at AWS reinvent thing where like every session I went to, they somehow dropped in the word AI, right? But I went to, I think a code whisper session where they actually showed me prompting and it's like, okay, that looks interesting. I'm gonna try it out. When I got back to Riot, we had a trial with Co Pilot and I started trying it out and I'm like, hey, this isn't too bad. Like, you know, I can do a two or three sentence prompt and generate a bunch of code that actually looks halfway reasonable and it helped reduce the toil.

Francis J.Nickels:

So for probably the last two years, I've been using AI and prompting and trying to figure out how to provide better context to AI so it generates better code for me. One of the tricks I discovered maybe a year ago is I had a situation where I had a function that took a couple strings as parameters, and I had prompted the AI to do some stuff and it went off and it generated some code. I didn't notice that it swapped the order of the parameters, and so it was passing in one value as a string into what should have been the second parameter. This bug got into my code and I spent maybe a day or two trying to figure out what happened to my code. Everything was going great with AI.

Francis J.Nickels:

Now it's all junk. And I looked, I found the bug and it's like, oh, the AI swapped the parameters. Like, why would it do that? And then the light bulb went on my head. It's like, you know what?

Francis J.Nickels:

I can use types in Go and I can call this parameter, the x y z thing and this other one, the a b c thing. And now the AI knows that the two parameters are what they are and it'll never swap them, right? Because it knows the typing. So once I start figuring that out, it's like, Oh, I just customized my types all over the place. Every function had its own type.

Francis J.Nickels:

I stopped using the native types and I found that AI got a lot better. Now more recently, I've been exploring spec driven development and design driven development, Ralph loops. It feels like the 1990s all over again. I am having so much fun just figuring out like how to leverage AI to generate more value faster. And it is, I'm excited to be working in technology right now.

Francis J.Nickels:

That there wasn't before, but I'm really excited to be working in it right now. And Go is my language of choice. So like, it's all, everything I'm writing is in Go.

Shay Nehmad:

The the highlight of what are you applying AI to, like which languages and which tools you've had before and how you use them, I think is something that a lot of people are realizing that, oh, well, I had this super custom, you know, dynamic thing that had a lot of context you had to hold in your head and a lot of these things that are not very clear, and I'm trying to put AI to work in that framework, it's having a really hard time, versus Go, which is very explicit. All the code looks the same, you know? Even it has it has one, hidden benefit. I'm sure it's less important now, but, I I was at a lecture at a Go for Con Israel around two years ago when these things, like, started getting into circulation. Like you said, you mentioned I caught the bug.

Shay Nehmad:

I love that approach where Natalie from Go Time, which is not used to be a podcast about Go, we all love, and now it's gone. So sad. She had a lecture about exactly that. She was, why is Go perfect for AI? Back then, it was the Go formatting uses tabs and not spaces, so it uses fewer tokens when you try to predict something.

Shay Nehmad:

I was like, oh my god. That's such a weird little edge case of it. But I think over time, people really agree with you. I don't know. Michi Laifenfeld on Twitter just recently said, oh, I'm using Go with AI, and it's a really good language for AI because of the type safety, because of the simplicity.

Shay Nehmad:

It doesn't have a lot of keywords, because all the code looks the same in the training set. And I I feel like other people are not having as much fun as we are, you know, if they're in a languages that are more open or more dynamic, because just AI is having a harder time there. I am wondering, you mentioned like expected development and whatever. Your services are like critical, right? If you push a bug that causes some like Riot to not have games for a day, that has a really big impact on the bottom line of the company.

Shay Nehmad:

Are you reviewing all the code that AI writes or are you at like YOLO mode? Code doesn't matter anymore. It's all about the spec.

Francis J.Nickels:

Yeah. No. So we had some really firm policies on that. No YOLOs allowed, because like I mentioned in the beginning, we care about reliability and uptime. I'm on the hook for anything I push to prod, right?

Francis J.Nickels:

Like this past year when I've been working on this project, I used to joke that if there's a global outage in the next six to twelve months, you know who caused it because I was touching essentially all the services that all the ingress traffic into all the services. So if I screwed something up and pushed something to prod that had a bug, I could very easily cause a global outage. So getting back to your thing about spectrum development, like what I'm what I'm exploring next week in our hackathon is I want to see if I can define I'm going to build a project and I'm going to go strictly off of the requirements and try to do a tight definition of those requirements. I'm then going to ask the AI to build the tool that I define in my requirements using Go, Rust, and maybe even Zig. Right?

Francis J.Nickels:

Like, I know Hashimoto's definitely into Zig. At least I think he is.

Shay Nehmad:

Yeah. He's a big Zig guy. I think it's their number one financial contributor.

Francis J.Nickels:

And I've I've only like, I've only dabbled a little bit in Rust, but I haven't touched Zig. But I know that from what I've seen AI do with Go, that it's really good at writing good code. I wanna be able to look at the same service stood up in three different languages based off the same specs and figure out like, how good is this AI? Can I build three different executables that are all built off the same specs that are functionally the same and what the differences are? That's the itch I wanna scratch, is test to see which language does better with There's also the question of which model to use.

Francis J.Nickels:

We have access to all these different models. It's like which model produces the better code? So that's the exciting part of things right now. It's like everything's in flux, it's changing regularly, and if you're not playing with it, you are not learning and you are falling further and further behind. The software engineer of tomorrow is going to have very different skillset than the software engineer of probably the last twenty or thirty years.

Francis J.Nickels:

But you still need to know all the foundations.

Jonathan Hall:

Yeah. That that that's pretty close to the way I see it. Like, it's it's not that what we've been doing becomes obsolete. It becomes the foundation we're building on. And I think it in a lot of ways, this is actually making good software engineering harder, not easier, in that, like, there's more training required to be able to do it properly.

Francis J.Nickels:

Yeah. The more interesting thing from an operation side, like, you know, so you generate all this code, Now it's in production. So when you get the, you know, the call at 03:00 in the morning that, you know, some service is not working, you know, today, a junior engineer who doesn't have a lot of tribal knowledge of the ecosystem that the service is running within, you know, may get pulled into that call and they are like, I don't know where to look and they have enough knowledge about it. It's like, okay, can AI solve that problem? The problem today or historically has been most of this knowledge is locked up in tribal knowledge in people's heads.

Francis J.Nickels:

This is where I think spec driven development really becomes invaluable, is the better documentation you have, the better the opportunity for using tooling to identify the source of the problem. As a senior engineer, when I get pulled into a triage call, I have years of experience working system, working on prior incidents. I know intuitively where to go, but how does AI get the knowledge and experience out of my head to actually be able to automate some of that incident response? That's another area that I'm quite fascinated with. It's like, how can we effectively and efficiently triage issues?

Francis J.Nickels:

And what sort of scaffolding needs to be in place for AI to be able to do a good job? You want to hand the AI the whole code source based and go figure out this issue. It's just going to chunk for too long. But you need some things where it's like, Hey, I've seen these five things happen. These metrics go off the rails and those are indicative of X, Y, and Z.

Francis J.Nickels:

Ideally the requirements, when you do software development, it's all about the exceptions. The normal use case is the easy part of coding. It's handling the exceptions and knowing where memory leak could happen or where some, you know, recursion that runs wild.

Shay Nehmad:

My my CTO calls that sense knowing where their bodies are buried. Looking at a PR and being like, you know what? I'm looking at this. It touches this table. I know this table is bigger than this developer expected.

Shay Nehmad:

I know this developer doesn't do performance testing normally. I'm going to block this PR, even though it seems reasonable. Let's just like wait on it. It's like just a sense of where the bodies are buried. Yeah.

Shay Nehmad:

Will say just to play devil's advocate, but also because it's my opinion, I think people like yourself who are very experienced, have acquired over time a sense for where the bodies are buried, have spent their fair share of Christmases in Knox, for example, connected servers, and just are generally experienced enough, are are in a very exciting space right now, which where, like, you can get back to being an individual contributor, you can try out all these things, they're very exciting, whatever. As a new software developer, it's really hard to get the fundamentals right, I feel, because it's sometimes you churn out slop and it kind of works well enough. Nobody figured out that you don't actually know what you're doing. And then in six months, you're going to have like unmaintainable code bases or really bad products. And personally, consumer of software, not just a producer of one, I've seen, I don't know, a lot of bugs recently, a lot of security incidents, a lot of bugs, a lot of things like going sideways, GitHub is down all the time, Duolingo is trying to push new lessons on me, they don't work.

Shay Nehmad:

And conversely, services that seem to be developed by AI pessimists, very disciplined developers that like hate on AI all the time, they somehow seem to be working alright. You know, Leechess has not been down once in the last six months versus Duolingo, which is having problems all the time. Two services that I like religiously use daily. You You know what I mean?

Francis J.Nickels:

The GitHub might just be there's so many more, you know, changes being committed because of AI.

Shay Nehmad:

You know? Yeah. Also, GitHub is its own thing. You know, they don't have CEO. They move to Azure.

Shay Nehmad:

It's like a whole big thing. Right? I'm not saying just to be clear, I'm not saying GitHub is because of AI coding. That would be totally dishonest of me. But I am saying I'm a pessimist about people being able to utilize AI at all experience level.

Shay Nehmad:

I'm looking at someone like yourself and I'm like, oh my God, this person has superpowers now. But it's not for everybody.

Francis J.Nickels:

Yeah. So to that point, like I've been mentoring college students for probably at least a dozen years now. I also recently began teaching part time at a local university. Teach a graduate course on essentially it's their capstone project for their gap graduate degree. And I know that, like, one of the common questions I get from students is like, you know, is this industry something I should even be pursuing?

Francis J.Nickels:

And I like my response to them is like, this is a fascinating time to be in this field, right? Like, you are starting your career with these tools that none of us had before. And early in your career, you learn a lot, right? And you have to try a lot of different things. So I am actually bullish on the opportunity for people to absorb how to use these tools that are still developing and also master the foundational stuff that they need to know.

Francis J.Nickels:

You've got to produce some slop first to realize, okay, this is slop. How how do I not write slop? Like, I I can I know my early programs look like crap? Even programs I wrote ten years ago, I look at it I go, oh my god.

Jonathan Hall:

Anything I wrote more than three weeks ago looks like crap.

Francis J.Nickels:

Exactly. So it is if there's one thing that I tell students is like, be prepared to be a lifelong learner, right? Like, I I am I am tickled pink that I am still learning stuff every day. If my whole corpus of knowledge was what I learned in the classroom forty years ago, I would probably be very bored. But the fact that there's new and exciting things to explore and master, I think is actually very promising.

Francis J.Nickels:

The people who produce Slop, guess what? They're going to have to realize I produce Slop and what do I need to learn and master so that I cannot produce Slop? That's where I find things like spectrum and development. Programmers are notoriously bad about documenting. I've been banging the door at where I work for probably a couple of years now saying, Hey, we need better internal documentation for the services that we have so that our internal customers know how to use our stuff better.

Francis J.Nickels:

I see spec driven development as a path to where we will have better documentation about what a service represents, how it works, how it handles different failure modes. That will only make AI stronger, but it'll also make the engineers who are architecting these systems stronger because they'll have a clear definition of how things work. It's even like when you explore a new programming language. If the documentation on it is good, it's easy to learn. If it's, you have to know all these secrets and do stuff that's not easy to uncover, like that language ends up being harder to master.

Shay Nehmad:

Totally. That's a good such a good, I think, balance of approach. It's not like a doomer, AI is useless or a doomer, there's no point in this profession anymore, but it's also not, oh, AI will do everything and I can just take that as it's just continue learning and this is a tool. That's I I think, a good approach.

Francis J.Nickels:

You had mentioned, you know, you got into game development and assembly code because, like, you thought that's how games were produced. Like, you know, if we were still all writing assembly code, like, think how like, where we'd be. We would not be where we are today. Like, This AI tooling thing is the next layer of abstraction from higher level programming languages. Heard some people say, Hey, maybe programming languages disappear, right?

Francis J.Nickels:

We go from specs straight to an executable and like nobody needs to look at the higher level programming languages. Like maybe Go becomes obsolete at some point. I don't think we're anywhere close to that right now, but I could imagine

Shay Nehmad:

It's an interesting thought experience.

Francis J.Nickels:

Yeah. But, you know, there'll still be higher level language, but you don't look under that hood anymore. The PR process and the test, you know, definition process will be strong enough. It's like, you know, I trust the code that AI built.

Shay Nehmad:

Well, I don't trust the code that I built three weeks ago, so it's going to be hard for me to trust a bunch of matrices. Yeah. But you might be right. You might be right. I think it's still far off to have that discussion in earnest, especially in products that either have, you know, critical uptime requirements such as games, like online games or industries that, I mean, that usually deal with highly regulated data and whatever.

Shay Nehmad:

But it's definitely an interesting thought experiment. Assuming Go doesn't become obsolete very, very soon, and there will be a need to use it like tomorrow ish, we have a stumper question that we usually ask every interviewee, but it changed over the years. This year it's, what is your favorite Go library or like Go trick or recommendation that isn't in the standard library? I'll be really interested to hear what's your answer to that. You actually shared before the interview today, you have a couple of answers you thought about for this one.

Francis J.Nickels:

Yeah. So like my easy answer to that, and it's because I write a fair number of CLIs that I maintain a fair number of CLI tools is the COBRA library, right? Like command line parser that's built in the standard library just leaves me wanting, and I found Cobra to be a cool tool. Tough one to master. Like, it it takes a little bit of a mind meld to sort of like figure out all the different things.

Francis J.Nickels:

There are bugs in it. I've even done I think I had a bug fix I did a couple years on it and I had another one I started on, but I never finished it, but it does the job for me. So I'm really happy with Cobra. And then an honorable mention for me is a tool that I think you guys had mentioned in a previous podcast. It's Excelize.

Francis J.Nickels:

Oh, yeah. It's a library for generating Excel data file formats. I had a tool I was building a couple of years ago and CSV or TSV was not going to be sufficient. I wanted to be able to create a tabular dataset from some stuff I was extracting and be able to provide a spreadsheet where I could pivot on different fields. But I need this data to update like weekly and, you know, doing a CSV and then having some manual process of merging into an Excel file just seemed arduous.

Francis J.Nickels:

So I went looking and lo and behold, this guy made this library that generates native Excel formats like, Hey, I can use this. And it solved my problem. I haven't used it since, but I recently looked at it and I see it's actually still live and looks like it's Yeah.

Shay Nehmad:

They just had a release February.

Francis J.Nickels:

There's 300 contributors on it. I think back when I was looking at it, there was like less than 20 contributors on it. It was just the one guy. I had issues with his library and I posted some questions to him and he actually got right on him and fixed my problems. So I was able to use his tool.

Francis J.Nickels:

So I'm a big fan of open source. I think open source is also a great way for young engineers to get exposed to what software engineering looks like, right? You don't need an internship at a big company to find out how software engineers work, getting exposed to creating a PR and getting feedback and tweaking your PR and finally getting your PR merged and doing that in an open source framework. It provides you experience and also provides you Resume Gravel for, look, I've been a contributor on this open source project for a year or so and as a hiring manager, I can go and look at their interactions with other developers and see how they might work on my team if I were looking to hire them. You don't see that regularly on residues, but for young men trying to break in, it's definitely a great venue for building up experience and having something that they can show.

Shay Nehmad:

That's cool.

Francis J.Nickels:

Yeah. It's better than showing like the hackathon project that you spent two days on. Like, I don't wanna look at your code from a two day hack. I want to I want to see the code that you want.

Shay Nehmad:

Real open source real open source work has a lot more cred because you got to pass a bar of the maintainer. You found a project where the like, you also show the fact that you are able to write software with other humans, which is becoming the harder part versus just generating some code that creates a portfolio website, which looks like 100% of the other students

Francis J.Nickels:

or Right. Or it was generated by AI, right? Like it's like, okay, did you do this Like, the AI did it. Like, I wanna see how you interact and understand these fundamentals.

Shay Nehmad:

Yeah. A 100%. Cool. If people wanna, talk to you or or or find you online or message you after the show, where they can can they find you?

Francis J.Nickels:

LinkedIn's usually the easier way.

Shay Nehmad:

Are you on our, Slack?

Francis J.Nickels:

I am on your Slack as well.

Shay Nehmad:

We we don't have to go through Microsoft software to talk to you, which is always a

Francis J.Nickels:

Okay. Alright. Yeah. If you're averse to Microsoft, like, I I think No.

Shay Nehmad:

I'm just kidding. I'm just kidding. I made so much money because I

Jonathan Hall:

want Google plus back, so I can use that.

Francis J.Nickels:

I I did sign up on that when that came out. But

Jonathan Hall:

Yeah. Was Was this you

Francis J.Nickels:

of the first half million signups on LinkedIn. And I have like one, you know, less than seven digit number for an ID on LinkedIn. I've I've found it invaluable over the course of my career. Yeah. Keeping track.

Francis J.Nickels:

Right. Keeping track of things.

Shay Nehmad:

Well, thank you so much for coming on the show. This has been fascinating. Honestly, it's a shame we don't have like long form interviews, like eight hours. You know what I mean?

Jonathan Hall:

We'll create a side channel for our Patreons to listen to those long conversations.

Shay Nehmad:

Maybe, maybe. But thank you so much for coming on the show, and I know you didn't really say it on the interview, but I really appreciate all the nice things you said about our podcast before we started recording. That's so awesome that we're able to, you know, provide an interesting show for you to listen to. I'm positively impressed about Riot Games right now. I had no opinion of the company, but now I'm like, I think I have a good opinion of the company.

Francis J.Nickels:

All right, Great. It's been my pleasure. And, yeah, I I really enjoy what you guys are doing with your podcast.

Shay Nehmad:

Thank you very much. Thanks so much.

Jonathan Hall:

We'll talk to you next time. Thanks, for listening.