1
00:00:00,090 --> 00:00:03,180
It's the idea of like build versus buy and all of the kinda stuff.

2
00:00:03,240 --> 00:00:05,640
It comes to a point where like, sure.

3
00:00:06,060 --> 00:00:09,390
This system is unstable, but unstable in a way that like you

4
00:00:09,390 --> 00:00:11,790
don't have to invest all of the resources, keeping the uptime,

5
00:00:11,790 --> 00:00:14,010
like all of the operational stuff, like all of the thing.

6
00:00:19,020 --> 00:00:20,670
Welcome to Screaming in the Cloud.

7
00:00:20,880 --> 00:00:25,770
I'm Cory Quinn, and I am joined today by a man of many talents.

8
00:00:26,009 --> 00:00:29,580
Ahmed Babar is a principal engineer at the New York Times.

9
00:00:29,700 --> 00:00:32,040
He's an AWS container hero.

10
00:00:32,070 --> 00:00:33,450
He's a cloud native and.

11
00:00:33,714 --> 00:00:36,265
Ambassador and a prolific public speaker.

12
00:00:36,415 --> 00:00:37,720
Ahmed, welcome to the show.

13
00:00:38,385 --> 00:00:39,765
Thank you Corey, for having me.

14
00:00:39,765 --> 00:00:42,045
I'm excited to see what we're gonna dive into.

15
00:00:42,135 --> 00:00:44,385
Uh, I know that you have a lot of questions,

16
00:00:44,385 --> 00:00:46,275
so I'm looking forward to hear some of them.

17
00:00:46,635 --> 00:00:48,644
We'll start with the, the direct insulting one.

18
00:00:48,644 --> 00:00:49,305
I suppose.

19
00:00:49,305 --> 00:00:52,905
You're an AWS hero, you're a cloud native ambassador.

20
00:00:52,965 --> 00:00:55,785
What, what got you down the path of, you know what I should do?

21
00:00:55,935 --> 00:00:56,504
That's right.

22
00:00:56,504 --> 00:00:59,535
Do volunteer work for giant entities that

23
00:00:59,535 --> 00:01:02,205
frankly could afford to pay people to do this.

24
00:01:02,205 --> 00:01:05,655
If you really think about it the right way, I, I mostly kid,

25
00:01:05,655 --> 00:01:07,785
I, Lord knows I've spent enough time in the community myself.

26
00:01:08,110 --> 00:01:08,980
How do you wind up there?

27
00:01:09,370 --> 00:01:14,050
Yeah, like to, to be honest, like I didn't know that I'm gonna end up there.

28
00:01:14,050 --> 00:01:17,740
Like, so a few years ago when I started my journey on, like when

29
00:01:17,740 --> 00:01:21,280
I came to the United States, I was like, sure, yeah, I'll try to

30
00:01:21,280 --> 00:01:24,370
solve a couple problems in a couple organizations here and there.

31
00:01:24,460 --> 00:01:28,990
And then all, all a sudden after some time in 2019, I remember

32
00:01:28,990 --> 00:01:32,350
this was my first public speaking opportunities that I had.

33
00:01:32,710 --> 00:01:34,420
It was like, it strike me as that like.

34
00:01:34,830 --> 00:01:38,910
I always thought that like, I don't know enough to share.

35
00:01:39,060 --> 00:01:42,030
Like that was like the really, the really tipping point to me.

36
00:01:42,390 --> 00:01:44,790
Like everything I do, like, yeah, everyone knows that.

37
00:01:44,820 --> 00:01:45,750
Everyone knows that.

38
00:01:45,840 --> 00:01:49,410
Until that moment and then I went in my first talk, I was like.

39
00:01:49,950 --> 00:01:52,170
Yeah, a lot of people didn't know what I'm gonna talk about

40
00:01:52,170 --> 00:01:55,800
and they liked it and they said like, this is great content.

41
00:01:55,890 --> 00:01:59,910
So from there, like I started to say like if some don't know about some

42
00:01:59,910 --> 00:02:03,600
things that I'm doing, why I'm not sharing, at least like I have it out there.

43
00:02:03,690 --> 00:02:08,310
And that ends up to be like, sure, I contribute to many open source community.

44
00:02:08,370 --> 00:02:10,139
I can teach people how to go there.

45
00:02:10,259 --> 00:02:12,090
And then like all of these things came like.

46
00:02:12,660 --> 00:02:15,540
Sure there's an ambassador program for the CCF F. Can I

47
00:02:15,540 --> 00:02:19,410
apply and see how can I explore the world from that space?

48
00:02:19,440 --> 00:02:20,730
Gives me a great opportunity.

49
00:02:20,850 --> 00:02:23,850
AWS Hero, it's kinda like pic, so it's kind of like a

50
00:02:23,850 --> 00:02:26,160
different story, but like also like I've been doing a

51
00:02:26,160 --> 00:02:28,440
lot of work with AWS, so that's what I've been picked.

52
00:02:28,590 --> 00:02:32,820
But what's really my interest here is to share more on what I

53
00:02:32,820 --> 00:02:36,480
have done, on what I heard about, on what I have seen better

54
00:02:36,480 --> 00:02:40,470
in my opinion, and see if that helps anyone on the ecosystem.

55
00:02:40,955 --> 00:02:45,510
It, it, it feels like you fall prey to the same trap that many of us do.

56
00:02:45,510 --> 00:02:49,620
Lord knows I still have to talk myself out of this, where I, I have this

57
00:02:49,620 --> 00:02:53,790
internalized perception that if I know something, therefore it's commonly known.

58
00:02:53,790 --> 00:02:55,410
Everyone basically knows this.

59
00:02:55,470 --> 00:02:57,480
But if I don't know something, that's the hard stuff.

60
00:02:57,480 --> 00:03:00,570
That's the interesting piece of it, and it's never true.

61
00:03:00,810 --> 00:03:04,020
Uh, similarly, I've, I've found that making a talk more broadly

62
00:03:04,020 --> 00:03:08,400
accessible to a larger number of people has never been the wrong decision.

63
00:03:08,610 --> 00:03:11,640
Because it's, it, it people, everything is new to someone.

64
00:03:11,640 --> 00:03:13,560
We live in a big world and a big space.

65
00:03:14,010 --> 00:03:15,540
You nailed it.

66
00:03:15,570 --> 00:03:20,130
It's that like concept in your head, like when you like drew a circle and

67
00:03:20,130 --> 00:03:23,310
then like you always keep circling around and I'm like, everyone knows it.

68
00:03:23,370 --> 00:03:27,090
Like talk to someone, like I talk to, like how many people I talk to.

69
00:03:27,095 --> 00:03:28,500
Usually it's not a lot.

70
00:03:28,500 --> 00:03:31,680
And then like you talk to someone and you're like, oh yeah, I know this feature.

71
00:03:31,680 --> 00:03:33,150
You talk to someone, I know this feature.

72
00:03:33,270 --> 00:03:34,410
But like then when you look.

73
00:03:34,785 --> 00:03:38,475
Over, like a lot of people don't know and sometimes actual,

74
00:03:38,475 --> 00:03:41,115
like even if you talk about the same topic over and over,

75
00:03:41,265 --> 00:03:43,755
some people may listen to that, not listen to the others.

76
00:03:43,755 --> 00:03:47,745
So sharing the same content sometimes in different ways, in different formats.

77
00:03:47,985 --> 00:03:51,075
Like what I also have seen resonate with people is that.

78
00:03:51,915 --> 00:03:54,105
I talk, I ha I'm not selling anything.

79
00:03:54,315 --> 00:03:57,705
Like no one have to listen to me because like I'm solving a problem.

80
00:03:57,825 --> 00:04:00,075
So is that also coming from like, I'm being an

81
00:04:00,075 --> 00:04:02,295
end user, tried something, sharing my thoughts.

82
00:04:02,415 --> 00:04:04,305
I'm not pushing you to buy my software.

83
00:04:04,305 --> 00:04:05,865
I'm telling you my software works.

84
00:04:06,285 --> 00:04:07,875
I tested something, I tried it.

85
00:04:07,875 --> 00:04:08,445
It works.

86
00:04:08,445 --> 00:04:11,385
You wanna use it, you wanna listen to it, you wanna correct me.

87
00:04:11,535 --> 00:04:12,975
It's, it's a community work.

88
00:04:12,975 --> 00:04:16,170
This is the feedback that I'm going to, but also like what I learned from that.

89
00:04:17,355 --> 00:04:18,675
Is by contributing.

90
00:04:18,975 --> 00:04:21,915
People might tell me like, oh, but have you looked into this?

91
00:04:21,945 --> 00:04:25,275
And that opened like a whole lot of can of worms

92
00:04:25,395 --> 00:04:27,795
where like, oh, you know, I didn't look into this.

93
00:04:27,795 --> 00:04:29,055
Let me look into it.

94
00:04:29,235 --> 00:04:33,465
And actually many of my talks I have people say, sure, that was a great talk

95
00:04:33,465 --> 00:04:36,345
and all the kinda stuff, but I have people said, have you looked into that?

96
00:04:36,405 --> 00:04:38,115
We tried this before and it didn't work.

97
00:04:38,175 --> 00:04:41,505
And that striked me as a great conversation to know like.

98
00:04:42,060 --> 00:04:43,110
I didn't look into it.

99
00:04:43,260 --> 00:04:44,190
Let me try.

100
00:04:44,310 --> 00:04:47,460
And then I start to look into it, and then it becomes a bigger thing.

101
00:04:47,520 --> 00:04:48,870
It's why I love conferences and the rest of

102
00:04:48,870 --> 00:04:51,150
the community where I, I'll talk to someone.

103
00:04:51,150 --> 00:04:53,820
The most recent thing that still irritates me that I went

104
00:04:53,820 --> 00:04:56,820
this long without knowing about it, is at Tuin, A-T-U-I-N.

105
00:04:57,030 --> 00:05:00,630
It's an incredibly awesome shell history that sinks between machines.

106
00:05:00,810 --> 00:05:02,730
Uh, I've discovered that, installed it everywhere.

107
00:05:03,075 --> 00:05:06,615
Cannot go back again to using the built-in nonsense,

108
00:05:06,615 --> 00:05:08,955
given how ephemeral most of my stuff tends to be.

109
00:05:09,135 --> 00:05:12,675
It, it's these weird things where, oh, well, why not build this tool?

110
00:05:12,825 --> 00:05:14,655
Like, and there's downsides to that too.

111
00:05:14,655 --> 00:05:19,365
After I first built out my original overly wrought newsletter publication

112
00:05:19,365 --> 00:05:22,395
system, someone said, well, why didn't you just use Curated Co?

113
00:05:22,455 --> 00:05:23,805
It's, but why?

114
00:05:23,805 --> 00:05:24,620
Why didn't I use what now?

115
00:05:25,305 --> 00:05:26,745
Because I didn't know it exists.

116
00:05:26,745 --> 00:05:29,025
This would've been handy several months ago.

117
00:05:29,295 --> 00:05:32,775
Uh, yeah, there are, there are always ways to do it in

118
00:05:32,775 --> 00:05:35,685
talking to people and getting the real skinny on what people

119
00:05:35,685 --> 00:05:38,145
think about how something works is incredibly valuable.

120
00:05:38,565 --> 00:05:38,775
Yeah.

121
00:05:38,835 --> 00:05:43,275
That this is, this is usually like how most of my learning has been over

122
00:05:43,275 --> 00:05:45,750
the year, and that like got me to a space where like, you know what?

123
00:05:46,590 --> 00:05:48,060
I experienced it.

124
00:05:48,150 --> 00:05:50,100
Let's talk about, let's see, like how it goes.

125
00:05:50,160 --> 00:05:51,180
Is it bad or good?

126
00:05:51,240 --> 00:05:53,490
It solved the problem for my own experience.

127
00:05:53,580 --> 00:05:55,860
And sometimes also it's interesting to choose affiliates.

128
00:05:56,040 --> 00:05:59,760
'cause like you wanna tell people like, what did you try and didn't work out?

129
00:05:59,910 --> 00:06:02,040
Because like you don't want them to sit in that trap.

130
00:06:02,040 --> 00:06:03,210
So like, either I'm learning something.

131
00:06:03,510 --> 00:06:07,740
But usually I try most of the times as much as I can to set my

132
00:06:07,740 --> 00:06:11,730
talks into like an experience that I have done into a real story.

133
00:06:11,820 --> 00:06:14,760
I don't wanna like bring a topic and just like talk about, sure,

134
00:06:14,849 --> 00:06:17,610
I can talk about Kubernetes, I can talk about like a WSI can talk

135
00:06:17,610 --> 00:06:21,330
about anything, but I usually try to big topics that a problem I

136
00:06:21,330 --> 00:06:24,990
try to solve or like a, a situation where I've been in that gives

137
00:06:24,990 --> 00:06:28,140
me like more, I don't wanna say credibility, but gives me more like.

138
00:06:28,364 --> 00:06:31,215
I'm in it, like usually I don't break much into the talk.

139
00:06:31,215 --> 00:06:34,034
I gives him a real story about what exactly happened.

140
00:06:34,364 --> 00:06:37,005
I mean, something I find is that documentation falls down terribly

141
00:06:37,094 --> 00:06:40,215
when it just tries to do a list of, and here's all the features

142
00:06:40,215 --> 00:06:43,815
it does and all here's an API reference for whatever reason.

143
00:06:43,875 --> 00:06:46,184
The thing I'm trying to do is never well documented

144
00:06:46,184 --> 00:06:49,125
in these things where, so I like experience reports.

145
00:06:49,125 --> 00:06:52,544
If I'm gonna build a to-do list app, to use an overdone example.

146
00:06:52,910 --> 00:06:53,210
Great.

147
00:06:53,210 --> 00:06:55,760
I wanna know how you use the tool to do it, what your steps

148
00:06:55,760 --> 00:06:59,150
were, how it wound up looking and you're driving to an outcome.

149
00:06:59,510 --> 00:07:02,030
Uh, I've also deeply appreciate the community stuff,

150
00:07:02,030 --> 00:07:04,760
especially the heroes folks on the AWS world because you are

151
00:07:04,760 --> 00:07:09,290
not beholden to AWS in the same way as an AWS employee is.

152
00:07:09,560 --> 00:07:13,820
If an AWS employee talks about aspects of AWS being complete

153
00:07:13,820 --> 00:07:17,720
crap, they're likely not going to be AWS employees for very long.

154
00:07:17,840 --> 00:07:20,690
Whereas the rest of the community, we talk about this because.

155
00:07:20,965 --> 00:07:22,375
It does have sharp edges.

156
00:07:22,375 --> 00:07:23,905
These things are painful.

157
00:07:24,205 --> 00:07:25,585
How do you split the difference there?

158
00:07:25,585 --> 00:07:28,495
Because it, on some level, it feels weird to go and speak at a company's

159
00:07:28,495 --> 00:07:31,705
conference and use their platform and then use that to drag them.

160
00:07:31,705 --> 00:07:35,005
I mean, I have a personal policy of not making people regret, inviting

161
00:07:35,005 --> 00:07:38,155
me to things, so I'm not gonna crap on them at their own conference.

162
00:07:38,594 --> 00:07:40,875
But I do sometimes feel like I have to strike a balance.

163
00:07:41,325 --> 00:07:41,565
Yeah.

164
00:07:41,565 --> 00:07:44,505
Like the balance is always, is like being honest and

165
00:07:44,505 --> 00:07:46,575
like showing what is the real value of something.

166
00:07:46,695 --> 00:07:48,974
Like I'm always, come on like many social medias

167
00:07:48,974 --> 00:07:51,525
and many platforms and say, that didn't work for me.

168
00:07:51,525 --> 00:07:52,905
That wasn't the right intention.

169
00:07:52,965 --> 00:07:56,685
There are meetings and spaces for things like what I should say.

170
00:07:56,745 --> 00:08:00,044
I've been saying over the years that a lot of people know this

171
00:08:00,044 --> 00:08:04,844
about me, that AWS user experience has been clunky all the time.

172
00:08:04,844 --> 00:08:05,985
They didn't master it.

173
00:08:05,985 --> 00:08:06,075
That

174
00:08:06,075 --> 00:08:07,305
is such a flattering way to put it.

175
00:08:07,945 --> 00:08:08,155
Yeah.

176
00:08:08,155 --> 00:08:08,785
Yeah.

177
00:08:08,875 --> 00:08:12,205
Like, like in a way, like, you know, like it's, it's been like

178
00:08:12,295 --> 00:08:16,615
ridiculous how many times I have seen like, oh, I have to go all the way.

179
00:08:16,675 --> 00:08:20,635
Like, I, I go and talks with service teams and sometimes and say like,

180
00:08:20,695 --> 00:08:24,145
you know what, like, why we have this three times on the same page.

181
00:08:24,205 --> 00:08:24,895
Like, why?

182
00:08:24,925 --> 00:08:27,115
Like, why, like they are reliable on something

183
00:08:27,145 --> 00:08:29,815
and, but they are not in, in something else.

184
00:08:29,875 --> 00:08:31,975
And that's where like always the balance comes in.

185
00:08:32,155 --> 00:08:34,615
But also like, it has to be like in, in our.

186
00:08:34,875 --> 00:08:38,534
I wanna give them feedback and I want it to be critical, but I want it to be

187
00:08:38,534 --> 00:08:42,015
like, I don't wanna say in a nice way, but I want it to be like honest feedback.

188
00:08:42,105 --> 00:08:44,865
I don't wanna embrace something that like other

189
00:08:44,865 --> 00:08:48,194
vendors has done for years and say, this is great.

190
00:08:48,255 --> 00:08:51,885
Like I would say like, it's great you have this now, but like

191
00:08:51,885 --> 00:08:53,265
what took so long?

192
00:08:53,265 --> 00:08:53,564
Yeah.

193
00:08:54,000 --> 00:08:58,110
Yeah, like, it, it, it's been long time like to get into something like that,

194
00:08:58,199 --> 00:09:01,260
but there are some innovations that I have seen is a space that deserves that.

195
00:09:01,439 --> 00:09:05,580
And like, what, what I said exactly about, I'm not

196
00:09:05,580 --> 00:09:08,580
behold to anything because I don't sell anything.

197
00:09:08,730 --> 00:09:11,370
So like, I'm not obligated to, to any of that.

198
00:09:11,370 --> 00:09:15,060
Like, I, I don't like AWS You're gonna what, like what's gonna happen?

199
00:09:15,060 --> 00:09:17,910
Like I'm not You gonna use AWS in the next three years.

200
00:09:17,910 --> 00:09:18,480
Sure, yeah.

201
00:09:18,480 --> 00:09:19,710
Like, don't work for the company.

202
00:09:19,710 --> 00:09:20,880
It's my honest opinion.

203
00:09:20,970 --> 00:09:22,110
I think like that's what.

204
00:09:22,455 --> 00:09:25,575
I should be doing, because I'm not doing this for AWS specifically.

205
00:09:25,575 --> 00:09:29,925
I know the intricacies of AWS in some cases, but I'm doing this for the people.

206
00:09:29,955 --> 00:09:32,925
Like if someone asked me my opinion today about like, we

207
00:09:32,925 --> 00:09:35,745
have this debate all the time, like funny story, uh, you

208
00:09:35,745 --> 00:09:38,445
probably know that, uh, uh, because you have seen a lot.

209
00:09:38,505 --> 00:09:42,795
Like when I talk to other people, I'm on the container, AWS hero space,

210
00:09:42,915 --> 00:09:45,675
if I talk to someone in the container, but like I'm have a favor.

211
00:09:46,350 --> 00:09:48,930
I favor EKS.

212
00:09:49,050 --> 00:09:50,670
Some others favor ECS.

213
00:09:50,790 --> 00:09:53,460
Like you have to seize the debate when like we

214
00:09:53,730 --> 00:09:56,640
trash some of the services sometimes for each other.

215
00:09:56,760 --> 00:09:59,910
Like say like, oh no, like EEC S is better than ES.

216
00:09:59,910 --> 00:10:01,410
I'm say no EKS is better.

217
00:10:01,530 --> 00:10:02,550
Like, and all of the kinda stuff.

218
00:10:02,580 --> 00:10:06,600
And it's at the end of the day, like it's fun situation to compare things, but

219
00:10:06,600 --> 00:10:10,740
like at least we have honest opinion about like where exactly is the use case.

220
00:10:10,860 --> 00:10:15,390
We laugh about, we kid about it a lot of time, but you ask me in one day.

221
00:10:15,445 --> 00:10:18,115
What is one of the services to use?

222
00:10:18,235 --> 00:10:22,105
I tell like if you are a small company, Kubernetes is not the right fit for you.

223
00:10:22,255 --> 00:10:23,935
Like either you're in a W that's shop or not.

224
00:10:23,935 --> 00:10:25,315
Like this is like irrelevant.

225
00:10:25,435 --> 00:10:28,465
But like if you are like using containers, if like it has to

226
00:10:28,465 --> 00:10:31,645
have like characteristics and criteria for like your decision.

227
00:10:31,705 --> 00:10:35,785
So like there's the fun talk, all of the tabs that we have, but

228
00:10:35,785 --> 00:10:39,235
there's also like the technological decisions that you need to make.

229
00:10:39,295 --> 00:10:41,095
And this is situation based.

230
00:10:41,185 --> 00:10:42,685
So it's not like, hey.

231
00:10:42,985 --> 00:10:47,155
All the way Go containers EKS, because that doesn't work.

232
00:10:47,155 --> 00:10:49,465
So I, I have never seen a solution as that fits all.

233
00:10:49,585 --> 00:10:52,705
At Duck Bill, we are building our product on top of ECS, which

234
00:10:52,705 --> 00:10:55,675
is, this makes sense for our scale and current constraints.

235
00:10:55,825 --> 00:10:57,565
We have a path forward and boom.

236
00:10:57,565 --> 00:10:58,495
Surprise sponsorship.

237
00:10:58,495 --> 00:10:59,035
That's right.

238
00:10:59,035 --> 00:11:02,335
This show is sponsored by duck bill hq.com.

239
00:11:02,365 --> 00:11:06,175
My employer, we have a platform now, rather than just handling the.

240
00:11:06,310 --> 00:11:08,589
Consulting side of the world is we have historically

241
00:11:08,589 --> 00:11:12,099
with contract negotiations for large entities, debating

242
00:11:12,099 --> 00:11:14,770
with AWS what the future might hold for both parties.

243
00:11:15,069 --> 00:11:18,069
Now we have software that we are systematizing part of this in.

244
00:11:18,219 --> 00:11:19,869
If that sounds relevant to what you're

245
00:11:19,869 --> 00:11:23,079
doing, please check us out@duckbillhq.com.

246
00:11:23,170 --> 00:11:27,849
And also we are hiring And also also Ahmed, one of the best

247
00:11:27,849 --> 00:11:31,449
parts about this timing is that yesterday, for the first time in.

248
00:11:32,445 --> 00:11:36,975
Since it first came out, I spun up an EKS cluster because I'm building

249
00:11:36,975 --> 00:11:40,035
some, a bunch of weird projects that I want to throw at a wall.

250
00:11:40,245 --> 00:11:43,335
All of my customers use EKS in some way, shape, or form.

251
00:11:43,485 --> 00:11:45,795
It's time for me to use it and it's gotten,

252
00:11:45,855 --> 00:11:47,715
from what I can tell, slightly better.

253
00:11:47,775 --> 00:11:51,105
It only took 10 minutes to spin up the EKS cluster

254
00:11:51,105 --> 00:11:53,865
instead of the 25 when I did several years ago.

255
00:11:53,865 --> 00:11:56,625
So it's improving bit by bit.

256
00:11:57,315 --> 00:11:58,155
What's your take on it?

257
00:11:58,545 --> 00:12:01,095
Let's not talk about the start of time for the cluster.

258
00:12:01,095 --> 00:12:04,725
That has been like a dilemma for a while.

259
00:12:04,725 --> 00:12:06,705
Like why this takes forever.

260
00:12:06,705 --> 00:12:10,665
Like I have seen the architecture for the control plane behind the scenes.

261
00:12:10,695 --> 00:12:11,895
I still like don't get it.

262
00:12:12,080 --> 00:12:14,720
Why it takes too long, because others have

263
00:12:14,720 --> 00:12:17,060
done it and it seems to be working for others.

264
00:12:17,060 --> 00:12:19,100
So I'm not sure like where is this coming from?

265
00:12:19,220 --> 00:12:22,760
I'm certain there are reasons and good reasons for it and honestly.

266
00:12:22,760 --> 00:12:25,640
And how often do you spin up or down your production cluster?

267
00:12:25,760 --> 00:12:28,880
Oh, I don't, but that's kind of the point in development.

268
00:12:28,910 --> 00:12:31,580
When I'm testing my infrastructure stuff, I want to smoke,

269
00:12:31,580 --> 00:12:34,880
test it in a test account, and that adds a tremendous

270
00:12:34,880 --> 00:12:37,970
burden to how long it takes to run through those tests.

271
00:12:38,580 --> 00:12:39,420
Please fix it.

272
00:12:39,420 --> 00:12:40,470
That's why I care.

273
00:12:40,740 --> 00:12:41,040
Yeah.

274
00:12:41,040 --> 00:12:41,760
I, exactly.

275
00:12:41,760 --> 00:12:43,830
I went through this use case like, and, and I agree

276
00:12:43,830 --> 00:12:47,310
with you, like how often you re a Kubernetes cluster.

277
00:12:47,310 --> 00:12:47,640
Sure.

278
00:12:47,670 --> 00:12:50,580
Not too much, but like when I need this for testing, when

279
00:12:50,580 --> 00:12:53,040
I need to mimic something, when I'm doing a demo, like I

280
00:12:53,040 --> 00:12:55,980
have to wait like 10 minutes for like a cluster to get up.

281
00:12:56,010 --> 00:12:57,300
But like, let's talk about like.

282
00:12:57,720 --> 00:12:58,110
Good.

283
00:12:58,110 --> 00:13:00,810
Other things that, like I haven't seen like in other, like there

284
00:13:00,810 --> 00:13:04,800
is new, uh, the ecosystem pattern of like, so let me tell you

285
00:13:04,800 --> 00:13:07,530
why I like Kubernetes in general, like the generality of it.

286
00:13:07,530 --> 00:13:11,670
It's just like, because it's a common pattern across multiple cloud

287
00:13:11,670 --> 00:13:16,260
provider, like I can get that flavor on, on a WSI can get this flavor

288
00:13:16,260 --> 00:13:20,460
and uh, as providers does a lot of the things behind the scene change.

289
00:13:20,520 --> 00:13:21,030
Sure.

290
00:13:21,150 --> 00:13:23,460
Instances, all of the kinda stuff like the, how

291
00:13:23,460 --> 00:13:25,470
they author each other, all of the kinda stuff.

292
00:13:25,500 --> 00:13:26,430
But at the end of the day.

293
00:13:26,475 --> 00:13:29,235
It's a deployment, it's a bud, it's a container.

294
00:13:29,295 --> 00:13:30,285
It's all shared.

295
00:13:30,375 --> 00:13:33,525
I can get a similar flavor into it, into my machine

296
00:13:33,525 --> 00:13:35,745
to test, which is relevant to what you're saying.

297
00:13:35,745 --> 00:13:40,005
So in my CI, I can spin up whatever, like Kubernetes thing

298
00:13:40,005 --> 00:13:43,455
on a Docker, whatever ecosystem to test something with.

299
00:13:44,115 --> 00:13:47,895
Problem was, it is like when you start having like despairs

300
00:13:47,955 --> 00:13:50,835
and like have solutions that like, sure I have some things

301
00:13:50,835 --> 00:13:53,925
that works for a cloud, but something also work for local.

302
00:13:54,045 --> 00:13:56,205
It's like becoming like a tangling effect

303
00:13:56,235 --> 00:13:58,125
and sometimes you cannot test the same stuff.

304
00:13:58,155 --> 00:14:01,395
So that's where like we have to come up with mockups, mockup

305
00:14:01,395 --> 00:14:04,545
APIs and see like, oh, now I have to call the E-K-S-A-P-I.

306
00:14:04,725 --> 00:14:06,135
Now I have to get my Bud Identity.

307
00:14:06,255 --> 00:14:07,155
All this kinda stuff.

308
00:14:07,275 --> 00:14:08,985
This is the sad part of it.

309
00:14:09,030 --> 00:14:10,140
The good part of it.

310
00:14:10,290 --> 00:14:12,030
There are like more capabilities that's

311
00:14:12,030 --> 00:14:14,430
coming into, like, into services like this.

312
00:14:14,520 --> 00:14:17,250
One of the things that like I really embraced when I have seen

313
00:14:17,250 --> 00:14:22,380
it is like the concept of manage, add-on, uh, or not add, add-on.

314
00:14:22,410 --> 00:14:25,380
They call it managed services now, whatever, but

315
00:14:25,380 --> 00:14:28,410
like it is a concept of having managed ARGUS complex.

316
00:14:29,100 --> 00:14:32,340
I, I've seen Argo runs and it's other controllers might

317
00:14:32,340 --> 00:14:35,730
not be, but like that's a good option if there's other

318
00:14:35,820 --> 00:14:38,880
community or open source projects that could run the same way.

319
00:14:38,880 --> 00:14:41,430
And it takes the complexity out of running the control plane.

320
00:14:41,610 --> 00:14:42,270
Loves that.

321
00:14:42,300 --> 00:14:43,290
That's a great idea.

322
00:14:43,380 --> 00:14:46,440
Removes burden if you know how to run things like that.

323
00:14:46,560 --> 00:14:48,985
So from that perspective, seems like it's growing.

324
00:14:49,900 --> 00:14:53,829
Does it do other better jobs like other providers?

325
00:14:53,829 --> 00:14:54,969
Maybe, maybe not.

326
00:14:54,969 --> 00:14:58,180
Like I have to put them in a benchmark to see like what they can do.

327
00:14:58,390 --> 00:15:00,280
There are use cases that I hear about.

328
00:15:00,780 --> 00:15:02,130
Obviously interesting to me.

329
00:15:02,460 --> 00:15:07,200
I dunno, like do I need to run a hundred thousand node on a single cluster?

330
00:15:08,340 --> 00:15:09,090
Never.

331
00:15:09,090 --> 00:15:10,830
Never had this use case in my entire life.

332
00:15:10,830 --> 00:15:11,250
Well, yes.

333
00:15:11,250 --> 00:15:12,720
If you're trying to get the AWS bill high

334
00:15:12,720 --> 00:15:14,640
score, how else are you planning on doing it?

335
00:15:14,940 --> 00:15:15,300
Sure.

336
00:15:15,300 --> 00:15:17,820
I don't have that money to spend that in my account.

337
00:15:17,910 --> 00:15:18,750
Oh good lord.

338
00:15:18,750 --> 00:15:20,310
You never do it with your own money.

339
00:15:20,310 --> 00:15:22,860
That's what employers are for or someone

340
00:15:22,860 --> 00:15:24,510
else, or clients in the consulting world.

341
00:15:24,625 --> 00:15:25,225
I digress.

342
00:15:25,405 --> 00:15:28,405
Uh, I've been running a test cluster at home for Kubernetes for two years.

343
00:15:28,405 --> 00:15:30,865
I had to build a conference talk out of it because I mouthed off on the

344
00:15:30,865 --> 00:15:33,925
internet seven years ago and said, no one's gonna care about Kubernetes.

345
00:15:33,925 --> 00:15:36,265
So I ended up having to give a talk called Terrible Ideas

346
00:15:36,265 --> 00:15:40,045
in Kubernetes, but I found it useful where now I can just

347
00:15:40,045 --> 00:15:42,745
write random nonsense or find it somewhere on GitHub.

348
00:15:43,045 --> 00:15:46,915
In a container, I can throw it onto the cluster, access it over my tail

349
00:15:46,915 --> 00:15:51,204
scale network, and I can just have a bunch of heterogeneous things running.

350
00:15:51,535 --> 00:15:54,324
Unfortunately, I've become a victim of my own success in that.

351
00:15:54,324 --> 00:15:57,535
Some of my team have seen some of the tools I've built that are useful

352
00:15:57,535 --> 00:15:59,694
for what they're doing, even though they're not, not coupled to.

353
00:15:59,775 --> 00:16:01,305
Client data because, my God.

354
00:16:01,515 --> 00:16:05,535
But then, oh, like I built a great image manipulator for marketing purposes.

355
00:16:05,540 --> 00:16:07,064
It, it has some advantages.

356
00:16:07,365 --> 00:16:09,975
And they said, they said, great, can I get a copy of that?

357
00:16:09,975 --> 00:16:13,875
Like, alright, time to build an internal cluster on this, but we're gonna do it.

358
00:16:13,875 --> 00:16:14,324
Right.

359
00:16:14,324 --> 00:16:18,944
And by, right, I mean, enterprisey, we are doing GI ops the whole way with

360
00:16:18,944 --> 00:16:24,314
Argo cd, we're using open tofu because Terraform gets really weird at scale.

361
00:16:24,870 --> 00:16:29,910
It is a wildly overbuilt solution for a single container at the moment.

362
00:16:30,360 --> 00:16:32,130
But something I found about these clusters is

363
00:16:32,130 --> 00:16:34,560
they never tend to stay single tenant for long.

364
00:16:34,620 --> 00:16:38,670
They, you, you start adding things to it, and in the fullness of time,

365
00:16:38,700 --> 00:16:41,880
this becomes really straightforward to start launching a bunch of internal

366
00:16:41,880 --> 00:16:47,100
corporate tools, which is handy, but the teething exercise of getting up and

367
00:16:47,100 --> 00:16:51,210
running with it, I'm glad this is not critical path for anything right now.

368
00:16:51,705 --> 00:16:53,415
'cause I don't know it well enough to support it.

369
00:16:53,595 --> 00:16:57,225
It is not like I, like, I, I recall the days that I thought about

370
00:16:57,225 --> 00:17:00,975
like, oh, do I have to like manage an enterprise cluster for like

371
00:17:00,975 --> 00:17:04,214
many use cases and do I have to run like all of the cube admin

372
00:17:04,214 --> 00:17:07,035
and join instances together and do all of that in the cloud?

373
00:17:07,035 --> 00:17:08,415
I was like, yeah, I don't wanna,

374
00:17:08,625 --> 00:17:11,115
because it depends on what laptop you ran it from.

375
00:17:11,145 --> 00:17:11,444
Oh.

376
00:17:11,474 --> 00:17:13,155
And then you talk to like, oh no, no.

377
00:17:13,155 --> 00:17:16,275
You're only supposed to run that from the CICD system.

378
00:17:16,275 --> 00:17:18,714
It's, that would've been terrific to put on the warning label.

379
00:17:19,210 --> 00:17:19,510
Yeah.

380
00:17:19,510 --> 00:17:22,030
So like all, all of that, like it solves a problem.

381
00:17:22,030 --> 00:17:24,849
It's like, you know, the whole cloud solves a problem for like,

382
00:17:24,849 --> 00:17:27,609
not have to care about hardware, but like, do I do my own stuff?

383
00:17:27,609 --> 00:17:28,210
Sure, yeah.

384
00:17:28,210 --> 00:17:30,190
My entire home automation system runs on

385
00:17:30,190 --> 00:17:32,230
K three s. That's where like, I'm running

386
00:17:32,230 --> 00:17:33,040
K three s myself.

387
00:17:33,040 --> 00:17:33,640
Home assistant.

388
00:17:34,180 --> 00:17:34,600
Yeah.

389
00:17:34,600 --> 00:17:35,080
Home system.

390
00:17:35,080 --> 00:17:35,350
Yeah.

391
00:17:35,380 --> 00:17:37,660
So I do not have that running on the cluster because that

392
00:17:37,660 --> 00:17:40,150
has gotten sizable and logic based enough that I have

393
00:17:40,180 --> 00:17:43,485
a, uh, I got a HP Mini PC that I put the whole thing on.

394
00:17:44,085 --> 00:17:47,205
Because it's, and again, with, with my wife, we're definitely proven

395
00:17:47,205 --> 00:17:49,635
the old trope that when you have a couple and someone's really

396
00:17:49,635 --> 00:17:52,995
into IO ot, the dynamic is one of you loves the fact that you're

397
00:17:52,995 --> 00:17:56,115
living in the future and the other one thinks the house is haunted.

398
00:17:56,205 --> 00:17:56,775
It's great.

399
00:17:57,015 --> 00:17:59,535
Yeah, that, that's exactly where I'm at right now.

400
00:17:59,535 --> 00:18:01,815
Like I have like, and I can tell you.

401
00:18:02,100 --> 00:18:05,459
I spent few days where like my wife would call me like

402
00:18:05,459 --> 00:18:07,679
and say like, Hey, the house lights are not turning on.

403
00:18:07,860 --> 00:18:10,199
I was like, um, I dunno, like what's happening?

404
00:18:10,199 --> 00:18:12,270
And she said like, all of a sudden it's not working.

405
00:18:12,419 --> 00:18:15,570
I was like, yeah, probably you have to restart the cluster somehow.

406
00:18:15,629 --> 00:18:17,159
And I'm like, go unplug it.

407
00:18:17,159 --> 00:18:18,480
Plug it again and it'll work.

408
00:18:18,540 --> 00:18:20,040
I was like, sure, yeah, that works.

409
00:18:20,040 --> 00:18:23,100
But now I'm hunted by my own clusters that I have to set it up.

410
00:18:23,159 --> 00:18:26,850
Sometimes I have to like upgrade it and do all of the work around it.

411
00:18:26,895 --> 00:18:28,815
But like to be honest, it works.

412
00:18:28,815 --> 00:18:31,245
Like I ran into like this is one of the things that you

413
00:18:31,245 --> 00:18:34,754
said, like setting it up one time was a complex stories

414
00:18:34,754 --> 00:18:37,004
that I have to get all of the things set up in my end.

415
00:18:37,125 --> 00:18:40,095
I have seen like how complex I have to bake images and do

416
00:18:40,095 --> 00:18:43,065
all of that to get like a small cluster in my home running.

417
00:18:43,215 --> 00:18:45,764
So imagine this like running this in an enterprise scale.

418
00:18:45,885 --> 00:18:49,514
Like I have to bake my images, do all of the work to get this.

419
00:18:49,605 --> 00:18:50,535
Now it's easier.

420
00:18:50,535 --> 00:18:54,705
Now it's just like a couple clicks and you get a cluster up that was like.

421
00:18:55,245 --> 00:18:56,235
Cool thing to have,

422
00:18:56,564 --> 00:19:00,375
I just discovered a few weeks ago from the person who wrote Atune, Ellie.

423
00:19:00,435 --> 00:19:04,725
As it turns out, uh, that K three s has a built in registry

424
00:19:04,725 --> 00:19:07,905
that is distributed across the nodes, which is awesome.

425
00:19:07,905 --> 00:19:10,334
It's, I have to, I can stop pulling the same image

426
00:19:10,334 --> 00:19:12,435
again and again, which is freaking wonderful.

427
00:19:12,840 --> 00:19:14,040
I didn't know about that.

428
00:19:14,040 --> 00:19:14,670
It's a command.

429
00:19:14,670 --> 00:19:16,470
It's a system command argument.

430
00:19:16,650 --> 00:19:18,510
Spiegel, S-P-E-G-E-L.

431
00:19:19,020 --> 00:19:21,600
It is built into K three s. You pass the server

432
00:19:21,600 --> 00:19:23,400
a command line parameter and you're done.

433
00:19:23,730 --> 00:19:24,150
Okay.

434
00:19:24,240 --> 00:19:25,410
I actually will look this up.

435
00:19:25,980 --> 00:19:26,220
Yeah.

436
00:19:26,280 --> 00:19:28,410
You see like that's why like I talked to you,

437
00:19:28,410 --> 00:19:30,510
I learned something, I'm gonna go implement it.

438
00:19:30,750 --> 00:19:33,870
Probably like my lights will not work tonight, but that's okay.

439
00:19:33,930 --> 00:19:35,520
It's, you know, it's a greater good.

440
00:19:35,640 --> 00:19:36,510
That's another trick.

441
00:19:36,510 --> 00:19:39,420
I switched all of the light switches I was using over to Lutron.

442
00:19:39,770 --> 00:19:42,080
Which is a little on the expensive side, but it's also

443
00:19:42,080 --> 00:19:44,090
what a lot of the smart home contractors build out.

444
00:19:44,179 --> 00:19:46,909
And what I love about them is if you don't hook it

445
00:19:46,909 --> 00:19:49,310
up to anything, it acts like a normal light switch.

446
00:19:49,459 --> 00:19:53,810
And when the system fails, the way it works is like a normal light switch.

447
00:19:53,810 --> 00:19:57,709
You push the button, the lights turn on, and suddenly I get yelled at less.

448
00:19:57,990 --> 00:19:59,520
I actually like this idea more.

449
00:19:59,520 --> 00:20:03,270
Like I ended up on that trend, not for all of my lights, but like, because I

450
00:20:03,270 --> 00:20:07,830
used the U Lights before and like the switch were like very like interesting.

451
00:20:07,920 --> 00:20:11,730
But then the Luron, this office is running on a Lutron switch and it actually

452
00:20:11,730 --> 00:20:14,910
like, allows me to do also three-way switches and different things and all

453
00:20:14,910 --> 00:20:18,870
the kinda stuff to mimic like a normal environment, but also like when wifi is

454
00:20:18,870 --> 00:20:22,950
working and everything is stable, when the cloud is running, it runs beautifully

455
00:20:23,280 --> 00:20:27,120
from a remote perspective, but that, you know, it's, it's a balance between.

456
00:20:27,400 --> 00:20:30,280
What do I need to do day to day and like how I tested things?

457
00:20:30,280 --> 00:20:33,760
I think like depends on like what I'm actually achieving for.

458
00:20:33,880 --> 00:20:39,190
I think, to be honest, my cluster is running up there and I, I barely touch it.

459
00:20:39,220 --> 00:20:41,050
Most of the cases, most of the time, like I

460
00:20:41,050 --> 00:20:42,940
don't need to touch it because it's working.

461
00:20:42,940 --> 00:20:44,050
It's an older upgrade.

462
00:20:44,290 --> 00:20:47,920
All of the kinda stuff that it's running effectively doing what I need

463
00:20:47,920 --> 00:20:51,430
to do, but when I need to swir a container on it, this is the easy thing.

464
00:20:51,550 --> 00:20:52,000
Like just.

465
00:20:52,305 --> 00:20:55,725
Log into it, throw a container, get out, and it's all working.

466
00:20:55,965 --> 00:20:56,715
So yeah,

467
00:20:56,985 --> 00:20:59,295
all my config lives in a GI repo that I just

468
00:20:59,295 --> 00:21:01,305
run, uh, Quebec to against for home stuff.

469
00:21:01,305 --> 00:21:04,785
I have not, I haven't gi opted yet, but it means that when I tear down the

470
00:21:04,785 --> 00:21:07,605
cluster and rebuild it as I have to every year and a half or so, 'cause it

471
00:21:07,605 --> 00:21:10,575
gets wonky, it's pretty easy to get the stuff I care about back and running.

472
00:21:11,145 --> 00:21:14,145
I have a backup, so my, like, I didn't get off it like just,

473
00:21:14,145 --> 00:21:17,534
I normally like would do anything for like my cloud stuff.

474
00:21:17,594 --> 00:21:17,715
Yeah.

475
00:21:17,715 --> 00:21:18,405
For the home stuff.

476
00:21:18,405 --> 00:21:20,655
It's like I'll run my own RSS aggregator.

477
00:21:20,745 --> 00:21:21,195
Terrific.

478
00:21:21,195 --> 00:21:21,584
Awesome.

479
00:21:21,584 --> 00:21:23,955
If that breaks, it's annoying.

480
00:21:24,014 --> 00:21:27,824
I have to get it back up and running, but none of my business stuff goes down.

481
00:21:27,824 --> 00:21:28,514
Nothing breaks.

482
00:21:28,514 --> 00:21:32,024
This is a different RSS system than the one that feeds the newsletter.

483
00:21:32,264 --> 00:21:35,745
That stuff all lives in AWS, like a grownup might put something there.

484
00:21:35,925 --> 00:21:38,385
It, it's also strange sometimes to look at.

485
00:21:38,615 --> 00:21:42,784
The monitoring for this and realize that my 11 node cluster that is all plugged

486
00:21:42,784 --> 00:21:47,375
into the same power strip has better uptime for a month than GitHub actions.

487
00:21:47,375 --> 00:21:50,284
And all right, that's, that's unfortunate, but okay.

488
00:21:50,584 --> 00:21:51,875
There's the other side of it too, that when

489
00:21:51,875 --> 00:21:53,794
it goes down, no one's coming to save me.

490
00:21:53,794 --> 00:21:55,925
I've gotta get it up and running myself.

491
00:21:56,294 --> 00:22:01,425
And not just wait for a vendor to, to fix it for me, it's a mixed bag.

492
00:22:01,425 --> 00:22:03,945
I, I don't know that there's necessarily one right way for this.

493
00:22:04,034 --> 00:22:06,105
It's, it's just the reality of it.

494
00:22:06,345 --> 00:22:08,715
We, we've forgotten on some level how to run hardware ourselves.

495
00:22:09,465 --> 00:22:13,004
This episode is sponsored by my own company, duck Bill.

496
00:22:13,300 --> 00:22:16,209
Having trouble with your AWS bill, perhaps

497
00:22:16,209 --> 00:22:18,699
it's time to renegotiate a contract with

498
00:22:18,699 --> 00:22:18,939
them.

499
00:22:19,270 --> 00:22:21,429
Maybe you're just wondering how to predict

500
00:22:21,429 --> 00:22:24,639
what's going on in the wide world of AWS.

501
00:22:24,730 --> 00:22:27,340
Well, that's where Duck Bill comes in to help.

502
00:22:27,550 --> 00:22:30,280
Remember, you can't duck the duck bill.

503
00:22:30,280 --> 00:22:32,949
Bill, which I am reliably informed by my

504
00:22:32,949 --> 00:22:36,355
business partner is absolutely not our motto.

505
00:22:36,695 --> 00:22:39,754
To learn more, visit doc bill hq.com.

506
00:22:40,755 --> 00:22:42,195
To be honest, it's a debate.

507
00:22:42,195 --> 00:22:44,955
It's a debate that I've been on with years.

508
00:22:45,405 --> 00:22:47,715
So let's talk about it from a software in

509
00:22:47,715 --> 00:22:49,665
general, not just like cloud perspective.

510
00:22:49,875 --> 00:22:51,405
A lot of solutions out there.

511
00:22:51,585 --> 00:22:53,805
Like you'll see like, oh, this solution provides me A,

512
00:22:53,805 --> 00:22:56,775
M, B, and C. Oh, but I can build A M, B and C and D.

513
00:22:56,955 --> 00:23:01,725
Sure, you can build it, but like the problem is not in the building anymore.

514
00:23:01,725 --> 00:23:03,045
It's a problem like few years.

515
00:23:03,165 --> 00:23:05,055
Like how you maintain it, how you keep it up

516
00:23:05,055 --> 00:23:06,915
and running, how you do all of that kinda work.

517
00:23:06,975 --> 00:23:08,355
It used to be day two problems.

518
00:23:08,355 --> 00:23:08,625
Great.

519
00:23:08,625 --> 00:23:09,675
Now it's like day 50.

520
00:23:10,440 --> 00:23:13,230
Like this is just like I start to think about also from

521
00:23:13,230 --> 00:23:15,510
a business perspective, like just a business mindset.

522
00:23:15,600 --> 00:23:18,120
Like what happened when like person maintains a system leaves

523
00:23:18,120 --> 00:23:20,790
or whatever, like a team or like something, or this technology

524
00:23:20,790 --> 00:23:24,210
gets old or you have to upgrade it or you have to run instances,

525
00:23:24,300 --> 00:23:27,900
or you leave it running in AWS for more than a year, in which case

526
00:23:27,900 --> 00:23:30,930
now it's extended support, which costs six times more for the.

527
00:23:31,060 --> 00:23:31,600
Cluster.

528
00:23:31,810 --> 00:23:32,200
Why?

529
00:23:32,200 --> 00:23:33,159
Because screw you.

530
00:23:33,250 --> 00:23:34,600
Another year goes past.

531
00:23:34,600 --> 00:23:38,800
Then they will blind upgrade you at a time of their choosing, not yours.

532
00:23:38,950 --> 00:23:40,840
So you're just kicking the can down the road, gaining

533
00:23:40,840 --> 00:23:42,879
nothing by it and banging through the nose for it.

534
00:23:42,879 --> 00:23:43,540
It's okay.

535
00:23:43,540 --> 00:23:45,310
That doesn't seem the most customer obsessed.

536
00:23:45,370 --> 00:23:46,750
This was always interesting.

537
00:23:46,750 --> 00:23:49,510
That part, like the extended support was always interesting

538
00:23:49,540 --> 00:23:53,169
because like they are trying to balance between like how to keep

539
00:23:53,169 --> 00:23:56,409
it sustainable for the team or like whatever the team is managing.

540
00:23:56,409 --> 00:23:58,389
That's from my perspective, but I also like.

541
00:23:58,705 --> 00:24:02,305
This is like sex X is a lot like, just like this is, it's a big number.

542
00:24:02,305 --> 00:24:05,125
Like when you try to do something like that and you're always read well

543
00:24:05,125 --> 00:24:09,535
like y, but also like if you look at a couple clusters, you don't pay much.

544
00:24:09,565 --> 00:24:11,665
Like for example, like the a b is a control plane.

545
00:24:11,665 --> 00:24:13,495
You don't bail like a lot of money for still

546
00:24:13,495 --> 00:24:15,505
money, but like you don't, don't bail a lot.

547
00:24:15,565 --> 00:24:18,445
I would like have a heart attack in some way if this is goes

548
00:24:18,445 --> 00:24:21,295
for my nodes or something like that, which is gonna be like

549
00:24:21,295 --> 00:24:23,665
more complicated than we're gonna have a conversation about.

550
00:24:23,890 --> 00:24:28,090
But again, the idea of like build versus buy and all of the kinda stuff,

551
00:24:28,150 --> 00:24:32,890
it comes to a point where like, sure, this system is unstable, but

552
00:24:32,890 --> 00:24:35,800
unstable in a way that like, you don't have to invest all of the resources,

553
00:24:35,800 --> 00:24:38,680
keeping the uptime, like all of the operational stuff, like all of the

554
00:24:38,680 --> 00:24:42,760
things, when I run something on my home, I understand the risk off.

555
00:24:42,760 --> 00:24:43,990
Like this is not working.

556
00:24:44,080 --> 00:24:47,110
Like I have built a land, it's not like critical to my life.

557
00:24:47,110 --> 00:24:50,110
Like my light's still gonna turn on like turn off.

558
00:24:50,110 --> 00:24:51,880
But like my Alex, I wouldn't say.

559
00:24:52,290 --> 00:24:55,020
Hey, like I can turn on the light from that.

560
00:24:55,020 --> 00:24:57,600
That's all the impact here or there.

561
00:24:57,810 --> 00:25:01,320
But like when I run a system and then I have to maintain it, there's a

562
00:25:01,320 --> 00:25:05,250
lot of operation overhead that I have to spend in maintaining the system,

563
00:25:05,250 --> 00:25:08,070
maintaining the infrastructure, maintaining everything behind that.

564
00:25:08,190 --> 00:25:10,020
So I always tend to tell people, like when

565
00:25:10,020 --> 00:25:11,880
someone asks like, should I build versus buy?

566
00:25:11,880 --> 00:25:13,590
It's just like, what do you have?

567
00:25:13,590 --> 00:25:17,490
Like are you building like a gigantic system and like you wanna do everything?

568
00:25:17,610 --> 00:25:19,530
Like I would rather like lean on like.

569
00:25:20,040 --> 00:25:20,760
Open source.

570
00:25:20,850 --> 00:25:23,580
What I have seen in my career in some way is

571
00:25:23,580 --> 00:25:26,515
that majority of the tech problem have been like.

572
00:25:27,135 --> 00:25:28,605
Solved in some way.

573
00:25:28,725 --> 00:25:30,945
So like you're gonna find the solutions that

574
00:25:30,945 --> 00:25:35,265
solve like 50, 60% out of your way of doing it.

575
00:25:35,655 --> 00:25:37,485
Don't rebuild it.

576
00:25:37,545 --> 00:25:40,905
Like if Kubernetes works for you, 80%, don't try to

577
00:25:40,910 --> 00:25:41,050
rebuild it.

578
00:25:41,050 --> 00:25:44,415
That's the rise of AI problem right there in a nutshell is

579
00:25:44,415 --> 00:25:47,895
that, well, I could just build my own custom solution on to

580
00:25:48,045 --> 00:25:51,285
out of spare, out of spare parts and that'll work and it will.

581
00:25:51,345 --> 00:25:54,255
Mostly for the exact use case you've defined and tested.

582
00:25:54,405 --> 00:25:57,975
As soon as the requirement changes, now you have a problem to work with.

583
00:25:58,125 --> 00:26:01,785
And for weird back of house single purpose apps, I do that all the time.

584
00:26:02,025 --> 00:26:04,755
But for stuff that matters, of course, I'm paying vendors.

585
00:26:04,785 --> 00:26:07,575
I pay for notion at work with a smile on my face

586
00:26:07,575 --> 00:26:09,555
for a bunch of reasons that should be obvious.

587
00:26:09,645 --> 00:26:12,975
I built my own newsletter publication system, rebuilt it finally the

588
00:26:12,975 --> 00:26:16,035
way I wanted to at the start of this year with the lessons learned.

589
00:26:16,035 --> 00:26:17,215
And this is the third generation of that's.

590
00:26:17,480 --> 00:26:17,930
System.

591
00:26:18,050 --> 00:26:21,200
It's much better than the previous generations, but I'm sure I'm

592
00:26:21,200 --> 00:26:24,050
gonna tear it down and replace it in a few years with something else.

593
00:26:24,050 --> 00:26:25,010
And that's okay.

594
00:26:25,370 --> 00:26:28,310
It's understand where the right approach is.

595
00:26:28,400 --> 00:26:31,790
Someone had a tweet a while back that it's interesting that Anthropic

596
00:26:31,850 --> 00:26:36,230
as a company uses a DP for payroll instead of revive coding their own.

597
00:26:36,320 --> 00:26:38,330
And the answer is, is because they're not insane.

598
00:26:38,420 --> 00:26:40,400
They understand that you're not just paying for a piece of

599
00:26:40,400 --> 00:26:43,760
software, you're paying for understanding the nuances of.

600
00:26:43,865 --> 00:26:47,795
Payroll law in a bunch of different jurisdictions in which you operate.

601
00:26:47,825 --> 00:26:52,085
Keeping up to date with legal changes and not having the Department of Labor

602
00:26:52,085 --> 00:26:55,355
kick your door off your a hinges three days after you miss a payroll run.

603
00:26:55,505 --> 00:26:56,525
It's the right move.

604
00:26:56,525 --> 00:26:58,925
It's not just the software, it's understanding

605
00:26:58,925 --> 00:27:00,470
the business context of what you're trying to do.

606
00:27:01,155 --> 00:27:01,725
A hundred percent.

607
00:27:01,754 --> 00:27:03,675
It's not because you can do it, you should do it.

608
00:27:03,675 --> 00:27:05,475
There's like a big, big app.

609
00:27:05,504 --> 00:27:09,195
Like if you all wanna try something out, if I wanna like build something

610
00:27:09,195 --> 00:27:12,585
really quickly, like have a demo, all of that kinda stuff, sure, go build it.

611
00:27:12,585 --> 00:27:14,024
Try like do whatever you want.

612
00:27:14,055 --> 00:27:16,245
When I think about what, when I think about long-term

613
00:27:16,245 --> 00:27:18,585
sustainability, like not everything is like rebuildable.

614
00:27:19,335 --> 00:27:23,055
Not because I can, I should, and this is where, where I stand by.

615
00:27:23,115 --> 00:27:26,024
Like if you solve the problem and then I look

616
00:27:26,024 --> 00:27:28,125
at like your solution and it, then it fits.

617
00:27:28,395 --> 00:27:29,085
Why not?

618
00:27:29,115 --> 00:27:33,255
Why not use this and add to it or like bake it into my like

619
00:27:33,255 --> 00:27:36,315
way of thinking rather than just like say, oh, it doesn't,

620
00:27:36,315 --> 00:27:39,045
it doesn't do all of the 10 things that I need to do.

621
00:27:39,105 --> 00:27:42,165
But it does eight, like does seven, it does five.

622
00:27:42,255 --> 00:27:43,365
It like it spilled.

623
00:27:43,485 --> 00:27:43,815
There's like.

624
00:27:44,360 --> 00:27:45,920
10 other people looking at it.

625
00:27:46,040 --> 00:27:48,500
'cause like think about it, like if I rely on the software,

626
00:27:48,500 --> 00:27:52,220
like let's pick any project in the open source in the ecosystem,

627
00:27:52,310 --> 00:27:55,310
and they're like, there's not always a single person use it.

628
00:27:55,430 --> 00:27:58,190
So there's many people use it, so someone has interest

629
00:27:58,190 --> 00:28:00,410
in doing that, but you build your own software.

630
00:28:00,705 --> 00:28:02,085
It's your only responsibilities.

631
00:28:02,085 --> 00:28:04,125
That's your things that you have to maintain.

632
00:28:04,304 --> 00:28:06,855
And saying yes to something means saying no to something else.

633
00:28:06,915 --> 00:28:07,725
Take your day job.

634
00:28:07,754 --> 00:28:09,165
You work at the New York Times.

635
00:28:09,405 --> 00:28:11,534
The New York Times does a bunch of different

636
00:28:11,534 --> 00:28:14,504
things Officially, I suppose you're a news outlet.

637
00:28:14,504 --> 00:28:17,835
Personally, I think that your job is to employ history's greatest monster,

638
00:28:18,014 --> 00:28:22,034
whoever it is that organizes and runs the connections puzzle every day, which

639
00:28:22,034 --> 00:28:25,879
vexes me like you would not freaking believe because I don't think in the right.

640
00:28:26,220 --> 00:28:29,160
Frame of reference sometimes, but at during, not through none

641
00:28:29,160 --> 00:28:32,190
of those perspectives is, oh, what does the New York Times do?

642
00:28:32,250 --> 00:28:32,880
That's right.

643
00:28:32,880 --> 00:28:34,410
You're a database company.

644
00:28:34,500 --> 00:28:36,330
You should build your own database.

645
00:28:36,630 --> 00:28:39,210
No, that is not where the value is.

646
00:28:39,300 --> 00:28:42,210
You have a website you should build and run your own web servers.

647
00:28:42,210 --> 00:28:43,890
That's something a fool would say.

648
00:28:44,070 --> 00:28:45,570
If I'm dealing with a bank.

649
00:28:46,180 --> 00:28:49,150
Handling the money, ensuring compliance, making sure

650
00:28:49,150 --> 00:28:51,250
that the, that the money is there when you say it is.

651
00:28:51,250 --> 00:28:52,420
That's the key job.

652
00:28:52,510 --> 00:28:56,350
An airline's job is to get people and planes and cargo from place to place.

653
00:28:56,380 --> 00:29:00,580
It is not to push the boundaries of computer science.

654
00:29:01,310 --> 00:29:04,100
Companies tend to lose sight of this, especially when engineers

655
00:29:04,100 --> 00:29:07,010
in some cases get carried with resume driven development.

656
00:29:07,130 --> 00:29:07,430
Yeah.

657
00:29:07,520 --> 00:29:12,020
That's where like scope and focus and specialty is

658
00:29:12,020 --> 00:29:14,420
one of the things that like anyone should look into.

659
00:29:14,570 --> 00:29:17,150
So I would rather like spend my time in my area of

660
00:29:17,150 --> 00:29:19,610
expertise, what I'm good at, like how I'm doing it.

661
00:29:19,790 --> 00:29:22,700
If I'm an engineer, if I wanna do a design, sure I

662
00:29:22,700 --> 00:29:25,400
can like do something quickly, but I don't necessarily

663
00:29:25,400 --> 00:29:27,920
have all of that understanding of how design work.

664
00:29:28,310 --> 00:29:30,889
Again, not because I can, I should.

665
00:29:30,980 --> 00:29:34,399
It's always like the idea of like, you should get to a point where

666
00:29:34,399 --> 00:29:37,580
you have an SM e. That's why they call it an SME in some way.

667
00:29:37,669 --> 00:29:39,290
That's where like people have studied things.

668
00:29:39,409 --> 00:29:42,800
If I'm asking for a serverless opinion in any way, I'm

669
00:29:42,800 --> 00:29:45,439
gonna go ask a serverless person who dealt with this.

670
00:29:45,445 --> 00:29:47,995
In a real production system, who knows when it

671
00:29:47,995 --> 00:29:50,425
breaks, who knows what are the bad things about it?

672
00:29:50,575 --> 00:29:52,465
Like a lot of people when we talk, say

673
00:29:52,465 --> 00:29:54,115
like Serverless is great, you can spin up.

674
00:29:54,115 --> 00:29:55,554
Sure, you can spin up.

675
00:29:55,615 --> 00:30:00,774
Have you ever run a serverless architecture that has like a thousand function?

676
00:30:00,955 --> 00:30:03,490
Let's talk about like how you govern all of them when you work together.

677
00:30:04,110 --> 00:30:05,100
That's a different story.

678
00:30:05,160 --> 00:30:09,480
Like that story like I saw in a demo, seeing like sure.

679
00:30:09,540 --> 00:30:10,380
A Lambda function.

680
00:30:10,380 --> 00:30:12,990
Now any function in any cloud system runs in like the

681
00:30:12,990 --> 00:30:15,510
matter of second ship, a container and it pops up.

682
00:30:15,840 --> 00:30:16,470
Great.

683
00:30:16,680 --> 00:30:19,139
Let's talk about how to govern this in a bigger system.

684
00:30:19,320 --> 00:30:20,460
That's a different story.

685
00:30:20,520 --> 00:30:24,720
So like that's why, again, back to my point of like when I give a talk,

686
00:30:24,780 --> 00:30:28,230
I talk about my experience, I talk about like the things that I explored.

687
00:30:28,350 --> 00:30:30,720
'cause like I have knowledge in that area.

688
00:30:30,720 --> 00:30:34,290
I have an understanding rather than just losing focus on what I'm trying to do.

689
00:30:34,290 --> 00:30:34,440
Right?

690
00:30:34,530 --> 00:30:37,825
I'm a former SRE and I have a radically different perspective.

691
00:30:38,395 --> 00:30:40,495
On environments depending on where they are.

692
00:30:40,764 --> 00:30:44,455
And I was always considerably one of the most stodgy, conservative

693
00:30:44,455 --> 00:30:48,175
curmudgeonly types when it came to things like databases and file systems.

694
00:30:48,175 --> 00:30:50,514
Because mistakes there are going to show.

695
00:30:50,725 --> 00:30:52,794
But in my test environment, ah, I have

696
00:30:52,794 --> 00:30:54,415
good backups of all the stuff I care about.

697
00:30:54,415 --> 00:30:55,375
Yeah, I'll go nuts.

698
00:30:55,375 --> 00:30:57,024
We'll do bleeding edge alpha thing.

699
00:30:57,024 --> 00:30:59,125
Oh, I guess that's why it's not GA yet.

700
00:30:59,155 --> 00:30:59,514
Whoops.

701
00:30:59,514 --> 00:31:00,580
Roll back and.

702
00:31:01,240 --> 00:31:03,220
I am fine with throwing things over the wall.

703
00:31:03,280 --> 00:31:07,720
I have a dedicated AWS account with no access to data that I have

704
00:31:07,720 --> 00:31:12,310
an EC2 box in upon which runs Claude Code in full permissions mode.

705
00:31:12,429 --> 00:31:15,399
It has a EC2 rule that gives its root, uh,

706
00:31:15,399 --> 00:31:17,679
administrative access to the entire AWS environment.

707
00:31:17,919 --> 00:31:21,370
It is called Superfund because it is both toxic and expensive and

708
00:31:21,399 --> 00:31:25,240
the only blast radius worst case to hear is it spikes my AWS bill.

709
00:31:25,485 --> 00:31:28,274
Which I can handle that if that's what comes down to it.

710
00:31:28,274 --> 00:31:30,915
Honestly, if I call in begging for forgiveness to the a Ws

711
00:31:30,915 --> 00:31:33,705
billing department, it'll become a company-wide holiday.

712
00:31:33,824 --> 00:31:35,324
Ladies and gentlemen, we got him.

713
00:31:35,504 --> 00:31:36,495
It'll, it'll be great.

714
00:31:36,885 --> 00:31:40,935
I think like the separation of concerns most of the time, like works in, in a

715
00:31:40,935 --> 00:31:44,564
lot of cases where like you need to understand where's your competency is at,

716
00:31:44,655 --> 00:31:44,955
right?

717
00:31:44,959 --> 00:31:46,695
Like, well, why not just do that in your production

718
00:31:46,695 --> 00:31:48,195
environment with like the customer database?

719
00:31:48,195 --> 00:31:49,514
Because I'm not insane.

720
00:31:49,514 --> 00:31:50,625
Thank you for asking.

721
00:31:51,165 --> 00:31:51,764
Exactly.

722
00:31:51,764 --> 00:31:53,955
Like that's why, like, uh, but like, it's also

723
00:31:53,955 --> 00:31:55,334
like, it's a, it's a situation where like.

724
00:31:55,680 --> 00:31:58,770
You should always think about like segregation, you should think about, like

725
00:31:58,770 --> 00:32:01,770
that's why like some people will like say, yeah, let's ship it to production.

726
00:32:01,770 --> 00:32:02,790
Have you tested it before?

727
00:32:02,805 --> 00:32:07,590
Oh, oh, I tested it, but like also like I can tell you a funny story, so.

728
00:32:08,280 --> 00:32:10,830
Some of the environments we shaped like in

729
00:32:10,830 --> 00:32:12,930
multiple places in multiple uh, environments.

730
00:32:13,110 --> 00:32:15,210
Like people are like, have you tested that before?

731
00:32:15,360 --> 00:32:16,320
Yes, I tested it.

732
00:32:16,500 --> 00:32:18,150
Have you tested on the same skill?

733
00:32:18,360 --> 00:32:18,840
No.

734
00:32:18,960 --> 00:32:21,810
Like the only test that in a specific environment.

735
00:32:21,870 --> 00:32:25,260
I was like, why you should always test with the same parameters like.

736
00:32:25,480 --> 00:32:28,810
Sometimes I, I worked in a company before where like,

737
00:32:28,899 --> 00:32:32,260
we're like, we're doing like some deployments and then one

738
00:32:32,260 --> 00:32:34,720
of the deployments we're like, oh, code is ready shipped.

739
00:32:34,720 --> 00:32:35,919
Everything is cool.

740
00:32:36,010 --> 00:32:38,050
Like it was a small company and so like

741
00:32:38,110 --> 00:32:40,000
all of a sudden shipped it to production.

742
00:32:40,320 --> 00:32:41,550
It doesn't work.

743
00:32:41,820 --> 00:32:45,540
It doesn't work because like you don't have the same parameters like that

744
00:32:45,540 --> 00:32:48,690
you are running this into, like, you're running one thing, like you're

745
00:32:48,690 --> 00:32:53,310
testing it with like curl like shipping, like a single a BI and then

746
00:32:53,310 --> 00:32:56,700
like you're testing it in production with sending like a 50,000 request.

747
00:32:56,790 --> 00:32:58,860
Like have you, have you done this?

748
00:32:59,040 --> 00:33:00,270
Oh, testing is real.

749
00:33:00,510 --> 00:33:00,899
Yeah.

750
00:33:00,960 --> 00:33:03,149
Met the standard expecta.

751
00:33:03,240 --> 00:33:04,415
Like all of these things like are.

752
00:33:04,965 --> 00:33:06,585
Actually like things that you have to think about or

753
00:33:06,585 --> 00:33:08,385
even canary deployment, because at some point

754
00:33:08,385 --> 00:33:10,455
of scale you cannot test at the same scale.

755
00:33:10,485 --> 00:33:13,995
Facebook was, they gave a lot of talks about this back when they had reasonable

756
00:33:13,995 --> 00:33:17,865
approaches to things and because they didn't have a spare billion users

757
00:33:17,865 --> 00:33:21,045
to run in the dev environment, so they started off by having the developer

758
00:33:21,045 --> 00:33:24,315
run it themselves and then a small gated list of internal use test users.

759
00:33:24,495 --> 00:33:27,435
And then it's like that effectively there were something like nine concentric

760
00:33:27,435 --> 00:33:31,785
circles from individual developer to the entire Facebook user base.

761
00:33:31,935 --> 00:33:35,235
And there was a scaled and measure, and they monitored the heck out of this.

762
00:33:35,235 --> 00:33:37,065
Like, Ooh, we're starting to see errors increase.

763
00:33:37,215 --> 00:33:38,385
Let's dial that back.

764
00:33:38,534 --> 00:33:42,675
Which works super well for Facebook, would work terribly for Stripe

765
00:33:42,735 --> 00:33:45,885
because every 500 error they get means someone didn't get paid.

766
00:33:45,885 --> 00:33:48,375
And that's a worse, that's a worse outcome than,

767
00:33:48,405 --> 00:33:50,355
oh, the cat picture didn't load fast enough.

768
00:33:50,565 --> 00:33:51,820
Yeah, that's, that's basically like.

769
00:33:52,504 --> 00:33:54,065
What they care about.

770
00:33:54,065 --> 00:33:57,845
Like some company cares about every single transaction, for example, like, and

771
00:33:57,845 --> 00:34:01,535
also like one of the thing that we, you mentioned here for strive, for example,

772
00:34:01,595 --> 00:34:05,165
like we, you don't, we don't know even like which transaction will fail.

773
00:34:05,225 --> 00:34:08,074
That might be like a very expensive transaction to fail and

774
00:34:08,074 --> 00:34:10,245
that will cause like the business to lose a lot of money.

775
00:34:10,755 --> 00:34:14,025
But like if someone didn't hit like on my comment on Facebook

776
00:34:14,085 --> 00:34:16,635
and then didn't work, like I'm not gonna get too offended, like

777
00:34:16,784 --> 00:34:16,995
Right.

778
00:34:16,995 --> 00:34:19,005
And there's also reputational damage too.

779
00:34:19,065 --> 00:34:21,975
I try to buy your book for $30.

780
00:34:21,975 --> 00:34:24,135
I send it and the transaction fails.

781
00:34:24,495 --> 00:34:28,425
Both you and I are gonna be upset with that, depending on how technical I am.

782
00:34:28,425 --> 00:34:31,574
And definitely for you, that's gonna flavor our impression of Stripe.

783
00:34:31,755 --> 00:34:32,955
It's okay.

784
00:34:32,955 --> 00:34:33,585
That's not great.

785
00:34:33,585 --> 00:34:34,514
It has to work.

786
00:34:34,574 --> 00:34:35,475
It must.

787
00:34:35,675 --> 00:34:38,165
Whereas with other, with other use cases, the, the

788
00:34:38,165 --> 00:34:40,055
restrictions are, are different, but the product that

789
00:34:40,055 --> 00:34:42,875
we're building is for business back of house users.

790
00:34:43,175 --> 00:34:45,875
Yes, we would like the site to be up when people are attempting

791
00:34:45,875 --> 00:34:49,715
to use it, but in the event that the site is down for an update

792
00:34:49,715 --> 00:34:52,010
or something for 20 minutes and has the maintenance page up it.

793
00:34:52,255 --> 00:34:52,765
It does not.

794
00:34:52,765 --> 00:34:54,055
It is not disastrous.

795
00:34:54,055 --> 00:34:58,315
It is not critical path for serving their customers that day.

796
00:34:58,615 --> 00:35:01,135
And I can see a future in which that potentially changes.

797
00:35:01,285 --> 00:35:04,255
In which case, our approach to uptime and responsibility

798
00:35:04,255 --> 00:35:06,865
and maintenance windows will no longer be a thing.

799
00:35:06,865 --> 00:35:09,535
We're going to be very cognizant of the needs of it.

800
00:35:09,835 --> 00:35:11,725
But not everything needs to be hyperscale.

801
00:35:11,725 --> 00:35:13,705
Not everything needs five nines of uptime.

802
00:35:14,305 --> 00:35:17,155
Understand the use case and the problem you're trying to solve for it.

803
00:35:17,275 --> 00:35:19,195
I'd rather just doing engineering fantasy.

804
00:35:19,754 --> 00:35:21,254
Build the thing that fits, solves the problem

805
00:35:21,435 --> 00:35:26,265
that the thing about yp sometimes, like strike me because like some uh, use

806
00:35:26,265 --> 00:35:29,835
cases I have seen in my best were like, talk about something like in consultant

807
00:35:29,835 --> 00:35:32,745
opportunity or like in anyone and say like, what's your system uptime?

808
00:35:32,745 --> 00:35:34,725
And he say like, oh, it's five nines.

809
00:35:34,815 --> 00:35:35,654
I was like, great.

810
00:35:35,714 --> 00:35:36,615
Why are you using?

811
00:35:36,915 --> 00:35:38,895
And I say like, this a BI is that a BS at ABIs at

812
00:35:39,134 --> 00:35:42,884
abi and 10 to one of them, like, oh, that's not five.

813
00:35:43,095 --> 00:35:45,105
And they're like, and it's critical for you.

814
00:35:45,254 --> 00:35:47,325
I'm like, yeah, but my system is five.

815
00:35:47,535 --> 00:35:47,924
I was like.

816
00:35:48,645 --> 00:35:52,575
How's that even work in the, like the, the back systems that you're

817
00:35:52,575 --> 00:35:56,805
using is not five nines, but like then you claim it's five nine.

818
00:35:56,865 --> 00:35:58,845
It, it's just like, it's, it's a very complex world.

819
00:35:58,875 --> 00:36:01,335
If you sincerely care about five nines of uptime on a

820
00:36:01,335 --> 00:36:04,305
service, you need to be in multiple regions to do it.

821
00:36:04,485 --> 00:36:07,185
Arguably multiple providers, though, I could be convinced of that.

822
00:36:07,190 --> 00:36:07,430
Otherwise.

823
00:36:08,134 --> 00:36:13,205
You cannot take third party dependencies because look, I can test in my account

824
00:36:13,205 --> 00:36:17,375
what happens if none of my stuff can reach S3 hypothetically, but I cannot test.

825
00:36:17,375 --> 00:36:20,345
What if my third party vendor dependencies can't reach

826
00:36:20,345 --> 00:36:24,305
S3 or their third party dependencies can't reach S3.

827
00:36:24,395 --> 00:36:27,754
The only way you test that is by S3 going down, which fortunately is not a

828
00:36:27,754 --> 00:36:32,915
common occurrence, but if you're serious about this must stay up at all times.

829
00:36:33,245 --> 00:36:36,365
You have to own so much of that availability piece yourself.

830
00:36:36,590 --> 00:36:40,100
Exactly, and just like where we have to think about like every beast that you

831
00:36:40,100 --> 00:36:43,820
put, so the more that like it's, it's all like at the end of the day, it's like.

832
00:36:44,520 --> 00:36:47,040
To me, like technology, like any other thing,

833
00:36:47,040 --> 00:36:48,960
it's like an architecture, it's a puzzle piece.

834
00:36:48,960 --> 00:36:50,850
It's like trade off somewhere.

835
00:36:50,850 --> 00:36:52,860
Like you get something, you lose something.

836
00:36:52,860 --> 00:36:55,380
It's not like, oh, hey, you have to get like an all

837
00:36:55,380 --> 00:36:57,780
optimum, we all strive for like the perfect system all

838
00:36:57,780 --> 00:37:00,600
over, but like, sure, you wanna build it, you wanna own it.

839
00:37:00,630 --> 00:37:03,300
Get your data center, get your stuff, make sure that

840
00:37:03,300 --> 00:37:05,700
you have redundancy, all of that kinda stuff at a cost.

841
00:37:05,910 --> 00:37:06,270
Sure.

842
00:37:06,270 --> 00:37:08,010
Or it goes the other way at a cost.

843
00:37:08,070 --> 00:37:11,560
Like it's, it's, it's one way of zr and then you build for what you need.

844
00:37:12,115 --> 00:37:14,575
And that's exactly what you have at the end

845
00:37:14,575 --> 00:37:16,105
of the day, but that's what I'm looking for.

846
00:37:16,105 --> 00:37:18,025
It's just like what is the right balance?

847
00:37:18,145 --> 00:37:21,445
I prefer sometimes to say like, these are, here are some

848
00:37:21,445 --> 00:37:24,265
Lego blocks, and then you build it, whatever like fits you.

849
00:37:24,265 --> 00:37:26,845
Like you want it tall, you want it short, you want it wide.

850
00:37:26,845 --> 00:37:28,075
You want it like large.

851
00:37:28,375 --> 00:37:31,855
What do you need is what you build based on your requirements.

852
00:37:32,305 --> 00:37:33,955
Here's the thing, your requirements.

853
00:37:34,275 --> 00:37:38,055
Usually change over time because this is what we have seen like today.

854
00:37:38,055 --> 00:37:40,125
You built, as you said, like I built for a single

855
00:37:40,125 --> 00:37:43,575
app tomorrow, 10 apps after tomorrow, a hundred apps.

856
00:37:43,635 --> 00:37:44,835
Like do you have that scale?

857
00:37:44,925 --> 00:37:46,725
Do you wanna get something to throw?

858
00:37:46,845 --> 00:37:49,035
And like you keep rearchitecturing every couple days.

859
00:37:49,080 --> 00:37:50,279
You want something plugable.

860
00:37:50,340 --> 00:37:53,190
That's why like usually like finding the right patterns.

861
00:37:53,310 --> 00:37:56,520
What other people have found, like, I think like a lot of people spend

862
00:37:56,520 --> 00:37:59,430
their times on Kubernetes figuring like, how did it make that work?

863
00:37:59,490 --> 00:38:00,540
It's, it's a spectrum.

864
00:38:00,540 --> 00:38:02,040
Like anything, there are trade-offs.

865
00:38:02,069 --> 00:38:04,880
The decisions you should make early on that will not hamstring you in the.

866
00:38:05,075 --> 00:38:05,615
Future.

867
00:38:05,705 --> 00:38:08,825
Almost every hyperscaler has had this problem before where

868
00:38:08,884 --> 00:38:11,975
we're just gonna build a small thing for back of house stuff.

869
00:38:11,975 --> 00:38:12,365
Great.

870
00:38:12,424 --> 00:38:15,395
We're gonna use the local time zone for the database entries.

871
00:38:15,544 --> 00:38:17,555
No, no, no, no, no.

872
00:38:17,555 --> 00:38:20,734
Talk to anyone who was at Google for about a decade and a half of

873
00:38:20,734 --> 00:38:24,095
time into there and use the phrase Google Standard time and watch

874
00:38:24,095 --> 00:38:28,240
them flinch because that is very painful to fix after the fact.

875
00:38:29,000 --> 00:38:30,950
It makes everything so much harder.

876
00:38:31,010 --> 00:38:35,510
So everything I build these days, even my dev box, it sits there running in UTC.

877
00:38:35,660 --> 00:38:38,180
If I want to know what time it is locally, great that

878
00:38:38,180 --> 00:38:40,790
my user account can change outta the presentation layer.

879
00:38:41,270 --> 00:38:41,780
Awesome.

880
00:38:41,780 --> 00:38:43,940
But the system itself must be UTC.

881
00:38:44,440 --> 00:38:46,000
That's where standards comes in.

882
00:38:46,150 --> 00:38:48,880
That's where like UTC is a common frame that like

883
00:38:48,880 --> 00:38:52,570
everyone's agree on and you convert based on your needs.

884
00:38:52,630 --> 00:38:56,320
Because I'm in whatever, I'm in New Jersey, someone else in California,

885
00:38:56,350 --> 00:39:00,340
we all can, we know what is a pattern in, but like if I start to ingest

886
00:39:00,640 --> 00:39:05,410
in my database data coming from like all local zones now, like I have

887
00:39:05,410 --> 00:39:08,950
and I have seen it in apps where like I go into an app and then I look

888
00:39:08,950 --> 00:39:13,090
at it and like, when is the last user has visited this app Tomorrow?

889
00:39:13,150 --> 00:39:13,480
I was like.

890
00:39:13,830 --> 00:39:17,970
What, what, what is today, like today is like absurd, but like

891
00:39:17,970 --> 00:39:21,300
someone visited the app in tomorrow, like how does this even happen?

892
00:39:21,360 --> 00:39:25,170
And because like they inserted their time zone from their local machine to

893
00:39:25,170 --> 00:39:28,680
the system and then you have a got wrong representation because now it's.

894
00:39:29,174 --> 00:39:30,315
This Edge case,

895
00:39:30,375 --> 00:39:35,535
the X-K-C-D-R-S-S feed always goes into the past for whatever reason.

896
00:39:35,535 --> 00:39:38,384
By about, I think eight and a half hours from your UTC

897
00:39:38,384 --> 00:39:41,145
time, something is not right, so it always pops up.

898
00:39:41,205 --> 00:39:44,714
I had to scroll back to find it yesterday when I was building out that EKS

899
00:39:44,714 --> 00:39:48,645
cluster with open tofu, I had Claude code do most of the, uh, to terraform slash

900
00:39:48,645 --> 00:39:53,865
tofu code, and I had to correct it where it's first, it put it in the 10.0.

901
00:39:54,015 --> 00:39:54,345
Great.

902
00:39:54,345 --> 00:39:58,275
That's gonna conflict with something somewhere because everyone uses that.

903
00:39:58,569 --> 00:39:59,890
In this case, the staging environment.

904
00:39:59,890 --> 00:40:00,279
Great.

905
00:40:00,279 --> 00:40:01,330
Put it somewhere else.

906
00:40:01,630 --> 00:40:03,730
Then it built a bunch of, uh, for the subnets

907
00:40:03,759 --> 00:40:05,860
slash 20 fours right next to each other.

908
00:40:06,040 --> 00:40:09,910
No, because when you run more than 255 containers,

909
00:40:09,910 --> 00:40:13,450
which can happen, sorry, 253 That's right.

910
00:40:13,450 --> 00:40:15,335
You've broadcast a network as well, that, that.

911
00:40:16,110 --> 00:40:17,670
Oh and the dns, which you can't get rid of

912
00:40:17,670 --> 00:40:19,350
inside of the subnet two, so that drops it.

913
00:40:19,350 --> 00:40:19,920
Two more.

914
00:40:20,160 --> 00:40:22,980
Great point being is at above a certain threshold, you have

915
00:40:22,980 --> 00:40:27,540
to renumber and that is painful build room to expand without

916
00:40:27,540 --> 00:40:30,270
having to move things around and you'll be much happier for it.

917
00:40:30,690 --> 00:40:33,029
Yeah, that is a problem that I had to solve in

918
00:40:33,029 --> 00:40:35,220
the many occasions and there's always a solution.

919
00:40:35,770 --> 00:40:37,210
It's like use a secondary sider.

920
00:40:37,299 --> 00:40:37,840
It's just like,

921
00:40:37,930 --> 00:40:39,400
just use IPV six.

922
00:40:39,400 --> 00:40:41,440
It's like grownups are speaking please.

923
00:40:41,560 --> 00:40:42,160
Sure.

924
00:40:42,190 --> 00:40:42,460
Yeah.

925
00:40:42,460 --> 00:40:45,580
Like this is, strikes me as like one of the most

926
00:40:45,640 --> 00:40:48,370
bogus standards that I have seen over so far.

927
00:40:48,549 --> 00:40:51,549
Not for anything, but because like we can't yet agree on it.

928
00:40:51,640 --> 00:40:54,370
Like I think like we are still living in the IPV four

929
00:40:54,400 --> 00:40:57,340
world, but like we want more, but we cannot get more.

930
00:40:57,340 --> 00:41:00,820
But not everything supports IV six, but so we are stuck in IV four.

931
00:41:00,910 --> 00:41:01,445
What's your take on it?

932
00:41:01,964 --> 00:41:04,095
IPV six is gonna save us all.

933
00:41:04,095 --> 00:41:06,165
They've been saying this since I was a child, and

934
00:41:06,165 --> 00:41:08,685
they'll be selling it to my grandkids as well.

935
00:41:08,865 --> 00:41:11,805
It's not a problem that is top of mind for anyone except

936
00:41:11,805 --> 00:41:15,194
the, the salt of the earth folk who keep the internet moving.

937
00:41:15,404 --> 00:41:19,995
So we are going to continue to ignore it until we can't anymore.

938
00:41:20,295 --> 00:41:22,035
And then don't worry, the AI will fix it.

939
00:41:22,530 --> 00:41:25,470
Yeah, AI will fix a lot of things and until like robots will not

940
00:41:25,470 --> 00:41:28,320
get, be able to get IBS and we are gonna be all stuck in that world.

941
00:41:28,350 --> 00:41:28,860
Exactly.

942
00:41:28,980 --> 00:41:30,900
So I wanna thank you for taking the time to speak with me.

943
00:41:30,960 --> 00:41:33,240
If people wanna learn more about what you're up to and

944
00:41:33,420 --> 00:41:35,310
how you view the world and catch your next conference

945
00:41:35,310 --> 00:41:37,800
talk, where's the best place for them to find you?

946
00:41:37,950 --> 00:41:39,180
Best place is linked in.

947
00:41:39,180 --> 00:41:41,700
That's where like I usually stay most up to date.

948
00:41:41,730 --> 00:41:43,740
If anyone wanna hit me on email, like they

949
00:41:43,740 --> 00:41:46,080
will find my links and all of my contact info.

950
00:41:46,080 --> 00:41:47,100
But if you wanna need.

951
00:41:47,355 --> 00:41:51,555
Anything or just check where I'm going next, like I'm going to Q coupon

952
00:41:51,555 --> 00:41:55,455
and Mr. Dam next month, like where I'm doing like all of the things.

953
00:41:55,515 --> 00:41:55,965
Sure.

954
00:41:56,055 --> 00:41:56,355
Yeah.

955
00:41:56,535 --> 00:41:57,435
LinkedIn is a way,

956
00:41:57,495 --> 00:42:00,165
I wanna say that maybe the first time in history that, uh, the,

957
00:42:00,315 --> 00:42:04,395
the phrase LinkedIn is the best place has ever been uttered because

958
00:42:04,395 --> 00:42:08,175
that is just, it is certainly a place I'm there a lot more myself.

959
00:42:08,175 --> 00:42:10,815
And we will of course put links to that into the show notes.

960
00:42:11,360 --> 00:42:13,850
Ahmed, thank you so much for being so generous with your time.

961
00:42:13,850 --> 00:42:15,170
I deeply appreciate it.

962
00:42:15,259 --> 00:42:15,950
Thank you, Corey.

963
00:42:15,950 --> 00:42:17,214
I really appreciate having.

964
00:42:18,015 --> 00:42:21,645
Me here and like looking forward to see you in many in-person events.

965
00:42:21,735 --> 00:42:22,455
Oh, I'll be there.

966
00:42:22,725 --> 00:42:25,545
Ahmed Babar, principal engineer at the New York Times,

967
00:42:25,545 --> 00:42:28,755
and AWS Community Hero and Cloud native ambassador.

968
00:42:28,755 --> 00:42:30,975
We're just stacking up the accomplishments these days.

969
00:42:31,305 --> 00:42:33,615
I am cloud economist, Corey Quinn, and.

970
00:42:33,705 --> 00:42:35,235
This is screaming in the cloud.

971
00:42:35,505 --> 00:42:37,935
If you've enjoyed this podcast, please leave a five

972
00:42:37,935 --> 00:42:40,215
star review on your podcast platform of choice.

973
00:42:40,424 --> 00:42:44,025
Whereas if you hated this podcast, please leave a five star review on your

974
00:42:44,025 --> 00:42:48,075
podcast platform of choice, along with an angry, insulting comment that I

975
00:42:48,075 --> 00:42:52,125
won't ever see because that podcast platform of choice runs on somebody's home.

976
00:42:52,125 --> 00:42:53,085
K three s cluster.