1
00:00:00,000 --> 00:00:11,120
Welcome to Fork Around and Find Out, the podcast about building, running, and maintaining

2
00:00:11,120 --> 00:00:15,920
software and systems.

3
00:00:15,920 --> 00:00:22,480
Hello, and welcome back to Fork Around and Find Out.

4
00:00:22,480 --> 00:00:26,760
I am your host, Justin Garrison, and with me today is Autumn Nash, as always.

5
00:00:26,760 --> 00:00:30,360
How's it going, Autumn?

6
00:00:30,360 --> 00:00:31,360
Surprising?

7
00:00:31,360 --> 00:00:32,360
Like, I don't know.

8
00:00:32,360 --> 00:00:33,360
The silence, I know.

9
00:00:33,360 --> 00:00:34,360
I know.

10
00:00:34,360 --> 00:00:35,600
There's a lot going on.

11
00:00:35,600 --> 00:00:40,280
Your daily distraction, or your monthly distraction from all the other bad news.

12
00:00:40,280 --> 00:00:43,600
Let's talk about outages and software.

13
00:00:43,600 --> 00:00:46,040
It seems so much better right now.

14
00:00:46,040 --> 00:00:50,720
It's just a lot, okay?

15
00:00:50,720 --> 00:00:55,240
Joining us on the podcast today, we have Ellie Ford, staff software engineer at DevZero.

16
00:00:55,240 --> 00:00:56,240
Thanks for coming on, Ellie.

17
00:00:56,720 --> 00:00:57,720
Of course.

18
00:00:57,720 --> 00:00:59,320
Thank you for inviting me on.

19
00:00:59,320 --> 00:01:06,400
I absolutely invited you on because your blue sky feed was for a time phenomenal for giving

20
00:01:06,400 --> 00:01:10,640
me continued insights into what's going on in data centers and hardware, because I really

21
00:01:10,640 --> 00:01:11,640
kind of miss it.

22
00:01:11,640 --> 00:01:15,840
Like, there's a lot of things that were just like, if you stay away from it for one or

23
00:01:15,840 --> 00:01:17,760
two cycles, like, you just, it's gone.

24
00:01:17,760 --> 00:01:20,400
Like, you just, you don't know anything that's going on anymore.

25
00:01:20,400 --> 00:01:24,160
You go to the cloud for four or five years and you're like, well, this must be state

26
00:01:24,160 --> 00:01:25,160
of the art, right?

27
00:01:25,200 --> 00:01:26,200
No.

28
00:01:26,200 --> 00:01:30,680
Like this is like the amount of performance and price per watt and everything else you

29
00:01:30,680 --> 00:01:35,640
can get by just buying some hardware is really changed over over time.

30
00:01:35,640 --> 00:01:39,480
Yeah, I think like one of the things that has been the most interesting is watching

31
00:01:39,480 --> 00:01:44,640
like the blue sky, you know, the company behind blue sky, like build their own infra and build

32
00:01:44,640 --> 00:01:48,400
it for kind of, you know, a tremendous scale.

33
00:01:48,400 --> 00:01:53,320
They are one of the biggest proponents of like hosting it your own because they too have

34
00:01:53,360 --> 00:01:58,400
realized and done the math and they're like, you know, to run this on AWS and with all

35
00:01:58,400 --> 00:02:04,040
of the egress pricing and all of, like, like all of the miscellaneous Bezos tax that you

36
00:02:04,040 --> 00:02:09,560
pay for, like it is so tremendously cheap to actually just go out and buy hardware and

37
00:02:09,560 --> 00:02:13,640
depreciate it over like seven years and just like stick it in a colo, right?

38
00:02:13,640 --> 00:02:17,280
It doesn't matter if you have like remote hands and like you have to pay for like people

39
00:02:17,280 --> 00:02:18,280
to staff it.

40
00:02:18,280 --> 00:02:22,960
It's just, it is so tremendously cheaper and you get more performance and you get more

41
00:02:23,000 --> 00:02:29,000
control and kind of at the, you know, you trade that off with like you have to do more

42
00:02:29,000 --> 00:02:30,000
capacity planning.

43
00:02:30,000 --> 00:02:33,800
You have to think more through like, okay, you know, we can only handle so much load

44
00:02:33,800 --> 00:02:37,320
on this hardware, but then you kind of get into things like performance tuning and things

45
00:02:37,320 --> 00:02:40,240
like that, which is so much fun.

46
00:02:40,240 --> 00:02:46,920
I mean, I'm a big proponent of people should budget there, like plan to some capacity and

47
00:02:46,920 --> 00:02:49,080
just make a budget and say, I'm not going over this.

48
00:02:49,080 --> 00:02:55,760
And that's like the cloud in general with infinite, infinite scaling credit card bills

49
00:02:55,760 --> 00:02:59,320
is more along the lines of not infinite scaling infrastructure, but you can spend as much

50
00:02:59,320 --> 00:03:02,600
money as you want on anything and never have a plan for it.

51
00:03:02,600 --> 00:03:08,800
There was, so we've been doing a lot of Kubernetes optimization work at my work lately.

52
00:03:08,800 --> 00:03:15,960
And there was a study by DataDog, I think, that showed that I think 80% of containerized

53
00:03:15,960 --> 00:03:19,040
workloads are over provisioned, right?

54
00:03:19,040 --> 00:03:25,400
So like, and it's, it has been crazy, like doing metrics and doing like a detailed rigorous

55
00:03:25,400 --> 00:03:28,760
analysis of like how much waste is actually happening inside of Kubernetes environments.

56
00:03:28,760 --> 00:03:30,560
And it's insane.

57
00:03:30,560 --> 00:03:32,600
And you're paying for all of that too.

58
00:03:32,600 --> 00:03:33,600
Is that new though?

59
00:03:33,600 --> 00:03:34,600
Not really.

60
00:03:34,600 --> 00:03:40,280
Like we used to buy servers and they'd sit there, you know, 10% utilization maybe.

61
00:03:40,280 --> 00:03:43,440
And then we'd like, oh, we need to utilize this more or less use VMs.

62
00:03:43,440 --> 00:03:46,800
And then we had a bunch of VMs that were 10% utilized and then we put it in containers

63
00:03:46,800 --> 00:03:48,360
and those are all, you know.

64
00:03:48,360 --> 00:03:51,520
In every case, I feel like we're just like, like try to densely pack these things more

65
00:03:51,520 --> 00:03:52,520
and more.

66
00:03:52,520 --> 00:03:53,520
And at some point you hide it, right?

67
00:03:53,520 --> 00:03:56,360
We're like, VMware had a memory sharing, right?

68
00:03:56,360 --> 00:04:00,120
You're like, I got 20 VMs on this, but I'm only using the RAM from one of them because

69
00:04:00,120 --> 00:04:03,640
it's all doing the same thing.

70
00:04:03,640 --> 00:04:05,600
Is the waste, is the waste a bad thing though?

71
00:04:05,600 --> 00:04:10,400
Like, I mean, so my background is, is a SRE, right?

72
00:04:10,400 --> 00:04:18,000
So I typically look at waste as capacity rather than actual waste, which very much flies contrary

73
00:04:18,000 --> 00:04:23,320
with like what the bean counters like to think and everything like that.

74
00:04:23,320 --> 00:04:28,880
Especially once you're talking about, you know, like reducing tail latency and like

75
00:04:28,880 --> 00:04:33,640
meeting strict SLAs, it is so much harder to do that when you are on a system that is

76
00:04:33,640 --> 00:04:35,560
like fully loaded.

77
00:04:35,560 --> 00:04:43,000
And one of the things that's been really interesting is like the Linux kernel isn't really well

78
00:04:43,000 --> 00:04:47,960
designed to handle large amounts of load, right?

79
00:04:47,960 --> 00:04:51,840
You think of things like the out of memory killer, the out of memory killer wants to

80
00:04:51,840 --> 00:04:56,840
preserve kernel functionality rather than user space functionality.

81
00:04:56,840 --> 00:05:01,160
So you will have the out of memory.

82
00:05:01,160 --> 00:05:05,400
I have a four by four that says OOM on it for my, it's my, it's my prop for out of memory

83
00:05:05,400 --> 00:05:06,400
killers.

84
00:05:06,400 --> 00:05:10,160
Yeah, it's just so funny.

85
00:05:10,160 --> 00:05:14,600
Like you will have to go out and you will like to make any system functional like Facebook

86
00:05:14,600 --> 00:05:16,080
has found this out the hard way.

87
00:05:16,080 --> 00:05:20,840
You have to have an early OOM daemon that kills processes before it even starts to trigger

88
00:05:20,840 --> 00:05:25,160
the kernel out of memory reaper because it will just like lock up that whole system and

89
00:05:25,160 --> 00:05:26,720
you'll have to power cycle it.

90
00:05:26,720 --> 00:05:32,320
Yeah, the work Facebook and Meta has been doing on a user space OOM killer is fascinating

91
00:05:32,320 --> 00:05:36,520
to me because it just shows that even though the kernel has all the power, it doesn't have

92
00:05:36,520 --> 00:05:40,000
all the context and information for like what's actually important.

93
00:05:40,000 --> 00:05:43,040
And we try to do that on the operating system I work on where it's like, how do we protect

94
00:05:43,040 --> 00:05:44,480
the things that we want to protect?

95
00:05:44,480 --> 00:05:46,680
It's like, we can't just do it in kernel.

96
00:05:46,680 --> 00:05:50,080
Like we have to have something that knows what's actually going on.

97
00:05:50,080 --> 00:05:55,000
Yeah, like you have like niceness values, which are supposed to do that.

98
00:05:55,000 --> 00:05:59,520
But if you've ever looked on like any regular Linux system, you'll see that like some processes

99
00:05:59,520 --> 00:06:04,760
are set as negative 20, which are like never kill and then others are like zero and like

100
00:06:04,760 --> 00:06:07,820
no one actually uses these kernel tunables.

101
00:06:07,820 --> 00:06:13,300
So you get, you quite often get stuck in this situation where you just run into out

102
00:06:13,300 --> 00:06:15,980
of memory and you can't do anything about it, right?

103
00:06:15,980 --> 00:06:18,900
So what sort of performance tuning have you been doing on Kubernetes?

104
00:06:18,900 --> 00:06:19,900
Fine.

105
00:06:19,900 --> 00:06:25,020
You know, you're just trying to eliminate the over provisioning or is it a latency or

106
00:06:25,020 --> 00:06:26,020
what is it?

107
00:06:26,020 --> 00:06:27,820
Not necessarily latency.

108
00:06:27,820 --> 00:06:33,020
So before we started doing the cost optimization work, you know, startups, we pivot all the

109
00:06:33,020 --> 00:06:34,020
time.

110
00:06:34,020 --> 00:06:36,060
We used to do cloud development environments before that.

111
00:06:36,060 --> 00:06:42,660
And a lot of the people at Dev Zero have really like they have worked on developer environments

112
00:06:42,660 --> 00:06:43,660
at Uber.

113
00:06:43,660 --> 00:06:45,260
We have people from Facebook.

114
00:06:45,260 --> 00:06:47,020
We have people from Carta.

115
00:06:47,020 --> 00:06:49,860
A lot of people who have done these developer environments.

116
00:06:49,860 --> 00:06:54,260
And one of the things that we were trying to do is like trying to make it snappy and

117
00:06:54,260 --> 00:06:58,820
responsive and actually functional, right?

118
00:06:58,820 --> 00:07:03,060
We wanted to do everything that an EC2 virtual machine could do, but then better and on your

119
00:07:03,060 --> 00:07:07,540
own infrastructure, which is not particularly easy.

120
00:07:07,540 --> 00:07:11,100
And it was in kind of like a different vein from like what Coder and everything does because

121
00:07:11,100 --> 00:07:15,260
Coder, like to their credit, it's more or less just the Terraform wrapper.

122
00:07:15,260 --> 00:07:18,740
It just provisions that environment for you and that does some like initial setup.

123
00:07:18,740 --> 00:07:21,260
It was, we were a lot more controlled.

124
00:07:21,260 --> 00:07:29,980
There were, I introduced micro VMs into the mix, which made things chaotic and so fun.

125
00:07:29,980 --> 00:07:32,980
I had to debug multiple hypervisor bugs.

126
00:07:32,980 --> 00:07:37,180
It was so gory.

127
00:07:37,180 --> 00:07:43,100
And you know, developers, we were trying to also solve like for developer efficiency

128
00:07:43,100 --> 00:07:47,820
at the same time and respond like responsiveness and everything like that, which basically

129
00:07:47,820 --> 00:07:52,100
meant that like we took it like Ubuntu distribution.

130
00:07:52,100 --> 00:07:54,900
We pinned the kernel at a specific version.

131
00:07:54,900 --> 00:07:59,260
We made our own kernel config with our own options to like slim it down as much as possible

132
00:07:59,260 --> 00:08:01,540
to reduce startup time.

133
00:08:01,540 --> 00:08:06,420
And then we had a custom PID one, like PID one that would run before anything else in

134
00:08:06,420 --> 00:08:11,700
the container that would copy all of the contents over to a persistent block device.

135
00:08:11,700 --> 00:08:14,220
It was, it was batshit crazy.

136
00:08:14,220 --> 00:08:18,220
I think that's kind of the struggle though.

137
00:08:18,220 --> 00:08:24,020
People want to create Linux distributions that have the least amount of what you need,

138
00:08:24,020 --> 00:08:29,220
you know, to do it because for one, it makes it more secure, two, it makes it faster.

139
00:08:29,220 --> 00:08:35,100
But try to figure out exactly what you need to be usable as the hard part balancing that

140
00:08:35,100 --> 00:08:38,700
and actually being usable.

141
00:08:38,700 --> 00:08:41,100
No, absolutely.

142
00:08:41,100 --> 00:08:43,340
It's, it is so much back and forth.

143
00:08:43,340 --> 00:08:45,340
It is, it is tuning.

144
00:08:45,340 --> 00:08:46,840
It is trial and error.

145
00:08:46,840 --> 00:08:51,500
It is, it is like trying 15 different kernel configs to find out which one works, which

146
00:08:51,500 --> 00:08:52,500
one doesn't.

147
00:08:52,860 --> 00:08:59,780
It is, it is an egregious amount of time and I have so, so much newfound respect for people

148
00:08:59,780 --> 00:09:05,540
who maintain distributions after that because I was like, this is, you could probably spend

149
00:09:05,540 --> 00:09:08,820
the rest of your life just tuning the Linux kernel.

150
00:09:08,820 --> 00:09:13,500
And not just that, but like Linux users are very specific about what they want.

151
00:09:13,500 --> 00:09:14,500
You know what I mean?

152
00:09:14,500 --> 00:09:15,700
Like it's, it's its own religion.

153
00:09:15,700 --> 00:09:16,700
Like people.

154
00:09:16,700 --> 00:09:17,700
You calling me out?

155
00:09:17,700 --> 00:09:18,700
What is this?

156
00:09:18,700 --> 00:09:19,700
Yes.

157
00:09:19,700 --> 00:09:20,700
Yes, I'm calling you out, sir.

158
00:09:21,100 --> 00:09:22,100
What do you mean?

159
00:09:23,500 --> 00:09:26,700
I got a new fun Linux distro this week was it Rocknix?

160
00:09:26,700 --> 00:09:30,420
Justin, every other week, he's like, I found a new distro and this is how I'm going to

161
00:09:30,420 --> 00:09:32,180
hurt my own feelings this week.

162
00:09:32,180 --> 00:09:34,740
Like, why are you like this?

163
00:09:34,740 --> 00:09:38,620
It's like you like paint or something for you and all our listeners.

164
00:09:38,620 --> 00:09:41,460
If you haven't compiled the Linux kernel the last couple of weeks, maybe you should.

165
00:09:41,460 --> 00:09:42,460
Maybe you should try.

166
00:09:42,460 --> 00:09:43,460
You see this?

167
00:09:43,460 --> 00:09:45,820
He's like, call these Linux isn't enough.

168
00:09:45,820 --> 00:09:48,820
I have to go with her.

169
00:09:48,820 --> 00:09:49,820
It's okay.

170
00:09:49,940 --> 00:09:51,440
I'm going to in high school.

171
00:09:51,440 --> 00:09:57,580
So just started out like, yeah, choosing violence, just choosing violence.

172
00:09:57,580 --> 00:10:02,540
Like, like I was, I was sitting in like AP human geography and I was compiling Firefox

173
00:10:02,540 --> 00:10:07,100
from source and like, laptop would go to a hundred percent, fans would kick on.

174
00:10:07,100 --> 00:10:09,180
It was, it was a sight to behold.

175
00:10:09,180 --> 00:10:12,580
I feel like you were doing the right thing because I was so bored in high school.

176
00:10:12,580 --> 00:10:15,380
At least she found something that was like fun to do.

177
00:10:15,380 --> 00:10:17,020
Like I was just sitting there doodling.

178
00:10:17,020 --> 00:10:18,020
I don't know.

179
00:10:18,580 --> 00:10:23,580
I think it's like, it's also hard too because there's no like, I think a lot of the other

180
00:10:23,580 --> 00:10:30,300
open source projects that everybody use have this like one united overall project and they

181
00:10:30,300 --> 00:10:34,780
might kind of branch off, but Linux distributions are different.

182
00:10:34,780 --> 00:10:40,620
Like there are so many sources of truth and there's so many like, there's so many different

183
00:10:40,620 --> 00:10:46,100
ways where people will pick the packages they want in their distribution and a lot of open

184
00:10:46,100 --> 00:10:50,980
source and the beauty of open source is maintaining things with a community, right?

185
00:10:50,980 --> 00:10:56,460
So you're not maintaining it by yourself and kind of like having that continuity over the

186
00:10:56,460 --> 00:10:59,140
packages that are built.

187
00:10:59,140 --> 00:11:05,340
And then trying to figure out what you want to continue to have is hard.

188
00:11:05,340 --> 00:11:09,740
It's even like, like when you're building an end user application for a Linux system,

189
00:11:09,740 --> 00:11:12,740
it's like, what versions of glibc do you need to support?

190
00:11:12,740 --> 00:11:13,740
How do you want to package it?

191
00:11:13,740 --> 00:11:14,740
Do you want to use RPM?

192
00:11:14,740 --> 00:11:15,740
Do you want to use dead?

193
00:11:16,380 --> 00:11:21,780
I feel like I have PTSD this week, good lord.

194
00:11:21,780 --> 00:11:29,220
I have learned the correct way is to statically link everything to Musil Libc and ship people's

195
00:11:29,220 --> 00:11:32,700
static binaries because everyone will just figure it out if you give them a static binary

196
00:11:32,700 --> 00:11:34,020
that just works on their system.

197
00:11:34,020 --> 00:11:39,820
But Linux distribution, like okay, when you're, Linux users are a special like, you have to

198
00:11:39,820 --> 00:11:40,820
know your customer, right?

199
00:11:40,820 --> 00:11:45,300
Like when you're building anything, you have to know your customer and like something about

200
00:11:45,340 --> 00:11:50,820
developers, especially Linux developers are like, I want to have all like, it's like people

201
00:11:50,820 --> 00:11:55,860
that like, there's people that use Macs in real life or some sort of like operating system

202
00:11:55,860 --> 00:11:58,660
in real life, but they're Linux developers, right?

203
00:11:58,660 --> 00:12:02,420
And then you've got the people who have to have Linux on everything, Justin.

204
00:12:02,420 --> 00:12:07,340
And then like, like, you know what I mean, like they're different flavors of like they

205
00:12:07,340 --> 00:12:14,100
want all the knobs, or I want it to be bare metal, like, and just do only what I want

206
00:12:14,180 --> 00:12:16,620
it to do and have nothing else, you know what I mean?

207
00:12:16,620 --> 00:12:19,500
Like, so it's like, what is your distribution for?

208
00:12:19,500 --> 00:12:21,740
Is it for the people that want just peer secure?

209
00:12:21,940 --> 00:12:27,700
There's so many different levels and you're building for like, who are, who's your audience?

210
00:12:27,900 --> 00:12:30,180
You know, like, if, yeah, absolutely.

211
00:12:30,220 --> 00:12:35,420
And that's like, like, that's why if I was to tell anyone, like, like any advice

212
00:12:35,420 --> 00:12:38,860
from building cloud developer environments for like the past, like our company's been

213
00:12:38,860 --> 00:12:40,380
doing it for the past three years, right?

214
00:12:40,660 --> 00:12:43,940
If there's anything we've learned, it's like, don't do it, don't do it.

215
00:12:43,980 --> 00:12:45,620
It's so bad, so awful.

216
00:12:45,620 --> 00:12:46,620
It's kind of fun too, though.

217
00:12:48,620 --> 00:12:52,220
Like when you were talking earlier about how you like, and I put this like cloud

218
00:12:52,220 --> 00:12:55,260
vm and I did this and I was just like, it's because you like chaotic things.

219
00:12:55,260 --> 00:13:01,260
Like I think we all have a little bit of like, you just kind of like the fire a little bit.

220
00:13:01,260 --> 00:13:05,060
And then you have to figure out like how much you're like risk tolerances for, you know,

221
00:13:05,060 --> 00:13:08,740
like, but another thing that you mentioned that I thought was interesting is you

222
00:13:08,740 --> 00:13:11,380
mentioned Colos and I've talked to people at Blue Sky who do that.

223
00:13:11,380 --> 00:13:14,820
And it's just, I don't think people realize that that's the intermediate, right?

224
00:13:14,820 --> 00:13:18,140
Like you have a cloud where you completely don't know what's going on.

225
00:13:18,140 --> 00:13:19,380
You have nothing to do with it.

226
00:13:19,940 --> 00:13:24,540
You've got Colos where you buy all the hardware, you can depreciate it, you ship

227
00:13:24,540 --> 00:13:26,420
it, and it's cheaper, right?

228
00:13:26,460 --> 00:13:30,700
But you still have some kind of like, they're your VMs, but somebody else

229
00:13:30,700 --> 00:13:31,580
may be maintaining it.

230
00:13:31,580 --> 00:13:33,020
You might have to pay for hands, whatever.

231
00:13:33,340 --> 00:13:36,700
And then you have completely on-prem, you are writing, running a data center.

232
00:13:36,700 --> 00:13:41,140
And I feel like we talk about on-prem, we talk about cloud, but nobody talks

233
00:13:41,140 --> 00:13:44,860
about Colos and how, like, because we're talking about on-prem, like everyone's

234
00:13:44,860 --> 00:13:45,540
going back to on-prem.

235
00:13:45,540 --> 00:13:48,380
People are not going back to truly writing your own data centers.

236
00:13:48,380 --> 00:13:48,820
Don't lie.

237
00:13:48,820 --> 00:13:49,660
Like, you know what I mean?

238
00:13:50,140 --> 00:13:54,620
Like people keep writing these articles for spicy, like takes and I'm like,

239
00:13:54,620 --> 00:13:56,700
bullshit, like that's not what we're doing right now.

240
00:13:56,740 --> 00:14:01,020
Like people are going to the middle option of like, not today Satan.

241
00:14:01,020 --> 00:14:05,540
I don't want to go back to holding a pager myself, you know, like, but buying

242
00:14:05,540 --> 00:14:06,900
my own hardware is cheaper.

243
00:14:06,900 --> 00:14:10,300
So like, I love that you brought that up because I feel like we, like.

244
00:14:10,780 --> 00:14:14,100
People are writing a lot of articles to be spicy, right?

245
00:14:14,100 --> 00:14:17,780
And like there is value in it being cheaper, but don't lie.

246
00:14:17,780 --> 00:14:19,180
Nobody's doing that kind of work.

247
00:14:19,300 --> 00:14:22,700
And the fact that we like, and that's the wild thing, because all of a sudden we

248
00:14:22,700 --> 00:14:27,180
now need DBAs, we need like people that run data centers.

249
00:14:27,180 --> 00:14:28,860
And not everybody has that talent.

250
00:14:28,860 --> 00:14:34,020
And most of the people that do are these huge companies that do nothing, but, you

251
00:14:34,020 --> 00:14:37,740
know, run people's Colos or the people that are, you know, have these big data

252
00:14:37,740 --> 00:14:38,220
centers.

253
00:14:38,340 --> 00:14:43,100
I find that really fascinating where the, the people that are making on-prem and

254
00:14:43,100 --> 00:14:45,340
data centers sound so cool are the cloud providers.

255
00:14:46,020 --> 00:14:49,580
It's like literally Amazon's like, we're making the best, biggest, coolest

256
00:14:49,580 --> 00:14:51,380
data center ever existed.

257
00:14:51,660 --> 00:14:53,420
And everyone's like, I want one of those, right?

258
00:14:53,420 --> 00:14:54,260
And like, well, let's just use ours.

259
00:14:54,300 --> 00:14:55,380
You can just, you can just pay for it.

260
00:14:55,380 --> 00:14:58,020
It's funny because it goes back to exactly what we're talking about Linux.

261
00:14:58,020 --> 00:14:59,820
It's the different levels of abstraction.

262
00:14:59,820 --> 00:15:01,020
How many knobs do you want to press?

263
00:15:01,140 --> 00:15:03,460
How many, like how much responsibility do you want here?

264
00:15:03,460 --> 00:15:05,780
It's like Macedon versus blue sky.

265
00:15:05,780 --> 00:15:08,700
People were like, yeah, and we're going to have our own stuff and we're going to

266
00:15:08,700 --> 00:15:08,980
do it.

267
00:15:08,980 --> 00:15:10,100
And then we got halfway through.

268
00:15:10,100 --> 00:15:11,420
We're like, this is terrible.

269
00:15:13,700 --> 00:15:15,620
I don't want this much responsibility.

270
00:15:15,620 --> 00:15:16,940
I just wanted it to be fun.

271
00:15:16,940 --> 00:15:19,860
And then we were like, but we don't want to go back to the evil place and the

272
00:15:19,860 --> 00:15:20,580
bad place.

273
00:15:20,820 --> 00:15:25,100
So then they were like, okay, the middle blue sky is the Colos of like social,

274
00:15:25,140 --> 00:15:26,500
of like social media, right?

275
00:15:26,500 --> 00:15:31,460
Like you got like, we don't want to completely be responsible because we do

276
00:15:31,460 --> 00:15:32,300
that all day.

277
00:15:32,300 --> 00:15:34,420
And nobody wants to run their own PDS.

278
00:15:35,380 --> 00:15:35,740
Did it?

279
00:15:35,900 --> 00:15:36,220
Okay.

280
00:15:36,660 --> 00:15:40,180
But did you remember earlier when I said that Justin chooses violence and he

281
00:15:40,180 --> 00:15:42,540
likes to be like, am I wrong?

282
00:15:44,020 --> 00:15:45,820
Fatal masochism is a disease.

283
00:15:45,820 --> 00:15:51,220
I'm telling you, you just got to pick what level of pain that you're just like

284
00:15:51,220 --> 00:15:51,980
excited about.

285
00:15:52,060 --> 00:15:53,780
I have to know how it works and I have to do it.

286
00:15:53,780 --> 00:16:00,220
So there's also this, he made it where he couldn't even like direct message me

287
00:16:00,220 --> 00:16:03,220
and talk to me on this guy for like a month.

288
00:16:03,820 --> 00:16:04,300
Okay.

289
00:16:04,300 --> 00:16:07,700
And then I was just like, Justin, every time I tried to at you and he's like,

290
00:16:07,700 --> 00:16:08,700
no, it's fine.

291
00:16:08,700 --> 00:16:09,900
What are you talking about?

292
00:16:09,900 --> 00:16:10,500
It was a feature.

293
00:16:10,500 --> 00:16:11,980
It was not fine.

294
00:16:12,020 --> 00:16:15,140
Ali, he didn't even like figure it out for a month.

295
00:16:15,260 --> 00:16:15,820
Okay.

296
00:16:16,700 --> 00:16:17,140
Yeah.

297
00:16:17,180 --> 00:16:21,300
So whenever someone talks about on-premise, I show them the picture of the OVH data

298
00:16:21,300 --> 00:16:22,060
center fire.

299
00:16:24,460 --> 00:16:26,540
The big one that just lit on fire.

300
00:16:26,780 --> 00:16:26,980
Right.

301
00:16:26,980 --> 00:16:28,900
I'm like, you want to deal with that?

302
00:16:29,100 --> 00:16:32,100
Not just that, but now people are like, I'm running a server in my grandma's

303
00:16:32,100 --> 00:16:32,540
basement.

304
00:16:32,540 --> 00:16:34,660
Please don't let your grandma's house on fire.

305
00:16:34,660 --> 00:16:38,980
Like grandma did not ask, are you hoping grandma paid this electricity bill?

306
00:16:38,980 --> 00:16:39,660
Like goodness.

307
00:16:39,660 --> 00:16:43,180
And that's, that is one of the things that I find super fascinating, especially

308
00:16:43,180 --> 00:16:47,380
about hardware, which I do want to talk about is how much cheaper,

309
00:16:47,380 --> 00:16:50,900
and quieter, and low power these, especially like the home lab systems,

310
00:16:50,900 --> 00:16:55,340
like the N100 chips are like, they just trickle power and can run a bunch of

311
00:16:55,340 --> 00:16:55,780
stuff.

312
00:16:55,780 --> 00:16:57,300
It's just super fascinating.

313
00:16:59,860 --> 00:17:02,180
We met Matthew works for them.

314
00:17:02,620 --> 00:17:03,940
Oxide, yeah.

315
00:17:04,020 --> 00:17:06,580
Oh my God, their servers are wild.

316
00:17:06,940 --> 00:17:10,820
I have never geeked out about a server so hard.

317
00:17:11,340 --> 00:17:14,060
I mean, usually like, ew, I don't want to touch those things.

318
00:17:15,020 --> 00:17:16,740
But that one, I was like, what?

319
00:17:16,740 --> 00:17:21,140
Like, I'm not buying one of these like, like, and then the tests that they do,

320
00:17:21,140 --> 00:17:23,220
they'll like go with their little mics and you're just like,

321
00:17:25,300 --> 00:17:31,060
I'm like, I have been begging the oxide people to like give me a rack for free.

322
00:17:31,060 --> 00:17:34,420
And I will like, I will do horrendous, awful things to it.

323
00:17:34,460 --> 00:17:35,860
I want to know how you break it.

324
00:17:35,900 --> 00:17:37,500
I want to know how you put it together.

325
00:17:37,500 --> 00:17:38,740
I wouldn't like, you know what I mean?

326
00:17:38,740 --> 00:17:39,980
I'm just like, you can do what?

327
00:17:40,620 --> 00:17:43,420
Like, do you know how many hood rat things you could run on that?

328
00:17:43,420 --> 00:17:45,580
And just so much fun.

329
00:17:46,020 --> 00:17:50,460
You know, I, I've been, we've been talking about, um, like the, the poly

330
00:17:50,460 --> 00:17:57,580
keel, poly keel oxide rack, one of, one of the, uh, you know, understated

331
00:17:57,580 --> 00:17:59,060
things about polyamory.

332
00:17:59,060 --> 00:18:02,300
I know like three people that would be on this like server.

333
00:18:02,300 --> 00:18:07,580
Like, oh, I can't wait to this episode goes live.

334
00:18:07,580 --> 00:18:12,620
I'm just going to be like, I talked about you at Charlotte.

335
00:18:14,420 --> 00:18:17,260
I'm going to be like, I love you so much.

336
00:18:19,620 --> 00:18:23,540
I'm always the friend that like, I told them I want a T-shirt friend of the

337
00:18:23,540 --> 00:18:26,500
poly keel, like poly keel.

338
00:18:27,340 --> 00:18:33,900
I'm just like, you guys keep adding like more people and I just love everybody.

339
00:18:35,900 --> 00:18:36,500
Yeah.

340
00:18:36,860 --> 00:18:42,460
Like the, so the thing is, like hardware is getting more, more like efficient,

341
00:18:42,500 --> 00:18:46,100
more powerful, but then at the same time, the AI craze is happening too.

342
00:18:46,220 --> 00:18:50,340
And that's like, I don't know if you've ever seen an A 100 while it's running.

343
00:18:50,620 --> 00:18:53,900
I don't know if you've ever had to work with an A 100 while it's running, but

344
00:18:54,100 --> 00:18:56,540
it is a passively cooled 400 watt card.

345
00:18:58,780 --> 00:19:01,540
So like, that's a lot of electrons.

346
00:19:01,740 --> 00:19:02,020
Yeah.

347
00:19:02,020 --> 00:19:05,500
That like it, it gets real hot, real toasty.

348
00:19:05,780 --> 00:19:09,300
And that's kind of been one of the funny things about like, I've been working

349
00:19:09,300 --> 00:19:11,780
with NVIDIA hardware for like the past year.

350
00:19:12,220 --> 00:19:15,900
God knows how long before that, um, and like a data center capacity.

351
00:19:15,900 --> 00:19:20,020
And like, no one doesn't like, no one tells you NVIDIA cards.

352
00:19:20,300 --> 00:19:21,700
They get so hot.

353
00:19:21,740 --> 00:19:25,340
They get so hot, so quick, even when they're idle, right?

354
00:19:25,740 --> 00:19:29,980
I have had an A 100 shut down on me because it was passively cooled

355
00:19:29,980 --> 00:19:32,020
and I didn't have any airflow going over it.

356
00:19:32,460 --> 00:19:36,100
And so there are like 3D printed brackets that you will have to get and you

357
00:19:36,100 --> 00:19:41,420
will have to point a whole 120 millimeter fan just straight at the fins to

358
00:19:41,420 --> 00:19:43,900
cool it down because otherwise it will shut down.

359
00:19:44,380 --> 00:19:44,780
Right.

360
00:19:45,580 --> 00:19:46,420
So it's been funny.

361
00:19:46,420 --> 00:19:49,500
Like CPUs are so efficient now, right?

362
00:19:49,900 --> 00:19:54,060
Um, especially once you look at like the arm space, what Apple's doing with

363
00:19:54,060 --> 00:19:56,620
like their M4 chips now, I think they're on.

364
00:19:56,860 --> 00:19:57,060
Yeah.

365
00:19:57,140 --> 00:20:01,660
Um, I got a new M4 laptop and I swear to God, I've never, it was the most

366
00:20:01,660 --> 00:20:06,860
beautiful thing I've ever opened in my, like the things that you can do on

367
00:20:06,860 --> 00:20:10,460
that laptop are just should be like, it's great.

368
00:20:10,460 --> 00:20:14,260
I when the, when the first like ends, like when they were first talking

369
00:20:14,260 --> 00:20:17,260
about switching over to arm, I was so skeptical charcoal ones.

370
00:20:17,580 --> 00:20:21,460
Oh God, they're so, when they were first talking about switching over to arm,

371
00:20:21,860 --> 00:20:23,180
I was like, so skeptical.

372
00:20:23,180 --> 00:20:24,700
I was like, no way in hell.

373
00:20:24,740 --> 00:20:25,420
They pulled it off.

374
00:20:25,460 --> 00:20:26,100
They pulled it off.

375
00:20:26,140 --> 00:20:26,700
I'm sorry.

376
00:20:26,860 --> 00:20:31,460
Like even, even working at a like, like language and runtime and seeing all

377
00:20:31,460 --> 00:20:34,820
the extra stuff you had to do for those M1 M4 trips, worth it.

378
00:20:35,100 --> 00:20:42,420
Well, I'm like, I, I swear to God, like I run exclusively Max.

379
00:20:42,540 --> 00:20:44,300
I had one laptop, right?

380
00:20:44,300 --> 00:20:45,780
It was a terabyte of RAM.

381
00:20:45,780 --> 00:20:48,180
It was like, oh, it's going to be my new gaming computer.

382
00:20:48,260 --> 00:20:51,100
And I would put one here and then I had an M4 laptop.

383
00:20:51,380 --> 00:20:52,940
This fan's like kicking on.

384
00:20:52,940 --> 00:20:53,820
It's dying.

385
00:20:54,020 --> 00:20:59,380
You'd run all the same stuff on an M4 and it's like, oh, like, it's

386
00:20:59,660 --> 00:21:03,380
you're just like, how like there, it's a year difference between these two computers.

387
00:21:03,420 --> 00:21:04,420
It's insane.

388
00:21:04,780 --> 00:21:07,700
Yeah, like, like things are so efficient now.

389
00:21:07,740 --> 00:21:10,100
Well, they have to be because it's the only way we're going to be able

390
00:21:10,100 --> 00:21:13,900
to continue to this whole AI race because there's no way they can continue

391
00:21:13,900 --> 00:21:17,540
to like use this much electricity and water, you know what I mean?

392
00:21:17,540 --> 00:21:20,420
So they're going to have to figure it out and they want to put it on everything.

393
00:21:20,780 --> 00:21:24,780
One of the things that nobody really talks about, it isn't really about the power.

394
00:21:24,780 --> 00:21:28,700
I mean, like the power is obviously one of the big constraints in like growing

395
00:21:28,700 --> 00:21:31,180
from the grid that much power is a huge concern.

396
00:21:31,420 --> 00:21:37,420
And like what, what I think XAI is doing is they have gas turbines at their plant

397
00:21:37,420 --> 00:21:41,060
now, which is like spitting, like spitting emissions into the air.

398
00:21:41,060 --> 00:21:43,860
Polluting, like a small under.

399
00:21:43,940 --> 00:21:48,380
Privilege community that they just keep pretending like they're not in line.

400
00:21:48,380 --> 00:21:49,380
Oh, yeah.

401
00:21:50,260 --> 00:21:55,500
Yeah, like, like the power is one thing, but no one talks about cooling, right?

402
00:21:55,820 --> 00:21:58,900
Like, well, that's why some of them are using water, but it's crazy.

403
00:21:58,900 --> 00:22:01,180
Like, did you see the study they did in Alaska?

404
00:22:01,180 --> 00:22:03,060
They were like, well, just go to the cold.

405
00:22:03,060 --> 00:22:04,700
I'm not Alaska and good drinker.

406
00:22:04,700 --> 00:22:08,500
They're like, we'll build them there and then we'll open the windows.

407
00:22:08,500 --> 00:22:11,140
And I was like, what are you doing to the environment by changing the

408
00:22:11,140 --> 00:22:13,940
temperatures? And then they were like, we'll build one under water.

409
00:22:14,140 --> 00:22:16,580
And it's not like we're already killing fish or anything.

410
00:22:17,460 --> 00:22:20,660
Matt has been doing the open the windows for a long time now, because they have

411
00:22:20,660 --> 00:22:23,020
the, the. Well, they're trying to build them in colder places.

412
00:22:23,020 --> 00:22:26,900
Yeah, well, I think one of the things like they built these data centers

413
00:22:26,900 --> 00:22:30,020
in what used to be colder places that are now getting warmer.

414
00:22:30,540 --> 00:22:34,100
Hey, we need to move closer to the poles because it's close to it's colder there.

415
00:22:35,300 --> 00:22:38,540
Yeah, there's going to be a lot of money for the first person to make a

416
00:22:38,540 --> 00:22:40,620
colo at Antarctica, right?

417
00:22:40,820 --> 00:22:45,180
So they're already trying, but I'm just saying, like, it's really working out

418
00:22:45,180 --> 00:22:47,740
from my home value because Washington's going to be California.

419
00:22:47,740 --> 00:22:49,820
Other one day, well, everything else is hot as balls.

420
00:22:49,820 --> 00:22:51,540
So it's going to be like, great, like.

421
00:22:54,340 --> 00:22:58,340
Now, question, though, if they build a data center or region at the South Pole,

422
00:22:58,340 --> 00:23:01,580
is that, is that a West or an East eliminated?

423
00:23:01,820 --> 00:23:07,580
Can we name it Santa or like, yeah, can I be like, can North Pole be the

424
00:23:07,620 --> 00:23:09,140
like, instead of like East one?

425
00:23:09,140 --> 00:23:11,380
Can it be like North Pole one?

426
00:23:11,380 --> 00:23:16,660
Cause like with like a little like candy, like a candy cane, like emoji.

427
00:23:16,660 --> 00:23:22,300
Emoji regions, that would be like, like East, like West, like, what is it?

428
00:23:22,300 --> 00:23:24,180
West. Let's test all your Unicode.

429
00:23:24,180 --> 00:23:29,500
West should be like a little, like, West should be like a little beach, you know?

430
00:23:29,940 --> 00:23:34,180
And like, like New York East one or two or whatever.

431
00:23:35,180 --> 00:23:36,980
Send us a blue.

432
00:23:36,980 --> 00:23:39,900
What should the emojis be for each region for regions?

433
00:23:42,060 --> 00:23:44,940
Dude, we're going to have, we're going to have a West coast,

434
00:23:44,940 --> 00:23:48,900
like East coast works if you're going to be great.

435
00:23:49,020 --> 00:23:50,140
I love it.

436
00:23:50,140 --> 00:23:53,460
I don't know, but I mean, it's, it's fascinating.

437
00:23:53,460 --> 00:23:55,260
Like you mentioned water cooling, though, right?

438
00:23:55,260 --> 00:23:57,020
It's actually going further than water cooling.

439
00:23:57,020 --> 00:24:01,060
It's like full immersion, like mineral, oil, immersion, cooling.

440
00:24:01,260 --> 00:24:03,300
But they're talking about under the water.

441
00:24:03,300 --> 00:24:04,180
We're right under the sea.

442
00:24:04,180 --> 00:24:05,740
Yeah, there was that.

443
00:24:05,740 --> 00:24:09,700
I think when I was at Facebook, when I was at Disney, Pixar had

444
00:24:09,700 --> 00:24:13,180
a immersion data center, which was fascinating because you walk in there,

445
00:24:13,180 --> 00:24:14,860
it's just quiet because there's no fans.

446
00:24:14,860 --> 00:24:15,980
There's no, it's just like, what's going on?

447
00:24:15,980 --> 00:24:18,980
Like it's hot in there too, because they run at a higher ambient temperature.

448
00:24:19,180 --> 00:24:23,020
So you're in the, in the data center and it's like 85 Fahrenheit or something like that.

449
00:24:23,020 --> 00:24:25,060
You're like, it's, it is muggy in here right now.

450
00:24:25,060 --> 00:24:25,700
What is going on?

451
00:24:25,700 --> 00:24:28,660
It's like, but it was, it was super cool because it was just like, yeah,

452
00:24:28,660 --> 00:24:32,900
we can run these, you know, harder, longer and we have less breakage on things.

453
00:24:32,900 --> 00:24:35,340
It wasn't all their data centers, but it was super fascinating just seeing

454
00:24:35,340 --> 00:24:37,820
how that was playing out in like a real environment.

455
00:24:38,220 --> 00:24:43,100
Yeah, I think, I think in California, I think I remember reading some of the

456
00:24:43,100 --> 00:24:49,740
data centers are there because like they need to prioritize capacity so much.

457
00:24:49,740 --> 00:24:52,740
And you, like the only way to get higher capacity inside of a data

458
00:24:52,740 --> 00:24:54,940
center like that is to go immersion, right?

459
00:24:56,100 --> 00:25:00,740
And so I think like some of the data centers in San Jose, at least last time

460
00:25:00,740 --> 00:25:06,180
I was reading are like fully immersion as well, because it's come so far

461
00:25:06,180 --> 00:25:09,780
and like a couple of years because they want to be able to push the constraints.

462
00:25:09,780 --> 00:25:10,380
And it's amazing.

463
00:25:11,060 --> 00:25:15,460
You, like, do you remember when it was just hobbyists, like the fish tank PC

464
00:25:15,460 --> 00:25:17,860
on like Linus tech tips back in the day?

465
00:25:17,900 --> 00:25:21,060
Do you remember, I remember going to PAX a couple of years ago and people were

466
00:25:21,060 --> 00:25:25,500
walking in with their like PCs underwater, but it was to be cool.

467
00:25:25,500 --> 00:25:27,020
Like it wasn't like, you know what I mean?

468
00:25:27,020 --> 00:25:29,140
Like, no, like you need it to like run.

469
00:25:30,940 --> 00:25:36,020
The bubbles coming off of this also the dedication to pull, to like drag a whole

470
00:25:36,020 --> 00:25:39,740
ass PC down to like Seattle, like props.

471
00:25:42,140 --> 00:25:43,780
It's, it's kind of funny.

472
00:25:43,780 --> 00:25:47,580
It's like, there's like, it's very much a push and pull factor.

473
00:25:48,500 --> 00:25:50,220
I think we're going to keep with that push and pull.

474
00:25:50,220 --> 00:25:52,340
It's always, it's like, oh, cloud.

475
00:25:52,340 --> 00:25:55,700
And then it's like data centers and like, there's always a limit, right?

476
00:25:55,700 --> 00:25:58,060
You go, you find the next limit and you work around it.

477
00:25:58,620 --> 00:26:01,740
Not even just the limit though, it's whatever people want to worship that day.

478
00:26:01,900 --> 00:26:08,020
You know, religion, like, and then you're just like, there, there's a real tools

479
00:26:08,020 --> 00:26:11,660
that you could just kind of weigh your options, but then it's just like, what,

480
00:26:11,660 --> 00:26:15,100
like, what, what direction is the religion going on today?

481
00:26:15,100 --> 00:26:19,540
And people are going to find their version of social media to argue to the

482
00:26:19,540 --> 00:26:23,500
depths, even though there could be like a middle argument and there's like a good

483
00:26:23,580 --> 00:26:25,580
use case for both, but you know, keep going.

484
00:26:26,340 --> 00:26:28,860
Cause it wouldn't be fun if you weren't arguing.

485
00:26:29,500 --> 00:26:31,420
No, I, well, yeah.

486
00:26:31,580 --> 00:26:35,540
And you know, like so many push and pull factors.

487
00:26:35,540 --> 00:26:41,020
I mean, like, because the internet is growing and continuing to grow, like we

488
00:26:41,020 --> 00:26:43,940
are getting faster and faster networks, right?

489
00:26:43,980 --> 00:26:48,420
I think they are now testing like 1.6 terabit optics.

490
00:26:48,700 --> 00:26:49,500
Just wild.

491
00:26:49,500 --> 00:26:53,060
Do you remember how long it took to like, download a movie back in the day?

492
00:26:53,100 --> 00:26:54,220
Do you remember being on like.

493
00:26:54,660 --> 00:26:58,260
I mean, I think one of the things that we're trying to talk about is like the

494
00:26:58,260 --> 00:27:04,460
history of computing, like back in 2013, 2014, like it was rare to see a 40 gigabit

495
00:27:04,460 --> 00:27:05,420
network even.

496
00:27:05,740 --> 00:27:11,100
I think like at the high end, people would have 10 gig now, but like, you can

497
00:27:11,100 --> 00:27:16,300
go talk to an ISP and get a dedicated 10 gig fiber line for about a thousand

498
00:27:16,300 --> 00:27:17,780
dollars a month right now.

499
00:27:18,180 --> 00:27:18,780
I know.

500
00:27:18,780 --> 00:27:24,140
Cause I just talked to like Zeo and, and astound they're a local ISP in

501
00:27:24,140 --> 00:27:25,100
the Seattle area.

502
00:27:25,540 --> 00:27:31,220
Um, and it's like, it is, it is so cheap to get 10 gig now.

503
00:27:31,820 --> 00:27:37,060
Um, and even on the hobbyist end, you can get a hundred gig equipment for under

504
00:27:37,060 --> 00:27:38,020
$2,000.

505
00:27:38,180 --> 00:27:38,580
Right.

506
00:27:38,900 --> 00:27:43,740
I used Melanox switch will only run you about $1,500.

507
00:27:43,780 --> 00:27:44,500
Don't tell me that.

508
00:27:46,540 --> 00:27:48,060
No, no, no, poor Beth.

509
00:27:48,060 --> 00:27:50,020
She's just going to have to put up with this now.

510
00:27:50,060 --> 00:27:52,100
She'd be like, Justin, what are you doing in the garage?

511
00:27:53,100 --> 00:27:58,420
I'll send you the eBay list because, because we just built a lab off of like a

512
00:27:58,420 --> 00:28:02,260
hundred don't help this on eBay.

513
00:28:02,260 --> 00:28:03,420
Literally yesterday.

514
00:28:04,420 --> 00:28:06,340
Have you seen his garage?

515
00:28:06,340 --> 00:28:07,660
Look behind him.

516
00:28:07,660 --> 00:28:09,460
He doesn't need any help.

517
00:28:10,340 --> 00:28:16,700
So the secret is, and before the tariffs, it was like the Chinese are

518
00:28:16,700 --> 00:28:21,540
incredibly good at recycling data center equipment and, and turning it into

519
00:28:21,540 --> 00:28:25,420
hobbyist grade equipment so you can get, um, Bluefield twos.

520
00:28:25,420 --> 00:28:28,660
They're, they're insanely cool, um, network cards.

521
00:28:28,660 --> 00:28:30,100
I can get into them a little bit more.

522
00:28:30,220 --> 00:28:32,620
They're a hundred gig capable, actually 200 gig capable.

523
00:28:33,020 --> 00:28:37,540
And like, you can get them for like $300 each, which is the same.

524
00:28:37,620 --> 00:28:38,620
Are those the ones you were running?

525
00:28:38,620 --> 00:28:40,700
They have like, you can run a full operating system on them.

526
00:28:40,700 --> 00:28:40,940
Oh yeah.

527
00:28:40,940 --> 00:28:41,260
Yeah, yeah, yeah.

528
00:28:41,260 --> 00:28:43,460
I run, I ran a blue sky PBS on one of them.

529
00:28:44,060 --> 00:28:48,500
Don't give it on a network card.

530
00:28:49,420 --> 00:28:50,980
I remember, I remember you posted that.

531
00:28:51,060 --> 00:28:55,260
And I spent the entire night watching YouTube videos about these cards because

532
00:28:55,260 --> 00:28:57,780
I didn't know anything about all these different hacks that people were doing.

533
00:28:57,820 --> 00:29:00,420
I'm not going to be able to add him again for like another month.

534
00:29:00,420 --> 00:29:01,420
Cause he's going to break it.

535
00:29:01,740 --> 00:29:04,380
So, so that's funny.

536
00:29:04,380 --> 00:29:08,780
This has been a whole research article or like whole research point of research for

537
00:29:08,780 --> 00:29:09,020
me.

538
00:29:09,020 --> 00:29:13,300
There's a blog post about this, uh, at my work blog.

539
00:29:13,780 --> 00:29:17,340
Um, I was going to say, if you're not on blue sky, are you talking about this

540
00:29:17,340 --> 00:29:17,780
somewhere else?

541
00:29:17,780 --> 00:29:19,860
Cause like then we'll lose all the information.

542
00:29:20,860 --> 00:29:23,540
Come back to us and just like, don't, just ignore everybody.

543
00:29:23,580 --> 00:29:26,380
Like, don't actually talk to people.

544
00:29:26,380 --> 00:29:28,580
Just post it and then just, just post.

545
00:29:29,660 --> 00:29:31,740
Come back, just drop the mic after.

546
00:29:34,140 --> 00:29:34,620
Yeah.

547
00:29:35,020 --> 00:29:41,140
Um, it's like, NVIDIA is doing some insanely cool stuff with these DPUs.

548
00:29:41,180 --> 00:29:44,220
The blue field, two cards, I mean, $300 each.

549
00:29:44,500 --> 00:29:49,260
They have a full arm VA core on them and they're like tied to a

550
00:29:49,260 --> 00:29:51,500
Kinect X seven network adapter.

551
00:29:51,660 --> 00:29:55,700
And just like it's in a really weird way where that arm chip has like full

552
00:29:55,700 --> 00:30:00,100
control over that network adapter and create like pseudo interfaces that show

553
00:30:00,100 --> 00:30:05,060
up on the host that are actually routed through that arm core out to the

554
00:30:05,060 --> 00:30:08,260
public internet through the ethernet port on the back or technically it's an

555
00:30:08,300 --> 00:30:09,100
SMV port.

556
00:30:09,460 --> 00:30:13,060
It's going to get to a point where we have so many of these fast ships that

557
00:30:13,060 --> 00:30:15,820
they will like just be everywhere.

558
00:30:15,820 --> 00:30:17,900
Cause it seems like, you know what I mean?

559
00:30:17,900 --> 00:30:20,260
Like they just keep building them faster and faster.

560
00:30:20,260 --> 00:30:22,980
So people are going to want the new ones and it's just like, we're just

561
00:30:22,980 --> 00:30:25,180
going to end up with this crazy surplus of chips.

562
00:30:25,620 --> 00:30:29,780
Well, not only just a crazy surplus, but just so many stacked on layers of

563
00:30:29,780 --> 00:30:31,700
like abstraction and complexity.

564
00:30:31,700 --> 00:30:36,900
I mean, I really don't envy someone who has to learn all of like the basics of

565
00:30:36,900 --> 00:30:38,180
PC hardware from scratch.

566
00:30:38,220 --> 00:30:40,660
And then now you're throwing on like DPUs.

567
00:30:40,860 --> 00:30:45,380
You're throwing on all of the extra complexity of like, oh, you know, how

568
00:30:45,380 --> 00:30:48,060
do you handle a hundred gig, gigabit line rates, right?

569
00:30:48,140 --> 00:30:49,180
Accelerator cards.

570
00:30:49,260 --> 00:30:53,740
My little brother was learning how to build his own PC like two years ago and

571
00:30:53,740 --> 00:30:58,460
he was trying to add more RAM and like get a new chip because he was running

572
00:30:58,460 --> 00:30:59,180
video games.

573
00:30:59,660 --> 00:31:03,820
And I remember like, you couldn't find like the crappiest card anywhere.

574
00:31:03,900 --> 00:31:04,900
Like it was wild.

575
00:31:05,380 --> 00:31:10,300
And I'm like, what are we going to do when they've got like, just at some point

576
00:31:10,300 --> 00:31:13,340
there's going to be too many chips because they keep making new ones, you

577
00:31:13,340 --> 00:31:15,340
know, what's going to be like, I wonder how that affects.

578
00:31:15,940 --> 00:31:19,820
One of the things I was thinking of just as I speak about hardware data

579
00:31:19,820 --> 00:31:25,780
centers is just that recycle factor of like companies handing out old hardware

580
00:31:25,780 --> 00:31:29,820
to employees that are interested in the amount of things that I learned over

581
00:31:29,820 --> 00:31:34,900
the years by having free access to random compute things that I didn't know

582
00:31:34,900 --> 00:31:35,740
what to do with or not.

583
00:31:35,740 --> 00:31:39,300
Like I've, I've taken my fair share of rackmount servers home, not knowing

584
00:31:39,300 --> 00:31:43,220
that I could, if I could power or cool it, but it was still interesting to be

585
00:31:43,220 --> 00:31:44,780
able to say that you didn't burn your out.

586
00:31:44,780 --> 00:31:47,940
So in the accident, this was in a dorm room, right?

587
00:31:47,940 --> 00:31:51,980
Like this is just taking out the whole door.

588
00:31:54,060 --> 00:31:55,140
There's someone burning popcorn.

589
00:31:55,140 --> 00:31:55,740
Don't worry about it.

590
00:31:57,540 --> 00:31:57,740
Yeah.

591
00:31:57,740 --> 00:32:00,900
Like this is, I feel like we, a lot of people miss out on that, right?

592
00:32:00,900 --> 00:32:05,780
Cause so many of these like college students and early career people are only

593
00:32:05,780 --> 00:32:08,980
working in cloud providers and they don't actually get access to know how

594
00:32:08,980 --> 00:32:13,140
this stuff works at a lower level, be able to freely break something and have

595
00:32:13,180 --> 00:32:16,740
constraints around like, I'm not going to, it's not going to accidentally charge

596
00:32:16,740 --> 00:32:20,980
me $10,000 if I, you know, like for loop this Lambda function, right?

597
00:32:20,980 --> 00:32:23,940
Like at some point, there's different trade-offs though, like a lot of times

598
00:32:23,940 --> 00:32:27,140
you get extra credits and you can experiment, but it'd be, give people

599
00:32:27,140 --> 00:32:28,900
access to both, you know what I mean?

600
00:32:29,820 --> 00:32:34,540
But I mean, is, is there anything like the constraints of a, here's a box

601
00:32:34,540 --> 00:32:39,180
that runs something like here's a, here's a DPU or even a GPU or an old desktop

602
00:32:39,180 --> 00:32:42,900
or something like the amount of home labs I've built on old desktops is just

603
00:32:42,900 --> 00:32:45,780
because I was like, I always had free access to them and it was fascinating

604
00:32:45,780 --> 00:32:49,860
how much I could learn within that constraints of like, oh, what can this do?

605
00:32:50,140 --> 00:32:51,900
So it's funny that you mentioned this.

606
00:32:51,900 --> 00:32:54,420
This is kind of one of the things that I've always wanted to do.

607
00:32:54,500 --> 00:32:56,980
Um, there's this project called cloud lab.

608
00:32:57,100 --> 00:33:03,180
It's run by, I think like the university of Georgia or, or like some, maybe Clemson,

609
00:33:03,180 --> 00:33:07,300
Clemson, yeah, Clemson, the university of Utah participate.

610
00:33:07,380 --> 00:33:11,580
And they basically built this like software defined data center where you

611
00:33:11,580 --> 00:33:15,060
can rent some of this hardware for like academic research use.

612
00:33:15,420 --> 00:33:19,060
And like, I just think like that's the type of things that needs to be in the

613
00:33:19,060 --> 00:33:20,540
world more, right?

614
00:33:20,980 --> 00:33:25,740
You need to give like so many people who don't have access, who can't

615
00:33:25,780 --> 00:33:28,380
afford this hardware, because a lot of it is so expensive.

616
00:33:28,420 --> 00:33:29,740
Like you need to give them access.

617
00:33:29,740 --> 00:33:33,300
Otherwise, like, how do you expect anyone to really get knowledgeable?

618
00:33:33,420 --> 00:33:36,900
I was going to say, cause like, I mean, I do think that having a box is great,

619
00:33:36,900 --> 00:33:41,020
but sometimes you may not have the room, you may not have one stable place to

620
00:33:41,020 --> 00:33:45,140
live, like there's so many different, like, I know people who were living in

621
00:33:45,140 --> 00:33:49,100
their car and using libraries, like to do development work.

622
00:33:49,100 --> 00:33:51,380
So like, I think there's a place for everything.

623
00:33:51,420 --> 00:33:55,180
I do think something that has no limits that you can break and fix is definitely

624
00:33:55,180 --> 00:33:56,140
different than the cloud.

625
00:33:56,540 --> 00:33:57,860
Yeah, no, absolutely.

626
00:33:57,860 --> 00:34:01,740
Like I, like me personally, like I've moved so much, right?

627
00:34:01,740 --> 00:34:07,900
I, I just finished up a move from North Carolina to Seattle and I, it is, I had

628
00:34:07,900 --> 00:34:11,660
to pack up all of the servers in my house that were just sitting there and

629
00:34:11,660 --> 00:34:12,820
shipped them across the country.

630
00:34:12,820 --> 00:34:15,220
And a lot of them got broken along the way.

631
00:34:15,500 --> 00:34:22,340
And it's just like, like, it is such a privilege to have so much space and so

632
00:34:22,340 --> 00:34:26,380
much, like all of that power available in order to be able to power this equipment.

633
00:34:26,380 --> 00:34:28,340
But there are a lot of people who don't, right?

634
00:34:28,780 --> 00:34:32,940
And I definitely do think that, like, there will always be a market in making

635
00:34:32,940 --> 00:34:33,860
these things accessible.

636
00:34:34,820 --> 00:34:35,180
Yes.

637
00:34:35,620 --> 00:34:38,220
Because like, there's just surpluses, right?

638
00:34:38,540 --> 00:34:42,460
Like, because of the way that this hardware is depreciated and like the way that

639
00:34:42,500 --> 00:34:46,980
it's written off, like to a company is basically useless, probably after like

640
00:34:47,380 --> 00:34:49,300
maybe three or four years, right?

641
00:34:49,340 --> 00:34:51,260
And just the advances they're going to get new.

642
00:34:52,180 --> 00:34:57,820
Yeah, and it's, it's been so cool because you will see, like a lot of Chinese

643
00:34:57,820 --> 00:35:03,940
companies that are solely like built around recycled or like secondhand, you

644
00:35:03,940 --> 00:35:08,620
know, enterprise hardware, which is like, it's no longer in fashion mainstream and

645
00:35:08,620 --> 00:35:12,540
like current data centers, but it's like still valuable, right?

646
00:35:12,860 --> 00:35:15,700
That's been, that's been like the fascinating thing to watch about deep

647
00:35:15,700 --> 00:35:16,700
seek, right?

648
00:35:17,420 --> 00:35:21,420
They basically, you know, their principles are all correct.

649
00:35:21,460 --> 00:35:22,820
They're using the right stack.

650
00:35:22,820 --> 00:35:24,260
They're doing the right things.

651
00:35:24,260 --> 00:35:27,540
I, I am so bullish on deep seek.

652
00:35:27,660 --> 00:35:32,940
If anyone is getting to AGI, which, you know, I don't know if that's even a possibility.

653
00:35:32,940 --> 00:35:37,380
I, I remain skeptical, but if anyone is going to do it, I definitely think it's

654
00:35:37,380 --> 00:35:39,540
going to be the Chinese, right?

655
00:35:39,900 --> 00:35:43,500
I think the fact that they were willing to put that clearly opens up.

656
00:35:44,180 --> 00:35:45,060
Yeah, yeah.

657
00:35:45,260 --> 00:35:46,220
That's ballsy.

658
00:35:46,260 --> 00:35:49,900
That's ballsy, you know, saying I know that we're really, really good because

659
00:35:49,900 --> 00:35:51,460
I'll let you have the code and see all.

660
00:35:53,140 --> 00:35:55,460
That's how, that's how you disrupt markets anyway, right?

661
00:35:55,460 --> 00:35:59,700
Like you make the thing that someone else finds valuable and just run it to zero.

662
00:35:59,700 --> 00:35:59,900
Right.

663
00:35:59,900 --> 00:36:01,140
Like that's what Kubernetes exists.

664
00:36:01,420 --> 00:36:06,420
As Google said, we wanted everyone to rethink how to deploy applications and we

665
00:36:06,420 --> 00:36:07,940
thought containers were better than VMs.

666
00:36:07,940 --> 00:36:09,740
And so we wanted to ruin Amazon's business.

667
00:36:09,740 --> 00:36:10,740
So Kubernetes exists.

668
00:36:10,940 --> 00:36:14,260
I don't think it's that, but I also think it's the Kubernetes community because

669
00:36:14,260 --> 00:36:17,020
that's probably the coolest community I've ever, well, sure.

670
00:36:17,060 --> 00:36:17,260
Yeah.

671
00:36:17,260 --> 00:36:18,780
Like, like the people.

672
00:36:19,980 --> 00:36:24,420
I think that they had to do things differently because they had other open

673
00:36:24,420 --> 00:36:26,660
source containerization things that exist.

674
00:36:26,660 --> 00:36:30,860
Me, so, but even Google, like Google had, let me container that containerize that

675
00:36:30,900 --> 00:36:38,740
for you and like these, there was, there, there was Mesa's, there was no mad where

676
00:36:38,740 --> 00:36:43,940
the two that were, were around at the time, Kubernetes, um, yeah, and it's funny

677
00:36:43,940 --> 00:36:50,540
too, if people contributed so much that that is the most diverse, like people

678
00:36:50,540 --> 00:36:56,420
with pink hair, people that are brown trans with just all the things.

679
00:36:56,540 --> 00:36:59,900
And I like, like you walk into a Kubernetes conference and I'm like, I

680
00:36:59,940 --> 00:37:02,780
want to run this stuff, even if I don't need to use it because I want to

681
00:37:02,780 --> 00:37:04,020
hang out with all these people.

682
00:37:04,060 --> 00:37:09,140
Like, yeah, like, like people like Ian Coldwater are my, like, are my role.

683
00:37:09,140 --> 00:37:13,860
Dude, like you actually, like, there's so many open source projects where

684
00:37:13,860 --> 00:37:17,180
they're just like old, grouchy people and you're like, oh, I like your stuff.

685
00:37:17,180 --> 00:37:23,740
But like, y'all, like Kubernetes, you're like, oh, it's painful, but the people are

686
00:37:23,740 --> 00:37:24,180
rad.

687
00:37:25,140 --> 00:37:30,340
It's, it's kind of like the untold secret that trans people tend to make up a

688
00:37:30,340 --> 00:37:34,060
lot of the backbone of most distributed systems, which is why I think it's

689
00:37:34,060 --> 00:37:36,100
funny when they're like, oh, we don't care about them.

690
00:37:36,100 --> 00:37:38,380
And I'm like, yeah, we tell your infrastructure to like breaks.

691
00:37:39,540 --> 00:37:40,700
Don't call anybody.

692
00:37:41,380 --> 00:37:42,780
They're not coming to help you.

693
00:37:43,100 --> 00:37:47,180
It's, it's like an unspoken fact that trans people and furries tend to make up

694
00:37:47,300 --> 00:37:49,580
most of this nation's critical infrastructure.

695
00:37:49,700 --> 00:37:53,260
Dude, it's, it's like when people are like, oh, we don't like immigration.

696
00:37:53,420 --> 00:37:58,700
I'm like, you shouldn't eat the most like unseasoned, like horrible food for

697
00:37:58,700 --> 00:37:59,660
the rest of your life.

698
00:37:59,660 --> 00:38:01,100
You're not allowed to have a taco.

699
00:38:01,100 --> 00:38:01,540
Okay.

700
00:38:01,580 --> 00:38:04,700
Like I just, nothing unless it's like the, just.

701
00:38:05,420 --> 00:38:08,740
I hope you eat the most bland food for the rest of your life and also no

702
00:38:08,740 --> 00:38:09,700
infrastructure for you.

703
00:38:09,900 --> 00:38:10,140
Like.

704
00:38:11,020 --> 00:38:14,620
So, so I grew up half Japanese in Salt Lake City, Utah.

705
00:38:14,900 --> 00:38:15,420
So.

706
00:38:15,540 --> 00:38:16,140
Oh, girl.

707
00:38:20,260 --> 00:38:25,260
So, um, Mormon too, Mormon too, should, should mention Mormon.

708
00:38:25,620 --> 00:38:25,980
Right.

709
00:38:25,980 --> 00:38:31,860
So I, I, I was an assumption made Salt Lake City that you got it.

710
00:38:31,860 --> 00:38:33,460
You have to clarify assumptions.

711
00:38:34,380 --> 00:38:34,620
Yeah.

712
00:38:34,620 --> 00:38:34,820
Yeah.

713
00:38:34,820 --> 00:38:35,340
You got it.

714
00:38:35,460 --> 00:38:35,740
Yeah.

715
00:38:35,740 --> 00:38:36,700
It was Mormon.

716
00:38:36,980 --> 00:38:39,380
It is, it is, it is shocking.

717
00:38:39,820 --> 00:38:42,540
Um, and then I moved to North Carolina right after that.

718
00:38:42,540 --> 00:38:44,780
And I was like, wow, food can taste good.

719
00:38:46,420 --> 00:38:49,740
It's like, it's like the, like, I don't know.

720
00:38:49,740 --> 00:38:54,420
It's like the great, I don't know, like it gets like, it's like the universe's

721
00:38:54,420 --> 00:38:56,620
way of being like, you don't deserve good food.

722
00:38:58,180 --> 00:38:59,540
The equalizing factor.

723
00:38:59,580 --> 00:39:03,860
You know, like I grew up a jove witness and I was like, all my friends who came

724
00:39:03,860 --> 00:39:08,180
from very, very, very, very religious backgrounds are all like so much fun.

725
00:39:08,180 --> 00:39:10,140
And I was like, you can tell we didn't get to do anything when we were little.

726
00:39:15,780 --> 00:39:16,380
Yeah.

727
00:39:16,540 --> 00:39:21,220
But it's, it's funny that you mentioned Kubernetes too, because if you really

728
00:39:21,220 --> 00:39:24,980
start looking at the background on like who's pushing a lot of the needle forward

729
00:39:24,980 --> 00:39:29,580
on a lot of this Kubernetes research too, isn't really the big cloud providers.

730
00:39:29,580 --> 00:39:34,700
It isn't really like, like the giants in the U S of a lot of the stuff that I've

731
00:39:34,700 --> 00:39:41,060
been looking at it, like overlay BD, um, is, it's, it's a fascinating project.

732
00:39:41,060 --> 00:39:44,340
If you've not looked into overlay BD, they basically made like a block device,

733
00:39:44,340 --> 00:39:48,780
which remotely streams in its content from a, a Docker registry.

734
00:39:48,780 --> 00:39:52,100
Like that's, that's I think a 10 cent project.

735
00:39:52,380 --> 00:39:52,780
Right.

736
00:39:53,340 --> 00:39:54,780
Um, GCP has that, right?

737
00:39:54,780 --> 00:39:56,660
GCP has a way to, to stream.

738
00:39:56,660 --> 00:39:57,060
Yeah, yeah, yeah.

739
00:39:57,060 --> 00:39:58,420
Like East RGZ.

740
00:39:58,820 --> 00:39:59,300
Yeah.

741
00:39:59,500 --> 00:40:07,260
It's, I, I like there, I think one of the main, main tanners behind container D is

742
00:40:07,300 --> 00:40:11,220
also like these Japanese Akiha Arosuda, right?

743
00:40:11,540 --> 00:40:12,340
Is this handle?

744
00:40:12,340 --> 00:40:18,980
I think, like there are people across the ocean who are kind of leading the way on

745
00:40:18,980 --> 00:40:23,420
a lot of this Kubernetes research and it's kind of fascinating.

746
00:40:23,540 --> 00:40:27,660
If you look at it all, like just all the way, that's why like I'm just like,

747
00:40:27,700 --> 00:40:32,220
when people say that like diversity work is like some sort of like you're doing

748
00:40:32,220 --> 00:40:33,780
it to be nice and help people.

749
00:40:33,780 --> 00:40:37,220
I'm like, um, have you ever heard of the bottom line and building things?

750
00:40:37,420 --> 00:40:40,500
Like, cause like this is not charity work out here.

751
00:40:40,820 --> 00:40:43,740
Like you are not doing anything for people.

752
00:40:44,100 --> 00:40:49,740
Like you, to get the best people that are like really into whatever they're

753
00:40:49,740 --> 00:40:54,500
building and they're just doing it, you have to go everywhere and allow everyone

754
00:40:54,500 --> 00:40:58,500
there, you literally get better products and more of a return on investment.

755
00:40:58,740 --> 00:41:02,260
Some of the smartest people I've ever worked with were high school dropouts.

756
00:41:02,380 --> 00:41:04,780
And that's, that's like a bizarre factor.

757
00:41:04,820 --> 00:41:06,460
I'm a college dropout myself, right?

758
00:41:06,460 --> 00:41:07,140
I didn't get to finish.

759
00:41:07,180 --> 00:41:11,780
Some of the best people, college dropouts, theater degrees, absolutely no

760
00:41:11,780 --> 00:41:13,180
real computer site.

761
00:41:13,220 --> 00:41:18,180
Like, and it's, it makes me sad because I've ran into so many times where we're

762
00:41:18,180 --> 00:41:21,620
like, you don't have real computer science to create or like, like, and I think

763
00:41:21,620 --> 00:41:26,220
that we've got this elitist vocab, you know, that if you don't sound exactly

764
00:41:26,220 --> 00:41:31,260
the same as other people that people think that you like, or if you didn't go

765
00:41:31,260 --> 00:41:36,180
to a fancy school and I'm like, some of the best technologists I know have

766
00:41:36,180 --> 00:41:41,020
invented half of these as vocab words and like, don't have that degree.

767
00:41:41,060 --> 00:41:42,380
Like that's ridiculous.

768
00:41:42,420 --> 00:41:47,100
Like, yeah, I think that's been one of the most interesting factors.

769
00:41:47,140 --> 00:41:54,180
Like, because, because of like open source technology is it is so, so like, it

770
00:41:54,180 --> 00:41:59,100
is so easy for people from underserved communities to like break into tech by

771
00:41:59,100 --> 00:42:01,060
just contributing to open source projects.

772
00:42:01,300 --> 00:42:02,500
That's how I got my break.

773
00:42:02,580 --> 00:42:05,940
It also helps you to be able to play with technology because it's all

774
00:42:05,940 --> 00:42:09,860
there in the open and there's usually good documentation or doctors will

775
00:42:09,860 --> 00:42:10,580
have a walkthrough.

776
00:42:10,740 --> 00:42:16,420
Yeah, it's, it's like, it's like, I really admire tail scale.

777
00:42:16,460 --> 00:42:21,980
I really admire oxide, how they all have their core basically open in public

778
00:42:21,980 --> 00:42:22,900
for people to look at.

779
00:42:23,140 --> 00:42:23,300
Right.

780
00:42:23,300 --> 00:42:26,060
A lot of people don't have the ability to do internships.

781
00:42:26,060 --> 00:42:28,780
Internships are especially like unpaid ones.

782
00:42:28,780 --> 00:42:29,980
That's for rich kids.

783
00:42:30,100 --> 00:42:36,100
Like, not everybody can do, not everybody can move to Seattle or New

784
00:42:36,100 --> 00:42:41,420
York or Silicon Valley and how are you supposed to pay the extra high rent for

785
00:42:41,420 --> 00:42:44,180
three months, even if you are making some like, you know what I mean?

786
00:42:44,180 --> 00:42:48,740
Like, people don't realize how much of the traditional process of getting into

787
00:42:48,740 --> 00:42:55,020
tech is gatekeeper by like your social economic class about where you're able

788
00:42:55,020 --> 00:42:56,500
to get like your education.

789
00:42:56,500 --> 00:42:57,460
Like there's so much of it.

790
00:42:57,460 --> 00:43:01,820
And I think open source is the great equalizer and 70% of infrastructure

791
00:43:01,820 --> 00:43:02,860
comes from open source.

792
00:43:03,940 --> 00:43:06,780
And that's the other thing I was saying with like the trickle down of free

793
00:43:06,780 --> 00:43:07,260
hardware, right?

794
00:43:07,260 --> 00:43:11,020
Like, cause that goes, it ends up going far and wide just beyond the person

795
00:43:11,020 --> 00:43:14,580
that gets it, like the amount of computers that I've, I've got from companies

796
00:43:14,580 --> 00:43:17,860
and then also given away to other people that I'm like, Oh, you should get into

797
00:43:17,860 --> 00:43:17,980
this.

798
00:43:17,980 --> 00:43:22,140
I play hockey with a, with someone and they work a service job and they seem

799
00:43:22,140 --> 00:43:23,900
really kind of apps to technology.

800
00:43:23,900 --> 00:43:25,580
And I'm like, Hey, I have a spare laptop.

801
00:43:25,580 --> 00:43:27,780
Do you want it so that we can do some programming classes?

802
00:43:27,780 --> 00:43:29,900
And they're like, I've never had a laptop before.

803
00:43:29,980 --> 00:43:33,220
I'm like, I always give my laptops away because I just definitely the kid that

804
00:43:33,220 --> 00:43:33,860
didn't have one.

805
00:43:34,140 --> 00:43:34,340
Yeah.

806
00:43:34,340 --> 00:43:38,100
And it totally has a second order effect of like, if you have, you have more

807
00:43:38,100 --> 00:43:42,460
time than ability at some point, you're like, just give me access to something.

808
00:43:42,740 --> 00:43:46,540
And I will bang my head against the desk until I figure something out in, in

809
00:43:46,540 --> 00:43:49,340
for systems work, for programming, for all sorts of things.

810
00:43:50,100 --> 00:43:54,180
We can't replace the desire to learn with more training, right?

811
00:43:54,180 --> 00:43:56,700
Like the, the amount of curiosity is one thing you can't train.

812
00:43:57,380 --> 00:43:57,580
Yeah.

813
00:43:57,580 --> 00:44:00,660
And like the, the amount like, Oh, we're just going to train up a new group

814
00:44:00,660 --> 00:44:01,620
of people to do this thing.

815
00:44:01,620 --> 00:44:02,500
I'm like, no, you're not.

816
00:44:02,500 --> 00:44:04,460
Like if they're not interested, they're not going to learn it.

817
00:44:05,100 --> 00:44:08,540
Not just that, but the rabbit hole and the research that you will go and you

818
00:44:08,540 --> 00:44:12,460
will learn when you are just like, want to solve a problem.

819
00:44:13,020 --> 00:44:16,860
Like that's why the best projects come from like somebody wanting to solve

820
00:44:16,860 --> 00:44:17,580
an issue that thing.

821
00:44:17,980 --> 00:44:18,980
Or prove someone wrong.

822
00:44:19,300 --> 00:44:23,420
Oh, oh my God, my life is spike driven development.

823
00:44:23,420 --> 00:44:24,340
I swear to God.

824
00:44:24,340 --> 00:44:25,340
Like, oh my God.

825
00:44:25,660 --> 00:44:25,980
No.

826
00:44:25,980 --> 00:44:29,580
So like this has kind of been one of the most frustrating things about

827
00:44:29,580 --> 00:44:31,780
watching modern computer science education.

828
00:44:31,820 --> 00:44:34,060
Like I, I am a colleague interviewing.

829
00:44:34,100 --> 00:44:35,100
Don't forget the interviewing.

830
00:44:35,220 --> 00:44:38,580
The, the interviewing process is frustrating, but like watching people

831
00:44:38,580 --> 00:44:42,740
who are so reliant on chat, GBT and like don't want to learn anything.

832
00:44:42,740 --> 00:44:46,140
It's so frustrating because I'm really scared for future developers.

833
00:44:46,140 --> 00:44:50,020
Like I, I think we're going to make people like dumber, but I think

834
00:44:50,020 --> 00:44:51,980
that AI could be useful in certain ways.

835
00:44:51,980 --> 00:44:55,220
But I just, it's so weird to me that they're shopping it as the black

836
00:44:55,220 --> 00:44:58,260
box of box of answers instead of this new way to learn.

837
00:44:58,260 --> 00:45:02,540
Like I, it's so weird to me that that would be so like also just the way that

838
00:45:02,540 --> 00:45:09,340
we, the fact that you can have AI do process on Apple phones and everything

839
00:45:09,340 --> 00:45:10,620
is processed in chip.

840
00:45:11,180 --> 00:45:12,260
That's fricking amazing.

841
00:45:12,300 --> 00:45:12,860
You know what I mean?

842
00:45:12,860 --> 00:45:17,220
Like there's like, there's so many cool ways that we could use this to help

843
00:45:17,220 --> 00:45:21,140
people and we're just, let's make them dumber.

844
00:45:22,380 --> 00:45:23,700
It's a really weird flex.

845
00:45:24,580 --> 00:45:28,700
I have, I have such a nuanced opinion about AI because on the one hand, I

846
00:45:28,700 --> 00:45:29,860
think it's so cool.

847
00:45:29,860 --> 00:45:32,340
Like this is the next frontier of natural language processing.

848
00:45:32,340 --> 00:45:36,060
Like you can talk to a computer and like literally just plain text.

849
00:45:36,060 --> 00:45:38,100
This can be your rubber ducky.

850
00:45:38,140 --> 00:45:42,300
It can help you like when you're a neuro-spicy person and you're trying

851
00:45:42,300 --> 00:45:45,100
to figure out how to interact with ready, but like, does this sound dumb?

852
00:45:45,140 --> 00:45:46,140
How do I rewrite this?

853
00:45:46,140 --> 00:45:47,460
Like, you know what I mean?

854
00:45:47,460 --> 00:45:51,500
Like here's all my neuro-spicy ideas that I'm too excited to make into words.

855
00:45:51,500 --> 00:45:52,820
Can you help me make it into work?

856
00:45:52,820 --> 00:45:58,380
Like, you know, like a structured like paragraph from my bullet list of ideas.

857
00:45:58,380 --> 00:46:03,260
Like, yeah, but like at the same time, like I, I spent so much time around my

858
00:46:03,260 --> 00:46:06,820
friends who are still in college who are just graduating and like, like chat

859
00:46:06,820 --> 00:46:11,660
GPT has completely changed how they kids don't want to write paragraphs.

860
00:46:11,700 --> 00:46:12,020
Yeah.

861
00:46:12,540 --> 00:46:14,420
They don't, they don't actually try.

862
00:46:14,420 --> 00:46:16,300
They just feed everything into chat.

863
00:46:16,300 --> 00:46:20,420
GPT use the output, like output, turn it in, maybe change some words here and

864
00:46:20,420 --> 00:46:21,660
there so to like avoid that.

865
00:46:21,660 --> 00:46:26,260
Yeah, I believe all the M dashes, which is funny because I feel like we did.

866
00:46:26,780 --> 00:46:30,220
Remember when they told people that you can't use an ID, you have to use Vim.

867
00:46:30,900 --> 00:46:34,460
And then they were like, Oh, well, like you can't copy and paste.

868
00:46:34,460 --> 00:46:37,540
You got to make your projects for like, people have been doing this shit for

869
00:46:37,540 --> 00:46:38,260
forever, right?

870
00:46:38,260 --> 00:46:41,580
But like, I think this is just like, it gives you such a finished product.

871
00:46:42,020 --> 00:46:46,820
That's so tailored to what you want before you could go copy some code, but

872
00:46:46,820 --> 00:46:50,340
you had to make it, you had to change enough to do what you want it to.

873
00:46:50,660 --> 00:46:55,420
You know, the one, the one radicalizing thing about learning about higher

874
00:46:55,420 --> 00:46:59,900
institutions and how education is actually conducted there is it isn't like

875
00:46:59,900 --> 00:47:03,220
not only is the curriculum more advanced, the cheating is more advanced, right?

876
00:47:03,500 --> 00:47:06,420
They get so creative on cheating.

877
00:47:06,420 --> 00:47:11,020
There are whole cheating rings formed and like disbanded on a daily basis.

878
00:47:11,020 --> 00:47:12,900
There are discord servers created.

879
00:47:13,220 --> 00:47:16,900
So every now and then I want to give them a little award like that dude who did

880
00:47:16,900 --> 00:47:21,180
the Trojan horse and the like, and he like bullied the maintainer into

881
00:47:21,180 --> 00:47:22,420
making him a maintainer.

882
00:47:22,460 --> 00:47:23,940
Like, should you do that?

883
00:47:24,100 --> 00:47:25,900
No, was it creative?

884
00:47:26,020 --> 00:47:27,700
You get an A plus for creativity.

885
00:47:27,700 --> 00:47:32,100
And I want to teach this in a computer science cyber class, but that was mean.

886
00:47:36,060 --> 00:47:44,100
Yeah, it's, you know, that is one of the things that I, it just, I, I think we

887
00:47:44,300 --> 00:47:47,060
don't do a good job of making people curious.

888
00:47:47,060 --> 00:47:50,340
And that's one of the things, but in a way that is curiosity and problem

889
00:47:50,340 --> 00:47:52,300
solving, but they're using it for bad.

890
00:47:52,300 --> 00:47:53,660
It's like when you have kids, okay?

891
00:47:53,900 --> 00:47:57,540
My kids use their intelligence to do little evil things.

892
00:47:57,540 --> 00:47:59,460
And I'm like, you could have done this the right way.

893
00:47:59,820 --> 00:48:01,580
And it would have took less brain power.

894
00:48:09,100 --> 00:48:13,140
Okay, I might have been talking crap about you earlier, but that was so rude.

895
00:48:14,740 --> 00:48:19,660
Oh, would you work with people that are your friends and they know you for real?

896
00:48:19,660 --> 00:48:20,420
It's so hard.

897
00:48:23,420 --> 00:48:25,180
Also, my kids worship him.

898
00:48:25,420 --> 00:48:26,020
I know.

899
00:48:26,500 --> 00:48:29,100
I was like, damn, and he's not wrong.

900
00:48:29,100 --> 00:48:35,020
So I don't even have a rebuttal to the heart.

901
00:48:38,860 --> 00:48:42,660
What do you think is interesting in hardware in the next five years?

902
00:48:42,740 --> 00:48:46,500
I was like, all these accelerators right now are like, everyone, I feel

903
00:48:46,500 --> 00:48:50,460
like everyone's just like throwing more electrons at AI right now, but it's

904
00:48:50,460 --> 00:48:52,020
not super interesting to me.

905
00:48:53,580 --> 00:48:57,540
Yeah, I think, I think one of the most fascinating things is like, people are

906
00:48:57,540 --> 00:49:01,540
now kind of revisiting the basics when it comes to the data center technology.

907
00:49:01,820 --> 00:49:02,860
Bye, Autumn.

908
00:49:03,820 --> 00:49:08,260
Um, it, and like, there's so much talk now about disaggregation, right?

909
00:49:08,460 --> 00:49:13,420
Because now you have GPUs, which are basically their own whole, like they

910
00:49:13,500 --> 00:49:15,060
consume the whole unit.

911
00:49:15,060 --> 00:49:18,820
They are basically the majority of the power unit of a lot of these rack units.

912
00:49:19,340 --> 00:49:23,420
Um, there's a lot more talk about, um, making them into their own standalone

913
00:49:23,420 --> 00:49:26,820
rack units and connecting over CXL.

914
00:49:26,860 --> 00:49:30,820
That's the standard that's coming out, I think, and it's going GA this year.

915
00:49:31,260 --> 00:49:36,020
Um, and people, you know, putting in more memory than what like these servers

916
00:49:36,020 --> 00:49:40,500
are physically capable of, um, like dynamically allocating and swapping

917
00:49:40,500 --> 00:49:43,980
around memory between servers without ever turning them off.

918
00:49:43,980 --> 00:49:47,180
It's, there's, there's so much cool research there.

919
00:49:47,260 --> 00:49:51,180
I had a call with both MemVirge and Liquid.

920
00:49:51,220 --> 00:49:57,180
Those are two startups that are, um, very hot and playing in the disaggregation

921
00:49:57,180 --> 00:50:00,180
space and it is insane what's going on there.

922
00:50:00,540 --> 00:50:05,980
It is, I, so what they're able to do is they basically realized,

923
00:50:06,020 --> 00:50:09,980
okay, PCIe is a switched, um, switched interconnect, right?

924
00:50:09,980 --> 00:50:13,620
It is, it is, if you squint, it's like a network protocol, right?

925
00:50:13,740 --> 00:50:14,820
And so they basically made it.

926
00:50:14,820 --> 00:50:20,780
So they break out the PCIe slot from inside of the computer into a copper

927
00:50:20,780 --> 00:50:26,060
like interconnect that can go 15 meters, 10 to 15 meters.

928
00:50:26,060 --> 00:50:31,380
And so you're able to have these individual thin, thin, um, you know,

929
00:50:31,460 --> 00:50:34,020
thin racks, thin, thin rack units.

930
00:50:34,140 --> 00:50:38,300
And then you're able to have all of these copper interconnects into, um,

931
00:50:38,500 --> 00:50:42,860
like a GPU box where you have like eight GPUs all in like a four U enclosure.

932
00:50:42,860 --> 00:50:44,620
And that's all it is, right?

933
00:50:44,700 --> 00:50:48,940
There's, you know, power, there's a power distribution system that's more

934
00:50:48,940 --> 00:50:53,620
equipped to handle all of that load for the, those GPUs, you know, or, or like

935
00:50:53,620 --> 00:50:56,980
even you could do it from like another rack unit over, right?

936
00:50:58,140 --> 00:51:00,540
That's been one of the fascinating things.

937
00:51:00,700 --> 00:51:05,780
When I, when I see like that in NVME over fabric and these things that are

938
00:51:05,780 --> 00:51:10,260
just like, let's just put everything as a network protocol and, and connect it

939
00:51:10,260 --> 00:51:13,580
over, over more generic cabling, right?

940
00:51:13,580 --> 00:51:15,100
This is basically what we're doing.

941
00:51:15,500 --> 00:51:19,740
And then I, when I envision it, like initially I think that like a rack,

942
00:51:19,740 --> 00:51:24,980
an entire rack is basically going to look like what a desktop PC looks like today.

943
00:51:24,980 --> 00:51:27,980
Like if I open my desktop PC, I got the CPU on this side.

944
00:51:27,980 --> 00:51:29,340
I got the storage over here.

945
00:51:29,340 --> 00:51:31,260
I got the GPU down here, whatever else.

946
00:51:31,260 --> 00:51:35,820
And like we're just doing that at the rack scale to say, Oh, it's fast enough

947
00:51:35,820 --> 00:51:40,500
to connect these things together with the network or some fabric connectivity.

948
00:51:40,500 --> 00:51:41,340
It's close enough.

949
00:51:41,340 --> 00:51:44,580
So now we can just like, I always envision like pulling off the side of a

950
00:51:44,580 --> 00:51:48,420
rack, like a big door and it's like a big PC where I'm looking at, okay, I'm

951
00:51:48,420 --> 00:51:51,260
going to change the CPU cooler here or something like that.

952
00:51:51,500 --> 00:51:53,940
But that, that seems like where we're going, where we want these, these

953
00:51:53,940 --> 00:51:57,620
specific units that do one thing.

954
00:51:57,620 --> 00:52:01,740
And we say, Oh, we need to swap the, you know, compute Bay two or something.

955
00:52:01,740 --> 00:52:01,940
Right.

956
00:52:01,940 --> 00:52:04,140
We got like, pull it out, hot swap a CPU out.

957
00:52:04,140 --> 00:52:04,380
Okay.

958
00:52:04,380 --> 00:52:05,580
We fix that, put it back in.

959
00:52:05,860 --> 00:52:06,620
It's back into.

960
00:52:07,020 --> 00:52:07,300
Yeah.

961
00:52:07,340 --> 00:52:08,660
That's, that's where we're going.

962
00:52:08,700 --> 00:52:13,180
I like, like, you know, that saying it's turtles all the way down.

963
00:52:13,380 --> 00:52:14,740
It's computers all the way down.

964
00:52:14,860 --> 00:52:15,100
Right.

965
00:52:15,140 --> 00:52:15,420
But yeah.

966
00:52:15,420 --> 00:52:20,100
And that's the thing I have a question about is at that point, every one of

967
00:52:20,100 --> 00:52:24,620
those components has an arm chip attached to it or something that is its

968
00:52:24,620 --> 00:52:25,660
operating system.

969
00:52:25,820 --> 00:52:26,180
Mm hmm.

970
00:52:26,980 --> 00:52:29,260
What is the operating system for the rack?

971
00:52:29,540 --> 00:52:29,860
Right.

972
00:52:29,860 --> 00:52:32,580
And I don't think Kubernetes is the right abstraction for that.

973
00:52:33,860 --> 00:52:38,020
I, I think like there has been some thought there.

974
00:52:38,100 --> 00:52:42,900
I think open stack was a very interesting player in that space.

975
00:52:42,940 --> 00:52:47,060
I would probably say they were the closest to get to get right, but it was

976
00:52:47,060 --> 00:52:48,580
just too complicated.

977
00:52:49,300 --> 00:52:54,540
Um, I, I been looking at other projects, open Nebula.

978
00:52:54,540 --> 00:53:02,900
They're, uh, you know, it's like no one has really figured that out.

979
00:53:03,380 --> 00:53:03,820
Right.

980
00:53:04,140 --> 00:53:09,620
I, I think the enterprise is kind of standardizing around Kubernetes as is

981
00:53:09,620 --> 00:53:14,220
right now, but it'll kind of be very interesting to see what comes out of this.

982
00:53:14,420 --> 00:53:18,620
Um, because I don't think Kubernetes provides the lower level abstractions

983
00:53:18,620 --> 00:53:23,460
that people want to manage that hardware, uh, especially if it's a, a per

984
00:53:23,500 --> 00:53:25,500
nick operating system, right?

985
00:53:25,500 --> 00:53:25,540
Yeah.

986
00:53:25,540 --> 00:53:28,940
So, so that's actually, it's actually funny that you mentioned this.

987
00:53:28,980 --> 00:53:33,380
Um, one of my research topics at work has been the Doka platform.

988
00:53:33,780 --> 00:53:38,780
Um, basically NVIDIA built a bunch of CRDs, uh, custom resource definitions on

989
00:53:38,780 --> 00:53:42,940
top of the Kubernetes API to like declaratively manage the DPU networking

990
00:53:42,940 --> 00:53:44,260
cards that we're talking about earlier.

991
00:53:44,700 --> 00:53:46,540
And it's so weird.

992
00:53:46,540 --> 00:53:50,660
It's like, it's like, yeah, you could do this, but should you?

993
00:53:50,820 --> 00:53:51,100
Yeah.

994
00:53:51,220 --> 00:53:57,140
Um, and the answer for me was kind of like, no, this, like, and, and like the

995
00:53:57,140 --> 00:54:01,300
way that they want you to set it up and, and like the way that you want to maintain

996
00:54:01,300 --> 00:54:05,020
it, it's like, you're running three Kubernetes clusters at the same time

997
00:54:05,020 --> 00:54:06,980
there, which is insane.

998
00:54:07,140 --> 00:54:07,420
Yeah.

999
00:54:07,460 --> 00:54:07,820
Right.

1000
00:54:07,980 --> 00:54:12,820
And there are like projects like Kamadi, which is what I use to manage, um, those

1001
00:54:12,820 --> 00:54:17,020
three Kubernetes clusters, which is so cool because it lets you like, it lets you

1002
00:54:17,020 --> 00:54:21,340
declaratively create Kubernetes clusters inside of Kubernetes clusters, which

1003
00:54:21,340 --> 00:54:28,420
is, which is, you know, uh, batshit and saying, but it's like, you know, this

1004
00:54:28,420 --> 00:54:33,380
doesn't feel like what we should be doing with Kubernetes because it wasn't

1005
00:54:33,380 --> 00:54:39,300
really designed for this is it like CRDs were really meant to be kind of, if you

1006
00:54:39,300 --> 00:54:42,820
need something more than just a deployment or replica set, something

1007
00:54:42,820 --> 00:54:45,140
like that, it still should be at the application layer, right?

1008
00:54:45,140 --> 00:54:45,660
It still should be.

1009
00:54:45,660 --> 00:54:50,540
If I think of the like OSI model, right, that's still at the top of the, like

1010
00:54:50,540 --> 00:54:54,900
CRDs are not down here in like the physical sort of like TCP area.

1011
00:54:54,900 --> 00:54:57,460
This is like, no, there needs to be something else that's managing everything

1012
00:54:57,460 --> 00:54:57,940
below that.

1013
00:54:57,940 --> 00:55:01,900
And, and to some degree, that's just the operating system has traditionally done

1014
00:55:01,900 --> 00:55:02,060
that.

1015
00:55:02,060 --> 00:55:06,540
I don't know that general purpose operating systems are the right thing for that either.

1016
00:55:06,740 --> 00:55:06,940
Right.

1017
00:55:06,940 --> 00:55:09,540
That's where we just start getting into like, what is firmware?

1018
00:55:09,540 --> 00:55:12,020
It's like, well, it's like a little operating system that does one thing.

1019
00:55:12,260 --> 00:55:12,860
Basically.

1020
00:55:13,340 --> 00:55:13,700
Yeah.

1021
00:55:13,700 --> 00:55:17,140
I mean, and it's like, you could probably cobble together like all of the

1022
00:55:17,140 --> 00:55:22,100
individual parts and individual software to like make like a whole rack scale, you

1023
00:55:22,100 --> 00:55:24,020
know, computer software.

1024
00:55:24,020 --> 00:55:28,940
But it's just like, you know, I spent so much time trying to get even like

1025
00:55:28,940 --> 00:55:30,500
net box working correctly.

1026
00:55:30,500 --> 00:55:32,100
And it's just, it's a headache.

1027
00:55:32,100 --> 00:55:38,420
Like there, no one has really thought of it that way as like a rack scale computer.

1028
00:55:38,660 --> 00:55:42,300
I think the closest people would probably be Oxide, honestly.

1029
00:55:43,580 --> 00:55:47,500
Because, you know, if there was anyone that was going to figure it out,

1030
00:55:47,500 --> 00:55:49,060
I would probably assume it's them.

1031
00:55:49,780 --> 00:55:54,300
Just because they're, you know, they're building out all of the key building

1032
00:55:54,300 --> 00:55:56,980
blocks and the way that you would expect, right?

1033
00:55:57,380 --> 00:55:59,500
Well, they're doing, I think they're doing it in such a different way.

1034
00:55:59,700 --> 00:55:59,900
All right.

1035
00:55:59,900 --> 00:56:03,220
Like they're, they're just saying, like, you know what, we don't need a BIOS.

1036
00:56:03,580 --> 00:56:04,820
Like it just doesn't exist.

1037
00:56:04,820 --> 00:56:06,980
Like we don't, we're not building that kind of machine.

1038
00:56:07,220 --> 00:56:11,140
We know everything about this hardware and how to initialize it when we plug it in.

1039
00:56:11,140 --> 00:56:14,340
So we don't need that general purpose BIOS UEFI stuff.

1040
00:56:14,340 --> 00:56:19,300
And it's just more complicated than it's worth to have that interface in the system.

1041
00:56:19,300 --> 00:56:20,260
So we'll get rid of it.

1042
00:56:20,380 --> 00:56:20,660
Yeah.

1043
00:56:20,700 --> 00:56:25,540
And in the more of, the more of those layers we can remove the better in a lot

1044
00:56:25,540 --> 00:56:30,860
of these cases where it's just like, actually, what is, what is valuable in this stack?

1045
00:56:31,940 --> 00:56:32,140
Yeah.

1046
00:56:32,140 --> 00:56:34,500
Well, I mean, disaggregation is cool, right?

1047
00:56:34,500 --> 00:56:36,020
It is, it is so cool.

1048
00:56:36,020 --> 00:56:37,940
It is so, so wild.

1049
00:56:37,940 --> 00:56:44,140
But at the same time, like there are limitations due to the speed of light.

1050
00:56:44,140 --> 00:56:45,860
A lot of, a lot of times, right?

1051
00:56:46,060 --> 00:56:50,580
There is an unavoidable amount of latency involved in like converting optical signals

1052
00:56:50,580 --> 00:56:54,260
to electricity, like two copper signals, right?

1053
00:56:55,660 --> 00:57:02,780
And it's like, it's just, it's going to be really fascinating to see, like, to see

1054
00:57:02,780 --> 00:57:07,500
all of this operationalize and see like maybe like 10, 15 years from now, like

1055
00:57:07,740 --> 00:57:10,020
were we correct about disaggregation, right?

1056
00:57:10,020 --> 00:57:11,100
Were we not correct?

1057
00:57:11,740 --> 00:57:16,020
And like the answer is kind of like, you have to try and find out, right?

1058
00:57:16,260 --> 00:57:16,540
Yeah.

1059
00:57:18,180 --> 00:57:25,180
But like from an SRE standpoint, disaggregation, it makes everything a black box.

1060
00:57:25,700 --> 00:57:27,740
Means you can't trust your storage devices.

1061
00:57:27,740 --> 00:57:29,460
It can't, you can't trust your network.

1062
00:57:29,900 --> 00:57:33,340
And one of the first things you learn about distributed systems is that the

1063
00:57:33,340 --> 00:57:35,980
network is not reliable, right?

1064
00:57:36,460 --> 00:57:40,820
Like there is a reason why TCP is basically the de facto norm.

1065
00:57:40,820 --> 00:57:44,740
And it is because packets are dropped pretty frequently, right?

1066
00:57:45,180 --> 00:57:49,660
And I mean, if like it has been a long, long time since I haven't trusted

1067
00:57:49,660 --> 00:57:54,940
a front side bus between the CPU and a PCI, like that has always worked for me.

1068
00:57:54,940 --> 00:57:56,620
Or most, most of my life that has worked.

1069
00:57:56,620 --> 00:57:59,780
And I've never had to think about maybe I need to debug this layer.

1070
00:57:59,980 --> 00:58:02,460
But when you go to a distributed system or disaggregation, you're like,

1071
00:58:02,460 --> 00:58:05,580
everything is network and now I don't trust any of those buses.

1072
00:58:05,700 --> 00:58:06,180
Yes.

1073
00:58:06,740 --> 00:58:07,180
Yeah.

1074
00:58:07,220 --> 00:58:13,260
It is, it is going to be so, so wild to see like all of this productionized.

1075
00:58:13,260 --> 00:58:18,300
Like, like, what does the SRE do 10, 15 years from now?

1076
00:58:18,300 --> 00:58:22,140
Like, are we teaching people how to like attach an oscilloscope and like

1077
00:58:22,140 --> 00:58:25,180
see the actual signal coming over the wire, like trace it?

1078
00:58:25,180 --> 00:58:31,060
Like, it kind of seems like where we're headed because, you know, one of the

1079
00:58:31,060 --> 00:58:36,700
things that I, I really pride myself on is like reverse engineering and

1080
00:58:36,700 --> 00:58:40,780
knowing every single stack and every single layer of like involved in like

1081
00:58:40,780 --> 00:58:43,180
an OS call or a sys call or something like that.

1082
00:58:43,420 --> 00:58:46,100
But then what's the like through disaggregation in the mix now?

1083
00:58:46,100 --> 00:58:48,660
Like, you can't even trust like an M map, right?

1084
00:58:48,980 --> 00:58:50,380
And M maps can fail.

1085
00:58:50,380 --> 00:58:51,380
And then what does that do?

1086
00:58:51,380 --> 00:58:53,020
Like software, right?

1087
00:58:53,500 --> 00:58:57,700
Like, it's, it's funny because like all of our hardware is advancing at such

1088
00:58:57,700 --> 00:59:01,380
a fast pace, but it doesn't feel like the software is keeping up.

1089
00:59:02,260 --> 00:59:02,540
Hmm.

1090
00:59:03,020 --> 00:59:10,140
And that, that has kind of been the, the like killer, killer, sorry.

1091
00:59:10,140 --> 00:59:11,100
What's like the term here?

1092
00:59:11,100 --> 00:59:13,660
It's like nailing the coffin for a lot of things.

1093
00:59:13,660 --> 00:59:17,140
Like you think of itanium, you think of Zion, fuck me.

1094
00:59:17,940 --> 00:59:23,380
Zion fi, I think, like Intel's experiments to try and make really cool hardware.

1095
00:59:23,700 --> 00:59:24,100
Right.

1096
00:59:24,340 --> 00:59:29,660
The software just never capped up for either to, for either itanium or Zion

1097
00:59:29,660 --> 00:59:32,620
fi, which led to both of them failing.

1098
00:59:32,980 --> 00:59:33,420
Right.

1099
00:59:34,100 --> 00:59:38,980
And so, you know, that's, that's kind of going to be really interesting to see

1100
00:59:39,020 --> 00:59:43,820
with Memvirgin liquid, because they're both betting on Kubernetes as being kind

1101
00:59:43,820 --> 00:59:45,340
of like the management plan for these things.

1102
00:59:45,660 --> 00:59:46,140
Interesting.

1103
00:59:47,740 --> 00:59:53,860
So like in some ways, like Kubernetes is, is not the right fit for any of

1104
00:59:53,860 --> 00:59:58,580
these, but it is what the enterprise is kind of standardizing on, because the

1105
00:59:58,580 --> 01:00:02,140
alternative is like proprietary vendor locked software.

1106
01:00:02,380 --> 01:00:02,620
Yeah.

1107
01:00:02,860 --> 01:00:08,140
Um, you know, like ESXi, that everyone was running on ESXi for a while until

1108
01:00:08,140 --> 01:00:12,020
Broadcom bought VMware and then was like, Hey, we're going to screw up all of

1109
01:00:12,020 --> 01:00:13,260
your licensing, by the way.

1110
01:00:16,620 --> 01:00:20,820
Uh, you mentioned provisioning there for a minute and I'm kind of curious, it's

1111
01:00:20,820 --> 01:00:24,380
always been like this passion of mine throughout my entire career.

1112
01:00:24,380 --> 01:00:28,020
I was just like, I, I love the, I love pixie booting things.

1113
01:00:28,100 --> 01:00:29,900
I love provisioning automated.

1114
01:00:29,900 --> 01:00:32,700
Like I can install a hard, like an operating system.

1115
01:00:33,620 --> 01:00:35,940
What's the, do you know what state of art, like state of the art, like for

1116
01:00:35,940 --> 01:00:37,100
provisioning systems these days?

1117
01:00:37,100 --> 01:00:41,420
Like when I did it, like I literally was like cobbler back in the day and a

1118
01:00:41,420 --> 01:00:45,900
satellite foreman, um, doing that sort of like red hat stack for quite a while.

1119
01:00:45,900 --> 01:00:49,460
And then I was like switching over to like when I did a core OS, like their whole

1120
01:00:49,500 --> 01:00:52,340
like install script was like a, it was just a bash script that just like

1121
01:00:52,340 --> 01:00:54,420
D deed an image to this kind of like, that's fascinating.

1122
01:00:54,420 --> 01:00:55,020
That's all you need.

1123
01:00:55,020 --> 01:00:55,820
I was like, yeah, that's it.

1124
01:00:55,820 --> 01:00:58,900
And, and I've kind of, I've, I've moved away from a lot of stuff.

1125
01:00:58,900 --> 01:01:01,020
I'm kind of curious what is, oh, net box.

1126
01:01:01,020 --> 01:01:01,740
That's what you mentioned.

1127
01:01:01,940 --> 01:01:04,500
Well, so net box is just like the network source of truth.

1128
01:01:04,500 --> 01:01:04,820
It is.

1129
01:01:04,820 --> 01:01:05,100
Okay.

1130
01:01:05,100 --> 01:01:07,940
That's just the, not IMDB, the CMDB, right?

1131
01:01:07,940 --> 01:01:11,340
So it's the database of like what's, what's there in the network.

1132
01:01:11,700 --> 01:01:14,180
So Nix is really interesting.

1133
01:01:14,580 --> 01:01:18,580
Nix is really interesting, but so far I haven't found a single person who's

1134
01:01:18,580 --> 01:01:24,420
gotten Nix to work as like that sort of like provisioning, you know,

1135
01:01:24,460 --> 01:01:26,340
Nix as the OS or Nix.

1136
01:01:26,420 --> 01:01:26,660
Yeah.

1137
01:01:26,660 --> 01:01:27,700
Nix OS, right?

1138
01:01:28,100 --> 01:01:31,740
I haven't seen a single person who has managed to make Nix work.

1139
01:01:32,180 --> 01:01:37,140
I think the state of the art there is still like puppet chef, Ansible.

1140
01:01:37,420 --> 01:01:42,420
Um, but I genuinely don't think that we've evolved beyond puppet or chef.

1141
01:01:42,420 --> 01:01:46,300
I mean, if you go to like the big hyperscalers, yeah, they, they totally

1142
01:01:46,300 --> 01:01:49,380
have a replacement for puppet and chef and everything like that.

1143
01:01:49,380 --> 01:01:52,300
And they definitely have like integrated solutions for all of these things.

1144
01:01:52,300 --> 01:01:59,940
But, um, yeah, you know, I think the people that I know who have made

1145
01:01:59,940 --> 01:02:05,260
advancements here are like packet before they got bought out by Equinex and

1146
01:02:05,260 --> 01:02:06,420
turned it to Equinex metal.

1147
01:02:06,460 --> 01:02:07,540
And now it gets shut down.

1148
01:02:07,540 --> 01:02:07,780
Yeah.

1149
01:02:09,700 --> 01:02:14,540
You know, like there hasn't been a lot of like, and this is something that

1150
01:02:14,540 --> 01:02:17,100
OpenStack was actually trying to do as well.

1151
01:02:17,220 --> 01:02:17,420
Right.

1152
01:02:17,420 --> 01:02:17,900
Ironic.

1153
01:02:18,260 --> 01:02:18,660
Yeah.

1154
01:02:18,660 --> 01:02:18,980
Yeah.

1155
01:02:18,980 --> 01:02:23,780
It's, it's like, it was, it was just a couple of years too ahead of its time.

1156
01:02:24,180 --> 01:02:29,020
And maybe now if someone tried to do OpenStack again, we'd be like, okay, you

1157
01:02:29,020 --> 01:02:32,620
know, and that's going to be really interesting to see also from like,

1158
01:02:33,180 --> 01:02:33,900
um, what was it?

1159
01:02:33,900 --> 01:02:34,780
Cidero too.

1160
01:02:35,300 --> 01:02:35,700
Right.

1161
01:02:35,740 --> 01:02:38,260
Like how is that Talos Linux stuff going to work?

1162
01:02:38,540 --> 01:02:38,780
Yeah.

1163
01:02:38,780 --> 01:02:44,340
And, and it's been interesting on like both sides of how things get provisioned

1164
01:02:44,340 --> 01:02:47,980
because I look at people that went to the cloud, just ignored the whole

1165
01:02:47,980 --> 01:02:52,100
how to provision things and it's just like boot the generic Ubuntu and, and

1166
01:02:52,100 --> 01:02:53,580
can, and cloud init the whole thing.

1167
01:02:53,860 --> 01:02:54,220
Right.

1168
01:02:54,220 --> 01:02:56,900
And like cloud init is the config management system, which is just such

1169
01:02:56,900 --> 01:03:01,980
a, a slow, difficult path to like actually make something work.

1170
01:03:01,980 --> 01:03:04,860
And then coming from like data centers where they're like, oh, we're running

1171
01:03:05,100 --> 01:03:06,740
puppet agent or chef or something.

1172
01:03:06,740 --> 01:03:11,380
And that's, uh, it is, is a, is a lot more powerful than cloud in its, but

1173
01:03:11,380 --> 01:03:15,060
also can, can quickly become more complex and harder to maintain.

1174
01:03:15,060 --> 01:03:15,540
Yes.

1175
01:03:15,740 --> 01:03:20,020
Um, and I, I keep telling friends at work, like people at work.

1176
01:03:20,020 --> 01:03:23,180
I'm like, I'm so spoiled by Talos now because like, I don't, I don't worry

1177
01:03:23,180 --> 01:03:26,940
about either of those things where it's like, I call an API and my OS gets

1178
01:03:26,940 --> 01:03:28,980
installed and that's weird.

1179
01:03:29,140 --> 01:03:32,780
I had to do a, I told you before we started recording, I'm doing performance

1180
01:03:32,780 --> 01:03:35,860
testing on a bunch of different platforms and operating systems.

1181
01:03:36,100 --> 01:03:39,620
And like every, I have a KVM and I'm mounting like two gig of Ubuntu

1182
01:03:39,660 --> 01:03:41,340
ISOs to it, to like install Ubuntu.

1183
01:03:41,340 --> 01:03:42,820
I'm like, this is so painful.

1184
01:03:43,100 --> 01:03:47,100
Like I can't believe it's no wonder that people wanted to get away from this.

1185
01:03:47,300 --> 01:03:51,340
Uh, because my, you know, my Talos Linux is like 80 megs in it.

1186
01:03:51,500 --> 01:03:53,100
It just like one API was okay.

1187
01:03:53,100 --> 01:03:54,140
It's installed, I'm ready to go.

1188
01:03:54,140 --> 01:03:55,740
And I'm like, oh, I'm still, it's still booting.

1189
01:03:55,860 --> 01:03:57,460
Like Ubuntu is still kind of going through that.

1190
01:03:57,460 --> 01:04:03,780
And I can't believe that nothing has changed that ecosystem in drastic ways.

1191
01:04:03,780 --> 01:04:09,540
I'm, I'm using things like, um, Silverblue, Bluefin for my desktop, which is, you

1192
01:04:09,540 --> 01:04:13,940
know, imaged base sort of installs, it's still large because it's still a

1193
01:04:13,940 --> 01:04:18,500
general purpose sort of Fedora installation, but the maintenance side of it has improved.

1194
01:04:18,500 --> 01:04:21,260
And so there's all these like weird areas where we're, we're kind of making

1195
01:04:21,260 --> 01:04:26,660
changes in some places and not a lot of people are rethinking the possibilities.

1196
01:04:26,780 --> 01:04:27,220
Yeah.

1197
01:04:27,300 --> 01:04:31,420
No, so many, so many of these like second order cloud providers, they're

1198
01:04:31,420 --> 01:04:32,940
still using cloud in it, right?

1199
01:04:33,540 --> 01:04:38,580
It is all, it is basically Libvert with cloud in it and they are using like the CD

1200
01:04:38,580 --> 01:04:43,220
data store that is it that, you know, and like you look at the cloud providers,

1201
01:04:43,220 --> 01:04:45,620
they're doing things like MMDS, right?

1202
01:04:45,660 --> 01:04:52,980
With the EC2 instances where you can call, I think 196.254.196.254 and it's like

1203
01:04:52,980 --> 01:04:58,940
this API where you get all sorts of metadata about your, um, like things

1204
01:04:58,940 --> 01:05:05,580
like that are probably the future and where we need to move towards.

1205
01:05:05,620 --> 01:05:10,500
And, you know, I just think like no one has really tackled that space.

1206
01:05:10,500 --> 01:05:15,620
And it's understandable why, but it's like, at the same time, like, you know,

1207
01:05:15,820 --> 01:05:19,060
you have a feeling like someone could do this better with like Nix or like

1208
01:05:19,060 --> 01:05:20,460
Talos or something like that.

1209
01:05:20,460 --> 01:05:23,820
It's just, you know, I just, I just don't have the time to do it.

1210
01:05:25,980 --> 01:05:28,620
Well, thank you so much for taking the time for this podcast.

1211
01:05:28,620 --> 01:05:29,500
This has been a lot of fun.

1212
01:05:30,340 --> 01:05:33,700
Uh, uh, if, if, if no one has seen it, I'm going to put it in show notes.

1213
01:05:33,740 --> 01:05:37,140
Uh, le.fm at your website is something that everyone needs to go to.

1214
01:05:37,140 --> 01:05:38,500
At least once experience.

1215
01:05:40,180 --> 01:05:41,620
I'm not going to throw out any spoilers yet.

1216
01:05:41,620 --> 01:05:42,580
Just, just click the link.

1217
01:05:44,340 --> 01:05:48,260
Uh, I hope that's not mobile friendly because it's not, it's not, right.

1218
01:05:48,340 --> 01:05:48,900
This is great.

1219
01:05:48,900 --> 01:05:49,100
Yeah.

1220
01:05:49,100 --> 01:05:50,740
Not optimized for mobile whatsoever.

1221
01:05:51,300 --> 01:05:53,380
If there's not horizontal scrolling, I don't want it.

1222
01:05:56,020 --> 01:05:58,660
So I know you're, uh, I don't know if people can reach out to you

1223
01:05:58,660 --> 01:06:02,060
somewhere, or at least, uh, where you could see where people are talking about,

1224
01:06:02,580 --> 01:06:06,660
uh, you know, I lurk, I lurk, I will, I will, I will see, I will see the

1225
01:06:06,660 --> 01:06:10,860
discussions, maybe, maybe I'll pop in on whatever new blue sky account I have at

1226
01:06:10,860 --> 01:06:11,420
the time.

1227
01:06:13,300 --> 01:06:13,940
Sounds great.

1228
01:06:13,940 --> 01:06:17,500
And I look forward to learning more about, uh, DPUs and, and going down, uh,

1229
01:06:17,540 --> 01:06:20,980
rabbit holes of, of YouTube videos of, I didn't know that was possible from

1230
01:06:20,980 --> 01:06:21,620
you in the future.

1231
01:06:23,500 --> 01:06:24,180
Have a good one.

1232
01:06:24,300 --> 01:06:24,460
Yeah.

1233
01:06:24,500 --> 01:06:24,900
Thanks.

1234
01:06:25,380 --> 01:06:27,780
And everyone, thank you for listening and we will talk to you again soon.

1235
01:06:28,660 --> 01:06:40,060
Thank you for listening to this episode of fork around and find out.

1236
01:06:40,380 --> 01:06:43,900
If you like this show, please consider sharing it with a friend, a coworker, a

1237
01:06:43,900 --> 01:06:45,740
family member, or even an enemy.

1238
01:06:45,820 --> 01:06:49,700
However we get the word out about this show helps it to become sustainable for

1239
01:06:49,700 --> 01:06:50,420
the longterm.

1240
01:06:50,620 --> 01:06:56,740
If you want to sponsor this show, please go to FAFO.fm slash sponsor and reach

1241
01:06:56,740 --> 01:06:59,780
out to us there about what you're interested in sponsoring and how we can

1242
01:06:59,780 --> 01:07:00,020
help.

1243
01:07:01,340 --> 01:07:04,580
We hope your system stay available and your pagers stay quiet.

1244
01:07:05,060 --> 01:07:06,220
We'll see you again next time.