1
00:00:00,120 --> 00:00:04,470
Saying this exact, uh, thing to many, many people in Amazon.

2
00:00:04,470 --> 00:00:07,710
Over the last while, um, my excellent account

3
00:00:07,710 --> 00:00:11,220
manager has been setting me up with various leaders.

4
00:00:11,640 --> 00:00:14,490
Uh, they've been asking for documents, they've been asking for examples.

5
00:00:14,610 --> 00:00:17,400
You know, they are hungry for this stuff, so I don't, don't doubt

6
00:00:17,460 --> 00:00:21,300
that there's no desire to be the leaders or to, to really satisfy

7
00:00:21,300 --> 00:00:23,370
their customers, but, you know, it's execution we care about.

8
00:00:23,580 --> 00:00:24,990
And when it comes down to it.

9
00:00:25,830 --> 00:00:27,330
We need excellent databases.

10
00:00:27,360 --> 00:00:29,430
We need the best databases to be able to

11
00:00:29,430 --> 00:00:31,320
ship world-class product to our customers.

12
00:00:36,360 --> 00:00:38,070
Welcome to Screaming in the Cloud.

13
00:00:38,160 --> 00:00:40,800
I'm Corey Quinn, and I'm here to correct an oversight.

14
00:00:40,835 --> 00:00:44,045
Because I have known Brian Scanlon for many years, and

15
00:00:44,045 --> 00:00:46,865
somehow he has slipped through the cracks and not been

16
00:00:46,865 --> 00:00:50,705
on this show previously, so let's go ahead and fix that.

17
00:00:50,855 --> 00:00:54,215
Brian Scanlon is a senior principal engineer at Intercom,

18
00:00:54,425 --> 00:00:57,695
where he has been for damn near 11 years at this point.

19
00:00:57,725 --> 00:00:59,225
Brian, welcome to the show.

20
00:00:59,255 --> 00:00:59,925
Thanks so much, Corey.

21
00:00:59,945 --> 00:01:01,655
It's great to finally be here.

22
00:01:02,065 --> 00:01:04,495
I know it's We save the best toward the end.

23
00:01:04,525 --> 00:01:05,245
Yeah, exactly.

24
00:01:05,245 --> 00:01:07,015
Except we're not ending the show anytime soon.

25
00:01:07,015 --> 00:01:08,575
Much to various people's chagrin.

26
00:01:08,605 --> 00:01:11,725
'cause most people want me to shut the hell up a lot more than I do,

27
00:01:12,055 --> 00:01:12,445
not me.

28
00:01:12,445 --> 00:01:13,885
I hope we keep going for a long time.

29
00:01:14,640 --> 00:01:17,070
Crying Out Cloud is one of the few cloud security

30
00:01:17,070 --> 00:01:19,500
podcasts that's actually fun to listen to.

31
00:01:19,590 --> 00:01:22,500
Smart Conversations, great guests, and Zero Fluff.

32
00:01:22,590 --> 00:01:25,590
If you haven't heard of it, it's a cloud and AI security

33
00:01:25,590 --> 00:01:29,940
podcast from Wiz Run by Clouds Sec. Pros for Clouds SEC Pros.

34
00:01:30,120 --> 00:01:32,280
I was actually one of the first guests on the

35
00:01:32,280 --> 00:01:34,230
show and it's been amazing to watch it grow.

36
00:01:34,290 --> 00:01:39,270
Make sure to check them out at wiz.io/crying-out-cloud.

37
00:01:40,095 --> 00:01:41,265
So Intercom.

38
00:01:41,384 --> 00:01:43,455
I have complicated feelings about the company.

39
00:01:43,455 --> 00:01:46,005
Originally I hated the thing 'cause I'm on a website trying to

40
00:01:46,005 --> 00:01:49,185
get something done and it pops up like freaking digital clippy.

41
00:01:49,185 --> 00:01:51,104
Hey, do you want to talk to a person?

42
00:01:51,375 --> 00:01:51,975
God no.

43
00:01:51,975 --> 00:01:52,755
I'm a millennial.

44
00:01:52,755 --> 00:01:53,025
Elder.

45
00:01:53,025 --> 00:01:55,575
Millennial, but still I don't want to talk to people.

46
00:01:55,600 --> 00:01:59,055
I. But they've come back around because it turns out the issue is not intercom.

47
00:01:59,055 --> 00:02:01,185
It was bad implementations thereof.

48
00:02:01,185 --> 00:02:03,915
Maybe you don't need to pop up, talk to a human on your

49
00:02:03,915 --> 00:02:06,675
landing page at the front of it, but when you're dealing with

50
00:02:06,675 --> 00:02:09,375
a support issue and you want to tag someone in and suddenly

51
00:02:09,375 --> 00:02:12,735
you're talking to a human right there, it's transformative.

52
00:02:12,915 --> 00:02:15,920
So yeah, it turns out that anything can be dumb if you hold it wrong.

53
00:02:16,840 --> 00:02:21,520
Yeah, I used to introduce Intercom to PE people to say, uh, yeah, we're one

54
00:02:21,520 --> 00:02:25,030
of those chatbots that pop up in the right hand bottom corner of your website.

55
00:02:25,030 --> 00:02:26,485
Except we're the good one, I think.

56
00:02:27,329 --> 00:02:29,760
There are a lot of bad implementations, and there you, you

57
00:02:29,760 --> 00:02:33,090
know, people use it for outreach and uh, for marketing and

58
00:02:33,090 --> 00:02:35,940
sales purposes, and, you know, they want messages in your face.

59
00:02:35,940 --> 00:02:39,750
But, uh, I think when it comes to basic customer support and

60
00:02:39,990 --> 00:02:43,140
actually putting humans in touch with humans and increasingly

61
00:02:43,140 --> 00:02:45,810
humans in touch with AI bots, that will answer your questions.

62
00:02:45,959 --> 00:02:47,519
I think intercom's, uh, definitely got.

63
00:02:47,855 --> 00:02:50,615
Uh, really good properties for, uh, in the marketplace.

64
00:02:50,885 --> 00:02:53,825
It's especially strange in this era that we are in

65
00:02:53,825 --> 00:02:58,145
where everyone is building AI chatbots and, okay.

66
00:02:58,145 --> 00:03:01,355
I have a whole laundry list of angry opinions on it, but where I, where I

67
00:03:01,355 --> 00:03:04,595
really get annoyed is when they don't admit that they're a chatbot upfront.

68
00:03:04,715 --> 00:03:07,595
And it irks me because by the time I finally break

69
00:03:07,595 --> 00:03:09,845
down and admit I have to talk to a human being.

70
00:03:10,355 --> 00:03:11,975
I have exhausted the documentation.

71
00:03:11,975 --> 00:03:14,135
It is not going to be something simple of,

72
00:03:14,135 --> 00:03:16,894
have you tried jiggling the handle instead?

73
00:03:16,894 --> 00:03:17,584
It's okay.

74
00:03:17,584 --> 00:03:19,475
There is now a weird corner case, which

75
00:03:19,475 --> 00:03:21,635
I'm very good at, blindly stumbling into.

76
00:03:21,964 --> 00:03:25,834
Uh, I need someone to go ahead and fix a thing on the backend or let

77
00:03:25,834 --> 00:03:29,405
me know that, that I wanna simply not possible with the platform or.

78
00:03:29,700 --> 00:03:32,700
Admit that your documentation is rubbish, and that's something that AI

79
00:03:32,700 --> 00:03:37,649
historically can't do, but forcing me to go through that filter to hit

80
00:03:37,649 --> 00:03:41,880
some arbitrary target of a fewest number of customer contacts, fastest

81
00:03:41,880 --> 00:03:46,380
resolution possible, and never let them talk to a human, has been maddening.

82
00:03:46,925 --> 00:03:50,825
Where are you right now on that whole gen AI spectrum from bot to human?

83
00:03:51,395 --> 00:03:57,155
So we are very much all in on, uh, gen ai, gen AI for, uh, chat bots.

84
00:03:57,395 --> 00:04:01,985
Um, so much so that, you know, in October, November, 2022 when

85
00:04:01,985 --> 00:04:05,735
chat GBT came out, uh, we reorientated the entire company.

86
00:04:05,735 --> 00:04:07,295
We saw that it was the future.

87
00:04:07,505 --> 00:04:09,155
Uh, we had actually been building ml.

88
00:04:09,235 --> 00:04:12,325
Chatbots, uh, and had them in the marketplace for a good few

89
00:04:12,325 --> 00:04:17,095
years prior to the current explosion of, uh, ML chatbots.

90
00:04:17,425 --> 00:04:22,225
Um, but the lifts or the improvement that we saw with GPT-3 0.5, and

91
00:04:22,225 --> 00:04:25,885
that came out, that was a okay, we need to change the entire company.

92
00:04:26,185 --> 00:04:29,545
Um, and we've changed the entire company and moves strongly

93
00:04:29,545 --> 00:04:32,965
towards what we think is like the best AI chat bot in the market.

94
00:04:33,205 --> 00:04:36,115
Uh, it needs to work well with humans and

95
00:04:36,115 --> 00:04:38,425
peop there's a transition as well also.

96
00:04:38,655 --> 00:04:40,515
There's just a lot of work that's to be done to get the

97
00:04:40,515 --> 00:04:43,484
quality really great so that people don't hate using it.

98
00:04:43,635 --> 00:04:46,875
One of the interesting things that we've seen is, uh, many of

99
00:04:46,875 --> 00:04:51,104
our customers who have good knowledge bases and who get high

100
00:04:51,104 --> 00:04:53,835
resolution rates, uh, from using the chatbots is that they'll

101
00:04:53,835 --> 00:04:56,744
find that customers actually just start asking more questions.

102
00:04:56,744 --> 00:05:00,674
Because if you can get really fast answers and you don't have to go around

103
00:05:00,674 --> 00:05:04,094
documentation sites and the chat bot's actually useful people get it.

104
00:05:04,195 --> 00:05:07,825
Addicted to this, they'll, they're gonna use it all the time, and weirdly

105
00:05:07,825 --> 00:05:11,665
enough, have increased the number of conversations coming in because people.

106
00:05:12,615 --> 00:05:15,615
In some cases, and we're seeing more, more and more of this

107
00:05:15,795 --> 00:05:18,375
where people will ask more questions because they're getting good

108
00:05:18,375 --> 00:05:21,105
fast answers, uh, which is a little bit counterintuitive, but

109
00:05:21,105 --> 00:05:24,135
I think we all always need to have ways to fall back to humans.

110
00:05:24,525 --> 00:05:26,715
Uh, of course, depends on the business and

111
00:05:26,715 --> 00:05:28,725
the volume and what makes sense for them.

112
00:05:28,725 --> 00:05:31,635
But you know, I don't think humans are going away and from what we've

113
00:05:31,635 --> 00:05:34,765
seen in the market as well, even for places that have adopted and are.

114
00:05:35,550 --> 00:05:38,790
Deflecting or answering large numbers of questions.

115
00:05:39,000 --> 00:05:42,570
We see that they're not reducing their support team sizes at all.

116
00:05:42,690 --> 00:05:45,990
They're putting their people on better, better questions, higher

117
00:05:45,990 --> 00:05:49,410
quality work, or just deeper work with customers, as well as like

118
00:05:49,410 --> 00:05:51,630
feeding the bot with better documentation, that kind of thing.

119
00:05:51,630 --> 00:05:53,430
So we're, we're seeing it as like actually

120
00:05:53,430 --> 00:05:55,470
a net positive into the customer experience.

121
00:05:55,620 --> 00:05:58,890
But, uh, there's definitely lots of bad implementations out there as well.

122
00:05:59,580 --> 00:06:01,440
It feels like AI assisted.

123
00:06:01,734 --> 00:06:05,995
Uh, support is on many cases better than pure AI support, and this is somewhat

124
00:06:05,995 --> 00:06:09,385
controversial among people who want to sell bots, but I find the chat bots are

125
00:06:09,385 --> 00:06:13,734
not necessarily a great interface just due to complete lack of discoverability.

126
00:06:13,734 --> 00:06:16,104
It's, it's the Alexa problem or the Siri problem,

127
00:06:16,104 --> 00:06:17,784
or whatever robot assistant you want to do.

128
00:06:17,965 --> 00:06:20,815
You ask it a. Question today, I was getting out of the

129
00:06:20,815 --> 00:06:23,335
shower and I asked one of the bots, or out of, I asked,

130
00:06:23,335 --> 00:06:25,615
tried it with both Siri and Alexa, neither one could do it.

131
00:06:25,825 --> 00:06:28,975
What is the partial pressor of oxygen at 10,000 feet above sea level?

132
00:06:29,035 --> 00:06:30,775
And they both drop the thing completely.

133
00:06:30,805 --> 00:06:32,695
'cause I have weird shower thoughts roll with it.

134
00:06:32,965 --> 00:06:36,235
But it's the, but the question then is, okay, it can't do it.

135
00:06:36,235 --> 00:06:36,665
Of course it couldn't.

136
00:06:37,345 --> 00:06:40,705
I will never ask that question again despite the fact that maybe

137
00:06:40,825 --> 00:06:43,465
it does know how to do that, it just didn't hear me properly.

138
00:06:43,465 --> 00:06:46,165
Or in two weeks it will be able to answer that question.

139
00:06:46,345 --> 00:06:49,195
But when you get, you ask a question and it can't answer it, you kind of

140
00:06:49,195 --> 00:06:52,765
feel dumb for having thought for a second that it might have been able to.

141
00:06:53,580 --> 00:06:57,630
The problem that the Alexa group has had forever, which is, is that humans use

142
00:06:57,630 --> 00:07:02,159
something like 98% of all the features they will ever use on their Alexa device.

143
00:07:02,159 --> 00:07:05,099
Within 90 minutes of setting it up the first time, play a

144
00:07:05,099 --> 00:07:09,390
song, set a timer, turn on lights, and that's about it For most

145
00:07:09,390 --> 00:07:12,539
people, it's just because it gains a bunch of these features.

146
00:07:12,705 --> 00:07:14,385
How do you tell people about that?

147
00:07:14,474 --> 00:07:16,875
Turns out that finishing every sentence with, by the

148
00:07:16,875 --> 00:07:20,565
way, pitching something unrelated just pisses people off.

149
00:07:20,745 --> 00:07:23,775
I think there will be a change as the products get better

150
00:07:23,775 --> 00:07:27,765
and as, uh, knowledge bases get better as customers.

151
00:07:27,974 --> 00:07:30,435
Our customers and, you know, customers of other chat bots

152
00:07:30,645 --> 00:07:33,735
know how to work with them better, that the expectations of.

153
00:07:34,320 --> 00:07:37,980
Of us as consumers of users of these things will improve or change

154
00:07:37,980 --> 00:07:41,010
over time and just not assume that these things are as brain

155
00:07:41,010 --> 00:07:43,500
dead as when we started interacting with them first time around.

156
00:07:44,130 --> 00:07:46,470
So I'm curious as far as what you've been

157
00:07:46,470 --> 00:07:49,410
up to from a technical perspective lately.

158
00:07:49,890 --> 00:07:50,310
We, uh.

159
00:07:50,385 --> 00:07:52,185
We've known each other for many years.

160
00:07:52,185 --> 00:07:53,505
You have been to my house for dinner.

161
00:07:53,505 --> 00:07:55,755
You're friends with my brother, which I think most

162
00:07:55,755 --> 00:07:57,945
listeners will be shocked to realize I have one of those.

163
00:07:58,094 --> 00:07:58,605
It's true.

164
00:07:58,664 --> 00:07:59,025
I do.

165
00:07:59,085 --> 00:07:59,865
He lives in Belgium.

166
00:07:59,925 --> 00:08:00,284
Great.

167
00:08:00,555 --> 00:08:04,545
The, but what I found that was so interesting and got me talking

168
00:08:04,545 --> 00:08:07,485
to you is I was recently talking to some of the fine folks over at

169
00:08:07,485 --> 00:08:10,784
Planet Scale, and they have talked about Intercom in general, and

170
00:08:10,784 --> 00:08:14,625
you in particular as being very pleased with their database offering.

171
00:08:14,925 --> 00:08:15,224
Now.

172
00:08:15,435 --> 00:08:18,735
I have talked to people in the past where I ask them questions about that.

173
00:08:18,765 --> 00:08:20,355
The response has been a wait.

174
00:08:20,355 --> 00:08:22,095
A company's saying, what about me now?

175
00:08:22,215 --> 00:08:23,325
So, right.

176
00:08:23,565 --> 00:08:25,275
You and I go back long enough that I can

177
00:08:25,275 --> 00:08:27,705
trust you not to bullshit me on these things.

178
00:08:27,705 --> 00:08:28,545
So, okay.

179
00:08:28,965 --> 00:08:31,095
Is it as good as they're telling me it is.

180
00:08:31,095 --> 00:08:36,195
And your response to still down to, if not better, you are a big champion.

181
00:08:36,195 --> 00:08:37,605
What planet scale is up to.

182
00:08:38,145 --> 00:08:38,925
Tell me more please.

183
00:08:38,925 --> 00:08:40,425
'cause I don't, I do a lot of things here.

184
00:08:40,455 --> 00:08:42,225
Scale is generally not one of them.

185
00:08:42,225 --> 00:08:44,055
For my own personal shit posting projects,

186
00:08:44,745 --> 00:08:45,165
yeah.

187
00:08:45,600 --> 00:08:46,680
Short answer is yes.

188
00:08:46,680 --> 00:08:49,230
Plan scale is great, but I'm gonna give you a long answer as well.

189
00:08:49,290 --> 00:08:49,920
So.

190
00:08:50,265 --> 00:08:54,975
Intercom is a Ruby on Rails monolith, and we really, really like this setup.

191
00:08:55,365 --> 00:08:59,955
Uh, we deploy our Ruby on Rails app onto EC2 computers, not

192
00:08:59,955 --> 00:09:04,545
stuffed away, and Docker containers, and using incomprehensible

193
00:09:04,575 --> 00:09:07,455
three letter acronyms for different parts of, uh, the, the setup.

194
00:09:07,725 --> 00:09:08,085
And we,

195
00:09:08,265 --> 00:09:09,825
oh, you can tear down and rebuild your stack on

196
00:09:09,825 --> 00:09:12,015
top of the latest, trendy thing every 18 months.

197
00:09:12,015 --> 00:09:12,075
Huh?

198
00:09:12,375 --> 00:09:12,645
Wow.

199
00:09:12,645 --> 00:09:14,535
Almost like you're not based in Silicon Valley.

200
00:09:15,070 --> 00:09:17,200
Yeah, maybe being in Dublin has its advantages.

201
00:09:17,200 --> 00:09:21,970
So, uh, yeah, we, we run really boring infrastructure and we

202
00:09:21,970 --> 00:09:25,000
have stuck with Ruby on Rails and it's largely been great for us.

203
00:09:25,060 --> 00:09:27,460
Uh, you know, you have to do a lot of work to scale it out to

204
00:09:27,460 --> 00:09:29,870
millions of lines of code, hundreds of developers working in it.

205
00:09:30,405 --> 00:09:34,635
But you at some stage you have to connect the thing to a database and

206
00:09:34,694 --> 00:09:39,345
database scaling has been a large part of the problems, uh, but also

207
00:09:39,345 --> 00:09:42,645
the joy of scaling Intercom in the 11 years that I've been there.

208
00:09:42,704 --> 00:09:46,214
When Intercom started off, we had a very simple, nice MySQL database.

209
00:09:46,214 --> 00:09:46,224
We.

210
00:09:47,040 --> 00:09:50,160
Then unfortunately we hit product market fit, having

211
00:09:50,160 --> 00:09:53,070
explosive growth, that was, uh, very challenging.

212
00:09:53,340 --> 00:09:55,590
Um, and even though we were based in RDS, you know,

213
00:09:55,590 --> 00:09:57,930
we were cloud first at the start from the very start.

214
00:09:58,230 --> 00:10:01,170
But, uh, native RDS couldn't deal or deal

215
00:10:01,170 --> 00:10:02,910
easily with, with what we were doing with it.

216
00:10:03,090 --> 00:10:06,510
When Aurora came along, we, uh, it honestly was a game

217
00:10:06,510 --> 00:10:09,330
changer for us and we jumped in very aggressively.

218
00:10:09,480 --> 00:10:10,980
We got to work with the Aurora team.

219
00:10:10,980 --> 00:10:13,350
We had some of the biggest tables on Aurora at the time.

220
00:10:13,620 --> 00:10:15,510
They would do all sorts of custom work for us.

221
00:10:15,820 --> 00:10:21,640
It was good fun, and just the Aurora architecture itself, the split between Com

222
00:10:21,790 --> 00:10:26,260
compute and storage, and how low latency the read replicas were and everything.

223
00:10:26,530 --> 00:10:28,330
This stuff just bought us years and years and

224
00:10:28,330 --> 00:10:31,030
years of scalability up to a certain point.

225
00:10:31,030 --> 00:10:34,420
And then we started having to do things like sharding some of our data.

226
00:10:34,760 --> 00:10:37,880
We had tables that were so large we couldn't mutate, we couldn't add new

227
00:10:37,880 --> 00:10:42,890
columns or, or, uh, do kind of other MI database migrations on this data

228
00:10:42,890 --> 00:10:46,430
because it was changing so fast, uh, without taking a lot of downtime.

229
00:10:46,580 --> 00:10:47,780
So we had to take action.

230
00:10:47,780 --> 00:10:49,970
And so we built our own kind of database charting

231
00:10:49,970 --> 00:10:52,760
system, again, built on top of Aurora, choosing.

232
00:10:52,835 --> 00:10:55,505
Technologies that we were very comfortable with.

233
00:10:55,865 --> 00:10:58,415
Um, and this bought us again, like years of scalability.

234
00:10:58,445 --> 00:11:02,015
It bought us the ability to be able to do database migrations on our

235
00:11:02,015 --> 00:11:06,905
tables, and life was good for another few years, but over time these

236
00:11:06,905 --> 00:11:11,045
like different sharding patterns and and way we were kind of using

237
00:11:11,045 --> 00:11:16,475
Aurora meant that we had 13 clusters connected to one application and.

238
00:11:17,075 --> 00:11:20,525
You get into these unfortunate situations where AWS will say, Hey, we

239
00:11:20,525 --> 00:11:23,825
gotta patch out and you really need to apply it to all of your clusters.

240
00:11:23,975 --> 00:11:24,485
Yes, we are.

241
00:11:24,485 --> 00:11:27,815
Downtime will be at some point during this broad window

242
00:11:27,815 --> 00:11:31,115
that is inconvenient for you and non-deterministic.

243
00:11:31,145 --> 00:11:35,375
So you've gotta be able to build a graceful degradation mode into

244
00:11:35,375 --> 00:11:38,495
your app from the, if you're using this technology, 'cause you've,

245
00:11:38,555 --> 00:11:41,855
one of the things you lose by a, with a managed database offering.

246
00:11:42,265 --> 00:11:46,675
Is the ability to be very granular around when and what gets applied where.

247
00:11:47,125 --> 00:11:47,455
Yeah.

248
00:11:47,455 --> 00:11:51,775
And even some of the upgrades that we would do, most of

249
00:11:51,775 --> 00:11:54,475
the time the cluster would come back in a minute or two.

250
00:11:54,505 --> 00:11:57,385
You know, not, not bad, but occasionally we'd have, you

251
00:11:57,385 --> 00:11:59,605
know, something would get stuck inside of a queue inside

252
00:11:59,605 --> 00:12:03,565
of Amazon or something, and it would be 20 minutes and.

253
00:12:04,064 --> 00:12:06,615
This kind of stuff starts to wear you down, especially

254
00:12:06,615 --> 00:12:08,655
when you've got so many clusters connected to your app

255
00:12:08,655 --> 00:12:10,875
and you are highly faced, highly fit, highly critical.

256
00:12:10,875 --> 00:12:14,505
As far as customer facing stuff, you're on the front of everyone's website.

257
00:12:14,564 --> 00:12:17,535
It, this is one of those areas where mistakes will show.

258
00:12:17,685 --> 00:12:18,375
Absolutely.

259
00:12:18,375 --> 00:12:19,360
And you know, we have.

260
00:12:20,280 --> 00:12:25,260
Loads of customers, uh, who have up to thousands of people whose

261
00:12:25,260 --> 00:12:29,910
job it is to be using intercom all day, replying to their customers.

262
00:12:30,209 --> 00:12:34,620
And it's, it's definitely no fun when they have their entire

263
00:12:34,620 --> 00:12:38,520
teams unavailable, not able to do their job because we.

264
00:12:39,145 --> 00:12:42,025
We're twiddling our tongues, waiting for an Aurora upgrade to complete.

265
00:12:42,295 --> 00:12:44,605
Well just do it outside of core business hours.

266
00:12:44,665 --> 00:12:45,715
I'm in San Francisco.

267
00:12:45,715 --> 00:12:46,795
Lots of people here use it.

268
00:12:46,795 --> 00:12:47,545
You're in Dublin.

269
00:12:47,545 --> 00:12:48,775
Lots of people there use it.

270
00:12:48,775 --> 00:12:51,564
You have customers in Australia, Japan, India, around the world.

271
00:12:51,564 --> 00:12:52,885
It is core business hours.

272
00:12:53,005 --> 00:12:54,954
Somewhere for someone.

273
00:12:54,954 --> 00:12:58,975
At any given point of the day, there is no, and now it is nighttime.

274
00:12:58,975 --> 00:13:00,270
The servers can take a nap now.

275
00:13:00,860 --> 00:13:01,460
Absolutely.

276
00:13:01,730 --> 00:13:04,520
This is not the DMV in the US or the social security

277
00:13:04,520 --> 00:13:07,250
website in the United States, which still blows my mind.

278
00:13:07,370 --> 00:13:10,790
It has a six hour maintenance window every night, like the last

279
00:13:10,790 --> 00:13:13,580
person outta the office, like turns off the mainframe or something.

280
00:13:13,670 --> 00:13:15,530
Probably some legacy batch job or whatnot.

281
00:13:15,530 --> 00:13:17,300
And there's good reason for it, but it feels

282
00:13:17,300 --> 00:13:19,160
like that the servers keep banker's hours.

283
00:13:19,550 --> 00:13:23,240
So we were aware of VI Tess and Planet Scale.

284
00:13:23,240 --> 00:13:29,870
So Vi Tess, uh, I'll like the brief introduction to VI Tess is it's a MySQL.

285
00:13:30,275 --> 00:13:36,155
Wrapper as such, or system that came out of YouTube about 12, 13 years ago.

286
00:13:36,155 --> 00:13:37,415
It's an open source project.

287
00:13:38,115 --> 00:13:41,715
It's got other large SaaS, B2B SaaS providers,

288
00:13:41,715 --> 00:13:44,085
like people like Slack, HubSpot, and

289
00:13:44,145 --> 00:13:46,635
the Slack folks have been a huge advocate of this.

290
00:13:46,660 --> 00:13:49,845
It it makes sense since they've, they given talks about this where

291
00:13:49,845 --> 00:13:53,865
effectively all, all of Slack is basically a giant MySQL database.

292
00:13:53,895 --> 00:13:57,075
Sharded heavily obviously, but yeah, it is.

293
00:13:57,075 --> 00:14:00,915
Every message is a line in a database, so yeah.

294
00:14:00,960 --> 00:14:04,800
Having the database work and not take a nap for a 20 minute upgrade

295
00:14:04,800 --> 00:14:08,700
at random times is kind of high on their list of mu it must do this.

296
00:14:08,760 --> 00:14:09,030
Yeah.

297
00:14:09,030 --> 00:14:11,010
And it's, you know, it's not just downtime, it's.

298
00:14:11,505 --> 00:14:14,055
We, we need to be able to do things like

299
00:14:14,355 --> 00:14:16,335
have we, we have to share a lot of data.

300
00:14:16,335 --> 00:14:18,375
And our, our customer's data is extremely charitable.

301
00:14:18,375 --> 00:14:20,535
We have, we're a multi-tenant application.

302
00:14:20,535 --> 00:14:23,895
We have lots and lots of our customer's data and they don't need to

303
00:14:24,105 --> 00:14:27,735
join across their, like, different workspaces or different customers.

304
00:14:27,944 --> 00:14:29,805
So we have very, very charitable data.

305
00:14:30,105 --> 00:14:31,275
Um, the other thing we do is.

306
00:14:31,590 --> 00:14:34,710
We struggle with connection pool management.

307
00:14:34,950 --> 00:14:38,430
We have hundreds of thousands of ruby on rails processes that all need

308
00:14:38,430 --> 00:14:41,820
to connect to databases that can only take 16,000 connections at most.

309
00:14:42,240 --> 00:14:47,850
And, uh, so we have to run a, a layer of proxy sql, uh, connection

310
00:14:47,850 --> 00:14:52,020
proxies in between our application and our Aurora database.

311
00:14:52,200 --> 00:14:55,230
Sometimes the proxy SQL layer goes wrong, and it's just

312
00:14:55,230 --> 00:14:57,450
another layer of complication that we don't want to think of.

313
00:14:57,600 --> 00:15:00,600
So we're aware of a test and it became increasingly.

314
00:15:01,230 --> 00:15:04,830
Clear that planet scale were the way to get Vitas and we have

315
00:15:04,830 --> 00:15:08,310
no interest in as well, or like we'd rather avoid running our

316
00:15:08,310 --> 00:15:10,860
own infrastructure or running our own high level services.

317
00:15:11,100 --> 00:15:14,580
If we can pay somebody to run a database for us, we will absolutely do that.

318
00:15:14,700 --> 00:15:16,800
Now I, I'm going to stop you there because

319
00:15:16,800 --> 00:15:19,440
historically that has been the entire.

320
00:15:19,795 --> 00:15:22,885
Rallying cry of cloud where Oh great.

321
00:15:22,885 --> 00:15:24,625
You don't wanna run servers yourself.

322
00:15:24,625 --> 00:15:28,195
To a point where some people have now gone so far around the bend that we view

323
00:15:28,315 --> 00:15:32,275
running servers in data centers as being a skillset set the ancients possessed,

324
00:15:32,275 --> 00:15:36,685
but has since been lost to modern humanity outside of three hyperscalers.

325
00:15:37,405 --> 00:15:40,975
No, but those folks have been with a default

326
00:15:40,975 --> 00:15:43,855
go-to for a lot of things for a lot of years.

327
00:15:44,410 --> 00:15:47,230
You are looking, but your answer was not to go

328
00:15:47,230 --> 00:15:48,910
and yell at the Aurora team to make it better.

329
00:15:48,910 --> 00:15:50,230
It was to look somewhere else.

330
00:15:50,320 --> 00:15:52,900
You know, we did talk to the Aurora team about the problems

331
00:15:52,900 --> 00:15:56,710
and certainly Amazon are going in the right direction with the

332
00:15:56,710 --> 00:16:00,940
likes of Aurora Limitless, which does have like native sharding.

333
00:16:01,545 --> 00:16:04,814
Um, it's ex, it is a Postgres setup, but it's,

334
00:16:04,814 --> 00:16:06,464
they're, they're thinking about it in the right way.

335
00:16:06,464 --> 00:16:08,775
But, and they do have things like RDS proxy,

336
00:16:08,775 --> 00:16:10,365
which could do some of these proxy things.

337
00:16:10,574 --> 00:16:13,665
So they do have these building blocks and they, some of the problems

338
00:16:13,665 --> 00:16:17,655
we could solve or maybe swap out with some Amazon managed services.

339
00:16:17,865 --> 00:16:19,365
But really we were looking for something,

340
00:16:20,055 --> 00:16:23,204
something a bit bigger and better and where.

341
00:16:23,710 --> 00:16:26,320
Actually serving queries, actually serving customers.

342
00:16:26,320 --> 00:16:30,280
Our customers is the problem of the provider that we have.

343
00:16:30,520 --> 00:16:34,660
We don't just want to be getting a proxy service from one part of

344
00:16:34,660 --> 00:16:39,430
the company and limited insights or no ability to go in and help us

345
00:16:39,430 --> 00:16:42,760
out with bad queries or give us insights into what's going wrong.

346
00:16:42,760 --> 00:16:43,660
You know, we wanna.

347
00:16:43,980 --> 00:16:47,850
We, we really need somebody who's like a partner who can go deeper

348
00:16:47,850 --> 00:16:51,990
into our problems and share our problems and not just be hands

349
00:16:51,990 --> 00:16:56,760
off with them, which, you know, at Amazon due to scale and due

350
00:16:56,760 --> 00:17:00,810
to the way they treat due security and a bunch of other reasons,

351
00:17:01,470 --> 00:17:05,069
uh, they don't act that way in their day-to-day operations.

352
00:17:05,069 --> 00:17:06,630
You can convince them eventually kind of to

353
00:17:06,630 --> 00:17:08,670
get into certain things, but it's certainly.

354
00:17:09,115 --> 00:17:13,915
They don't have one small solution that it fixes all the problems that we

355
00:17:13,915 --> 00:17:16,615
want, like connection, pooling, sharding, fast failovers and everything.

356
00:17:16,795 --> 00:17:19,974
They're kind of just vending, a bunch of building blocks and maybe it's just

357
00:17:20,035 --> 00:17:24,240
because they're two pizza teams the whole way down and but planet scale are I.

358
00:17:25,020 --> 00:17:29,520
A good, healthy, up and coming company who we liked the look of.

359
00:17:29,520 --> 00:17:32,310
We liked the way that they were talking about providing

360
00:17:32,310 --> 00:17:35,460
managed VI tests in their into companies like us.

361
00:17:35,700 --> 00:17:38,520
I think what we liked about Planet Scale was like they were clearly

362
00:17:38,520 --> 00:17:42,030
building for companies like us using a technology built for.

363
00:17:42,449 --> 00:17:43,080
Exactly.

364
00:17:43,080 --> 00:17:44,100
Customers like us.

365
00:17:44,399 --> 00:17:49,620
Um, and you know, the, the kind of way like the, they

366
00:17:49,620 --> 00:17:52,770
kind of like a one stop shop white glove service.

367
00:17:53,070 --> 00:17:55,770
You just show up, send your queries at their database and

368
00:17:55,770 --> 00:17:58,709
they'll do the rest as opposed to you need to assemble

369
00:17:58,709 --> 00:18:01,590
a, a. A variety of building blocks and hope for the best.

370
00:18:01,649 --> 00:18:03,659
The one challenge I see coming outta the planet scale

371
00:18:03,659 --> 00:18:05,370
folks, they have, they have amazing talent there.

372
00:18:05,370 --> 00:18:07,649
Richard Crowley I've known for years is phenomenal.

373
00:18:07,649 --> 00:18:10,139
Sam Lambert is the CEO and he is there.

374
00:18:10,199 --> 00:18:10,770
Kidding, kidding.

375
00:18:10,949 --> 00:18:13,949
Uh, they have a bunch of terrific folk working there, but I find that the

376
00:18:13,949 --> 00:18:17,399
way that their position of the stories they tell are aligned perfectly.

377
00:18:17,429 --> 00:18:20,459
For folks like you, you are deep in the weeds.

378
00:18:20,459 --> 00:18:21,600
You know this stuff cold.

379
00:18:21,600 --> 00:18:24,870
You have been running hyperscale systems for many years.

380
00:18:25,650 --> 00:18:26,400
Terrific.

381
00:18:26,640 --> 00:18:29,220
There are a lot more people that look like me, by which I

382
00:18:29,220 --> 00:18:32,430
mean dumb out in the universe than there are people like you.

383
00:18:32,430 --> 00:18:36,420
So making it a a broader mass market appeal seems like it's not the story

384
00:18:36,420 --> 00:18:40,050
they're telling at the moment, which is kind of a shame because based on

385
00:18:40,050 --> 00:18:43,350
the stories I've had with you and others at the conversations around this.

386
00:18:43,710 --> 00:18:46,860
They're, they're solving a problem that meets an awful lot of people.

387
00:18:47,250 --> 00:18:49,350
It a awful lot of people's problems.

388
00:18:49,560 --> 00:18:52,800
Uh, I will also say that this reinforces a belief I've had

389
00:18:52,800 --> 00:18:56,370
for many years, which is as val, as things move up the stack,

390
00:18:56,670 --> 00:18:59,850
the value and the margins increase by being able to do it.

391
00:19:00,120 --> 00:19:02,969
Amazon has got the low level infrastructure stuff on lock.

392
00:19:02,969 --> 00:19:06,360
No one is gonna build a better VM platform than they're doing.

393
00:19:06,629 --> 00:19:08,879
Uh, their reliability is untouchable.

394
00:19:08,969 --> 00:19:11,370
They have all kinds of great baseline foundational

395
00:19:11,370 --> 00:19:14,010
services, but every time they try to move up the stack

396
00:19:14,010 --> 00:19:17,129
into applications or things a little further up the chain.

397
00:19:17,770 --> 00:19:19,300
They fail miserably.

398
00:19:19,330 --> 00:19:23,980
They've never yet built a good user interface on anything Amazon has ever done.

399
00:19:23,980 --> 00:19:27,190
We all learned to use their website, not because it's good, because we

400
00:19:27,190 --> 00:19:30,460
have to, and what we're seeing with things like planet Scale is they're

401
00:19:30,460 --> 00:19:33,610
now the, the rest of the industry is starting to erode some of those

402
00:19:33,610 --> 00:19:37,930
things and come further down faster than Amazon is able to go up the stack.

403
00:19:38,170 --> 00:19:40,240
I, it's not just things like planet Scale.

404
00:19:40,240 --> 00:19:43,240
We see it with Snowflake, Databricks, uh, a whole bunch

405
00:19:43,240 --> 00:19:45,639
of other folks out there that are doing these things.

406
00:19:45,639 --> 00:19:48,129
People are using Confluence, uh, sorry, confluent

407
00:19:48,430 --> 00:19:51,520
instead of running their own Kafka clusters or MSK.

408
00:19:51,790 --> 00:19:55,720
It's, it's, those companies are eroding AWS, they're charging

409
00:19:55,720 --> 00:19:59,470
more in some cases, but delivering vastly superior value.

410
00:19:59,889 --> 00:20:01,985
And this tells me in the future.

411
00:20:02,675 --> 00:20:05,314
Unless, so, they're gonna come out with something I can't foresee.

412
00:20:05,614 --> 00:20:10,235
Amazon is going to become the equivalent of the layer one backbone providers.

413
00:20:10,534 --> 00:20:11,675
The, they're gonna be next.

414
00:20:11,705 --> 00:20:15,995
Like if NTT goes down, the internet isn't working so well today and we're all

415
00:20:15,995 --> 00:20:18,844
having a bad time, but most people don't know what the hell that company is.

416
00:20:19,530 --> 00:20:21,810
Everything and the, all the value rides on top of them,

417
00:20:21,899 --> 00:20:24,030
and I think that's Amazon's future given their course.

418
00:20:24,149 --> 00:20:24,330
Yeah.

419
00:20:24,330 --> 00:20:27,270
We've seen this isn't a pattern that is, we've only

420
00:20:27,270 --> 00:20:29,490
seen with, with the move from Aurora to Planet Scale.

421
00:20:29,610 --> 00:20:32,940
We had the exact same with a move from Redshift to Snowflake.

422
00:20:33,750 --> 00:20:36,720
Uh, and again, we worked closely with Amazon.

423
00:20:36,720 --> 00:20:39,480
We tried to resolve our Redshift stability problems with them.

424
00:20:39,899 --> 00:20:41,850
They gave us a bunch of things to do, but ultimately.

425
00:20:42,584 --> 00:20:46,425
When we were moved to Snowflake, not only was the technology

426
00:20:46,425 --> 00:20:49,725
just that bit better, they were just able to ship for

427
00:20:49,725 --> 00:20:53,415
us or a bit more responsive on solving for our needs.

428
00:20:53,564 --> 00:20:56,205
But that bit hungrier of where we didn't feel like we

429
00:20:56,205 --> 00:20:59,264
were just one out of a million customers for Redshift.

430
00:20:59,594 --> 00:21:04,544
Um, with Snowflake, we got stuff turned around quickly and the thing has been.

431
00:21:05,100 --> 00:21:07,800
Pretty awesome as well and just kind of left Amazon behind.

432
00:21:08,070 --> 00:21:13,020
I think once things are business critical for us, uh, and

433
00:21:13,020 --> 00:21:16,060
they're higher, higher level applications, uh, I think it's.

434
00:21:16,980 --> 00:21:19,710
We're at a point now where we'd be considering taking

435
00:21:19,710 --> 00:21:22,290
it off Amazon rather than trying to fix it on Amazon.

436
00:21:22,770 --> 00:21:24,960
Something that could become important could be,

437
00:21:25,080 --> 00:21:29,040
say, DDoS has became really problematic for us.

438
00:21:29,159 --> 00:21:30,420
At the moment, we just use waf.

439
00:21:30,450 --> 00:21:31,710
You know, we use the Amazon stack.

440
00:21:31,710 --> 00:21:32,310
It's fine.

441
00:21:32,310 --> 00:21:32,909
It's not that.

442
00:21:33,125 --> 00:21:34,024
Big a deal for us.

443
00:21:34,415 --> 00:21:38,344
But if we really had to nail the DDoS problem, I'd probably go to

444
00:21:38,344 --> 00:21:41,975
CloudFlare and wouldn't, wouldn't stick around with Amazon for too long.

445
00:21:42,245 --> 00:21:46,594
Kind of on the understanding that I think Amazon probably do a reasonably

446
00:21:46,594 --> 00:21:49,655
good job and will, you know, they'll take support tickets and whatnot.

447
00:21:49,895 --> 00:21:52,715
You, if you're going to go a DDoS, you definitely need to talk to your provider.

448
00:21:52,715 --> 00:21:54,485
There's no real way around that.

449
00:21:54,514 --> 00:21:54,695
Yeah.

450
00:21:54,695 --> 00:21:58,294
And their team is excellent, but the, the customer touch

451
00:21:58,294 --> 00:22:02,615
points, they're, they're not really, I'm sorry, but they aren't.

452
00:22:02,725 --> 00:22:07,915
And, you know, AWS support, it can be tough to get listened to at times.

453
00:22:07,915 --> 00:22:10,135
Like I've done On-Call, a lot of on-call and

454
00:22:10,225 --> 00:22:13,465
opened a lot of, uh, issues with AWS support.

455
00:22:13,705 --> 00:22:16,885
And even just knowing how to open a case, it's like it's pretty difficult.

456
00:22:17,125 --> 00:22:19,495
Whereas if I'm opening a support case with some

457
00:22:19,495 --> 00:22:22,584
of our providers, like Open AI or incident.io or.

458
00:22:22,675 --> 00:22:25,345
Or Snowflake or, or or planet scale.

459
00:22:25,465 --> 00:22:27,625
Very often it's little more than a message in a

460
00:22:27,625 --> 00:22:30,325
Slack channel and all of their automation kicks in.

461
00:22:30,325 --> 00:22:32,665
You get routed to the right person very, very quickly and

462
00:22:32,665 --> 00:22:35,034
they're able to tell us very quickly if it's our problem or their

463
00:22:35,034 --> 00:22:37,945
problem, as opposed to what Amazon of where they're gonna try and

464
00:22:37,945 --> 00:22:41,185
catch you out, asking you like which region your problem is in.

465
00:22:41,185 --> 00:22:43,465
And uh, that can be frustrating at times again.

466
00:22:43,920 --> 00:22:45,690
It's a problem of scale and I kind of get it.

467
00:22:46,020 --> 00:22:48,690
Um, but the experience is way more tailored to

468
00:22:48,690 --> 00:22:52,140
our needs from smaller, hungrier c uh, companies.

469
00:22:52,140 --> 00:22:53,010
And our experience.

470
00:22:53,100 --> 00:22:56,610
This episode is sponsored by my own company, the

471
00:22:56,610 --> 00:22:59,890
Duck Bill Group, having trouble with your AWS bill.

472
00:23:00,245 --> 00:23:03,185
Perhaps it's time to renegotiate a contract with them.

473
00:23:03,395 --> 00:23:05,495
Maybe you're just wondering how to predict

474
00:23:05,495 --> 00:23:08,584
what's going on in the wide world of AWS.

475
00:23:08,735 --> 00:23:11,705
Well, that's where the Duck Bill group comes in to help.

476
00:23:12,125 --> 00:23:14,375
Remember, you can't duck the duck bill.

477
00:23:14,375 --> 00:23:16,475
Bill, which I am reliably informed by my

478
00:23:16,475 --> 00:23:19,145
business partner is absolutely not our motto.

479
00:23:20,280 --> 00:23:22,620
I, I think that you're right and it's kind of sad.

480
00:23:22,710 --> 00:23:27,660
It, it also, if I'm reading trends, it feels like Amazon is moving away

481
00:23:27,660 --> 00:23:30,990
on the AWS side, at least from product led growth and speaking explicitly

482
00:23:30,990 --> 00:23:35,010
to large enterprises and okay, maybe it's the right answer for them.

483
00:23:35,010 --> 00:23:37,680
Lord knows they have better strategic insight into their customers and

484
00:23:37,680 --> 00:23:40,380
their needs and their growth patterns than I do sitting in the cheap seats.

485
00:23:40,770 --> 00:23:44,010
But what attracted me to it was the fact that I could get started with these

486
00:23:44,010 --> 00:23:46,950
things for pennies and so much of what they're coming out with these days.

487
00:23:47,360 --> 00:23:49,070
A prerequisite enterprise support, which

488
00:23:49,070 --> 00:23:52,400
starts at $180,000 a year and ends nowhere.

489
00:23:52,400 --> 00:23:53,120
It never ends.

490
00:23:53,120 --> 00:23:56,300
It grows as an unbounded growth problem like an AWS bill itself,

491
00:23:56,660 --> 00:23:59,990
and that is that that rules out a lot of things that I'd want

492
00:23:59,990 --> 00:24:03,080
to kick the tires on unless I start taking hostages again.

493
00:24:03,380 --> 00:24:06,500
You know, the fire hose of AWS.

494
00:24:06,985 --> 00:24:09,775
Updates and launches and stuff.

495
00:24:10,075 --> 00:24:12,895
I think the hit rate for me of where I see something that

496
00:24:12,895 --> 00:24:15,505
I'm actually gonna try out or where, where I'm thinking,

497
00:24:15,685 --> 00:24:18,385
Hey, they're nailing this, they're solving our problems.

498
00:24:18,685 --> 00:24:20,905
Um, I think that's gone down over the years.

499
00:24:21,025 --> 00:24:22,975
I'm sure they're crying into their money.

500
00:24:23,665 --> 00:24:25,285
Uh, I'm sure they're, they're, they've

501
00:24:25,285 --> 00:24:26,905
got some pretty good businesses out there.

502
00:24:27,030 --> 00:24:30,745
But, uh, for the kind of mid-range cost.

503
00:24:31,199 --> 00:24:34,169
Like tech first company, it seems like they're, they're not the

504
00:24:34,169 --> 00:24:36,870
lead, the leaders that they used to be because I think with the

505
00:24:36,870 --> 00:24:39,689
likes of Aurora and Redshift, maybe they had like early mover

506
00:24:39,689 --> 00:24:42,689
advantage 'cause they had obviously access to cloud services.

507
00:24:43,350 --> 00:24:46,260
Before the cloud existed, uh, they were able to build like

508
00:24:46,320 --> 00:24:50,340
really great, um, cloud specific services on top of that.

509
00:24:50,490 --> 00:24:53,639
But I think they've been outpaced by hungrier competitors at this point.

510
00:24:53,639 --> 00:24:55,379
And, you know, it's good for us.

511
00:24:55,379 --> 00:24:58,710
We're able to take advantage of these and so I'm kind of happy to do that.

512
00:24:58,710 --> 00:25:02,430
But, um, I think it's, I'm kind of quietly sad for, for Amazon as well.

513
00:25:02,430 --> 00:25:02,460
I.

514
00:25:03,990 --> 00:25:04,650
I am too.

515
00:25:05,220 --> 00:25:07,200
Everyone seems to think I have an axe to grind against

516
00:25:07,200 --> 00:25:09,840
AWS but it comes from being close to them for so long.

517
00:25:09,840 --> 00:25:11,520
I, I don't hate the company.

518
00:25:11,520 --> 00:25:14,550
If I did what I did for a company I hated, that's a pathology

519
00:25:14,550 --> 00:25:17,010
and I need a diagnosis and probably a restraining order.

520
00:25:17,340 --> 00:25:18,690
Uh, it's, I like what they do.

521
00:25:18,690 --> 00:25:20,130
I want them to be better than they are.

522
00:25:20,130 --> 00:25:22,350
I want the offerings to improve over time.

523
00:25:22,680 --> 00:25:25,200
I just, I don't see, that's the direction it's going in the way

524
00:25:25,200 --> 00:25:28,155
that it once was, and it brings me no joy whatsoever to say that.

525
00:25:28,995 --> 00:25:31,604
Um, I mean, one of the good things about Amazon is that

526
00:25:31,935 --> 00:25:34,209
they do want to hear this stuff like saying this exact.

527
00:25:34,965 --> 00:25:37,845
Uh, thing to many, many people in Amazon.

528
00:25:37,875 --> 00:25:41,115
Over the last while, um, my excellent account

529
00:25:41,115 --> 00:25:44,655
manager has been setting me up with various leaders.

530
00:25:45,045 --> 00:25:47,895
Uh, they've been asking for documents, they've been asking for examples.

531
00:25:48,045 --> 00:25:50,805
You know, they're hungry for this stuff, so I don't, don't doubt

532
00:25:50,865 --> 00:25:54,705
that there's no desire to be the leaders or to, to really satisfy

533
00:25:54,705 --> 00:25:56,775
their customers, but, you know, it's execution we care about.

534
00:25:57,015 --> 00:25:58,255
And when it comes down to it.

535
00:25:59,280 --> 00:26:00,750
We need excellent databases.

536
00:26:00,780 --> 00:26:02,850
We need the best databases to be able to

537
00:26:02,850 --> 00:26:04,770
ship world-class product to our customers.

538
00:26:05,040 --> 00:26:06,330
And I think that that's important.

539
00:26:06,570 --> 00:26:09,150
It's, it, it's the, it's the need the customers have.

540
00:26:09,150 --> 00:26:10,830
And if the cloud provider won't give it to

541
00:26:10,830 --> 00:26:12,390
them, they will find ways to meet that need.

542
00:26:12,390 --> 00:26:13,200
It's what they do.

543
00:26:13,650 --> 00:26:16,380
Uh, a last topic that I want to get into, it's been a recurring

544
00:26:16,380 --> 00:26:18,990
theme throughout the years on this show, which is where does the.

545
00:26:19,300 --> 00:26:23,260
Generation come from because people like you and people like me who came up

546
00:26:23,260 --> 00:26:27,070
being, you know, support folk in uh, the early days back when this was all

547
00:26:27,070 --> 00:26:30,220
an open field and no one really knew how computers were supposed to work.

548
00:26:30,250 --> 00:26:33,520
Not that we do now, but we lie to ourselves and would, we gathered

549
00:26:33,520 --> 00:26:36,129
experience and came from those places to where we are now.

550
00:26:36,430 --> 00:26:38,379
That door has been firmly shut.

551
00:26:38,440 --> 00:26:40,899
That is not a path that.

552
00:26:41,475 --> 00:26:43,065
Is open, at least for me.

553
00:26:43,695 --> 00:26:44,504
Where do you come from?

554
00:26:44,534 --> 00:26:46,575
How did you get to the place that you are now?

555
00:26:46,754 --> 00:26:50,595
Yeah, so the fun part about my career, I've had just so much

556
00:26:50,595 --> 00:26:55,220
luck and fortune and random timing things, uh, that have.

557
00:26:55,650 --> 00:26:57,180
It worked out reasonably well.

558
00:26:57,180 --> 00:26:59,250
I don't think my career has been too bad to date, but

559
00:26:59,250 --> 00:27:04,830
it all started in 1997 or so when I went to university.

560
00:27:04,860 --> 00:27:09,390
And, uh, we had a, what was then called a networking society,

561
00:27:09,570 --> 00:27:13,050
which was basically a bunch of students running a few Unix boxes.

562
00:27:13,230 --> 00:27:16,140
And this was kind of in the pre-social media age where.

563
00:27:16,760 --> 00:27:20,990
We didn't have WhatsApp for, even Facebook or anything to talk to each other.

564
00:27:21,230 --> 00:27:25,070
So the obvious thing that we did back then was we had a

565
00:27:25,070 --> 00:27:29,540
large proportion of the people in our university log on to a

566
00:27:29,540 --> 00:27:33,320
Unix shell, uh, on a bunch of servers run by, uh, students.

567
00:27:33,905 --> 00:27:37,415
Uh, who, who, and we all, we struggled to keep these things online.

568
00:27:37,415 --> 00:27:41,675
It was pretty tough running these kind of Solaris uh, servers.

569
00:27:41,675 --> 00:27:43,805
And we had like instant messaging.

570
00:27:43,805 --> 00:27:45,695
We had these wrappers around, right?

571
00:27:46,205 --> 00:27:49,085
If you're old enough, you might remember, right?

572
00:27:49,175 --> 00:27:51,754
And we had like really healthy news groups and IRC and stuff like that.

573
00:27:51,754 --> 00:27:53,315
So we had this like super, super.

574
00:27:53,815 --> 00:27:57,805
Awesome community of people who are partially of people who are interested in

575
00:27:57,955 --> 00:28:01,825
doing cool stuff with tech, learning about Unix, learning about networking.

576
00:28:01,885 --> 00:28:05,665
And we had a lot of users and just like we were the largest society

577
00:28:05,665 --> 00:28:09,205
on campus, we two great parties and it was pretty, pretty cool.

578
00:28:09,385 --> 00:28:14,695
Um, but totally co uh, totally coincidentally, or like through Fortune.

579
00:28:15,385 --> 00:28:18,445
Uh, there were also some people who've ended up being like, really?

580
00:28:19,080 --> 00:28:24,090
Notable in the tech community since the likes of say, John Looney, Tanya Riley,

581
00:28:24,120 --> 00:28:29,460
Colin McCarthy, and the list goes on of people who kind of started off their

582
00:28:29,460 --> 00:28:34,380
careers in technology just tinkering around on these Unix boxes back in college.

583
00:28:34,710 --> 00:28:35,760
Um, and.

584
00:28:36,270 --> 00:28:41,610
My career largely up until maybe when I joined Intercom, um, it was all

585
00:28:41,610 --> 00:28:46,020
about getting doors open by knowing people, uh, through that community,

586
00:28:46,169 --> 00:28:50,850
staying in touch with them, you know, doing, um, doing things, whether

587
00:28:50,850 --> 00:28:54,240
it was our local Linux user group or different activities like that.

588
00:28:54,450 --> 00:28:59,610
Um, but really having, uh, good fortune from meeting a bunch of early, I guess,

589
00:28:59,669 --> 00:29:04,350
Unix tinkerers or CISA bins back in the day who all then kind of grew into, uh.

590
00:29:04,765 --> 00:29:06,535
Working in various places in the industry

591
00:29:06,535 --> 00:29:10,345
and, um, but my own, uh, where I went after.

592
00:29:10,860 --> 00:29:14,429
Tinkering around Unison College was into Solaris technical support.

593
00:29:14,669 --> 00:29:17,340
Then that moved into like real cis admin work.

594
00:29:17,639 --> 00:29:23,550
Then later into like running, uh, building out nationwide broadband networks in,

595
00:29:23,580 --> 00:29:27,750
um, in, in Ireland and connecting every school in the state and building out ISP

596
00:29:27,750 --> 00:29:31,679
services and, and all the, it was like a mix of classic system in, and a bunch

597
00:29:31,679 --> 00:29:36,240
of automation increasingly automated as, as things got better in, in technology.

598
00:29:36,690 --> 00:29:39,060
Then for a while as well, I was in this small

599
00:29:39,060 --> 00:29:42,420
bookseller called Amazon prior to, um, Intercom as well.

600
00:29:42,660 --> 00:29:46,140
Um, but I guess I had like a bit of a classic, well, what I consider to

601
00:29:46,140 --> 00:29:51,240
be a classic mo move kind of up the stack from cis admin help desk to

602
00:29:51,270 --> 00:29:54,300
writing more software, maybe a bit of management leadership, and then

603
00:29:54,300 --> 00:29:57,900
ultimately into the kind of tech leadership area that I'm in at the moment.

604
00:29:59,340 --> 00:30:00,480
Like, where do people come from?

605
00:30:00,480 --> 00:30:04,530
I mean, like, we certainly don't have the pipeline of lots of people

606
00:30:04,530 --> 00:30:07,650
sitting around like building ISPs or building hosting providers.

607
00:30:07,740 --> 00:30:09,630
I think these are all like really solve problems.

608
00:30:09,630 --> 00:30:13,020
You don't have that kind of tinkering or just hands-on work

609
00:30:13,020 --> 00:30:16,770
that you need to, to build the, to build and use these services.

610
00:30:16,770 --> 00:30:19,110
And so that, that kind of gateway.

611
00:30:19,500 --> 00:30:23,610
Into, uh, running services, infrastructure services, networking and all

612
00:30:23,610 --> 00:30:27,899
that, uh, doesn't, it's isn't obviously there as much at the moment, I think.

613
00:30:27,990 --> 00:30:30,570
No, that's the problem is the, the provider, the cloud providers

614
00:30:30,570 --> 00:30:34,320
abstracted so much of that away that I know a lot of folks

615
00:30:34,320 --> 00:30:37,830
at hyperscale born in the cloud environments like Intercom.

616
00:30:37,950 --> 00:30:39,870
I'll even ask you this, this is not.

617
00:30:39,930 --> 00:30:43,140
Necessarily, please stab coworkers in the back, but

618
00:30:43,140 --> 00:30:46,470
looking at your, at your technical team across the board,

619
00:30:46,800 --> 00:30:49,590
what is the depth of networking knowledge at Intercom?

620
00:30:49,890 --> 00:30:51,420
Oh, I think I am the networking team.

621
00:30:53,490 --> 00:30:54,330
And you're no slouching it.

622
00:30:54,330 --> 00:30:56,880
I wanna be very clear, but I gave a keynote at Nano about

623
00:30:56,880 --> 00:31:00,660
this last year where this is a, this is a perennial problem.

624
00:31:00,720 --> 00:31:03,930
I was talking to folks at AWS about this, where a lot of your

625
00:31:03,930 --> 00:31:06,815
customers do not have a deep bench of networking knowledge and.

626
00:31:07,180 --> 00:31:09,970
They make the very reasonable response of, well, that's not true.

627
00:31:09,970 --> 00:31:12,010
We were talking to a customer this morning and they

628
00:31:12,010 --> 00:31:15,220
had, they were as good at this as we are by easily.

629
00:31:15,460 --> 00:31:16,030
I'm like, great.

630
00:31:16,180 --> 00:31:19,780
Just outta curiosity, what sector was that customer in?

631
00:31:19,870 --> 00:31:20,710
Oh, they're a telco.

632
00:31:20,710 --> 00:31:20,980
Why?

633
00:31:21,195 --> 00:31:21,284
Hmm.

634
00:31:22,215 --> 00:31:23,655
Wonder if that has anything to do with it.

635
00:31:23,804 --> 00:31:23,925
Hmm.

636
00:31:23,985 --> 00:31:24,615
Imagine that.

637
00:31:24,794 --> 00:31:27,885
But these board in the cloud, standup companies don't do networking

638
00:31:27,885 --> 00:31:30,314
because you don't need to know networking until suddenly.

639
00:31:30,314 --> 00:31:34,304
You very much need to know networking, but you can go an entire career

640
00:31:34,335 --> 00:31:36,675
weaseling your way between the cracks without having to pick it up.

641
00:31:37,020 --> 00:31:39,300
You know, some of our recent hires, we've, we've we're

642
00:31:39,300 --> 00:31:42,419
fortunate enough to be close enough to a fairly large Amazon

643
00:31:42,419 --> 00:31:45,750
office, and we've hired a good few people from Amazon support.

644
00:31:46,080 --> 00:31:48,419
Um, so maybe it's shifted up the stack.

645
00:31:48,510 --> 00:31:51,719
You know, it's no longer people who are building ISPs or hosting providers.

646
00:31:51,719 --> 00:31:54,574
It's a. People who work for larger providers do in

647
00:31:54,574 --> 00:31:57,094
kind of entry level tech roles or support type roles.

648
00:31:57,304 --> 00:32:00,365
So there's, there's something of maybe le looks like the old pipelines.

649
00:32:00,365 --> 00:32:03,334
It's not the exact same, definitely different shape of people.

650
00:32:03,334 --> 00:32:06,544
Um, with, uh, they tend to be actually better at coding

651
00:32:06,544 --> 00:32:09,935
than I was back in, but when I was at their uh, level.

652
00:32:09,935 --> 00:32:11,135
But, um.

653
00:32:12,235 --> 00:32:16,285
The, it does seem like it's, uh, with AI coming in as

654
00:32:16,285 --> 00:32:18,895
well, it seems like there's gonna be a good bit of change.

655
00:32:19,254 --> 00:32:21,565
Uh, so like where people, like what skills

656
00:32:21,565 --> 00:32:25,195
people use and grow and need in their careers.

657
00:32:25,495 --> 00:32:28,975
Um, there, you know, there's concerns at the moment that the

658
00:32:28,975 --> 00:32:33,145
use of AI and engineering and to write code, we'll, like remove

659
00:32:33,145 --> 00:32:36,865
the need for junior engineers or like, just we'll, we'll, uh,

660
00:32:36,895 --> 00:32:40,105
maxim or benefit largely senior engineers or people who can.

661
00:32:40,935 --> 00:32:42,165
Guides the agent.

662
00:32:42,735 --> 00:32:44,655
Uh, m uh, LLM.

663
00:32:45,389 --> 00:32:50,340
Coding tools rather than, you know, working your way up by working,

664
00:32:50,370 --> 00:32:53,669
by working on small problems and building and shipping things.

665
00:32:53,970 --> 00:32:57,690
Um, so I think in tech, you know, there's a lot of change.

666
00:32:57,899 --> 00:33:01,200
Certainly there's many entry paths, uh, like the one that I take,

667
00:33:01,200 --> 00:33:04,169
which I think are gone, um, since some kind of replacements, but.

668
00:33:04,580 --> 00:33:07,250
Do worry about, like especially in areas like networking

669
00:33:07,250 --> 00:33:10,159
and uh, low level Unix and stuff that we're not seeing

670
00:33:10,159 --> 00:33:12,980
the kind of depth or knowledge that, uh, we used to have.

671
00:33:12,980 --> 00:33:15,064
And I don't think I'm just being bitter and old about that.

672
00:33:15,169 --> 00:33:16,820
I think it is pretty useful stuff to know.

673
00:33:17,419 --> 00:33:17,780
Oh it is.

674
00:33:18,050 --> 00:33:19,909
Uh, one thing I wanna point out, 'cause this is a recurring theme

675
00:33:19,909 --> 00:33:22,939
that I see a lot where you mentioned a few extraordinary names of

676
00:33:22,939 --> 00:33:25,550
people who are terrific and in the space that have been formative

677
00:33:25,550 --> 00:33:28,250
influences, would it surprise you to know that when I've spoken to

678
00:33:28,280 --> 00:33:31,010
multiple of those people, your name comes up in the same context?

679
00:33:31,865 --> 00:33:34,115
People don't realize that we all learn from each other.

680
00:33:34,235 --> 00:33:37,175
It's, it's one of those things where, oh yeah, those people are smart.

681
00:33:37,175 --> 00:33:38,615
I'm just an idiot sitting here.

682
00:33:38,615 --> 00:33:40,685
It, it's a common pattern and I think we

683
00:33:40,685 --> 00:33:44,315
internalize it pretty well, but it shows there.

684
00:33:44,345 --> 00:33:46,925
There's one other aspect I want to get into about Intercom.

685
00:33:47,105 --> 00:33:49,145
I was going to mention earlier, but we, we

686
00:33:49,145 --> 00:33:50,495
got, we've done this conversational path.

687
00:33:50,705 --> 00:33:52,175
It's one of those interesting things about it.

688
00:33:52,505 --> 00:33:56,475
Uh, I. Wound up focusing my skillset, which is not that dissimilar

689
00:33:56,475 --> 00:34:00,495
to yours on AWS bills, because I wanted a specific expensive

690
00:34:00,495 --> 00:34:03,314
problem eight years ago when I was getting started down this path.

691
00:34:03,554 --> 00:34:04,155
Nine years now.

692
00:34:04,155 --> 00:34:04,635
My God.

693
00:34:05,024 --> 00:34:08,264
And what the reason I did it was I was down

694
00:34:08,264 --> 00:34:10,725
to this or IAM like did I know a lot about?

695
00:34:10,725 --> 00:34:11,835
I am at the time, no.

696
00:34:11,835 --> 00:34:13,815
But I didn't know that much about AWS Bills either.

697
00:34:14,165 --> 00:34:16,775
Turns out when you focus on things, you could pick up a lot.

698
00:34:17,105 --> 00:34:19,235
But the reason I went with bills is because

699
00:34:19,235 --> 00:34:21,695
there is never a 2:00 AM billing emergency.

700
00:34:21,875 --> 00:34:25,445
I've had enough horrifying on-call experiences in my career that

701
00:34:25,445 --> 00:34:30,335
I am effectively done with it inter all companies across the board

702
00:34:30,364 --> 00:34:33,290
have on-call because they need this stuff to work in various ways.

703
00:34:33,310 --> 00:34:36,154
And you don't have every team have representatives at

704
00:34:36,215 --> 00:34:39,185
every hour around the clock in a follow the sun rotation.

705
00:34:39,935 --> 00:34:42,455
Intercom takes a unique approach to this, to my understanding.

706
00:34:42,605 --> 00:34:43,205
Tell me about it.

707
00:34:43,325 --> 00:34:46,895
Yeah, so this is one of the things that I'm most proudest

708
00:34:46,895 --> 00:34:50,585
of at Intercom, and to be clear, it's not all my work and.

709
00:34:51,765 --> 00:34:55,725
Arguably, I didn't initiate it, but I was a big influence on it, and I've

710
00:34:55,725 --> 00:34:58,665
certainly spent a lot of time, a lot of time running about it, and more

711
00:34:58,665 --> 00:35:01,335
importantly, talking about it in public and taking load of credit for it.

712
00:35:01,695 --> 00:35:05,685
But we have an on-call system where we use volunteers

713
00:35:05,685 --> 00:35:08,595
rather than conscripts, and this means that we.

714
00:35:09,545 --> 00:35:13,925
Put people on call out of office hours, um, not because

715
00:35:13,925 --> 00:35:18,425
they happen to be on a certain team or are on a a, a rota or

716
00:35:18,875 --> 00:35:22,115
know something about maybe networks or systems or anything.

717
00:35:22,385 --> 00:35:27,485
Uh, we ask for people to volunteer to join this rotation.

718
00:35:27,665 --> 00:35:30,485
And so we generally have about six or seven people, uh, in

719
00:35:30,485 --> 00:35:34,715
this rotation, and we compensate them for their time on call.

720
00:35:34,715 --> 00:35:37,925
So the way we do it is it's, you're on call for a week.

721
00:35:38,715 --> 00:35:40,515
Uh, not in in office hours.

722
00:35:40,515 --> 00:35:44,205
The teams who own the alarms that are firing will get those alarms at that time.

723
00:35:44,415 --> 00:35:46,845
But outside of office hours, if you're on call in

724
00:35:46,845 --> 00:35:49,425
this volunteer team, you get the page for that.

725
00:35:49,785 --> 00:35:53,055
Um, and, but of course you can't just say these things like, let's

726
00:35:53,055 --> 00:35:56,985
have a volunteer based on call and hope that it works out for the best.

727
00:35:57,165 --> 00:35:59,775
We have to put in place a bunch of things, uh,

728
00:35:59,835 --> 00:36:02,115
both on the technical and social side of things.

729
00:36:02,360 --> 00:36:05,180
To make sure that this thing was sustainable, that people would

730
00:36:05,180 --> 00:36:07,850
feel like it, the work was valued and not just because of the

731
00:36:07,850 --> 00:36:10,160
compensation, but that the work was rewarding and you might

732
00:36:10,160 --> 00:36:13,160
actually learn something and maybe even enjoy doing on-call work.

733
00:36:13,580 --> 00:36:17,150
So we insisted on all teams writing runbooks

734
00:36:17,150 --> 00:36:19,490
for every alarm that can page somebody.

735
00:36:19,730 --> 00:36:22,640
Most importantly, we treat every single page like

736
00:36:22,640 --> 00:36:25,610
a heart attack, kind of using charity majors.

737
00:36:25,670 --> 00:36:26,120
Uh.

738
00:36:26,335 --> 00:36:27,205
Quote here.

739
00:36:27,475 --> 00:36:31,855
Um, and so this means that say the next morning after a page goes off

740
00:36:32,125 --> 00:36:35,935
in the middle of the night or whatever, our teams take it seriously.

741
00:36:35,965 --> 00:36:38,635
In fact, they take it more seriously than as if they

742
00:36:38,635 --> 00:36:41,065
had paged somebody in their own team out of bed.

743
00:36:41,245 --> 00:36:44,755
When you're paying, paging somebody you don't know or who's remote from you.

744
00:36:45,510 --> 00:36:48,600
At a bed in the middle of night because you're, you set up a bad alarm

745
00:36:48,600 --> 00:36:51,360
or because your thing fell over, you feel you're not guiltier about that.

746
00:36:51,630 --> 00:36:54,180
Oh, whenever I page anyone I start the call, I'm sorry to

747
00:36:54,180 --> 00:36:57,240
wake you, but because it's just a little, a little politeness

748
00:36:57,240 --> 00:37:00,060
and courtesy can move mountains, but please continue

749
00:37:00,330 --> 00:37:01,680
without too much effort.

750
00:37:01,920 --> 00:37:06,210
We just got excellent buy-in from the teams who own these different areas of the

751
00:37:06,210 --> 00:37:09,750
product and you know, could be building a lot and a lot of stuff can go wrong.

752
00:37:09,990 --> 00:37:14,700
But we were able to hold a high bar for pages being.

753
00:37:15,004 --> 00:37:17,375
Actually something that a human needs to do and then

754
00:37:17,555 --> 00:37:20,765
giving that person the tools to actually fix the problem.

755
00:37:20,975 --> 00:37:23,915
We have some technical reasons why this stuff is easier for us than it

756
00:37:23,915 --> 00:37:27,484
is for compared to other companies, such as having a large Ruby on Rails

757
00:37:27,484 --> 00:37:30,305
monolith, as opposed to every single team having their own bespoke tech stack.

758
00:37:30,605 --> 00:37:34,325
So that stuff helps us, but it's more the culture and

759
00:37:34,504 --> 00:37:39,634
how we also reward and, uh, give shout outs to people.

760
00:37:39,845 --> 00:37:41,825
You know, everybody from the CEO down.

761
00:37:42,260 --> 00:37:47,135
At Christmas whenever Addie kind of time, we always make sure to, uh, not just.

762
00:37:47,820 --> 00:37:51,150
Pay the people the money for the time that they spend on call out of hours.

763
00:37:51,360 --> 00:37:54,870
But it's recognized socially and in also in

764
00:37:54,870 --> 00:37:56,790
things like promotions and things like that.

765
00:37:57,000 --> 00:37:59,760
It's something that's really, uh, valued in, in the organization.

766
00:38:00,030 --> 00:38:02,370
So we've had this in place now for seven or eight years.

767
00:38:02,370 --> 00:38:05,310
It's hard to remember exactly how long it's been sustainable.

768
00:38:05,760 --> 00:38:07,590
One of the biggest problems we've had is so many people

769
00:38:07,590 --> 00:38:11,220
want to join us, um, that pe so people actually like it.

770
00:38:11,670 --> 00:38:13,650
And we've, we've also built.

771
00:38:14,250 --> 00:38:18,330
People, uh, like we've, we've made people better operators.

772
00:38:18,330 --> 00:38:20,910
We've made people actually enjoy and learn and

773
00:38:20,910 --> 00:38:22,620
learn more about what happens in the company.

774
00:38:22,859 --> 00:38:26,400
Um, and it's been actually a, a great long-term recruitment for my own kind of

775
00:38:26,400 --> 00:38:29,790
infrastructure oriented teams where people get a taste of this kind of work.

776
00:38:30,150 --> 00:38:33,270
Um, they might just be a product engineer from some random part

777
00:38:33,270 --> 00:38:35,670
of the business, but then when they see this work and they see.

778
00:38:35,715 --> 00:38:37,845
They actually see what's going on under the hood.

779
00:38:38,145 --> 00:38:40,635
Uh, they, they ask to join our team full time.

780
00:38:40,815 --> 00:38:42,855
Um, there's other stuff we have to do as well.

781
00:38:42,855 --> 00:38:45,975
There you have to have a way for the person

782
00:38:45,975 --> 00:38:49,665
to, uh, who's on call to bring in an expert.

783
00:38:49,695 --> 00:38:52,125
Like, so We have an Instant Commander program as well,

784
00:38:52,125 --> 00:38:54,615
and there's support there so that people don't feel like

785
00:38:54,615 --> 00:38:57,135
they're isolated on their own outta the PagerDuty playbook.

786
00:38:57,315 --> 00:38:57,675
Yeah.

787
00:38:57,675 --> 00:39:01,785
And, uh, when it comes down to it as well, look, not everybody can

788
00:39:01,785 --> 00:39:04,425
solve every kind of problem and we'll just go to the back phone.

789
00:39:04,425 --> 00:39:07,905
We'll page in as many people as we need to to solve a problem, which

790
00:39:07,905 --> 00:39:11,205
is, even if you had 10 people on call, you might need to do that anyway.

791
00:39:11,415 --> 00:39:13,065
And so, uh, this has been great.

792
00:39:13,065 --> 00:39:15,405
I think having a single person on call for a business

793
00:39:15,405 --> 00:39:18,675
decide of intercom, it's, it can be challenging at times.

794
00:39:18,765 --> 00:39:21,405
Um, but we've never been at the point of where we've decided

795
00:39:21,705 --> 00:39:24,900
or been at any risk of things falling apart or having to inf.

796
00:39:25,504 --> 00:39:28,384
Put multiple teams and lots of people on call, keeping

797
00:39:28,384 --> 00:39:32,345
things down to one person on call ruins fewer lives.

798
00:39:32,674 --> 00:39:36,395
Uh, we all get a better quality of life and, uh, doing

799
00:39:36,395 --> 00:39:38,855
this sustainably gives us something that, uh, we can

800
00:39:39,064 --> 00:39:42,064
really feel like we're making a difference in our work.

801
00:39:42,335 --> 00:39:45,214
And that the work just isn't feeding the robots.

802
00:39:45,214 --> 00:39:46,174
It's like high quality.

803
00:39:46,174 --> 00:39:46,625
It's good.

804
00:39:46,625 --> 00:39:48,589
We're learning and setting.

805
00:39:52,080 --> 00:39:54,450
Not just tolerating low quality alarms and stuff like that.

806
00:39:54,750 --> 00:39:56,070
Yeah, that's the important part.

807
00:39:56,070 --> 00:39:58,440
It's if it wakes you up, you're empowered to fix it

808
00:39:58,440 --> 00:40:00,900
or turn off the alarm or just thresholds or something.

809
00:40:00,900 --> 00:40:03,180
It's, it's the human element of it.

810
00:40:03,180 --> 00:40:05,550
It's the fact that this is a, you are

811
00:40:05,550 --> 00:40:07,620
compensated for doing it as a volunteer thing.

812
00:40:07,620 --> 00:40:10,050
It's not as part of your job responsibilities.

813
00:40:10,200 --> 00:40:10,560
Yes.

814
00:40:10,560 --> 00:40:12,240
I know you have a six month old who's having

815
00:40:12,240 --> 00:40:15,120
trouble sleeping, get up anyway, it none of that.

816
00:40:15,120 --> 00:40:17,250
It's a human approach to it, and that is

817
00:40:17,250 --> 00:40:19,320
something this industry has lacked historically.

818
00:40:20,070 --> 00:40:20,280
Yeah.

819
00:40:20,285 --> 00:40:20,305
And.

820
00:40:21,049 --> 00:40:21,830
I've been.

821
00:40:22,770 --> 00:40:25,950
Spreading the good word about this, uh, trying to influence other

822
00:40:25,950 --> 00:40:29,310
places to, to, to improve things and not just accept the status quo.

823
00:40:29,339 --> 00:40:31,980
The interesting part has been having conversations with co different

824
00:40:31,980 --> 00:40:34,859
people from different companies who, who are interested in doing

825
00:40:34,859 --> 00:40:38,700
this, but they have all sorts of other issues, like whether it's.

826
00:40:38,910 --> 00:40:42,660
Many, many tech stack or different compliance approaches or just

827
00:40:42,660 --> 00:40:45,569
other socio tech technical problems that can make it difficult.

828
00:40:45,690 --> 00:40:48,870
I think we we're probably on easy mode in intercom.

829
00:40:48,870 --> 00:40:52,649
We, there's a lot of, we did design it for our culture and our technology stack.

830
00:40:52,950 --> 00:40:55,259
Uh, not everyone can do it so easily, but I

831
00:40:55,259 --> 00:40:57,960
would encourage everybody to like not accept.

832
00:40:58,120 --> 00:41:00,190
Again, like you said, status quo around.

833
00:41:00,460 --> 00:41:02,920
Um, just because you're on a certain team, you

834
00:41:02,920 --> 00:41:05,140
need to carry a pager and be always on call.

835
00:41:05,290 --> 00:41:08,230
I think being on call a lot does reduce the quality

836
00:41:08,230 --> 00:41:09,880
of your life, even if you're not being paged.

837
00:41:10,180 --> 00:41:13,120
Um, and so being deliberate about that as well as recognizing the work,

838
00:41:13,120 --> 00:41:16,810
I think it's very important and it just gives you a great story that

839
00:41:16,900 --> 00:41:19,029
shows you that you actually care about the people who work for the

840
00:41:19,029 --> 00:41:22,685
company, uh, as opposed to just being part of some machine, uh, that.

841
00:41:23,355 --> 00:41:25,484
Needs to satisfy the, the computers,

842
00:41:26,774 --> 00:41:27,765
which is important.

843
00:41:27,855 --> 00:41:31,725
There's a human piece of it, and that's the, that's the thing that gets lost.

844
00:41:31,725 --> 00:41:33,105
It's not just a technical problem.

845
00:41:35,145 --> 00:41:38,294
I, I wanna thank you for taking the time to speak with me about all of this.

846
00:41:38,384 --> 00:41:41,129
If people wanna learn more, where's the best place for 'em to find you?

847
00:41:42,660 --> 00:41:46,365
I am kind of on X Twitter, but not really anymore.

848
00:41:46,365 --> 00:41:49,335
I mean, I, I'm not so good luck.

849
00:41:50,145 --> 00:41:51,435
I'm sort of on there.

850
00:41:51,435 --> 00:41:55,634
I mean, you can type in Brian Scanlon and I, I repost work stuff, I guess.

851
00:41:55,904 --> 00:41:58,904
Um, I'm on blue Sky, but not as much as I was on Twitter.

852
00:41:58,965 --> 00:42:00,795
Again, it's like you can type my name, which

853
00:42:00,795 --> 00:42:03,075
is probably a healthy thing, but yeah, I hear you.

854
00:42:03,285 --> 00:42:04,154
Yeah, I don't know.

855
00:42:04,154 --> 00:42:05,745
I'm kind of sad about those things.

856
00:42:05,745 --> 00:42:07,140
I mean, I'm on LinkedIn, but.

857
00:42:07,630 --> 00:42:08,470
Who uses LinkedIn?

858
00:42:09,310 --> 00:42:09,790
Oh God.

859
00:42:09,790 --> 00:42:14,170
I maintain that LinkedIn remains the world's largest porn site because

860
00:42:14,170 --> 00:42:17,740
it's where business people go to pleasure themselves on the internet.

861
00:42:17,890 --> 00:42:22,030
That is the best description I've got of it, and I have no tolerance for it.

862
00:42:22,450 --> 00:42:23,320
So I don't know.

863
00:42:23,320 --> 00:42:24,820
Maybe the best place to find me will be if

864
00:42:24,820 --> 00:42:26,440
you set up a Unix server and we all just.

865
00:42:27,029 --> 00:42:29,129
Log on and use right to talk to each other.

866
00:42:29,460 --> 00:42:30,600
I figured you'd put up a personal website.

867
00:42:30,600 --> 00:42:31,830
Just have an intercom chat with me.

868
00:42:31,830 --> 00:42:33,629
Box in the corner that pops up.

869
00:42:33,629 --> 00:42:36,149
'cause you know it's not like you're doing anything else these days, right?

870
00:42:36,330 --> 00:42:36,720
That works.

871
00:42:36,720 --> 00:42:38,069
You can find me on intercom.com.

872
00:42:38,069 --> 00:42:39,660
I am the other person on the side.

873
00:42:40,529 --> 00:42:41,009
There we go.

874
00:42:41,009 --> 00:42:41,339
There we go.

875
00:42:41,370 --> 00:42:44,339
I will include links to all of this in the show notes.

876
00:42:44,549 --> 00:42:46,980
Brian, thank you so much for taking the time to speak with me.

877
00:42:46,980 --> 00:42:47,700
I appreciate it.

878
00:42:47,930 --> 00:42:48,380
That's been great.

879
00:42:48,380 --> 00:42:49,010
We should do it again,

880
00:42:49,550 --> 00:42:49,970
we should.

881
00:42:50,390 --> 00:42:53,720
Brian Scanlon, senior principal engineer at Intercom.

882
00:42:53,960 --> 00:42:57,050
I'm cloud economist Corey Quinn, and this is Screaming In the Cloud.

883
00:42:57,320 --> 00:42:59,660
If you've enjoyed this podcast, please leave a five

884
00:42:59,660 --> 00:43:01,790
star review on your podcast platform of choice.

885
00:43:01,820 --> 00:43:04,610
Whereas if you've hated this podcast, please leave a five star

886
00:43:04,610 --> 00:43:07,400
review on your podcast platform of choice along with an angry,

887
00:43:07,400 --> 00:43:10,425
insulting comment, but that that platform will not be one of

888
00:43:10,425 --> 00:43:12,920
Amazon's 'cause That's way too far up the stack for them to do well.