1
00:00:00,000 --> 00:00:02,392
Databricks introduced Genie Spaces.

2
00:00:02,612 --> 00:00:06,475
It's a way to skip SQL and query your data using natural language.

3
00:00:06,596 --> 00:00:13,942
For business users, this is a game changer because you no longer need to wait for an
analyst or a BI ticket to get insights.

4
00:00:13,942 --> 00:00:15,783
You can just ask the AI.

5
00:00:15,963 --> 00:00:23,769
To make this possible, you need to feed that agent with a semantic layer, information
about how to calculate your metrics and how to piece data together.

6
00:00:24,030 --> 00:00:26,411
In Databricks, these are called metric views.

7
00:00:26,692 --> 00:00:28,174
We tested them end to end.

8
00:00:28,174 --> 00:00:33,234
Stefan built metric views, dbt integrations, and a gd space on top of that.

9
00:00:33,394 --> 00:00:36,394
In this episode, he walks you through all of it.

10
00:00:36,394 --> 00:00:41,810
My name is Jonny, knowledge lead here at Dataminded, and welcome to technology
explorations in data and AI.

11
00:00:50,814 --> 00:00:54,477
Hi everyone, welcome to technology explorations at Dataminded.

12
00:00:54,477 --> 00:00:58,500
In this series, we give you a quick look into new or interesting technologies.

13
00:00:58,500 --> 00:01:01,062
The topic of today is Databricks.

14
00:01:01,062 --> 00:01:04,024
How can you talk to your data in a Databricks environment?

15
00:01:04,024 --> 00:01:05,765
For this topic, I've invited Stefan.

16
00:01:05,765 --> 00:01:07,016
Welcome, Stefan.

17
00:01:07,696 --> 00:01:08,845
Hello Jonny.

18
00:01:08,920 --> 00:01:15,547
All right, before we dive into the demo and all the details, could you quickly introduce
yourself, you do at Dataminded, what's your role,

19
00:01:17,388 --> 00:01:25,724
um I'm working as a team lead at Dataminded, basically helping companies and clients to
build more data centric organizations and everything what's needed for that.

20
00:01:25,724 --> 00:01:35,627
I work at different levels on the engineering level, uh implementing use cases, a bit of
platform, but also on a strategical and a more technical level.

21
00:01:35,627 --> 00:01:42,159
So sometimes a lot of switching, but I like to keep in touch with the engineering part as
well.

22
00:01:42,159 --> 00:01:43,132
Yeah,

23
00:01:43,132 --> 00:01:44,714
Okay, Stefan, let's get started.

24
00:01:44,714 --> 00:01:47,929
So you will demo us today Databricks and how to talk to your data.

25
00:01:47,929 --> 00:01:50,132
What does it look like talking to your data?

26
00:01:51,069 --> 00:01:56,234
Well, the idea is that you have data set or a use case, let's say.

27
00:01:56,235 --> 00:02:02,337
So for example, here I took a data set from the internet.

28
00:02:02,337 --> 00:02:10,571
It's a data set describing all Lego sets and all parts and pieces since, I think, the 70s
or so.

29
00:02:10,571 --> 00:02:12,903
It's from the re-brickable site.

30
00:02:12,903 --> 00:02:14,807
this is the website.

31
00:02:14,807 --> 00:02:18,747
So it has an inventory of all the different sets with...

32
00:02:18,747 --> 00:02:21,598
with all pictures and so on.

33
00:02:21,598 --> 00:02:25,132
What I took here as a source is, the download page.

34
00:02:25,132 --> 00:02:32,756
So basically you can download all the data as full zip files and ingest them into your
data platform.

35
00:02:32,756 --> 00:02:38,352
And actually it gives a pretty neat data set because everybody knows LEGO, And it's pretty
structured.

36
00:02:38,352 --> 00:02:42,305
have a set there, you have some parts,

37
00:02:42,305 --> 00:02:45,728
sometimes you have more parts, sometimes you have less parts and so on and so on.

38
00:02:45,728 --> 00:02:48,569
So it's pretty nice data set to work with.

39
00:02:48,820 --> 00:02:52,938
And you turn that into a talk to your data data set, right?

40
00:02:53,255 --> 00:02:54,406
Yeah, correct.

41
00:02:54,406 --> 00:02:58,369
So what we did is for this video, we worked on Databricks.

42
00:02:58,369 --> 00:03:01,051
So we ingested the data onto Databricks.

43
00:03:01,051 --> 00:03:03,492
We built a semantic layer on top.

44
00:03:03,492 --> 00:03:05,413
I think we will come back to that later.

45
00:03:05,413 --> 00:03:06,507
But that's basically...

46
00:03:06,507 --> 00:03:11,132
not exposing the raw data, but building some more information on top.

47
00:03:11,132 --> 00:03:17,060
Like for example, if you would need to count the amount of parts in a set, you would need
to take the sum.

48
00:03:17,060 --> 00:03:20,513
if you want to have the distinct amount of parts, you need to do a distinct.

49
00:03:20,513 --> 00:03:24,146
These are the types of things that live within a semantic layer.

50
00:03:24,261 --> 00:03:32,275
Based on that semantic layer, we have built a talk to your data application, let's say,
and in Databricks, that's called a Genie Space.

51
00:03:32,275 --> 00:03:36,479
And we connected to the data assets in Databricks.

52
00:03:36,479 --> 00:03:40,221
And based on that, you can ask questions about the data.

53
00:03:40,221 --> 00:03:43,522
So for example, can you...

54
00:03:44,319 --> 00:03:45,967
Give me the...

55
00:03:46,219 --> 00:03:48,565
distinct amount of sets.

56
00:03:49,278 --> 00:03:50,913
that were released.

57
00:03:51,165 --> 00:03:52,216
Okay, yeah.

58
00:03:53,127 --> 00:03:54,548
Let's see if that works.

59
00:03:54,548 --> 00:03:59,806
something you could quite easily write as a SQL query, but now we just do it in natural
language.

60
00:03:59,812 --> 00:04:00,633
Yep.

61
00:04:00,884 --> 00:04:02,749
And so what is it doing now behind the scenes?

62
00:04:02,749 --> 00:04:09,527
I guess it's making a query to query the model that we will show later, but to query the
model you prepared.

63
00:04:09,970 --> 00:04:10,540
Yeah, indeed.

64
00:04:10,540 --> 00:04:15,835
So what's, let's say the difference with the, let's say a very simple chat GPT, right?

65
00:04:15,835 --> 00:04:22,249
You could hypothetically copy all your data into a chat window and then just ask the
question.

66
00:04:22,390 --> 00:04:25,162
That's typically not something very smart to do.

67
00:04:25,162 --> 00:04:25,462
Why?

68
00:04:25,462 --> 00:04:28,497
Because you will push a lot of data to models.

69
00:04:28,497 --> 00:04:29,848
It will be expensive.

70
00:04:29,848 --> 00:04:33,991
And for a model, it will also be difficult to really like understand what's inside.

71
00:04:34,133 --> 00:04:39,186
Here the model has access to the data, but it can also query data.

72
00:04:39,186 --> 00:04:46,681
It can create SQL queries on the data or it can query the data because the model also
knows SQL, right?

73
00:04:46,681 --> 00:04:51,759
So it can reason, it can translate your question into a query.

74
00:04:51,759 --> 00:04:59,310
a targeted query, for example, because here I would assume that it can generate the query
to get a distinct amount of sets.

75
00:04:59,310 --> 00:05:05,257
So actually it's a lot smarter than just, let's say, pushing all your data to the model.

76
00:05:05,346 --> 00:05:07,028
Otherwise you would consume all the tokens.

77
00:05:07,028 --> 00:05:09,630
Tokens are also getting quite expensive.

78
00:05:10,011 --> 00:05:12,614
And so it would also be quite unreliable.

79
00:05:12,614 --> 00:05:15,498
And here you take advantage of the technology that's there.

80
00:05:15,498 --> 00:05:20,824
Data still lives in Databricks, it's easy to query and you use the model to create a
query.

81
00:05:20,824 --> 00:05:23,532
And in this case, I see also a visualization.

82
00:05:23,532 --> 00:05:26,825
Yeah, indeed I didn't ask for it here, but okay, it's nice that we have it.

83
00:05:26,825 --> 00:05:29,788
Yeah, and it gives some key trends.

84
00:05:29,788 --> 00:05:31,560
You see the trend is going upward.

85
00:05:31,560 --> 00:05:33,021
Yeah, that makes sense.

86
00:05:33,129 --> 00:05:35,963
Does this match your feeling of datasets per year?

87
00:05:36,214 --> 00:05:40,594
Yeah, think there were a few more difficult years for LEGO.

88
00:05:41,094 --> 00:05:50,234
I think it might be indeed the second half of the 2000s, where you see indeed a bit of a
decline, but then there was a sharp increase again since 2010.

89
00:05:50,234 --> 00:05:52,640
So I think, yeah, feeling-wise it makes sense.

90
00:05:52,640 --> 00:05:54,455
Yeah, okay, nice.

91
00:05:54,455 --> 00:05:58,846
This is really useful if you want to get a quick, ad-hoc question and get an immediate
visual.

92
00:05:58,846 --> 00:06:00,327
This really helps.

93
00:06:00,578 --> 00:06:10,054
And I also see we get some interpretation like in the bottom part, we get like an analysis
or an analyst would give you as a high level summary, we get like some trends.

94
00:06:10,649 --> 00:06:11,590
yeah, indeed.

95
00:06:11,590 --> 00:06:16,350
Now, for this agent, it only has access to the data, right?

96
00:06:16,350 --> 00:06:20,335
And in the data, no company history is there, only the data is there.

97
00:06:20,335 --> 00:06:24,218
So I think these things are more interpretations from the agent.

98
00:06:24,289 --> 00:06:32,899
I think maybe as a next step you could also give it access to, I don't know, more
descriptive information about the company, to complement this information as well.

99
00:06:32,899 --> 00:06:34,785
But it uses something for grounding.

100
00:06:34,785 --> 00:06:37,665
see like it has this one footnote attached to it.

101
00:06:37,665 --> 00:06:39,559
Is that referring to something?

102
00:06:40,137 --> 00:06:40,800
This one?

103
00:06:40,800 --> 00:06:45,708
the blue ones after each sentence it has like a small footnote thingy.

104
00:06:46,151 --> 00:06:48,711
so it uses the data to give that.

105
00:06:48,711 --> 00:06:49,092
Yeah.

106
00:06:49,092 --> 00:06:53,298
So I think here it's made a query.

107
00:06:53,298 --> 00:06:56,741
It made a query for the amount of distinct sets per year, right?

108
00:06:56,741 --> 00:06:58,343
You can also see the code.

109
00:06:58,343 --> 00:07:04,107
So it is indeed like a count distinct from the semantic layer.

110
00:07:04,107 --> 00:07:04,928
Yeah.

111
00:07:05,179 --> 00:07:06,400
Okay, very nice.

112
00:07:06,651 --> 00:07:09,595
And so this is one of the ways we can now query data.

113
00:07:09,595 --> 00:07:11,477
But this does not come out of the box.

114
00:07:11,477 --> 00:07:17,354
Maybe we can dive a bit deeper like what brought you here and what is the overall idea
behind this.

115
00:07:17,943 --> 00:07:22,808
So as mentioned in the beginning, yeah, there is a lot of data in companies.

116
00:07:22,808 --> 00:07:29,784
The question is how do we get information from that data and how can we talk to our data?

117
00:07:30,101 --> 00:07:34,885
We see in many customers that often simple questions take a long time to answer.

118
00:07:34,885 --> 00:07:39,149
But it's something that we want to, let's say, make easier at Dataminded.

119
00:07:39,149 --> 00:07:44,373
We want to be able to allow our customers to have a fast time to value if there is a
question.

120
00:07:44,373 --> 00:07:47,575
It should not be difficult to get the answers to the question.

121
00:07:47,575 --> 00:07:48,319
What we...

122
00:07:48,319 --> 00:07:54,898
typically used as a data architecture, that's the data product architecture or data
product thinking.

123
00:07:54,898 --> 00:07:59,403
The idea behind that is the fact that we have, let's say, confined scopes of

124
00:07:59,657 --> 00:08:00,168
data.

125
00:08:00,168 --> 00:08:08,032
So we are not integrating all data in a very big model because that will typically become
a bottleneck for answering questions later on.

126
00:08:08,032 --> 00:08:14,996
We want to link data sets, confined data sets to specific business entities.

127
00:08:14,996 --> 00:08:19,319
can be specific owners, can be teams, can be the partners.

128
00:08:19,319 --> 00:08:23,240
So we want to bridge the gap between the data and between the business.

129
00:08:23,240 --> 00:08:24,561
think that's basically the

130
00:08:24,561 --> 00:08:25,963
core idea.

131
00:08:25,963 --> 00:08:30,847
And there are like many ways to do that and there are many different steps, but that's the
core idea.

132
00:08:30,847 --> 00:08:33,010
Here you see the typical lineage view.

133
00:08:33,010 --> 00:08:35,453
So you start from source applications.

134
00:08:35,453 --> 00:08:39,096
Source applications will result in source aligned products.

135
00:08:39,096 --> 00:08:43,860
So this ball typically represents the data product, You might have multiple source aligned
products.

136
00:08:43,860 --> 00:08:44,220
products.

137
00:08:44,220 --> 00:08:53,928
Important is the fact that for these products we know who are the owners in the business,
who are the key users of those applications, who in the business knows this data very

138
00:08:53,928 --> 00:08:54,178
well.

139
00:08:54,178 --> 00:08:56,030
It's important to have a close connection to that.

140
00:08:56,030 --> 00:09:01,333
From the source line products we can create integrated products or aggregated products if
it's needed.

141
00:09:01,333 --> 00:09:08,799
If we see that combination of certain sources we are using a lot, okay, maybe it makes
sense to create an integrated product from that.

142
00:09:08,799 --> 00:09:09,651
And then

143
00:09:09,651 --> 00:09:12,414
And on the right you see more the consumer line products.

144
00:09:12,414 --> 00:09:15,297
And these are products that are fulfilling a business need.

145
00:09:15,297 --> 00:09:19,110
So they are an answer to a specific business need.

146
00:09:19,110 --> 00:09:27,401
for example, for us the Lego data, we can create a consumer line product that is, let's
say, targeted at a specific need.

147
00:09:27,401 --> 00:09:36,006
And also in this part, the semantic layer lives mostly because it's something that's
typically not close to the source.

148
00:09:36,006 --> 00:09:43,350
It's more towards business metrics like the KPIs and the metrics and so on.

149
00:09:43,350 --> 00:09:46,137
So this is, let's say, the structure that we're using.

150
00:09:46,137 --> 00:09:50,443
Here I'm referring to an article that a colleague of ours wrote,

151
00:09:50,443 --> 00:09:51,994
on semantic layers.

152
00:09:51,994 --> 00:09:54,965
So where can you use a semantic layer?

153
00:09:54,965 --> 00:09:58,658
As we mentioned, it's mostly, let's say, on the right side, mostly on the downstream side.

154
00:09:58,658 --> 00:10:01,460
I think in the article he proposed multiple options.

155
00:10:01,460 --> 00:10:07,903
You can create more kind of a centralized semantic layer if your context requires that.

156
00:10:07,903 --> 00:10:17,229
For example, if it's important to have metrics that are reproducible over a lot of
business domains, OK, maybe then you need to centralize the semantic layer.

157
00:10:17,229 --> 00:10:18,479
But sometimes it's also not

158
00:10:18,479 --> 00:10:18,849
need.

159
00:10:18,849 --> 00:10:28,942
is sometimes you just for a specific use case you want to have a specific metric and you
have specific business logic and in that case you can have the semantic layer as an output

160
00:10:28,942 --> 00:10:31,245
of your uh data product.

161
00:10:31,245 --> 00:10:33,076
ah

162
00:10:33,076 --> 00:10:40,253
much do you put inside the data product and how much do you then get centralized as one
big layer and finding a balance there.

163
00:10:40,603 --> 00:10:41,845
yeah, indeed, indeed.

164
00:10:41,845 --> 00:10:45,416
So there's no, let's say, one answer fits all.

165
00:10:45,416 --> 00:10:48,649
It really depends on the context you are working within.

166
00:10:48,649 --> 00:10:55,033
For our example, we basically used a separate data product to generate semantic layer.

167
00:10:55,033 --> 00:10:57,834
And from there, we made some analytics.

168
00:10:57,834 --> 00:11:01,495
So this is for our demo, how we approached it.

169
00:11:01,747 --> 00:11:02,678
Okay, very clear.

170
00:11:02,678 --> 00:11:10,985
And so what you will construct in the demo or what have you constructed for the agent was
something that resembles the thing on the right.

171
00:11:10,985 --> 00:11:18,492
You built a data product that harvests the data from Lego website or from the dump of the
Lego website.

172
00:11:18,492 --> 00:11:21,034
And then on top of that, you build a semantic layer.

173
00:11:21,034 --> 00:11:24,236
Now, could you tell a bit more, what is that semantic layer?

174
00:11:24,236 --> 00:11:25,407
What is inside of that?

175
00:11:25,553 --> 00:11:29,278
I will show you, let's say the basics of the semantic layer.

176
00:11:29,278 --> 00:11:33,684
So yeah, this is Databricks documentation site.

177
00:11:33,684 --> 00:11:36,738
And this is an example of a semantic layer.

178
00:11:36,738 --> 00:11:39,570
So you can see it's a SQL statement.

179
00:11:39,777 --> 00:11:44,070
On Databricks, a semantic layer is implemented as a metric view.

180
00:11:44,070 --> 00:11:46,842
So that's like technology specific implementation.

181
00:11:46,842 --> 00:11:50,874
I think in other data products, it might have a different name.

182
00:11:50,874 --> 00:11:52,995
um

183
00:11:52,995 --> 00:11:54,409
terminology that they use.

184
00:11:54,409 --> 00:11:56,746
think in Snowflake it's called semantic view.

185
00:11:56,746 --> 00:11:58,930
Here they call it metric view.

186
00:11:58,970 --> 00:12:00,201
A metric view, yeah, indeed.

187
00:12:00,201 --> 00:12:04,400
So it's a specific asset within the Databricks Unity catalog.

188
00:12:04,400 --> 00:12:06,102
So you also created...

189
00:12:06,424 --> 00:12:11,827
you use a SQL statement to create it, create or replace view with metrics.

190
00:12:11,827 --> 00:12:14,769
And then basically there's a YAML SQL to create it.

191
00:12:14,769 --> 00:12:22,074
But the basic ideas for many of these, let's say implementations on different data
platforms, it's often pretty similar.

192
00:12:22,074 --> 00:12:26,097
You start from a certain source, main source.

193
00:12:26,097 --> 00:12:28,579
you might join data together.

194
00:12:28,579 --> 00:12:30,289
You might join different tables together.

195
00:12:30,289 --> 00:12:33,762
We'll show you later how we did that for our data set.

196
00:12:33,762 --> 00:12:36,395
And you define a bunch of dimensions.

197
00:12:36,395 --> 00:12:37,956
This one shows orders.

198
00:12:37,956 --> 00:12:42,260
So for example, you have an order status, have some dates, keys.

199
00:12:42,260 --> 00:12:44,862
This is typical dimensional information.

200
00:12:44,862 --> 00:12:47,925
And then you define a bunch of facts or measures.

201
00:12:47,925 --> 00:12:49,926
And measures are basically...

202
00:12:50,275 --> 00:12:55,778
the definition of the fact, but also on how do you calculate a measure based on a fact.

203
00:12:55,778 --> 00:12:59,259
So for example, if you want to have a count, you need to do a count.

204
00:12:59,259 --> 00:13:06,035
If you want to have a total revenue, you need to calculate the sum of a specific column.

205
00:13:06,035 --> 00:13:11,945
So this is basically a fact plus how do you calculate a metric from a fact.

206
00:13:12,065 --> 00:13:15,127
I think this is the pattern that you see in most semantic layers.

207
00:13:15,127 --> 00:13:17,878
Also, what's important is the fact that you have comments here.

208
00:13:17,878 --> 00:13:20,219
This is of course nothing special to a semantic layer.

209
00:13:20,219 --> 00:13:23,133
You also have this in a table or in a view.

210
00:13:23,133 --> 00:13:32,222
But this also creates extra metadata that can be used by humans or by, let's say, agents
like the Databricks Genie Space.

211
00:13:32,739 --> 00:13:33,180
Yeah.

212
00:13:33,180 --> 00:13:41,709
And this will then be used to interpret or to translate a natural language query also and
to match that with specific calculations, I guess.

213
00:13:42,275 --> 00:13:43,416
Yeah, correct.

214
00:13:43,416 --> 00:13:49,319
So suppose we would ask the agent, give me the total revenue filtered on specific years.

215
00:13:49,319 --> 00:13:52,380
The agent will say, okay, I will look in the semantic layer.

216
00:13:52,380 --> 00:13:55,282
I see there's a column called total revenue.

217
00:13:55,282 --> 00:13:57,343
It already tells me how I need to calculate it.

218
00:13:57,343 --> 00:14:05,248
So the translation step for the agent becomes less to get the final query.

219
00:14:05,248 --> 00:14:05,814
Yeah.

220
00:14:05,814 --> 00:14:06,486
Yeah.

221
00:14:06,486 --> 00:14:11,094
And you mentioned that this is applying to one table we saw at the top.

222
00:14:11,496 --> 00:14:13,831
It's also possible to have multiple tables then.

223
00:14:13,831 --> 00:14:15,296
Okay.

224
00:14:15,288 --> 00:14:18,119
is an example And here you can see the joins.

225
00:14:18,119 --> 00:14:22,979
So for example, you select a source, which are the orders.

226
00:14:23,559 --> 00:14:29,859
Basically, it's kind of a fact table and you will join this with dimensions, with the
customer dimension.

227
00:14:29,859 --> 00:14:32,699
And here you say how you will join it.

228
00:14:32,819 --> 00:14:35,359
And you can also chain the joins.

229
00:14:35,479 --> 00:14:38,583
So for example, you have your join to the customer and here you have

230
00:14:38,583 --> 00:14:41,476
join from customer to nation, right?

231
00:14:41,476 --> 00:14:42,978
And then you say how you join it.

232
00:14:42,978 --> 00:14:49,246
you have like this type of chain joins that will be combined in one metric view.

233
00:14:49,246 --> 00:14:50,066
Yeah.

234
00:14:50,317 --> 00:14:54,265
Okay, in this case you would have view that spans multiple tables.

235
00:14:54,265 --> 00:15:00,760
Would you then also need to create one for the other tables or is that just one metric
view that you offer to your agent in this case?

236
00:15:01,746 --> 00:15:05,008
That really depends on the business logic.

237
00:15:05,008 --> 00:15:09,891
I think a metric view you can compare with a star schema.

238
00:15:09,891 --> 00:15:16,536
So if you have a typical star schema, as you know, you have dimension tables and you have
effect table.

239
00:15:16,536 --> 00:15:22,980
This is also more or less represented in a metric view because you have your dimensions
and you have your metrics and your measures.

240
00:15:23,232 --> 00:15:28,348
If you would in reality create multiple star schemas, I think you also need to have
multiple metric views.

241
00:15:28,348 --> 00:15:33,593
One reason for doing this is if you have...

242
00:15:33,805 --> 00:15:37,488
facts or measures with different granularity, for example.

243
00:15:37,488 --> 00:15:44,792
If you have very similar effects with same granularity, maybe you can just add them
together in the same metric view.

244
00:15:44,792 --> 00:15:49,240
But if you have multiple granularities, yeah, you cannot combine them.

245
00:15:49,240 --> 00:15:50,381
That's an example.

246
00:15:50,381 --> 00:15:55,644
Another example I will show you later is that in our data set we have two metric views.

247
00:15:55,644 --> 00:16:03,068
One is for the parts itself and the other one is for the minifigs because a minifig is

248
00:16:03,734 --> 00:16:05,866
an essential different thing than a part.

249
00:16:05,866 --> 00:16:09,201
A part has a color, a minifig has different attributes.

250
00:16:09,201 --> 00:16:19,700
So we could try to force it in the same metric view, but here I think it's better to
separate it because then you have like two clear metric views with two different purposes.

251
00:16:19,864 --> 00:16:22,947
It depends also on the questions you would want to ask, right?

252
00:16:22,988 --> 00:16:27,258
The minifig questions would be a bit different than the normal brick questions, I imagine.

253
00:16:27,258 --> 00:16:27,589
Yep.

254
00:16:27,589 --> 00:16:28,321
Yep.

255
00:16:29,054 --> 00:16:34,130
I can first shortly show the data that we use.

256
00:16:34,132 --> 00:16:35,673
so here you see the data model.

257
00:16:35,673 --> 00:16:41,277
So basically one box here is it's a table that you can download from the Rebrickable site.

258
00:16:41,277 --> 00:16:43,377
And what you see here, you see

259
00:16:43,377 --> 00:16:45,411
few important tables, the sets.

260
00:16:45,411 --> 00:16:48,536
So you can see a certain set has typically a code.

261
00:16:48,756 --> 00:16:50,837
There is a team linked to a set.

262
00:16:50,837 --> 00:16:54,260
So you have like the classic sets, you have the Technic sets.

263
00:16:54,260 --> 00:16:56,460
set is linked to an inventory.

264
00:16:56,521 --> 00:17:02,246
An inventory has a version and is linked to a set of parts.

265
00:17:02,246 --> 00:17:04,887
And the parts is, what is the part number?

266
00:17:04,887 --> 00:17:06,268
What's the color?

267
00:17:06,782 --> 00:17:09,273
What's the quantity and is it spare?

268
00:17:09,273 --> 00:17:20,588
For example, if you would have like, let's say four by four squares, plates, and you have
10 of them of the same color, they would be in the same inventory parts.

269
00:17:20,667 --> 00:17:28,050
And if you would then have like the same part, but then a different color, it would be a
different record in this table.

270
00:17:28,050 --> 00:17:29,267
That's how it's separated.

271
00:17:29,267 --> 00:17:31,911
have different inventory IDs as well.

272
00:17:32,380 --> 00:17:34,142
Yeah, yeah, indeed.

273
00:17:34,142 --> 00:17:37,744
And here you can see then multiple other dimensions.

274
00:17:37,744 --> 00:17:39,505
A part can have a category.

275
00:17:39,505 --> 00:17:42,386
I don't know, blades or bricks or whatever.

276
00:17:42,386 --> 00:17:44,088
A part can also have a color.

277
00:17:44,088 --> 00:17:47,742
The RGB values and also is it transparent part?

278
00:17:47,742 --> 00:17:48,353
Yes, no.

279
00:17:48,353 --> 00:17:50,494
there are also relationships between parts.

280
00:17:50,494 --> 00:17:51,392
So you have like...

281
00:17:51,392 --> 00:17:53,354
child-parent relationships and so on.

282
00:17:53,354 --> 00:18:01,737
And here you can see, coming back to the discussion we had before, minifigs are actually a
different type of thing.

283
00:18:01,737 --> 00:18:08,341
So a minifig is also linked to a set, but it just has a name and it has a number of parts.

284
00:18:08,341 --> 00:18:09,603
It does not have a color.

285
00:18:09,603 --> 00:18:11,304
It does not have a category.

286
00:18:11,304 --> 00:18:12,784
So it is a different entity.

287
00:18:12,784 --> 00:18:17,546
And that's why in our case, I also separate it into a different metric view.

288
00:18:17,676 --> 00:18:21,330
Okay, so how does that come together in Databricks?

289
00:18:21,330 --> 00:18:24,143
So here you see our Databricks workspace.

290
00:18:24,143 --> 00:18:25,376
So we're working in a.

291
00:18:25,376 --> 00:18:27,376
basic Databricks workspace.

292
00:18:27,876 --> 00:18:32,196
We have a catalog here where we're working in.

293
00:18:32,676 --> 00:18:35,956
In that catalog we have multiple schemas.

294
00:18:35,996 --> 00:18:41,916
How we typically work is that for a data product we have two schemas.

295
00:18:41,916 --> 00:18:45,496
We have the internal schema and we have an external schema.

296
00:18:46,096 --> 00:18:49,256
The internal schema contains really internal data.

297
00:18:49,256 --> 00:18:54,480
So for example, if you would download files and you need to

298
00:18:54,480 --> 00:18:57,847
store them for one or the other reason, you would do it in the internal schema.

299
00:18:57,847 --> 00:19:03,698
And then if you expose data-specific tables, then you would put it in the external schema.

300
00:19:03,767 --> 00:19:08,765
Yeah, the eternal one you never want to expose beyond the team that works on the data
product.

301
00:19:08,992 --> 00:19:09,693
indeed.

302
00:19:09,693 --> 00:19:14,157
And the external one should be quality controlled, needs to be documented and so on and so

303
00:19:14,157 --> 00:19:15,718
So here you can see the source.

304
00:19:15,718 --> 00:19:22,252
So we have a source aligned re-breakable download product where we have downloaded the
different tables.

305
00:19:22,252 --> 00:19:24,704
These tables correspond to the model I showed you earlier.

306
00:19:24,704 --> 00:19:25,364
sorry.

307
00:19:25,364 --> 00:19:28,275
And we have, there's also an external layer.

308
00:19:28,275 --> 00:19:30,486
So again, here super simple.

309
00:19:30,486 --> 00:19:34,238
We just have the tables in turn and we expose them one on one.

310
00:19:34,238 --> 00:19:34,889
externally.

311
00:19:34,889 --> 00:19:36,350
This is a source line product.

312
00:19:36,350 --> 00:19:40,951
On top of that, we have built a consumer lined semantic layer product.

313
00:19:40,951 --> 00:19:47,054
Like in reality, you would never call a product semantic layer, I think, but for this
demo, that's okay.

314
00:19:47,054 --> 00:19:50,034
In the internal layer, there is nothing really.

315
00:19:50,034 --> 00:19:54,920
There's just like a one-on-one view on the source-aligned product.

316
00:19:54,920 --> 00:19:59,302
But in the external layer we created the different metric views.

317
00:19:59,302 --> 00:20:04,147
So you see we split it here the part and the minifig metric views.

318
00:20:04,147 --> 00:20:07,411
We also see here an underscore dbt, I will show you that later.

319
00:20:07,411 --> 00:20:20,925
We created the metric views with let's say native Databricks syntax and we also created it
with dbt semantic layers which is a dbt entity that is now also available for Databricks.

320
00:20:20,925 --> 00:20:25,889
if we look at the agent demo you gave before, this is what you gave to the agent, right?

321
00:20:25,889 --> 00:20:27,141
These semantic models.

322
00:20:27,141 --> 00:20:27,597
Okay.

323
00:20:27,597 --> 00:20:28,458
Correct.

324
00:20:28,458 --> 00:20:33,606
So here I gave, I think, the part and the minifig model I gave to the agent.

325
00:20:33,858 --> 00:20:36,122
If you look here, what is inside.

326
00:20:36,122 --> 00:20:38,965
So if you have a metric view on Databricks.

327
00:20:39,558 --> 00:20:46,352
you do not have the classical view where you can sample the data or where you get just the
list of columns.

328
00:20:46,352 --> 00:20:46,663
No.

329
00:20:46,663 --> 00:20:52,806
You have the measures specified and you have the dimensions specified.

330
00:20:52,806 --> 00:21:01,052
So the measures are basically as I showed before, the different measures you can calculate
and see it as calculations based on top effects.

331
00:21:01,052 --> 00:21:03,893
And the dimensions are the different dimensions as...

332
00:21:04,074 --> 00:21:08,116
as shown earlier on the documentation website.

333
00:21:08,116 --> 00:21:19,952
So for example here you see the set number, it's a dimension, you have a name, release
year, of course it's important, the team ID, team name, and maybe team ID is not so

334
00:21:19,952 --> 00:21:28,257
relevant because it's maybe a more technical dimension, but the name for sure, maybe you
want to search for specific words in the teams, and so on and so on.

335
00:21:28,257 --> 00:21:29,968
So typically you will have quite a

336
00:21:29,968 --> 00:21:30,850
of dimensions.

337
00:21:30,850 --> 00:21:32,129
We have here 16.

338
00:21:32,129 --> 00:21:44,090
And here you have the measures, like the amount of sets, the total quantity of parts, the
amount of distinct parts, and for example the spare quantity, was the total amount of

339
00:21:44,090 --> 00:21:44,520
parts.

340
00:21:44,520 --> 00:21:48,351
um

341
00:21:48,351 --> 00:21:53,377
box shows you bit more on how it's defined then, the info box at the end of the line.

342
00:21:54,239 --> 00:21:58,970
yeah, so that has the expression that we saw in the YAML files.

343
00:22:05,232 --> 00:22:04,905
Because indeed, you see there's an expression always and that you will also see that in
the definition of the metric views.

344
00:22:04,905 --> 00:22:06,937
So this is shown here.

345
00:22:06,937 --> 00:22:09,150
um

346
00:22:09,150 --> 00:22:11,663
that we are limited to these kind of measures?

347
00:22:11,663 --> 00:22:13,505
Because you defined four here.

348
00:22:13,505 --> 00:22:21,878
What if I want to have something else, like the average number of parts in a set or
something like that, which is a different kind of measure.

349
00:22:22,152 --> 00:22:24,264
Yeah, that's a very good question.

350
00:22:24,264 --> 00:22:25,855
Yes, indeed, you're limited.

351
00:22:25,855 --> 00:22:29,528
So I think that's a bit of the flip side of metric views.

352
00:22:29,528 --> 00:22:37,493
Metric views or semantic layers are very powerful because you can give a lot of business
information and KPIs with that.

353
00:22:37,493 --> 00:22:42,715
But the amount of information you give is also limited to the amount of information you
give, right?

354
00:22:42,715 --> 00:22:48,120
So if somebody wants to calculate completely different metrics that you didn't think
about,

355
00:22:48,120 --> 00:22:50,124
Yeah, you would need to add them here.

356
00:22:50,124 --> 00:22:56,441
you can very well think of questions that cannot be answered by this metric view.

357
00:22:56,441 --> 00:23:02,304
Where I work now, we also have a data set, for example, with a lot of history inside.

358
00:23:02,405 --> 00:23:06,447
And this history is very difficult to capture in a metric view.

359
00:23:06,783 --> 00:23:09,456
while it is very relevant for a lot of use cases.

360
00:23:09,564 --> 00:23:10,295
Okay, very nice.

361
00:23:10,295 --> 00:23:15,478
So you basically made sure you had these semantic layer metric views in place.

362
00:23:15,478 --> 00:23:21,123
You fed that to the agent and you had a way to calculate those metrics views, right?

363
00:23:21,590 --> 00:23:22,422
Yeah, indeed.

364
00:23:22,422 --> 00:23:24,565
So let's switch to the code.

365
00:23:24,565 --> 00:23:30,720
So here we have, let's say, organized our repository for data products.

366
00:23:30,720 --> 00:23:35,444
We have here the consumer reliant layer and here we have the semantic layer data product,
right?

367
00:23:35,444 --> 00:23:39,327
Again, not a good name to use in real life,

368
00:23:39,856 --> 00:23:46,572
I used two ways of, let's say, implementing or deploying these metric views.

369
00:23:46,572 --> 00:23:49,445
Let's start maybe with the most straightforward one.

370
00:23:49,445 --> 00:23:55,110
It's one where we just execute the Databricks SQL code.

371
00:23:55,110 --> 00:23:57,572
How we often work is that we have...

372
00:23:57,572 --> 00:24:03,865
a bunch of assets in a data product or let's say the see it as the logic and the code.

373
00:24:03,865 --> 00:24:08,367
It can be Python files, can be SQL files, it can be dbt projects.

374
00:24:08,589 --> 00:24:11,121
we deploy these with asset bundles.

375
00:24:11,121 --> 00:24:18,064
For example, here you can see the job file where we execute a specific file.

376
00:24:18,064 --> 00:24:21,398
So that's how we, deploy this file to data.

377
00:24:21,401 --> 00:24:23,993
and we can schedule it or whatever.

378
00:24:23,993 --> 00:24:27,854
But in the end, what we are doing is we're just executing this code.

379
00:24:27,854 --> 00:24:30,734
And this is basically a notebook task.

380
00:24:30,734 --> 00:24:34,095
So it's like Python code.

381
00:24:34,095 --> 00:24:38,216
And the SQL statement, it's basically in here.

382
00:24:38,216 --> 00:24:40,517
So we say create or replace view.

383
00:24:40,517 --> 00:24:41,817
We have some

384
00:24:41,952 --> 00:24:46,845
some parameters in here to make sure it ends up in the right catalog, in the right schema.

385
00:24:46,845 --> 00:24:50,068
But this is the same as we saw in the documentation.

386
00:24:50,068 --> 00:24:56,886
So here you see, for the minifigs, we start from the sets, which is our data set.

387
00:24:56,991 --> 00:25:03,462
We join with the themes yes, because it's always important to understand the themes Then
we join with the inventories.

388
00:25:03,462 --> 00:25:09,094
And from inventories we go to the minifigs, inventory minifigs, and we go to the minifigs.

389
00:25:09,094 --> 00:25:16,558
So if you look here, basically what we are doing is we are going from here to inventories
and then to the inventory minifigs and minifigs.

390
00:25:16,558 --> 00:25:21,501
These are basically the tables that we are joining and transforming into this metric view.

391
00:25:21,753 --> 00:25:23,395
So this is basically the SQL statement.

392
00:25:23,395 --> 00:25:33,761
You see it's create or replace view and then there is specific syntax you see with the $2
signs and then we end up with YAML syntax.

393
00:25:34,302 --> 00:25:36,393
Yeah, yeah, indeed.

394
00:25:36,393 --> 00:25:40,206
Yeah, that's how Databricks implemented it.

395
00:25:40,206 --> 00:25:43,608
We have a second metric view that's a bit more complex.

396
00:25:43,608 --> 00:25:45,011
That's for the sets.

397
00:25:45,011 --> 00:25:46,011
We do the same thing.

398
00:25:46,011 --> 00:25:47,252
We start from the sets.

399
00:25:47,252 --> 00:25:57,086
We do the tour to the teams and then we go from inventory, inventory, inventory parts and
then we go for colors, parts and the part categories.

400
00:25:57,086 --> 00:26:01,007
But this all corresponds to the, let's say the chain of tables you see here.

401
00:26:01,007 --> 00:26:04,488
um Yeah, correct.

402
00:26:04,488 --> 00:26:08,094
So actually I think as a result of this, also makes...

403
00:26:08,094 --> 00:26:10,594
your data set easier to understand.

404
00:26:10,594 --> 00:26:17,234
Because if you see this, okay, it's understandable, but you need to know a bit about the
data, you need to do multiple joins.

405
00:26:17,774 --> 00:26:24,274
And in these metric views, these joins are done for you already.

406
00:26:24,274 --> 00:26:30,834
So I think it makes it bit easier to use the data as well, even if you're not an agent, if
you're a human, right?

407
00:26:31,111 --> 00:26:40,371
And so this is essentially when you deploy this bundle these scripts are executed as
different steps and they perform the SQL operations.

408
00:26:41,277 --> 00:26:44,810
Yeah, correct, So here I'm in the semantic layer folder.

409
00:26:44,810 --> 00:26:46,692
So we are in the correct data product.

410
00:26:46,692 --> 00:26:48,895
We have the asset bundles.

411
00:26:48,895 --> 00:26:49,705
defined here.

412
00:26:49,705 --> 00:26:51,716
It's with the Databricks.yml.

413
00:26:51,716 --> 00:26:53,777
There are some variables defined.

414
00:26:53,777 --> 00:27:01,803
We have a link to the targets file which defines the workspace we are deploying to.

415
00:27:01,803 --> 00:27:04,007
And then we are taking these jobs.

416
00:27:04,007 --> 00:27:05,507
We will deploy jobs.

417
00:27:05,507 --> 00:27:07,058
And we can also deploy storage.

418
00:27:07,058 --> 00:27:08,750
This is something else.

419
00:27:08,750 --> 00:27:09,600
So let's do this.

420
00:27:09,600 --> 00:27:10,671
can...

421
00:27:10,678 --> 00:27:13,129
to a Databricks bundle deploy.

422
00:27:13,380 --> 00:27:21,543
And now it's basically copying a bunch of files to our Databricks workspace.

423
00:27:21,803 --> 00:27:29,027
And then in a second step, it's deploying these files as Databricks resources.

424
00:27:29,027 --> 00:27:31,004
So that's kind of a...

425
00:27:31,004 --> 00:27:34,689
deployment mechanism internal to Databricks.

426
00:27:34,689 --> 00:27:39,034
But that's how you can deploy resources in Databricks.

427
00:27:39,269 --> 00:27:46,253
Okay, so the effect of this operation is that they're actually executed as well and the
semantic, no, not executed.

428
00:27:46,253 --> 00:27:47,044
not executed.

429
00:27:47,044 --> 00:27:52,004
Probably you can configure it somewhere, but for now they are just deployed.

430
00:27:52,004 --> 00:27:55,070
So the assets are in Databricks.

431
00:27:55,321 --> 00:27:59,407
So for example, if we go to the jobs, you can see.

432
00:27:59,659 --> 00:28:02,001
the semantic layer, for example here.

433
00:28:02,001 --> 00:28:05,942
So you see I did quite some trials before.

434
00:28:06,462 --> 00:28:11,766
But there are basically a bunch of tasks here defined.

435
00:28:11,766 --> 00:28:17,129
And the ones that I showed, like create set minifig, that's now a task, and that's in
here.

436
00:28:17,129 --> 00:28:19,869
And it's basically executing.

437
00:28:20,231 --> 00:28:26,971
Python file which is uploaded to Databricks now and if you click on this you will see the
code.

438
00:28:27,331 --> 00:28:31,071
So this is basically the code that we had in the...

439
00:28:31,071 --> 00:28:38,971
So this is a bit like kind of small detour to a deployment mechanism but that's how we
typically deploy.

440
00:28:39,011 --> 00:28:50,031
If we want to run this, suppose that we want to run one metric view, we can just run this
one.

441
00:28:50,282 --> 00:28:52,124
and now it starts running this.

442
00:28:53,186 --> 00:28:55,550
Yeah, yeah, correct.

443
00:28:55,550 --> 00:28:59,996
And also I selected one task here, so typically you will run everything together.

444
00:28:59,996 --> 00:29:01,780
Now I just run one task

445
00:29:01,780 --> 00:29:06,482
It's using serverless compute now, so it should go quite fast.

446
00:29:06,482 --> 00:29:08,844
But this is basically the code that it's executing.

447
00:29:08,844 --> 00:29:11,776
Okay, so when this is done, the metric view will be created.

448
00:29:11,776 --> 00:29:12,717
Yeah, yeah.

449
00:29:12,717 --> 00:29:18,843
There is a second way we can create the symmetric views, and that's by using dbt.

450
00:29:18,843 --> 00:29:22,625
So also here you can see, created the dbt project.

451
00:29:22,963 --> 00:29:27,125
We like to use dbt because it's like a nice framework to do your transformations in.

452
00:29:27,125 --> 00:29:31,389
You can do a lot of, let's say, configurations scaffolding.

453
00:29:31,389 --> 00:29:32,350
It's very nice to use.

454
00:29:32,350 --> 00:29:36,493
So also we wanted to see, can we define this metric views in dbt?

455
00:29:36,493 --> 00:29:38,074
And yeah, answer is yes.

456
00:29:38,074 --> 00:29:40,333
But not in, let's say,

457
00:29:40,333 --> 00:29:42,174
full release yet.

458
00:29:42,174 --> 00:29:47,101
So you have dbt-core and then dbt-core has a bunch of adapters.

459
00:29:47,101 --> 00:29:57,992
So for example for Databricks you have the dbt-databricks adapter which allows us to run
dbt on Databricks and there is currently one version here it's in a pre-release that's

460
00:29:57,992 --> 00:30:02,109
this version that has support for metric views.

461
00:30:02,109 --> 00:30:08,494
it will go into a full release soon but for the moment this still in a pre-release.

462
00:30:08,494 --> 00:30:09,555
How does that work?

463
00:30:09,555 --> 00:30:12,319
I can show you here in the models.

464
00:30:12,319 --> 00:30:13,659
external layer.

465
00:30:13,659 --> 00:30:16,621
So we define a dbt model as we always do.

466
00:30:16,621 --> 00:30:20,982
But we say that the materialization is metric view.

467
00:30:21,233 --> 00:30:24,677
And for the rest is the same as the Databricks language.

468
00:30:24,677 --> 00:30:31,899
So like the create view with metric statement is here handled by the dbt Databricks
adapter.

469
00:30:31,899 --> 00:30:33,564
uh

470
00:30:33,564 --> 00:30:46,164
essentially you paste in the YAML in here, but you can also use then the dbt macros that
you have access to and you generate YAML, which is then interpreted by this uh adapter and

471
00:30:46,164 --> 00:30:47,944
it makes you a Metafiew.

472
00:30:49,427 --> 00:30:57,234
Would it be similar to just write plain SQL and then like you did before and just paste it
in a dbt file?

473
00:30:57,486 --> 00:30:59,468
No, I don't think that will work.

474
00:30:59,468 --> 00:31:04,140
Because dbt takes care of the, let's say, the create or replace view type statements.

475
00:31:04,140 --> 00:31:06,914
And that's something that we cannot put in here.

476
00:31:06,914 --> 00:31:16,679
But you see that the syntax here, I think this is like very, very similar to the YAML part
of the native Databricks SQL statements.

477
00:31:16,724 --> 00:31:19,016
Would this then require specific maintenance?

478
00:31:19,016 --> 00:31:22,370
This seems like somebody needs to maintain this, keep it up to date.

479
00:31:22,370 --> 00:31:24,912
This particular metric view.

480
00:31:25,163 --> 00:31:26,114
Yeah, correct.

481
00:31:26,114 --> 00:31:34,686
It's like I would say any other query or table you have living on your data platform that
needs to be maintained.

482
00:31:34,686 --> 00:31:37,650
This one is typically closer to business side.

483
00:31:37,650 --> 00:31:40,433
So I think in an ideal case, if you have...

484
00:31:40,590 --> 00:31:50,243
business people, be it business analysts or business data engineers that know very well
the business logic, the ideal thing is that they are able to maintain this.

485
00:31:50,243 --> 00:31:57,222
And I think in that sense, it's also these metric views, okay, it's YAML, so there's a
pretty low threshold, I think.

486
00:31:57,222 --> 00:31:57,875
Yeah.

487
00:31:57,875 --> 00:31:59,716
to start working on this.

488
00:31:59,716 --> 00:32:04,528
Well, if you would have complex SQL joins or let's say even Python code, it's more
difficult.

489
00:32:04,528 --> 00:32:11,444
So I think here the knowledge you need to work with this is a lot more shifted towards the
business knowledge.

490
00:32:11,444 --> 00:32:13,895
Like what exactly do you want to calculate?

491
00:32:13,895 --> 00:32:15,027
That's the main question.

492
00:32:15,027 --> 00:32:18,416
Not necessarily that all the technicalities behind.

493
00:32:18,416 --> 00:32:24,856
If you can give a YAML editor that takes care of spacing in a nice way, I think this is
possible.

494
00:32:25,107 --> 00:32:33,087
But this does mean historically these models, semantic models, metric views, they were
built inside of BI tools, right?

495
00:32:33,087 --> 00:32:37,847
They were built as part of a Power BI package or whatever, or workbook.

496
00:32:38,207 --> 00:32:43,827
This is now shifting more towards the left, let's say, towards the actual data pipelines.

497
00:32:43,947 --> 00:32:46,947
And it seems that could cause some friction, no?

498
00:32:47,447 --> 00:32:56,479
Because people would need to still learn new technologies, new concepts, a new way of
working Do you see any friction on that when you speak to people?

499
00:32:57,277 --> 00:33:00,219
where I'm working, not really.

500
00:33:00,219 --> 00:33:02,731
I think it depends a lot on the context.

501
00:33:02,731 --> 00:33:10,027
If you have business analysts that have been working in the same tool for like decades,
yeah, of course you will have friction.

502
00:33:10,027 --> 00:33:17,429
but if you have, let's say people who are open to learning new technologies, then there
will be less friction, of course.

503
00:33:17,429 --> 00:33:23,154
But that's, think, also a bit like how the data world works, things are evolving all the
time.

504
00:33:23,154 --> 00:33:28,979
um And you need to keep up with technology and I think you need to do the right thing.

505
00:33:28,979 --> 00:33:36,265
And in this case, the evolvement of AI and the large language models.

506
00:33:36,638 --> 00:33:47,718
I think are a clear push to having this type of logic more on your data platform rather
than in a BI layer where it's like just used for one specific visualization in a report

507
00:33:47,718 --> 00:33:48,882
and so on.

508
00:33:48,882 --> 00:33:49,532
Yeah, exactly.

509
00:33:49,532 --> 00:33:53,085
That, that feels like locked in into that one Power BI dashboard.

510
00:33:53,085 --> 00:33:56,118
Well, now we pull it back to the data product itself.

511
00:33:56,118 --> 00:33:59,374
So it can be used by multiple of these dashboards.

512
00:33:59,552 --> 00:34:03,106
And so you did this in dbt and to deploy it on Databricks.

513
00:34:03,106 --> 00:34:05,057
Does it also follow the bundle principle?

514
00:34:05,057 --> 00:34:08,093
Do you do also Databricks bundle deploy?

515
00:34:09,337 --> 00:34:10,619
that's similar.

516
00:34:10,619 --> 00:34:15,402
basically with this deploy, we also deployed the dbt part, by the way.

517
00:34:16,943 --> 00:34:20,587
Yeah, it's just, yeah, it's one product.

518
00:34:20,587 --> 00:34:23,148
And basically it's a different task.

519
00:34:23,148 --> 00:34:28,774
So you see, we called it the dbt build task and we need to give some information.

520
00:34:28,774 --> 00:34:30,495
We need to give it the...

521
00:34:30,757 --> 00:34:42,603
dbt project directory and we need to give the commands so we say dbt build we give it some
extra parameters here that depends on your setup Databricks recognizes this as a dbt task

522
00:34:42,603 --> 00:34:47,000
so it will execute let's say the dbt parsing and the dbt

523
00:34:47,000 --> 00:34:53,871
running part, and then after it will execute the SQL commands as well on Databricks.

524
00:34:54,512 --> 00:34:55,372
So.

525
00:34:55,697 --> 00:35:01,374
So it follows the same deployment and then you also do the run and you get the same model
that you showed us earlier.

526
00:35:01,572 --> 00:35:02,844
Yeah, indeed.

527
00:35:02,844 --> 00:35:07,007
Here we also have the dbt task.

528
00:35:07,007 --> 00:35:09,608
This is basically, you call it dbt build task.

529
00:35:09,608 --> 00:35:11,529
It is a dbt task.

530
00:35:11,529 --> 00:35:14,771
So it is recognized as a dbt task in Databricks.

531
00:35:14,771 --> 00:35:15,456
Yeah.

532
00:35:15,456 --> 00:35:16,812
And so this you can run as well.

533
00:35:16,812 --> 00:35:19,474
And then out comes the model.

534
00:35:19,705 --> 00:35:21,036
Yeah, yeah, indeed.

535
00:35:21,036 --> 00:35:24,543
Could you show the end result again, Stefan?

536
00:35:25,000 --> 00:35:27,578
So here you have the semantic layer.

537
00:35:27,578 --> 00:35:29,391
Yeah, they're completely equivalent.

538
00:35:29,391 --> 00:35:39,392
And so we spent some time on how to get from the source, the Lego bricks data that you
have to make a source line product on top of that, the semantic data product.

539
00:35:39,553 --> 00:35:43,332
How do you then create that Genie Space or that agent on top?

540
00:35:43,332 --> 00:35:45,093
That's actually super easy.

541
00:35:45,093 --> 00:35:49,196
You can just create a Genie Space.

542
00:35:49,196 --> 00:35:52,720
Actually, it already directly asks you connect to your data.

543
00:35:52,720 --> 00:35:55,901
And it has a predefined filter for metric views.

544
00:35:55,901 --> 00:35:59,303
So as you see, it's a bit opinionated towards metric views.

545
00:36:00,523 --> 00:36:01,475
So let's...

546
00:36:01,475 --> 00:36:04,862
have like non-metric fields you could have normal tables attached to it.

547
00:36:04,862 --> 00:36:05,664
Okay.

548
00:36:05,664 --> 00:36:07,566
These are the non-DBT versions.

549
00:36:07,566 --> 00:36:08,907
Now it created a new space.

550
00:36:08,907 --> 00:36:12,989
So yeah, you can change the name if you want.

551
00:36:20,410 --> 00:36:22,750
So you see here the space.

552
00:36:23,001 --> 00:36:25,181
What is needed for space?

553
00:36:25,432 --> 00:36:28,592
You need to have an owner, of course.

554
00:36:29,352 --> 00:36:38,492
It needs to be connected to a warehouse because as we mentioned in the beginning, the
agent will typically interpret your question and write SQL code to...

555
00:36:38,534 --> 00:36:43,329
to get the data, so it needs to execute the SQL code somewhere, therefore you need a
warehouse.

556
00:36:43,329 --> 00:36:48,032
People using this warehouse, using this space of course also need to have access to this
warehouse.

557
00:36:48,032 --> 00:36:52,160
um It needs to have access to data.

558
00:36:52,160 --> 00:36:54,004
You can add more data here as well.

559
00:36:54,004 --> 00:37:00,226
And then you have a bunch of metadata or let's say instructions for the space.

560
00:37:00,226 --> 00:37:02,320
So you can give it general instructions

561
00:37:02,320 --> 00:37:07,975
like explaining what is Lego or maybe indeed giving some company information.

562
00:37:07,975 --> 00:37:10,217
You can provide joins.

563
00:37:10,217 --> 00:37:14,761
So as you mentioned before, here you can like propose some joins that are useful.

564
00:37:14,761 --> 00:37:22,829
You can also provide standard SQL statements like a filtering and then you can give it
extra metadata.

565
00:37:22,829 --> 00:37:26,238
and these are example SQL queries you can give it.

566
00:37:27,155 --> 00:37:27,936
this is interesting.

567
00:37:27,936 --> 00:37:32,602
think in Snowflake, for example, you ship that as part of your semantic model.

568
00:37:32,602 --> 00:37:42,965
Here you ship that as part of the agent, because you can plug in tables in the agent and
add the joins, add the examples, not as part of the model, but as part of the agent

569
00:37:42,965 --> 00:37:44,096
instructions.

570
00:37:44,648 --> 00:37:53,668
Yeah, yeah, I have the feeling that the Genie Space is not super opinionated, right?

571
00:37:53,668 --> 00:38:01,328
If you create it, then it directs you a bit towards metric views, but you don't need to
use metric views.

572
00:38:01,328 --> 00:38:06,728
can also use tables and give some instructions on how to join tables.

573
00:38:07,318 --> 00:38:10,277
Cause in the UI you could just say these tables, they're joined like this.

574
00:38:10,277 --> 00:38:11,943
Here are some example queries.

575
00:38:11,943 --> 00:38:12,535
There we go.

576
00:38:12,535 --> 00:38:15,112
Okay.

577
00:38:15,408 --> 00:38:16,970
So there are different approaches.

578
00:38:16,970 --> 00:38:20,878
I don't think there is like one approach that's clearly the best you need to follow.

579
00:38:20,878 --> 00:38:24,692
But there are different ways of achieving your result.

580
00:38:24,692 --> 00:38:27,293
And what I was then also wondering is about the access model.

581
00:38:27,293 --> 00:38:30,095
You mentioned you need to give people access to the agent.

582
00:38:30,095 --> 00:38:34,937
Do they also need access to the underlying data then or is access to the agent enough?

583
00:38:35,298 --> 00:38:36,138
Okay.

584
00:38:36,138 --> 00:38:41,061
So you need to have the agent permissions and in this case, the semantic model data
product.

585
00:38:42,102 --> 00:38:43,042
Okay.

586
00:38:43,462 --> 00:38:46,584
So the agent is doing the query with your credentials.

587
00:38:46,851 --> 00:38:52,526
Yes, uh And you also need to have access to the correct compute.

588
00:38:52,526 --> 00:38:56,150
So there is compute needed to execute the queries upon.

589
00:38:56,150 --> 00:38:58,101
And yeah, there you also need to have access.

590
00:38:58,101 --> 00:38:58,753
Yeah.

591
00:38:58,753 --> 00:39:00,014
Yeah, okay.

592
00:39:00,014 --> 00:39:06,829
So you need to ideally automate part of that because if you want to give people access to
an agent, yeah, you have to have several things in place.

593
00:39:06,829 --> 00:39:09,882
Okay, got it.

594
00:39:13,210 --> 00:39:20,113
so maybe to pick in on that, because indeed we show here, you can make a genius space in
like 10 seconds, but to maintain it and to make it available to like say, broader

595
00:39:20,113 --> 00:39:23,186
audience, I think these are, this is another level, right?

596
00:39:23,186 --> 00:39:28,190
So there you need to take a few more steps, but that's outside of the scope of today.

597
00:39:28,142 --> 00:39:30,944
Maybe one last thing I prepared.

598
00:39:30,944 --> 00:39:34,928
It's pretty small, but we also created a query.

599
00:39:34,928 --> 00:39:39,817
We created basically a kind of analytics data product to get specific information.

600
00:39:39,817 --> 00:39:44,479
for example, what we wanted to know is the amount of distinct sets per year.

601
00:39:44,580 --> 00:39:49,744
And we made that based on the source aligned data and based on the

602
00:39:50,209 --> 00:39:51,370
semantic model.

603
00:39:51,370 --> 00:39:52,330
Here you can see the query.

604
00:39:52,330 --> 00:39:59,873
So for the semantic model, it's basically selecting a release here and then you measure
the set count, the distinct set count.

605
00:39:59,873 --> 00:40:05,098
And for the plain query, it's more or less the same here because it's a pretty simple
question.

606
00:40:05,098 --> 00:40:07,108
You need to do a count distinct, right?

607
00:40:07,108 --> 00:40:10,500
So the count distinct, it's something that's already in the metric view.

608
00:40:10,488 --> 00:40:11,269
Yeah.

609
00:40:11,492 --> 00:40:12,013
Okay.

610
00:40:12,013 --> 00:40:16,726
If you would have to summarize your experience up to now, what would be your main
conclusions?

611
00:40:16,977 --> 00:40:21,877
Well, that you can, let's say, use the semantic layer.

612
00:40:22,160 --> 00:40:28,763
in our existing setup on Databricks, would say, with data products with CI, CD
implemented.

613
00:40:28,763 --> 00:40:36,038
Now here you see we do a manual deploy, but you can also do this with the GitHub actions
or DevOps pipelines.

614
00:40:36,038 --> 00:40:45,671
So we can basically integrate the semantic layers pretty straightforward into our existing
systems, our existing setup, our existing architecture.

615
00:40:46,081 --> 00:40:56,902
like interested to go further on this and to understand how the Genie Space is really
interact with the data because we saw now the metric views.

616
00:40:56,902 --> 00:40:58,253
Okay, it's interesting.

617
00:40:58,253 --> 00:41:02,340
It gives like an extra layer on top of the data.

618
00:41:02,340 --> 00:41:10,999
But do you need to use it or can we also do like with tables with some join proposals and
some extra comments?

619
00:41:10,999 --> 00:41:13,990
Is that not like a better approach for the agents?

620
00:41:13,990 --> 00:41:17,349
I think that's interesting follow-up work to do.

621
00:41:17,347 --> 00:41:17,918
Yeah.

622
00:41:17,918 --> 00:41:18,419
Okay.

623
00:41:18,419 --> 00:41:18,689
Yeah.

624
00:41:18,689 --> 00:41:22,546
From what I've seen today, I've seen, you can do talk to your data in Databricks.

625
00:41:22,546 --> 00:41:27,043
You have your data already set up and all you need to add is either a semantic model.

626
00:41:27,043 --> 00:41:31,980
You can use dbt for that or the more native approach, if I may call it so.

627
00:41:32,309 --> 00:41:33,691
to publish those models.

628
00:41:33,691 --> 00:41:36,054
You hook them up to the agent and you're good to go.

629
00:41:36,054 --> 00:41:39,457
Of course, these Genies Spaces still require more exploration.

630
00:41:39,457 --> 00:41:45,164
But next to that, I also saw in the agents, if you want to use a UI, you can also click it
still together to experiment.

631
00:41:45,164 --> 00:41:47,046
You can add tables, you can add the joins.

632
00:41:47,046 --> 00:41:50,069
That's also still an option if you quickly want to explore.

633
00:41:50,227 --> 00:41:52,807
Okay, and then one final question remains Stefan.

634
00:41:52,807 --> 00:41:55,483
Of course I want to know what's your favorite Lego set?

635
00:41:55,483 --> 00:41:57,264
That's a difficult one.

636
00:41:57,504 --> 00:42:02,866
I'm always in favor of the the buildings, like the modular buildings.

637
00:42:02,866 --> 00:42:08,849
is one every year, I think, even on the 1st of January, there is a new one released.

638
00:42:09,830 --> 00:42:12,350
So I would need to pick one of them.

639
00:42:12,571 --> 00:42:17,613
I think at one point in time there was the 10th year anniversary, which was a bit bigger.

640
00:42:18,433 --> 00:42:20,134
So I will go for that one.

641
00:42:20,534 --> 00:42:22,677
Assembly square, that's it.

642
00:42:22,880 --> 00:42:22,999
nice,

643
00:42:22,999 --> 00:42:23,802
Okay cool.

644
00:42:24,181 --> 00:42:25,842
Thanks a lot for the demo.

645
00:42:25,842 --> 00:42:31,924
Very cool to see how Databricks works and how you can attach agents to your data using
these metric views.

646
00:42:32,138 --> 00:42:32,858
All right.

647
00:42:32,858 --> 00:42:34,758
Thanks a lot everybody for watching.

648
00:42:34,818 --> 00:42:38,757
I hope you enjoyed this on Databricks and we'll see you next time.

649
00:42:38,757 --> 00:42:39,552
Bye bye.