Michael: Hello and welcome to PostgresFM, a weekly show about all things PostgreSQL.
I am Michael, founder of pgMustard, and I'm joined as always by
Nik, founder of Postgres.ai.
Hey, Nik
Nik: Well, Michael
Michael: Hello, and we have a special guest today, Tudor
Golobenco from Xata, CTO there.
Really nice to have you, Tudor.
Thanks for joining us
Tudor: hi Michael, hi Nick, hi everyone.
Thank you so much for having me
Michael: It is our pleasure.
So to kick us off, I wonder if you could give us a bit of the
background story of Xata and take us through to where we are today
Tudor: Yeah, so I will actually start with what it is today because we did a
change once, like a pivot or like a-- I call it more like a technical pivot.
So it's a Postgres platform which has as differentiating features a very fast
copy-on-write branching, so it can create a branching in a matter of seconds, even
for databases that, that are terabytes in sizes, and scale to zero which means that
if a database is inactive, the compute part shuts down, and then on the next
connection, it automatically wakes up.
And this wake up is very fast.
We're targeting one second.
So that makes it very convenient, especially for developer workflows.
So these non-production use cases, things like if you want a Postgres branch for
each pull request or if you want to create an, sort of an ephemeral database for each
agent run or for your agent and so on.
Now, these are our more common use cases.
But on the Xata platform we can and do run production workloads as well.
So the differentiating features are on the developer experience if you
want, but you can use it as a normal Postgres hosting provider as well.
So that's what Xata is today.
It didn't really start like that.
The initial idea five years ago my partner Monica Sarbu, she wanted to do
a database that's really easy to use.
It was a lot of inspiration from Airbase or spreadsheets and because they were
like like developer using Airbase as a database or Google Sheets as a database.
So the initial idea was to make one that's as easy as to use as a
spreadsheet but that scales up so you don't have to switch at some point.
So having features from both.
And we did that for a while with some moderate success.
But at some point, we decided that it's pretty hard to build a
long-term business on that audience.
And it n-never really took off as a primary database
for, like for more production
Nik: This…
Tudor: use as teams and things like this.
Nik: Apologies for interrupting, because you're packing a lot of
stuff in one a very short time.
I just wanted to pause a little bit here and it- this reminds me a bit IO, right?
Th- they also like drag and drop CSV file.
You don't think about schema or anything, but it's Postgres behind it.
So you had similar thing, right?
Tudor: Yeah, but p.io were actual Postgres.
Ours, we were actually hiding Postgres behind our own API.
So it was a REST API that you interacted with data a bit like
PostgREST or something like this.
Nicer, I would say, but but that, that type of interfaces.
Yeah
Nik: So it's I like this idea a lot but Bit.io was closed.
I remember like a very short notice, like two weeks notice or something.
People were like outraged.
That was yet another not fully Postgres platform, but I have something,
like I store my data somewhere and but then it's closed and I have
very short notice to grab my data.
And you are telling us right now that also this model didn't work well, right?
In terms of-
Tudor: yeah.
Yeah, like there's definitely companies that do-- that address
this sort of audience successfully.
Like I'm thinking of Supabase, for example, right?
But they have like millions of developers and such, and out of those millions,
you're going to have a small minority that will become large databases, so
they will pay a lot and so on, right?
And we had a few tens of thousand.
I think at the peak we had 50,000 users.
Turns out of those, there's not enough paying customers
to sustain such a business.
It didn't work for
Nik: Thank you for sharing.
This is super interesting.
50,000 it's actually a lot.
But
Tudor: yeah.
It was quite a lot.
We had the growth in the number of users, but that didn't
reflect in the growth in revenue.
Nik: Many of them are not paying.
Most
Tudor: Yeah.
Nik: Yeah
I feel you well.
I understand this problem also, like from different kinds of businesses.
Yeah.
It's hard to find business model on this.
Unless you're Supabase.
This is a good comparison.
Thank you so much.
It's super.
Like before th-this recording I said we were, we're going to focus
on technical stuff, but the start is very different than what…
and I like it.
Yeah.
Thank you so much.
Michael: I think it's really useful context to then dive into the technical
side because I remember one of your posts that I thought was really good
about the architecture which I guess might have been the architecture all
along, but it explained how you could offer a free plan and how the free
plan didn't actually cost you very much because of your architectural choices.
And that, that was a really nice post from the business perspective but also
from a technical perspective, like understanding how those trade-offs
actually help you as a business.
Tudor: Yeah, we-- So on the new platform, we don't have a free tier.
this is again, the business side, and our experience with the old platform.
So we call the old platform Zeta Lite.
It's now closed.
We migrated the users that we could and so on, but it's now
closed, so now it's only Zeta
but the old platform had a had a free tier and when doing the new platform, we
were not very eager to add another one.
Although with our current architecture, we can do it again because we have
this scale to zero, which is also very efficient from that point of view.
But we're not very eager because it adds a lot of noise and like I was saying earlier
you have some growth, but as long as…
If that doesn't really result in in the type of business that
you want, you get a bunch of feedback that's not really useful.
And it confuses things a little bit more on the business side.
We might come back to that decision and maybe eventually
we'll add again a free tier plan, but currently we don't have one.
Nik: Yeah, let's shift to more technical stuff because I know you
use CloudNativePG, you have branching, you have very interesting open
source tools like pgStream developed and others like pgRole, right?
I'm very curious to talk with you about any of them.
Tudor: Yeah, maybe we start at the Xata Store.
It's our storage engine that makes a lot of this possible.
And then we can go about how we use CloudNativePG to get
the very fast wake-up times.
That's also, I think, interesting.
Nik: So copy on write and branching,
right?
Tudor: yeah.
so we do copy-on-write branching, but we do it at the underlying volume level.
So because our feature set is pretty similar with Neon, right?
Also like copy-on-write branching and then scale to zero with
fast wake-up times, these
Nik: decouple storage and compute, right?
Tudor: Yeah, the
Nik: And you re- and you recently made everything open source,
Apache 2.0 or no, or not everything
Tudor: it not quite everything, but enough to run the whole platform yourself.
But it won't scale to the same levels that we scale our own platform, and you won't
get as fast times as with our platform.
I can go into more details about exactly why, but we have op-open
source enough to be, to be running it yourself and to be useful.
But it's not quite everything open source in there
Nik: Cool.
So if we talk about storage, like what's interesting there
Tudor: Yeah, so Zeta Store, we call it like the Zeta Store and it's-- this
is like one of the proprietary pieces.
And we have quite a long history into that.
We tried a bunch of products and Kubernetes tools for managing
storage, the storage layer.
As in the end, we decided to build our own, but also use a bunch of ZFS
functionality, so it's based on that.
The way it works is, and I'm not sure how familiar you are with ZFS, it's
Nik: let me interrupt you here.
my company, Postgres.ai, has Database Lab Engine, which implemented
database branching in 2019 before Neon and before Xata, right?
And I have a lot of questions related to your experience with database
branching and how you convince users that it's really needed.
Because we do it seven years and some of them are solved very easily,
but some of them still struggle and are like, "Oh, production data,
we need to change it," and so on.
Let's talk about this separately, but I just to give you context
that you probably don't have.
Tudor: yeah.
Yeah
Nik: And it's on ZFS.
Yeah.
Tudor: Oh, are you using ZFS?
I thought you are using BTRFS.
Uh, no
Nik: it was not stable when we started, and I don't know
Tudor: Gotcha.
Okay.
I was actually remembering
wrong then
Nik: It was actually interesting.
When I created it first, it was out came out of ex-experience with GitLab.
We just needed to give backend engineers capabilities to experiment
and see ex-explain plans not touching a few database experts, right?
And it's, it was great.
It worked great.
And then we scaled it and the problem always was that first of all, production
data how to protect this, right?
And also when Neon started working with branching, it's very different.
So we do it very locally, and we achieve static cost for all clones and branches.
So you have, you pay once and you have everything, like mu-multi-tenancy
in one virtual machine, basically.
You have very different model, so I'm very curious how it works and what's your…
what do you observe how people use it?
And because my honest take, I spent a lot of time in this area,
market is still not understanding that branching is super powerful.
And I hope AI will
Tudor: I think--
Nik: but yeah.
Tudor: I think it's it's it's growing pretty nicely.
But yeah the, like the use cases that we see most often are branch per PR.
So they do PR and have some sort of preview environments.
This could be, like Vercel or their own stuff or different
other platforms for ephemeral environments or preview environments.
But then they want also the database to be part of that with the data, right?
So then a platform like ours makes that possible.
You have the database with the data.
It doesn't have to be the exact production data.
And the way we typically recommend people to do is we tell them,
"Look, keep production where it is."
Let's say it's RDS.
And then we have, this is why we develop pgStream.
We create a replica of it.
And pgStream can copy the data once.
So for small databases, we recommend people to set up a nightly job
or a weekly job or something like this because it's simpler.
For large databases, pgStream can also do the initial snapshot and then
keep it up via a logical replication.
Nik: And DDL?
And
Tudor: and DDL,
Nik: you solve DDL.
I looked at it briefly, and I'm very curious, like, how… if
you do it and what's your scale?
Is it working really well?
If it's working really well, why it doesn't go to core
of Postgres in that form?
Tudor: Yeah.
I think there's some work in progress to make it in in, In co-
Nik: it-- there are several, there, there are several ways, and if you
check recent activities it's not super progress-- it's not progressing
Tudor: And not progressing, yeah.
actually a little bit of backstory on, on pgStream.
We were doing in the old product, so Zetalight, we were doing, we were
offering this REST API, and then most of it was powered by Postgres.
But we also had a free text search with, BM25 and relevance and so on.
And that was actually searched from Elasticsearch.
So we wanted to have this replication between Postgres and Elasticsearch.
And we could do, because we had like our own API for modifying the schema and such,
you could do it in both at the same time.
But that's also tricky because if you replicate data but the schema
changes come out of order, then again you have a problem because
it's not going to fit the schema, the classical CDC type of problems, right?
So it's actually better if the schema changes are in the same
replication stream as the actual data.
It's better that way.
So that's why we did this event hook approach, listening for schema
changes and then apply them that.
There, with Zetalight we had this advantage that we, you couldn't
do arbitrary schemas in Postgres.
It it was a particular set of data types and so on.
It was a little bit more into our control, so we knew we could
test all possible schema changes and make sure they replicate
correctly and these sort of things.
But then we generalized this approach a little bit more.
Nik: Yeah.
So that's interesting.
So you have limited set of what's happening, right?
And then using event triggers, you, Yeah.
Tudor: Yeah, that's our approach.
And of course, it needs-- we need to make sure it's in the right
place in the replication stream.
And the way we do it is we write in a kind of, in a shadow table.
We write what has changed and which w- with what altered statement that
happened, and that gets put into the logical replication stream.
And then pgStream reads it from there and applies it on
the target in the right order.
That's the general approach.
Nik: And I'm very curious at what scale it is working and are you
fully satisfied or there are some like plans to improve or something?
Tudor: Yeah.
We have people using pgStream at like four or five terabytes of data.
And the initial snapshot takes like 24 hours or something like this.
And then And it b- that depends on a number of factors or something like
this, and then it's kept up to date.
But then it, like it, it's from company to company, and sometimes
we have to work on pgStream a little bit , to make it work for them.
But in the meantime, we had we it had enough battles, let's say that
now we're feeling like, it covers already the majority of, uh, of,
cases pretty well.
Nik: Yeah.
Battle proven already, right?
Tudor: Yeah, to some degree.
Like there, there are still problems.
The CDC logical replication
Nik: oh yes
Tudor: is still quite a pla- I w- I would love if DDL would be supported in core
Nik: everyone waits for it s- but it's super not trivial as I saw
last time when I checked this topic.
That's cool.
I like s-small non-non-technical side note side question.
These pieces are… Like you mentioned some proprietary stuff
about p- pgroll and pgstream
Tudor: Yeah, these are fully open source,
Nik: yeah, that's and Apache 2.0, right?
Tudor: Yes.
yeah.
Yeah, and especially pgStream, it's important for us to be open
source because we recommend people to run it in their environment
because it also does anonymization.
This is the, you know, like kind of what we started from.
It's important to do anonymization so that when you do this copy-on-write
branching, the data is already anonymized and you have branches, very fast branches
that don't contain the sensitive data.
Like it's up to you what what transformation rules you set and so on.
But in many cases, it's like 80% of the data is fine as it is, and you have
some data that it's sensitive and you need to transform it and make sure it's
not in any developer copy and so on.
And this I think is the, is the perfect balance of things if you want.
But it does depend on this replication story.
And we want to run it in their environment so they have full control over it.
That's why it needs to be open source really.
Nik: Yeah.
this answers my question
Michael: on the anonymization front, I saw you acquired a company.
I saw some blog posts I think you wrote.
Was it beginning of this year?
Is that what you've then ro- rolled into this
Tudor: Yeah, partially partially.
The it's called Privacy Dynamics, this company, and they were, like, really
deep into the anonymization front.
it's like there's levels to this stuff.
In pgStream we already had the anonymization, which was
simple column rule-based.
Like you can say, okay, this column, you can hash it like this
deterministically and, it's hashed.
But then like for example, for HIPAA compliance and so on, you need to do
the type of anonymization where if you, le-let's say there's patient's data and
you have a woman that's 46 years old and so on, and has this email address
or something like this, and if you just anonymize the email address, but it's
the only woman in the data set with those characteristics then you haven't
really anonymized anything because you can, like even if the name is not
there, there's enough information, unique information there that you
can still work backwards what it is.
So they doing these sort of things where they see exactly combining the
data set into groups and figuring out, okay, we need to anonymize this
data so that no group is really small.
Every group has more than 10 data points and such, so you
cannot find the individual.
And this is like more like, like HIPAA compliant sort of things and such.
So that's not, that's also something that's not in pgStream at the moment.
Maybe we will roll out more of those features in there that we set up
more like manually for customers that actually need this complex stuff.
Nik: Yeah, that's great.
So these two pieces are, they're interesting themselves.
They could they could be separate episodes, honestly.
This is what I… Yeah, because if we dive deeper, there are so many levels.
As I said I'm in that space as well.
We usually use some other tools, like we didn't build.
And we-- I remember also there was a company, Tonika.
Yeah I'm curious what happened to it.
Als-also
Tudor: They still exist, I think, right?
Nik: Yeah.
So a lot of… if you look to what they build, you understand
that so many layers go there.
It's like
Tudor: yeah.
Absolutely.
Yeah.
And Tonic AI I think they still exist, but it's just proprietary and they
open source like a simple version of it, but it doesn't scale at all.
So it's,
Nik: and important to also remember, like use cases are
very different for branching.
If you just want to… Like you said you don't recommend to take production and if
it's like simple development it's better to have some like preview environments.
It can be some synthetic data, for example, right?
You don't need to synchronize it all.
But If you bring data for a various like closer to production testing,
and especially if it's re- related to performance testing, I'm-- I, I was
always concerned about what plans we will be getting because if you change
data, I stop trusting those plans.
Tudor: Yeah, of course.
Nik: So I usually try to convince people to keep like red data a- as is production
data and just limit access and consider it as part of production and still have
branching and test it closer to production because in… this is the only way you can
have really realistic performance testing.
What you think about this?
Tudor: Yeah.
So there, there is this deterministic hashing which basically means if you
hash a value, it will always hash to the same thing, which is also
useful for referential integrity.
Let's say, like if you have a foreign key on a string column or something
like this, then you want it to anonymize the same in both places.
So that helps a little bit with this.
But yeah, with the Postgres planner, there can be a lot of nuances around
Nik: we, as I said, this could be whole episode.
If we dive deeper, like when you say hashing, okay pgStream takes
data from RDS, understand this.
You have ability, like you copy anyway, right?
But if you talk about your own platform, you have physical data directory,
Tudor: You're right.
Yes.
Yeah
Nik: W- when you touch it with any update, with anything, it's better,
like you have, like we have think cloning branching on physical level.
And if you start touching it, you already will Mi- Michael will tell
better what, what will happen with plans because he builds pgMustard, right?
Which is focusing only on, on the plans.
So if you physically - change something, you like, different
number of blocks and so on, it's already different plans, right?
Tudor: Yeah
Nik: Relpages can… reltuples numbers will become different.
Statistics might start different.
Even if you don't, if you say update some row set column equals the same
column ID something, logically nothing changed, but physically it changed.
And eventually if you massive scale, it can drift plans
Tudor: Yeah, and logical replication already like kind
of throws that away because
Nik: Right.
But if it's, if we go back to ZFS and thin cloning, the copy-on-write,
I, I-- my point was always like, let's not touch the data and we will
have ability to have this, the exact
Tudor: and you can you can still do that.
You can have… like if you run production on Xata what we typically
recommend is have a prod project and then something like a staging project
or something like this, and pgStream between them to do the anonymization.
But then the prod project still supports copy-on-write branching,
and that would be like ZFS
Clone type of thing.
And you can use that but you would only use this, use it when you
actually need it for like performance,
Nik: And like I, I see Michael wants to ask something.
But I will ask very last question because I sit on this topic seven years already,
so I like, I'm super curious to find guys who have like also spent time there.
Last question from me.
Unit economics.
If we want to build like pipelines which will test performance plans, right?
If we keep data in Zetta production and build these pipelines will
each every CI pipeline lead to provisioning on the whole node?
Will I be paying separately for whole… For if I run
10 pipe- or 100 pipelines, you… I need to pay for every one, right?
Or it's
Tudor: Because we do the separation of storage and compute, right?
And on the storage side, there's this thin copy, right?
So you only pay for the difference.
That's on the
Nik: so it basically doesn't change.
If I don't change data, it's the same price.
I
Tudor: Right.
With, of course, some gotchas there as well, but at the
high level, let's say that.
And on the compute side they would we use Kubernetes so that would be-- they will
be starting on different compute nodes potentially, but not each new branch
will be a new node necessarily, right?
They will fill existing nodes and such.
And there you only pay if you use our Xata Cloud platform because
you also have bring your own cloud, which is a little bit different.
But if you use our cloud platform you only pay for the time the branch is active.
And with the scale to zero that that, it's like for development it makes a
lot of sense because it wakes up in a second when you connect to it, right?
And you can set like a five minutes expiration time.
This is like a, a lot of our customers do.
It's like you pay almost nothing.
I actually have a blog post.
You can run thousand… It's just how the math works.
You can run a thousand branches for one dollar basically a tenth of a cent
each if they run for five minutes.
Nik: yeah, that's great.
So this is much better than what I observe with thing cloning
at, in Aurora and Neon, right?
Because there is absolutely it, it's linear scaling.
If you wa- if you want many pipelines in CI to test, you
amplify your spending budgets.
And this limits, This is super important because this makes engineers to
stop testing enough, and they start
Tudor: Yeah.
Yeah.
And it discouraged to have a branch for each PR, which is like
Nik: so that's the kudos for that.
This is cool.
I, as I understand, it's not like a constant price, but definitely you've
thought about this and because this is super important to make testing
more like intensive and complete.
Tudor: Yeah.
Yeah.
So definitely Neon ch-- Neon, like you get 20 branches for free.
You still pay the compute.
They're not completely free.
You would pay for the compute but above 20 branches you pay a fixed amount for
each of them, even if they are inactive.
So we don't do that.
We don't charge that.
That's why you can have a thousand branches with only $1, which I don't
know, maybe in a year we'll have another call and I will say that
does- that doesn't work economically.
Nik: I you start to add nuances.
I know which exactly which nuance.
If you have some branch or clone or like it holds snapshot, and
if production runs away into the future, you keep… You start paying
for all data to be stored, right?
Tudor: Yeah, Exactly.
Nik: That it is.
I know this problem very well
Tudor: Yeah like sometimes hard to explain because you create a branch,
its cost is zero on the storage, and then you delete all the data into it,
and then the-- you would expect-- you wouldn't expect the cost to go up, but
that's actually what happens because now it's different from the parent, right?
So it's like, "Hey, why am I paying more by deleting data," right?
That, that can be confusing
Nik: this is exactly what we also ha-had and still having.
Yeah, that's cool.
Cool.
Yeah.
Yeah, Michael, I, I apologize you had questions.
I, I'm just super excited to talk about
Michael: I was gonna ask more, but actually now I'm thinking,
I think this is great in terms of using Xata for your developer,
environments, let's say like pre-production those early maybe per
agent, per branch, that kind of thing.
But I think you've also got some interesting stuff on the developer
experience of running Postgres in production, like pgRole for example.
But I suspect you'll have opinions on a lot of the other things that
are currently quite hard to do as a developer running Postgres in production.
So I'd, I wondered about getting your opinion on some of those.
So the big one for me, I think still that's difficult for people
to manage is major version upgrades.
Like that, like things like that, I wonder if you have any takes or any,
any interesting features that most managed services don't offer or things
that you want to do in the future to make things like that easier?
Tudor: Yeah, we did put a lot of effort into the schema changes part of it.
It felt to us like one of the biggest pain points in, in Postgres.
There's a lot of gotchas there as well, like you do a schema change and it works
on your small database that you test with, but then you run it in production and it
locks the whole table for a long time.
So then you have to do this process and orchestrate multiple
PRs, change the application, do that, do the backfill, and so on.
So that w- that's how, like how we started pgroll.
It's also related to the old platform, which again, it was all about making
the database as easy as possible to use.
so then we had an API to do schema changes.
And that was not as, you couldn't do as much as as Postgres can do.
But everything that you did there was guaranteed to be safe.
You could never lock your database and such.
But then again, you hit the problem that people want to do those things.
So it's like that Postgres functionality was not added for nothing.
And then you end up having to support everything, which was,
whi- which was our problem.
So yeah, pgroll is is like it's an attempt to make that as as simple as possible.
It's working.
It also has some, some gotchas that currently we're not super focused on
it because we're focused primarily on the Xata platform itself with copy
and write branching and supporting many agents and these sort of things.
That's where our focus is.
So we didn't have that much time for pgroll lately.
Yeah, that's wh- that's where we are.
Nik: So now focus is to support to give database to agents, right?
To-- This is
Tudor: Right.
And that can be in two ways.
One is during development, like you give like I was saying
earlier, branches per PR.
Now with agents, you create a lot more PRs, and we're seeing, like we're seeing
companies that have two engineers, and they have 250 open PRs, and they're
actually all making progress and such.
We see people that that don't do reviews.
It's only AI reviews, and they get merged automatically.
The tests are running.
A lot of these things are happening now and will probably happen more, right?
So that's one.
And the second way is like AI platforms that need a Postgres database
as part of their product right?
And then they want to spin up like an FML database for that application, which could
be an experiment that they throw away.
So then you want these cheap databases that spin up and down
very dynamically and scale a lot,
scaling to
Nik: Yeah.
Yeah, these guys even don't need HA at all, right?
It's just single node no failover
Tudor: Yeah, exactly.
Exactly.
U-usually you don't need replicas for those use cases, yes
Nik: Yeah, so no high availability needed.
Maybe just backups and that's it, right?
Like you also feel this like new demand
Tudor: yeah.
Yeah, and we do take advantage of that because the way we solve
these very fast wake-up times is by doing what we call warm pools.
We keep CNPG clusters ready to go.
And then when there's a connection for a hibernated branch or database we take
one already provision clustered from from the pool, and we connect it to the
right ZFS volume on the ZetaStore storage node, and then serve it to the user.
And that's how, that's how it happens so fast.
But this means the clusters in the pool have to all look the same because
if they don't look the same and and they have, they have different
RAM and so on, it's obvious, but also different Postgres settings.
If that causes a restart, then you did nothing because you
need to restart it then too.
So we take advantage of that.
The cluster pools have no, , no read replicas.
It's-- They're very fast, but it is what it is,
Nik: how fast it is?
Tudor: So like we're targeting a second.
pro-- Yeah, provisioning a, a new cluster is the simplest operation
because we just take one of the, from the pool and give it.
It's it's, yeah, also second.
But it's that most of the cost there is really the control plane.
We have a single global control plane region, so we have to go to it and so on.
But that's fast.
The more-- The wake-up is actually more interesting because there you cannot
just take one from the pool and serve it.
You have to take one of the, from the pool, connect it to the right
volume, and then serve it, right?
And this is what the, this is what the Xata store enables be- and this
separation of storage and compute to some degree And have a bit of a trick
that Cloud Native PG doesn't support any of this, so we are maintaining kind of
our fork of it to enable some of that.
But there's the instance manager in in Cloud Native PG, and we modified it to
essentially wait for the PGDATA folder to show up, and until then it's waiting.
So the pod is completely provisioned.
It's just waiting for that folder to show up before letting Postgres start.
And behind the scenes, we mount it in the right place, and as soon as
that shows up there, the instance manager will let Postgres start.
And Postgres starts in 300 milliseconds or so.
If the, it depends as well if the, it, if the checkpoint was
done and then things like this.
But if it doesn't have to do the recovery, it's actually really fast.
But what's expensive is provisioning the whole stuff in Kubernetes
around it, the pod and the init containers and these sort of things.
that's where the time was going.
So that's why we do all that thing, and then the connection
the, the work that we do at the connection time is actually minimal.
That's why it's so fast.
Nik: That's impressive.
That's cool.
I have a
tricky question.
Oh.
okay, Michael first
Michael: I maybe have the same tricky question.
I wondered about your-- i-in a development environment, you probably
want the same extensions as production, or you might you might need them
in order to test various things.
And also, you might have changed a lot of configuration parameters.
I'm wondering how…
maybe it's not quite relevant to the same pooling system, but does that, that system
you just described limit your use of extensions and changing of configuration
Nik: I just want superuser.
That's
Tudor: Oh, okay.
two different questions.
Let me start with the one from Michael.
Yeah it does.
Because the, the problem is you cannot keep a cluster pool.
This is how we call them.
You cannot keep a cluster pool for each combination of, like you have instance
size, and then you s- have a Postgres setting or list of - extensions and so on.
Preload libraries is more relevant.
Because the extensions, they can be on the image, you just
don't enable them or enable them.
That's fine.
That's not a problem.
But if you need different set of preload libraries, that is a problem.
So we just chose to enable more more common stuff in the
preload libraries by default.
And then you have to, like you have to keep in mind that if you are not
on the happy path and we don't have a cluster pool for you, it still works.
It's just slower.
It will take a few seconds then s- six to 10 seconds instead of one
if you're not on the happy path.
And then the other thing is that we're-- we, we haven't done
yet, but we want to do, which is auto-scaling for the cluster pools.
Because some some configurations, if you want, are going to be a lot
more common than others, right?
The default settings that we have, you do a, a cluster pool
of 100 waiting to go, right?
So that you can do 100 at once and you don't-- y- it's not overloaded.
If you have a more complex one, it could be just one is enough, right?
Because you start that one, and then you immediately provision
the replacement for it.
That will take 10 seconds, . As long as someone else doesn't do the same operation
in those 10 seconds, you're okay.
So it's enough to keep one around.
But making this -- via static configuration is definitely annoying.
So that's why we want to do some sort of auto-scaling based on that.
So depending on the demand that we've seen historically, we would make these
pools smaller or bigger, and that would automatically optimize costs for us.
That, that's something that we haven't done yet but want to look into it.
Yeah, superuser is like I happen to know your opinion on that.
We don't give superuser to to our user.
We give the, give up s-
Nik: But will you think about it at least?
Tudor: Yeah, the, the issue is because we run on Kubernetes
it's pretty hard to secure it at
that level.
If you can run copy program on
Nik: Let's wrap up this question.
I have much more interesting and we don't have a lot of time.
So
Tudor: Mm-hmm.
Nik: question to you, tricky one.
If you started this platform to build this once again right now from scratch,
would you choose Kubernetes still?
Tudor: We do use a lot of stuff from Kubernetes and it's good.
Yeah, I'm happy with-- Overall,
I'm
Nik: most important.
Okay, cool.
Are you
Tudor: I'm happy
Nik: cloud native PG?
Tudor: I'm-- Yeah, you know, it's like that's more on the fence, I would say
for me.
Nik: opinion about it
Tudor: I know your opinion about it.
Nik: Yeah and more is coming actually.
Literally this weekend I didn't work only on this PG SimCity
Tudor: Oh, that's awesome by the way.
Nik: Yeah, thank you.
It's, it was a simple one prompt.
It was unexpected.
but this thing like what's happening, we have a le- a, a winning leader among
Kubernetes operators, which is completely like ignoring the facts like about
split brains and data loss and so on.
Do you use synchronous replication if you have multi-node setup?
Tudor: It's configurable.
It's
Nik: It's configurable, but it's not on def- by default,
Right?
It's not d- it's not on by default.
And it w- which means I have demonstration of split brain is still possible
in multiple f- in multiple ways.
Michael: Tudor, do you mean you let your users configure it?
Tudor: yeah.
We can enable it for them
Nik: And in Cloud Native it's not enabled by default, it's configurable.
It's the same.
Yeah, that's the problem.
And so haven't you already… Like maybe I'm over like maybe we don't need proper
HA, split brain protection and so on.
Like maybe this is like it's okay to run this thing and if your users,
if you don't know about your users suffering may- at your scale, that
this is good enough already, right?
Like it's okay.
What's your take on this?
Tudor: No, I think it's definitely a problem that like I'm, yeah I'm
also a little bit disappointed by the response to that issue.
I was expecting, like to be closed sooner.
I understand it's tricky because in CNPG there's no there's no proxy layer.
So they rely on Kubernetes services to expose things and to move them around.
That makes it, I think, a little bit more complex for them
compared to Patroni, for example.
The funny thing is we do have a gateway
Nik: it's not complex, it's impossible.
It's impossible.
You can… There is no consensus algorithm used.
There is only quorum, and there is mixing of concepts like quorum
is needed to handle transactions.
It's not n- it's not for deciding who is the primary.
So I have demonstrations.
You can have multiple-- You have two primaries.
You can have split brain, you can have data loss, and more to be published soon.
I'm pissed off because m- because of their popularity clients keep asking us.
I literally have right now two asks from two different companies to
explain, should we use CloudNativePG because it's super popular.
And now, like I meet you and you use it at scale.
So you don't have problems with no-nobody lost data, no-nobody
dealt with split brains.
Do we have… M-maybe we have already good cloud resources which don't experience
like outages, like maybe th-this is it.
Maybe we don't need the HA replicas may-maybe even, right?
Like it's already good enough to leave a single node.
So there is no such a problem as HA, high availability, no more such problems.
Or you had such issues like split brains or data losses because
there's no consensus algorithm
Tudor: Yeah, it's like I, like I would have to go back and look how often.
Like I do remember maybe one case where to be honest, we couldn't
figure out exactly what happened.
in these things there's always a combination of factors.
But I do remember once we thought, like this might actually be the
split brain scenario hitting us.
So like I, I think it's a bug really.
Nik: Unfortunately, in their DNA, they like, it's like this is a basic decision.
If you read recent post.
I have an answer unpublished.
I'm going to publish it, so more stuff is
Tudor: It's coming
Nik: And I sent you a LinkedIn request.
Please accept it.
Let's talk, let's keep in touch on this topic because I want the
industry understand that consensus algorithm is needed, and either
CloudNativePG needs to be, to change or people should stop using it.
Or maybe we like, again, like one node is enough.
We don't need the HA, everything is good.
AWS and Google Cloud they stopped being unreliable as it was 10 to 15 years ago.
Tudor: Yeah, on, on our side, we're already I told you we are on a fork of it.
It's not that I would, I would not recommend people to use our fork
because we would, we, we-- It's open source as well, but we didn't make
it with the goal of becoming, like a CNPG successor or anything like that.
But we did we did go a bit in a different direction when it comes to backups.
For example, we're using pgBackRest, and we made it in the operator itself
rather than the plug-in solution, which has a number of disadvantages.
So we have the, like we have the precedent of us going a little bit in different
directions regarding these things.
Also, we do have a gateway anyway.
We need a gateway, like a SQL gateway.
So then, like the main benefit that you are proxy-less actually
doesn't really apply to us.
So it might, that might be also something that we would choose to
do a little bit different there.
Nik: Yeah.
And by default CloudNative PG has a Barman or, for backups or what?
Tudor: I- it's a plugin that's-- It used to be, Barman in the core, but
then now it's Barman as a plugin.
I think as of right now it's the only, though, plugin that exists.
I think there's one for pgBackRest,
but in…
Nik: So if you replace and if you pl- if you have pgBackRest, this
next step is just to add Patroni
Tudor: Yeah, you're right
Nik: And then my concerns are fully resolved.
I'm okay.
Sensor algorithm works well and so on.
But I understand it can be super challenging at your scale because if
you have a lot of smaller clusters and so on ETCD or something, like you
sh- you need to properly engineer it.
I think it's done by some guys, but it's not straightforward to
not-- like, because we don't…
If we have a lot of small clusters, we cannot afford ETCD for everyone, right?
We need to have shared ETCD, and there are some
limitations
remember.
Tudor: exactly.
Nik: But it's like it would be natural step.
Like next time, like you bump into a couple of other weird cases and maybe you
should add Patroni, something like this.
Tudor: Yeah.
Yeah.
but that's, like there's the advantage that we don't really do this yet, but it
does make sense to have a bit of different configuration for your development
branches versus your production branches.
Because the the thousands of branches and so on, the really large numbers don't
even have replicas like you're saying, because that like the point is that
you shut them down, they come back up.
The storage is separated and it's secure no matter what.
It doesn't depend on Postgres at all.
So then, you just don't use read replicas at all for those use cases.
And the production ones are just much fewer and then you can
Nik: And computer's smaller, right?
So like less bu- shared buffers is small, everything like this right?
But then you can fool the planner and make it work as to choose
the same plans as in production.
This is magic I realized some time ago, and it's super cool that you
can s- have a tiny me- like memory small, shared buffers is tiny, but
you say effective cache size is huge.
Other p- planning parameters are huge, some other things, and
you get exactly same plans as in
production.
Yeah, and then you need to just adjust you, you need s- small mind
shift and engineer minds to make them focus on buffers numbers mo- more
because timing might be different.
This is what we talk about buffers three years already or four years?
Yeah
Michael: I'm super conscious of time,
but Tudor, is there any, any last things or shout-outs you
wanted to give on the Xata side?
Like I'm very aware we haven't even talked about DeltaX,
which looks really promising.
Tudor: Maybe we do another one in a few,
Nik: Yeah, let's have part two
quite soon.
Yeah.
It's, it was cool.
And thank you for coming.
I enjoyed to- a lot.
I think we do similar things, like in terms of where we ha- headed.
I think everyone should use database branching.
It's super cool, we will have all the links in show notes, right?
And so I encourage everyone to try out if you haven't.
And I guess you just need to tell your agent to try to go and use it, right?
That's it.
Tudor: Yeah, pretty much, yes.
Nik: That's cool.
Thank you for coming.
Let's have part, part two soon.
Michael: Yeah.
Thanks so much, Judah
Tudor: Thank you.
It was really nice.
Thanks both