Michael: Hello and welcome to PostgresFM, a weekly show about
all things PostgreSQL.
I am Michael, founder of pgMustard, and I'm joined as usual by
Nik, founder of Postgres.AI.
Hey, Nik.
Nikolay: Hi, Michael.
And let's welcome our guest.
Michael: Yeah, we are joined by a very special guest, Sugu, who
is a co-creator of Vitess, co-founded PlanetScale, and is now
at Supabase working on an exciting project called Multigres.
So welcome, Sugu.
Sugu: Thank you.
Glad to be here.
Michael: All right.
It's our pleasure.
So it's my job to ask you a couple of the easy questions to start
off.
So what is Multigres and why are you working on it?
Sugu: Multigres is a Vitess adaptation for Postgres.
It's been on my mind for a long time, many years, and we've even
had a few false starts with this project.
And I guess there is a timing for everything and finally the
timing has come.
So I'm very excited to get started on this finally.
Michael: Yeah timing is an interesting one it feels like for many
years I was looking at PlanetScale and Vitess specifically very
jealously thinking you can promise the world you can promise
this you know horizontal scaling with a relational database for
OLTP and it's it you know all of the things that people want
and we didn't really have a good answer for it in Postgres.
But all of a sudden, in the last few months, it seems almost,
there are now 3, 4 competing projects all doing it.
So why now?
Why is it all happening now?
Sugu: I would say, I think there is a timing for every new idea
to gather momentum.
Like the idea may be good, but the timing may be bad for one reason
or another.
But once the timing is good, then it is obvious that it is the
right time.
And it feels like now is that time.
Like I would say, for example, PayPal, there have been previous
attempts at moving money online, but it never worked.
But PayPal just came at the right time and therefore took off
during that time.
Many such examples where, you know, some ideas came too early
and didn't take off.
But for Postgres, this feels like it's the right time.
Nikolay: That's interesting.
Let me reveal something.
We met in 2018, I think, at Percona conference.
And in 2019, we met again in Nutanix at the meetup.
Go meetup, right.
And discussions about Vitess for Postgres started.
And a couple of times we had a call, I tried to involve a couple
of guys to and from from my understanding it never worked because
You could not do it yourself being busy with other thing I guess
my sequel related and the guys are looking at the complexity
other guys looking at the complexity of related and, and didn't
proceed.
And actually, in 1 case, it was, they decided to build from scratch.
It was, it was SPQR project.
It's live and there's sharding for Postgres.
Sugu: Yeah, Andrey Borodin.
Nikolay: Yeah, Andrey Borodin.
Borodin.
Yeah. Yeah.
So and other folks also involved.
And so for me, it was disappointing that it doesn't work.
And at some point I saw the message in Vitess, I think, that
we are not going to do it.
So like, don't expect.
Sugu: I feel so bad because I was so excited about doing it.
And then I realized, oh my God, you know.
Nikolay: But now you started and last week PlanetScale decided
to support Postgres.
So what's happening?
Like, I don't understand.
Just, just right time.
Right.
Yeah.
Enough, enough number of companies using Postgres, which really
needed.
Sugu: At least 1 horse will win.
Yeah.
Nikolay: So yeah, it's great.
But, but yeah, long, long story to this point.
Michael: Yeah.
Sometimes when there are multiple projects there's kind of slight
differences in philosophy or approach or trade-offs, willing
to trade 1 thing off in relation to another.
And I saw your plan, I really liked that you used, you mentioned
building incrementally.
So Vitess is a huge project, lots and lots of features, But
I've heard you talk in the past about building it quite incrementally
while at YouTube.
You know, it didn't start off as complex as it is now, obviously,
and you did it kind of 1 feature at a time.
And it sounds like that's the plan again with Multigres.
Is that different to some of the other projects?
Or what do you see as your philosophy and how it might differ
slightly to some of the others?
Sugu: I think my philosophy is that I would say I don't want
to compromise on what the final project is going to look like.
So any path that deviates me from hitting the target on the final
project, I do not want to take.
But any shortcuts that can get us early results, that we can
do, but only those that ensure that it doesn't risk what we become
eventually, which is a project that should feel native as if
it was for Postgres, by Postgres kind of thing.
I want it to be a pure Postgres project.
Nikolay: In this case, I must ask, in this case, are you sure
you're still good with choice of Go language?
Because sometimes we see projects which have really like sub
millisecond latency on average, like 100 microseconds, for example,
on average.
And Go definitely will bring a few hundred microseconds of latency
overhead.
So usually it's not a big deal, but maybe in some cases it's
some deal, right?
Are you happy with Go?
Yeah, I mean- Because you were 1 of the first big Go language
users building Vitess, as we know from various interviews and
so on.
Yeah.
So it's still a good choice because now there is Rust, right?
Sugu: Yes.
I would say, by the way, when we started Go, compared to where
Go is today, it was a nightmare.
Nikolay: Yeah, like 10 milliseconds.
Sugu: Like, yeah, like 10 milliseconds or something round trip
is what we were paying for.
Those days we had hard disks, by the way.
So that's another 3 to 5 milliseconds just within the database.
But things are a lot better now.
And at this point, the way I would put it is, Like the trade-offs
are in favor of Go, let's put it that way.
Mainly because there is a huge amount of existing code that you
can just lift and port.
And rewriting all of that in Rust is going to just delay us.
And at least in Vitess, it has proven itself to scale for, you
know, like multi, hundreds of terabytes.
And the latencies that people see are, they are not affected
by a couple of hundred microseconds.
So I think the, and plus there's, there's this inherent acceptance
of this network latency for storage and stuff.
And if you bring the storage local, then this actually wins out
over anything that's there.
Nikolay: This is exactly what I wanted to mention.
Yeah, I see PlanetScale right now, they came out with Postgres
support, but no Vitess.
And I'm very curious how much it will take for them to bring
it and compete with you.
It's another interesting question.
But from past week, I see my main impression is like, main stake
is on local storage.
And this is great because local storage for Postgres, we use
it in some places where we struggle with EBS volumes and so on,
but it's considered like not standard, not safe, blah blah blah.
There are companies who use it, I know.
I know myself, right?
And it's great.
Like today, for example, Patroni and since Postgres 12 we don't
need to restart nodes when we have failover.
So if we lose node, forget about node, we just failover and so
on.
And with local storage, not a big deal.
But now I expect with your plans to bring local storage, it will
become more, like I expect it will be more and more popular,
and that's great.
So you shave off latency there and keep Go, which brings 200.
Sugu: Good compromise.
Effectively, it's a win because 1 network hop completely eliminates
language level overheads.
Nikolay: Sounds good.
Maybe your goal will improve additionally.
Michael: I wanted to go back, Sugu, you mentioned not wanting
to compromise on feeling Postgres native.
That feels to me like a really big statement, coming from Vitess
being very MySQL specific, saying you want to be Postgres native
feels like it adds a lot of work to the project.
It feels like a lot to me.
What does it mean?
Is that about compatibility with the protocol?
What does it mean to be Postgres native?
Sugu: There's 2 answers.
1 is, why do we still think we can bring Vitess if it was built
for MySQL?
And how do you make it Postgres native?
That's because of Vitess's history.
For the longest time, Vitess was built not to be tied to MySQL.
It was built to be a generic SQL 92 compliant database.
That was actually our restriction for a very long time until
the MySQL community said, you need to support all these MySQL
features, otherwise we won't
Nikolay: move forward. CTEs, right?
Common Table Expressions, with, right?
It's I guess SQL 99 feature, not 92.
Sugu: Yeah, I think the first parser I built was SQL 92, which
is the most popular 1 that I know of.
So that's answer 1.
Answer 2 is more with the behavior of Postgres.
What we want is completely mimic the Postgres behavior right
from the beginning.
Basically, in other words, we plan to actually copy or translate
what we can from the Postgres engine itself, where that behavior
is very specific to Postgres.
And the goal is not compatibility just at the communication layer,
but even internally, possibly even, you know, recreating bugs
at the risk of recreating bugs.
Nikolay: In this case, it's like, so there is Citus in the hands
of Microsoft got everything open sourced.
So before, resharding was only in paid version, now it's in free
version, open source, so it's fully open source.
And they put Postgres in between, so they don't need to mimic
it, they can use it, right?
And latency overhead is surprisingly low, we checked it.
Well, it's C, but it's whole database in between.
But it's sub millisecond, so it's acceptable as well.
I think it's half a millisecond or so in our experiments with
simple select 1 or something, SELECT, just like.
So don't you think it's like quite, like in comparison, it's
quite a challenging point when you say I'm going to mimic a lot
of stuff, but they just use Postgres in between?
Sugu: Yeah, yeah, I think there's a difference in architecture
between our approach between Multigres versus Citus.
I think the main difference is it's a single coordinator for
Citus.
And there is some bottleneck issue with that.
If you scale to extremely large workloads, like that goes into
millions of QPS, hundreds of terabytes.
So having that single bottleneck, I think would be a problem
in the future.
Whereas,
Nikolay: yeah.
I understand you can put multiple, like you can have multiple
of nodes there and also you can put a PgBouncer to mitigate
the connection issues.
So it can scale as well.
Sugu: That's good.
That's not, that's something I haven't known before.
So, yeah, it's possible then that they may also have something
that can viably scale for OLTP.
Nikolay: Yeah.
So we're still exploring this and more benchmarks are needed.
Actually, I'm surprised how few and not comprehensive benchmarks
there are published for this.
Sugu: Yeah, what I know of Citus is probably what you told me
when we met.
So I was about 5 years old.
Michael: Yeah, yeah.
Another big difference and this is typically Nikolay's question
is on the license front I think you've picked about as open a
license as you could possibly pick which is not the case I think
for many of the other projects.
So that feels to me like a very Supabase thing to do and also
in line with what Vitess did and that seems to me like a major
advantage in terms of collaborating with others, other providers
also adopting this or working with you to make it better.
Is like, what's your philosophy on that side of things?
Sugu: My philosophy is, my metric for success is adoption.
Yeah.
And the only way to have a project be adopted is to have a good
license, a license that people are confident to use.
That has been the case from day 1 of Vitess.
We actually first launched with BSD license, which is even more
permissive than Apache.
And then when we moved the project to CNCF, they said, oh, no,
no, we need Apache license, which is why we converted it to that
1.
So adoption has always been, yeah.
Nikolay: Why do they say it?
Do you know?
Sugu: Why CNCF wants Apache?
I think Apache is a pretty good license.
They just made it a policy.
I mean, had we asked to keep the BSD license, they would have
allowed us, but we didn't feel like it was a problem to move
to Apache.
Nikolay: Yeah, and I remember you described like, when you did
it, like in YouTube, you thought about external users.
You need external users for this project to grow.
I guess at Google, AGPL is not popular at all, we know.
Sugu: Oh, banned.
Nikolay: Yeah, yeah, banned.
And Citus is AGPL,
Sugu: which is interesting.
Nikolay: Also, compared to Citus, I think you have chances to
be compatible with RDS and other managed Postgres, to work on
top of them, unlike Citus, which requires extension and so on,
right?
Sugu: Correct.
Correct.
Yes.
This was actually something that we learned very early on, wanting
to work.
We made a 5 line change on MySQL just to make Vitess work initially.
And it was such a nightmare to keep that binary up, to keep that
build running.
Fork.
Yeah, to keep that fork alive.
So we decided, no, it's like, we are going to make this work
without a single line of code of change in MySQL.
And that actually is what helped Vitess move forward.
Because people would come in with all kinds of configurations
and say, make it work for this.
So in this case, actually, we'll probably talk about the consensus
part.
That is 1 part that we think it is worth making a patch for Postgres,
and we're going to work hard at getting that patch accepted.
But I think what we will do is we will also make Multigres work
for unpatched Postgres for those who want it that way.
Except they will lose all the cool things about what consensus
can give you.
Nikolay: I'm smiling because we have so many variations of Postgres
these days.
I would expect people coming not only with different configurations
of Postgres, but also with various flavors like Aurora.
We have a client who just migrated from regular Postgres to AlloyDB.
Hello from Google Cloud.
And they already sharded on application side, but imagine they
could come to you and say, let's support AlloyDB now.
It's almost Postgres, right?
So yeah, so these things might happen as well.
Sugu: Don't they claim full compatibility with Postgres?
I thought you Not full,
Nikolay: but but most of it.
It's, they did interesting stuff in memory like, like column
storage and memory for tables.
It's row storage on disk, but column storage in memory.
But it looks like kind of Postgres, and we actually even had
some questions answered from my team unexpectedly because we
don't normally work with AlloyDB, but it looks like Postgres.
So I could imagine the request, let's support AlloyDB as well.
Anyway, but my question, I remember featuring the test that we
work with RDS and managed MySQL.
Did those features, like has this feature survived?
Sugu: No, actually later we decided that at least, we call it
actually managed versus unmanaged.
Managed meaning that means that Vitess manages its databases.
And unmanaged means that the database is managed by somebody
else, Vitess just acts as a proxy to serve queries.
At some point of time, we realized that supporting both is diluting
our efforts.
And that's when we decided, okay, you know, it's not worth it
to try and make this work with every available version that exists
out there in the world.
And we said, okay, we will do only managed, which means that
we will manage it ourselves.
And if you want, we'll build the tools to migrate out of wherever
you are, and we'll make it safe, we'll make it completely transparent.
In other words, you deploy Vitess on both and then we'll migrate
your data out without you having to change your application.
But then Vitess can be more intentional about its features,
more opinionated about how clusters are to be managed.
And we were able to commit to that because at that point, Vitess
had become mature enough.
People were completely trusting it.
They actually preferred it over previous other managed solutions.
So it wasn't a problem at that time.
Nikolay: Yeah, it's not a surprise.
That's why I asked.
Because you talk about local disks, backups, HA, a lot of stuff,
right?
And basically-
Sugu: Yeah, 5 nines is like what Vitess shoots for.
And like most big companies that run Vitess do operate at that
level of availability.
Nikolay: So what's the plan for Multigres going to support?
So only managed version, right?
Sugu: Only, So it would be, yes, it would be only managed versions
because I believe that the cluster management section of Vitess
will port directly over to Postgres, which means that you will,
once it goes live, it will be coming with batteries included
on cluster management, which should hopefully be equal to or
better than what is already out there.
So I don't see a reason why we should try to make it work with
everything that exists today.
Nikolay: So it means there is no, like, this is the same like
with Citus, it doesn't work with RDS on 1 hand, but on another
hand, we, like, I don't see it's only a sharding solution.
It's everything, which is great.
I mean, it's interesting, super interesting.
So a lot of problems will be solved.
And I expect even more managed services will be created.
I don't know how it will continue, like in terms of Supabase,
because of the very open license and so on.
But also I expect that many people will think, reconsider their
opinion about managed.
We had episode about this.
This is my usual opinion about managed services because they
hide superuser from you, they don't provide you access, it's
hard to troubleshoot problems.
In this case, if problems are solved in this and this gives you
a new way to run Postgres, so if many problems solved, it's great.
If you want
Sugu: to do so,
Nikolay: for example.
Sugu: Yeah, if you may not know, the initial focus of Vitess
is actually solving these problems first.
Sharding was actually came much later, like protecting the database,
making sure that they survive abusive queries.
Basically, that's what we built Vitess for initially.
And the counterpart of taking away power from the user, like
you said, is 1 is, well, we now know exactly how to make sure
that the cluster doesn't go down.
And 2, we countered that by building really, really good metrics.
So when there is an outage, you can very quickly 0 in on a query.
If a query was responsible, Vitess will have it on top of it,
like on the top of the line, saying that this is a query that's
killing your database.
So we built some really, really good metrics, and which should
become available in Postgres, probably from day 1.
Nikolay: That's interesting.
I didn't see, maybe I missed, I didn't see in readme you were
writing right now in the project.
Sugu: There's a last section called observability.
I missed that.
I need to revisit.
Nikolay: We're actually building something there as well for
regular Postgres.
I'm very curious, I will definitely revisit this, interesting,
okay. So yeah, great.
Yeah.
Michael: Also, I feel like this is quite a big difference on
the, at least with Citus in terms of the philosophy or at least
the origin story.
I feel like that started much more with OLAP-focused features
in terms of distributed queries and parallelised across multiple
shards and aggregations and columnar, and loads of things that
really benefit OLAP workloads, whereas this has come from a philosophy
of let's not worry about optimizing for those cross shard queries,
this is much more let's optimize for the single shard very very
short quick OLTP queries and let's make sure we protect it against
abusive queries.
So it feels like it's coming, architecturally, it's coming from
a very different place of what to optimize for first.
Sugu: And historically, that was YouTube's problem.
Surviving the onslaught of a huge number of QPS and making sure
that 1 single QPS doesn't take the rest of the site down.
Michael: Yeah.
Yeah.
Perfect. Makes loads of sense.
So actually, before we move on too much from that, what, where
do you see sharding as becoming necessary?
Like, is it just a case of a total number of QPS or like writes
per second type thing.
We've talked about sharding in the past and talked about kind
of a max that you can scale up to perhaps in terms of writes,
in terms of WAL.
WAL per second I think was the metric we ended up discussing.
Are there other reasons or kind of bottlenecks that you see people
getting to that sharding then kind of makes sense as it's now
time or you should be considering at this point?
Sugu: Well there is a physical limiting factor which is the single,
if you max out your single machine, that is your Postgres server,
then that's the end of your scale.
There is nothing more to do beyond that.
And there are a lot of people already hitting those limits from
what I hear.
And the sad part of it is they probably don't realize it.
As soon as that limit is hit, in order to protect the database,
they actually push back on engineering features indirectly, saying
that, you know, this data, can you make it smaller?
Can you somehow lower the QPS?
Or could you put it elsewhere?
Nikolay: Let's stop showing this number on front page.
Sugu: Yeah, yeah.
And it affects the entire organization.
It's a very small, it's a very subtle change.
But the entire organization slows down.
Like we experienced that at YouTube when we were at our limits.
We like the default answer from a DBA was always no.
We used to even kid, no?
The answer is no.
What's your question?
And when we started sharding, it took us a while to change our
answer to say that, you know, bring your data I like it we can
scale as much as you want. Believe it or not, we went from 16
shards to 256 in no time.
And the number of features in YouTube exploded during that time
because there was just no restriction on how much data you wanted
to put.
And coming back here, the upper, like reaching the limit of a
machine is actually something you should never do.
It's very unhealthy for a large number of reasons, like even
if there is a crash, like how long is it going to take to recover?
Like the thing that we found out is once you can shard, it actually
makes sense to keep your instances way, way small.
So we used to run like 20 to 50 instances of MySQLs per machine.
And that was a lot healthier than running big ones.
For a couple of reasons.
One is, if you try to run so many threads within a process, that
itself is a huge overhead for the machine.
And it doesn't do that very efficiently, whereas it does it better
if you run it as smaller instances, I think.
It's more of a feeling, but I don't know if there is proof or
whatever.
But like Go, for example, wouldn't do well.
Go, I think, beyond a certain memory size or beyond a certain
number of goroutines would start to slow down, would not be
as efficient as it was before.
Mainly because the data structures to keep track of those threads
and stuff, they are growing bigger.
But more importantly, on an outage, a smaller number of users
are affected.
If you have 256 shards and 1 shard goes down, it is 1 256th of
the outage, right?
And so the site looks a lot healthier, behaves a lot healthier.
There's less panic if a shard goes down.
So people are, you know, a lot less stressed managing such instances.
Nikolay: Right, I wanted to mention that this discussion was
with Lev Kokotov, PgDog, which is a competitor as well, a new
sharding tool written in Rust.
And we discussed that there is a big limitation when Postgres...
So replication, physical replication has limitation because it's
single threaded process on standby.
If we reach like somewhat like 150, 200, 250 megabytes per second,
depending on core and also number of, not number, structure of
tuples and so on.
We hit 1 single CPU, 100%, 1 process, and it becomes bottleneck
and replica standbys, they start lagging.
It's a big nightmare because you usually by that time, but that's
like at high scale you have multiple replicas and you will float
a lot of read only queries there.
And then you don't want, don't know what to do except as you
described let's remove this feature and slow down development
and this is not not fun at all.
So what I'm trying to do here is trying to move us to discussion
of replication not physical but logical.
I noticed your plans involve heavily logical replication in Postgres.
But we know it has, it's improving every year.
So like when we started the discussion 5, 6 years ago, it was
much worse.
Right now it's much better.
Many things are solved, improved, but many things still are not
solved.
For example, schema changes are not replicated, right?
And sequences, there is work in progress, but if it's committed,
it will be only in Postgres 19, not in 18.
So it means like long wait for many people.
So what are your plans here?
Are you ready to deal with problems like this?
Pure Postgres problems, you know?
Sugu: Yeah, yeah.
If you ask me, I think the Postgres problems are less than what
we faced with MySQL.
Nikolay: I wanted to involve physical as well, because this great
talk by Kukushkin, which describes very bad anomalies when data
loss happens and so on.
Let's talk about this.
Sugu: Yeah, we should talk about both.
I think overall the Postgres design is cleaner, is what I would
say.
Like you can feel that from things.
Like the design somewhat supersedes performance, which I think
in my case is a good trade-off, especially for sharded solutions,
because some of these design decisions affect you only if you
are running at, you know,
Nikolay: if
Sugu: you're pushing it really, really hard, then these design
decisions affect you.
But if your instances are small to medium size you won't even
know and then you benefit from the fact that these designs are
good.
So I actually like the approaches that Postgres has taken with
respect to the WAL as well as logical replication.
And by the way, I think logical replication theoretically can
do better things than what it does now, and we should push those
limits.
But yes, I think the issue about schema not being as part of
logical replication, it feels like that is also a theoretically
solvable problem except that people haven't gotten to it.
I think there are issues about the transactionality of DDLs which
doesn't even exist in MySQL.
So at least in Postgres, it exists in most cases.
There are only a few cases where it is not.
And for such things like a proxy layer, like Multigres or Vitess,
it's a no problem for them because you should say, oh, OK, this
particular construct is not transactionally safe, well, then
we'll even prevent you from doing it transactionally because
we don't want you to get the wrong impression.
We'll let you do it non-transactionally, and we know that it's
non-transactional, and therefore, we can do something about it,
right?
Those abilities don't exist previously, But eventually, if it
becomes transactional, then we can actually include it in a transaction.
Nikolay: Yeah, just for those who are curious, because there
is a concept, all DDL in Postgres is transactional.
Here we talk about things like creating this concurrently because
we had discussion offline about this before recording.
So yeah, creating this concurrently can be an issue, but you
obviously have a solution for it.
That's great.
Sugu: The way I would say it is we have dealt with much worse
at with MySQL.
So this is much better than what was there then.
Sounds good.
Yeah.
Nikolay: Okay, good.
And let's talk about physical replication because I saw you are
going to use it.
So each shard is going to have a standby.
And yeah, so with quorum commit, right?
So like we don't lose data because data is on local disks by
default, as I understand, like ephemeral in cloud.
So we want to be sure that data is written to at least 2 places,
for example, or 3, right?
Configurable, of course.
Here, this interesting talk by Kukushkin, he presented it recently
at an online conference by Microsoft, describing that synchronous
replication in Postgres is not what you think.
Sugu: Correct.
So, right.
Correct.
Nikolay: What are you going to do about this?
Sugu: Well, I was just chatting with someone and essentially
synchronous replication is theoretically impure when it comes
to consensus.
I think it's provable that if you use synchronous replication
then you will hit corner cases that you can't handle.
And the most egregious situation is that it can lead to some
level of definitely split brain, but in some cases it can even
lead to downstream issues.
Because it's a leaky abstraction, it's a leaky implementation,
there are situations where you can see a transaction and think
that it is committed.
And later, the system may fail.
And in the recovery, may choose not to propagate that transaction,
or may not be able to, and it's going to discard that transaction
and move forward.
Nikolay: But this is the same as with asynchronous replication,
it's the same, we're just losing some data, right?
Sugu: Exactly, yeah, it is the same as asynchronous replication,
Yes.
Nikolay: It's not split-brain, it's just data loss.
Sugu: It's data loss, correct.
It's data loss, but for example, if you are running like a logical
replication off of 1 of those, then that logical replication
may actually propagate it into an external system and now you
have corrupted downstream systems that don't match the source.
So those risks exist and at Vitascale people see this all the
time, for example, and they have to build defenses against this
and it's very, very painful.
It's not impossible, but it's very hard to reason about failures
when a system is behaving like this.
So that is the problem with synchronous replication.
And this is the reason why I feel like it may be worth patching
Postgres.
Because there is no existing primitive in Postgres on which you
can build a clean consensus system.
I feel like that primitive should be in Postgres.
Nikolay: I now remember from Kukushkin's talk, there is another
case when a primary transaction looks like not committed because
we wait a replica, but the replica somehow is like lost connection
or something.
And then we suddenly, and client thinks it's not committed because
commit was not returned.
But then it suddenly looks committed.
It's like not data loss, it's data and loss
Sugu: somehow.
Boom.
Nikolay: Like suddenly, and this is not all right as well.
And when you think about consensus, I think you are very good
describing these things like concept and distributed systems.
It feels like if you have 2 places to write, definitely there
will be corner cases where something will go off if you don't
use two-phase commit, right?
Sugu: Correct.
Nikolay: And here we have this.
But when you say you're going to bring something with consensus,
it immediately triggers my memory how difficult it is and how
many attempts it was made to bring pure HA to Postgres, just
to have auto-failover.
All of them failed.
All of them.
And they say, let's be outside of Postgres.
So here maybe it will be similar complexity to bring these 2
inside Postgres.
Is it possible to build it outside this thing?
Sugu: It is not possible to build it outside.
Because if it was, that is what I would have proposed.
The reason is because building it outside is like putting band-aid
over the problem.
It will not solve the core problem.
The core problem is you've committed data in 1 place, and if
that data can be lost, and there is a gap when the data can be
read by someone, causes is the root cause of the problem.
That is unsolvable.
Even if you later, later raft may choose to honor that transaction
or not.
And that becomes ambiguous, but we don't want ambiguity, right?
Nikolay: What if we created something extension to commit, like
make extendable to talk to some external stuff to understand
that can be finalized or something?
I don't know, consensus.
Correct. Sugu: Correct.
So if you, essentially, if you reason through about this, your
answer will become a two-phase system.
Nikolay: Yeah.
Sugu: Without a two-phase system.
Nikolay: Which scares me.
But as I told you, a two-phase commit in the Postgres OTP world
is considered really, really slow and the rule is let's just
avoid it.
I see your enthusiasm and I think, I couldn't find good benchmarks,
0, published.
Sugu: This is not two-phase commit, by the way.
This is two-phase synchronization.
Nikolay: I understand.
It's not, in two-phase commit, it's like more communication happens.
I understand this.
Sugu: So two-phase synchronization, the network overhead is exactly
the same as full sync, because the transaction completes on the
first sync.
Later it sends an acknowledgement saying that yes, I'm happy,
you can commit it, but the transaction completes on the first
sync, so it will be no worse than full sync.
Nikolay: Yeah, compared to current situation when primary commit
happens, but there is a lock which is being held until- Correct,
Sugu: it is the same cost.
Nikolay: Yeah, it is the same cost.
We wait until standby, And for user it looks like when lock is
released, it thinks, okay, commit happens.
But the problem with this design, if, for example, standby restarts,
lock is automatically released and commit is here and it's unexpected.
This is data on loss, right?
So you are saying we can redesign this, network cost will be
the same, but it will be pure.
Yeah, that's great, I like this.
I'm just thinking, will it be acceptable?
Because bringing autofailover is not acceptable.
There was another attempt last year from someone and with great
enthusiasm, let's bring autofailover inside Postgres.
Actually, maybe you know this guy, it was Konstantin Osipov who
built a Tarantool database system.
It's like memory.
He was X MySQL in performance after Zaitsev.
Zaitsev was X MySQL then Osipov was MySQL.
And so Konstantin came to Postgres saying let's build this.
Great enthusiasm, but it's extremely hard to convince such big
thing to be in core.
So if you say it's not big thing, this already...
Sugu: So I can, it's, I'll probably have to explain it in a bigger
blog, but essentially, now that I've studied the problem well
enough, the reason why it's hard to implement consensus in Postgres
with the write-ahead log is because they are trying to make Raft work with
the write-ahead log.
And there are limitations about how the Raft, how commits work
within Postgres that mismatch with how Raft wants commits to
be processed.
And that mismatch, so far, I have not found a way to work around
that.
But, the variation of Raft can be made to work.
Nikolay: Interesting.
Sugu: The way the, I don't know if you know about my blog series
that I wrote when I was at PlanetScale.
It's an 8 part blog series about generalized consensus.
People think that Raft is the only way to do consensus, But it
is 1 of a thousand ways to do consensus.
So that blog series explains the rules you must follow if you
have to build a consensus system.
And if you follow those rules, you will get all the properties
that are required by a consensus system.
So This 1 that I have, the design that I have in mind, follows
those rules, and I am able to prove to myself that it will work,
but it's not Raft.
It's going to be similar to Raft.
I think we can make Raft also work, but that may require changes
to the write-ahead log, which I don't want to do.
So this system I want to implement without changes to the write-ahead log
as possibly a plugin.
Nikolay: Yeah.
Well, now I understand why you could, like another reason you
cannot take Patroni not only because it's Python versus Postgres
But also because you need another version of consensus algorithm
Sugu: Correct, correct.
Nikolay: And among those hundred Thousand millions of ways.
Sugu: By the way Patroni can take this and use it because it's
very close to how FullSync works.
Nikolay: Good.
Okay.
Michael: I was just thinking, watching Alexander Kukushkin's
talk, he said a couple of things that were interesting.
1 is that he was surprised that this hasn't happened upstream.
So you definitely have an ally in Kukushkin in terms of trying
to get this upstreamed, but also that he thinks every cloud provider
has had to patch Postgres to, in order to offer their own high
availability products with Postgres, each 1 has had to patch
it.
And they are having to, you mentioned earlier today how painful
it is to maintain even a small patch on something.
Nikolay: I don't think it's every, I think it's Microsoft for
sure, knowing where Kukushkin works at.
Yeah.
But maybe more, not every.
Yeah,
Michael: all I mean is that there are a growing number of committers
working for hyperscale and hosting providers.
So I suspect you might have more more optimism for consensus
or at least a few allies in terms of getting something committed
upstream so I personally think there might be growing chance
of this happening even if it hasn't in the past for some reason.
Sugu: Yeah, I feel like also being new to the Postgres community,
I am feeling a little you know, shy about proposing this upfront.
So what I am thinking of doing is at least show it working, show
it working at scale, have people gain confidence that this is
actually efficient and performant and safe.
So I also plan to, I don't know if you've heard of FlexPaxos,
which is actually, in my opinion, a better way to handle durability,
because today's cloud environments are a lot more complex, and
a simple majority-based quorum is actually very hard to configure
if your needs are different, which actually FlexPaxos does handle.
It's actually something I'm a co-inventor of, of some sort.
And this blog post...
Nikolay: I only heard the name.
That's it.
So I like...
Yeah. Can you explain a little bit?
Not super difficult.
Sugu: Oh sure.
Yeah.
So the...
Actually, let me explain what is the reason why.
So FlexPaxos was published a few years ago, about 7 years ago
or so.
And you'll see my name mentioned there, which I feel very proud
of.
And the block series that I wrote is actually a refinement of
FlexPaxos.
And that actually explains better why these things are important.
The reason why it's important is because people think of consensus
as either a bunch of nodes agreeing on a value, right?
That's what you commonly hear.
Or you think of like reaching majority, reaching quorum is important.
But the true reason for consensus is just durability.
When you ask for a commit and the system says, yes, I have it,
you don't want the system to lose it.
So instead of defining quorum and all those things, define the
problem as it is and solve it the way it was asked for is, how
do you solve the problem of durability in a transactional system?
And the simple answer to that is, make sure your data is elsewhere.
Michael: Yeah, I love how simple you make it.
Sugu: Yeah, if you make sure your data is elsewhere, if there
is a failure, your challenge is to find out where the data is
and continue from where it went.
And that is all that consensus is about.
And then all you have to do is have rules to make sure that these
properties are preserved.
And Raft is only just 1 way to do this.
So if you look at this problem, if you approach this problem
this way, you could ask for something like, I just want my data
to go across availability zones.
As long as it's in a different availability zone, I'm happy.
Or you can say, I want the data to be across regions, or I want
at least 2 other nodes to have it, right?
So that's your Durability requirement.
But you could say, I want 2 other nodes to have it, but I want
to run 7 nodes in the system, or 20 nodes.
It sounds outrageous but it is actually very practical.
In YouTube we had 70 replicas but only 1 1 node the data have
to be in 1 other node for it to be durable and we were able to
run this at scale.
The trade-off is that when you do a failover you have a wild
goose chase looking for the Transaction that went elsewhere but
you find it and then you continue.
And so that is basically the principle of this consensus system.
And that's what I want to bring in Multigres.
While making sure that the people that want simpler majority
based quorums to also work using the same primitives.
Michael: Just quickly to clarify, when you say the wild goose
chase, is that because it was 1 of 70, but different transactions
could have gone to different of the 70 or it's always the same
1, but you have to know which 1 that is
Sugu: No, it could be anyone the way we we ran it the way we
ran it It is 1 it could not be at any given point of time There's
only 1 Primary which means that there is only 1 Transaction that
you have to chase down.
Michael: The latest 1.
Sugu: The latest 1, yes.
Michael: Yeah, makes sense.
Sugu: Yeah.
There was a time when we found that Transaction in a different
country.
So we had to bring it back home and then continue.
It was once it happened in whatever the 10 years that we ran.
Nikolay: It's interesting that talking about sharding, we need
to discuss these things, which are not sharding per se, right?
So it's about HA inside each shard, right?
Sugu: It's actually like what I would call healthy database principles,
which is, I think, somewhat more important than sharding.
Nikolay: Yeah.
Yeah.
Michael: It is true that it is to do with it being a distributed
system, right?
And that is because it's sharded, no?
Sugu: I think they are orthogonal.
Okay.
Yeah, I think sharding, like you can do sharding on anything,
Right?
Like you can do sharding on RDS.
Somebody asked me, like, what about Neon?
I said, you can do sharding on Neon too.
It's like you put a proxy in front and then it does the sharding.
But the problem with sharding is it is not just a proxy.
That's what people think of it when they first think of the problem
because they haven't looked ahead.
Once you have sharded, you have to evolve.
You start with 4 shards, then you have to go to 8 shards.
And the evolution is not linear this way.
Actually, it's an exponential growth because 4, 8, 16.
But at some point of time, it changes because your sharding scheme
itself will not scale.
Like if you, for example, are in a multi-tenant workload and
you say shard by tenant, at some point of time, a single tenant
is going to be so big that they won't fit in an instance.
And that we have seen.
And at that time, we had to change the sharding scheme.
So how do you change the sharding scheme?
Slack had to go through this, where they were a tenant-based
sharding scheme, and a single tenant just became too big.
They couldn't even fit 1 tenant in 1 shard.
So they had to change their sharding scheme to be user-based.
They actually talk about it in 1 of their presentations.
And Vitess has the tools to do these changes without actually
you incurring any kind of downtime, which again, Multigres
will have.
I keep talking about Vitess, but these are all things that Multigres
will have, which means that you are future-proofed when it comes
to.
And these are extremely difficult problems to solve.
Because when you're talking about changing the sharding scheme,
you are basically looking at a full crisscross replication of
data.
And across data centers.
Nikolay: Yeah, and also, like I know, it has version 3, right?
It was when you changed, basically created a new planner, right,
to deal with arbitrary query and understand how to route it properly
and where to execute it, which is it a single shard or it's global
or it's like different shards and so on.
Like what's, are you going to do the same with Postgres?
I think yes.
Right.
Sugu: So that's the part that I'm still on the fence.
By the way, the v3 now has become Gen 4.
It's actually much better than what it was when I built it.
The problem with v3 is that it is still not a full query.
It doesn't support the full query set yet.
It controls supports like 90% of it, I would say, but not everything.
On the temptation side, there's the Postgres engine that supports
everything.
So I'm still debating how do we bring the 2 together?
If it was possible to do a simple git merge, I would do it.
But obviously this 1 is in C, this was in Go.
And the part that I'm trying to figure out is how much of the
sharding bias exists in the current engine in Vitess.
If we brought the Postgres engine as is, without that sharding
bias, would this engine work well for a sharded system?
Nikolay: So this looks like CytoSort, if you bring up the whole
Postgres.
There's a library, libpg_query, by Lukas Fittl, which basically
takes the parser part of Postgres and brings it to...
And there is a Go version of it as well.
Sugu: Oh, libpg_query you said?
Nikolay: Yeah, yeah, yeah, I will send it to you.
So many, many, many, many systems use it when we need to parse.
Yeah, 1 day I told it to SPQR guys about this, and eventually
I think they ended up using it.
I think Lev Kupotov uses it, Pijadok also uses it, the Rust version.
Sugu: Is it like 100% Postgres compatible?
Nikolay: Well, it's based on Postgres source code.
So parser is truly broad, but it's not whole Postgres.
So maybe you should consider this.
Right?
If you're thinking about parsing, I mean, queries and so on,
but I'm very curious, I also noticed you mentioned routing, like
read-only queries routed to replicas automatically.
And this concerns me a lot because many Postgres developers,
I mean, who use it, users, they use PL/pgSQL functions, all PL..
Python functions, anything, which are writing data.
And the standard way to call function is select, select function
name.
So understanding that this function is actually writing data
is not trivial.
Right, and we know in pgpool, which I, all my life I just avoid.
I touched it a few times, decided not to use at all because it
tries to do a lot of stuff at once and always considered like,
no, I'm not going to use this tool.
So pgpool solves it like saying okay like let's build a list
of functions which are actually writing or something like this.
So it's like patch approach you know workaround approach.
So this is going to be a huge challenge, I think, if you...
Sugu: Yeah, yeah.
Nikolay: For automatic routing, it's a huge challenge.
Sugu: Yeah, I think this is the reason why I think it is important
to have the full Postgres functional engine in Multigres, because
then these things will work as intended is my hope.
What we will have to do is add our own sharded understanding
to these functions and figure out what does it mean to call this
function, right?
If this function is going to call out to a different shard, then
that interpretation has to happen at the higher level.
But if that function is going to be accessing something within
shard, then push the whole thing down and just let the push the
whole SELECT along with the function down and let the individual
Postgres instance do it.
Nikolay: Yeah, but how to understand?
Function can contain another function and so on.
It can be so complex in some cases.
It's also funny that there is still there is actually Google
Cloud, CloudSQL supports it, kind of language, it's not language,
called PL/Proxy, which is sharding.
For those who have workload only in functions, this can route
to proper shard.
It was created at Skype.
It still exists, but not super popular these days.
But there is a big requirement to write everything in functions.
In your case, if you continue, like, I would expect in some case
you would say, okay, don't use functions.
But I'm afraid it's not possible.
Like, I love functions.
Actually, Supabase loves functions because they use Postgres,
right? Postgres, like, it provokes you to use functions.
Sugu: Oh, really?
Oh, yeah, yeah.
Actually, I saw that.
Yeah.
So in Vitess, I feel like this was a mistake that we made, which
is if we felt that anything, any functionality that you used
didn't make sense.
Like if I were you, I wouldn't do this, right?
Because it's not, it won't scale.
It's a bad idea.
You know, it's like those we didn't support.
We didn't want to support.
You said, no, we will never do this for you because we'll not
give you a rope long enough to hang yourself.
Basically, that was our philosophy.
But in Multigres, we want to move away from that, which means
that if you want to call a function that writes, have at it.
Nikolay: Just put a comment, it's going to write something.
Sugu: Yeah.
If you want a function that calls a function that writes have
at it.
If we cannot, like the worst case scenario for us is we don't
know how to optimize this.
And what we will do is we'll execute the whole thing on the VT
gate side.
Nikolay: There's another, I remember there is interesting solution
in, I think in AWS RDS proxy, which when they, as I know it,
maybe I'm wrong, when they needed to create a global, it's called
I think Aurora Global Database maybe or something like this.
So there is a secondary cluster living in a different region
and it's purely read-only, but it accepts writes.
And when write comes, this proxy routes it to original primary,
waits until this write is propagated back to replica and responds.
Sugu: Oh, wow.
Yeah.
I don't think that feature, I don't know how even that feature
can be supported.
Nikolay: No, no, it's just some exotic, interesting solution
I just wanted to share.
Maybe, you know, if we, for example, if you originally route
a write to a replica, then somehow in Postgres you understand,
oh, it's actually a write.
Sugu: Okay, Yeah, so maybe 100% is theoretically impossible to
support.
Nikolay: Yes, it's okay.
It's super exotic.
Okay.
Sugu: But I think if people are doing things like that, it means
that they are trying to solve a problem that doesn't have a good
existing solution.
Nikolay: Exactly.
Sugu: So if we can find a good existing solution, I think they'll
be very happy to adopt that instead of whatever they were trying
to do.
Nikolay: Well, this is just multi-region setup.
I saw not 1 CTO which wanted it for like dealing with Postgres
like say we are still single region we need to be to be present
in multiple regions in case if 1 AWS region is down right it's
also okay yeah so availability and business characteristics so
yeah anyway okay yeah it's it's exotic but but interesting still
yeah So
Michael: you've got a lot of work ahead of you, Sugu.
Sugu: I feel like we barely covered like 1 of so many topics.
Nikolay: Let's touch something else.
Maybe it's a very long episode, but it's worth it, I think.
It's super interesting.
What else?
Sugu: I think the other interesting 1 would be 2PC and isolation.
Nikolay: Isolation from what?
Sugu: Like the 1 issue with the sharded solution is that, again,
this is a philosophy for the longest time in Vitess we didn't
allow 2PC.
You said you shard it in such a way that you do not have distributed
transactions.
And many people lived with that.
And some people actually did
Nikolay: not adopt Vitess.
Let me interrupt you here, because this is like the most, the
best feature I liked about Vitess, it's this materialized feature
when data is brought.
Sugu: Oh yeah, materialized is another topic.
That's actually a better topic than 2PC.
Nikolay: Well, yeah, because this is your strength, right?
So this is like, I love this idea, basically distributed materialized
view, which is incrementally updated.
Sugu: Yes, yes, yes.
Nikolay: That's great.
We need it in Postgres ecosystem.
Just maybe as a separate project, even, you know, like we lack
it everywhere.
So yeah, this is how you avoid distributed transactions basically,
right?
Sugu: No, this is 1 way to avoid it.
1 way, yeah.
Like there are 2 use cases where materialized views are super
awesome.
You know a table that has multiple foreign keys, that has foreign
keys to 2 different tables is the classic use case, where the
example I gave was a user that's producing music and listeners
that are listening to it, which means that the row where I listen
to this music has 2 foreign keys, 1 to the creator and 1 to the
listener.
And where should this row live?
Should this row live with the creator or should this row live
with the listener is a classic problem.
And there is no perfect solution for it.
It depends on your traffic pattern.
But what if the traffic pattern is 1 way in 1 case and another
way in another case?
There is no perfect solution.
So this is where in Multigres what you could do is you say okay
in most cases this row should live with the creator let's assume
that right so then you say this row lives with the creator and
we shard it this way, which means that if you join the creator
table with this event row, it'll be all local joins.
But if you join the listeners table with this event row, it's
a huge cross shard while this chases.
So in this case, you can say materialize this table using a different
foreign key, which is the listeners foreign key into the same
sharded database as a different table name.
And now you can do a local join with the listener and this event
table.
And this materialized view is near real-time, basically the time
it takes to read the WAL and apply it.
And this can go on forever.
And this is actually also the secret behind resharding, changing
the sharding key.
This is essentially a table that has real-time present with 2
sharding keys.
If you say, oh, at some point of time, this is more authoritative,
All you have to do is swap this out.
Make 1 the source, the other is a target.
You've changed your sharding key.
Actually, the change sharding key works exactly like this for
a table.
Nikolay: Distributed denormalization technique.
This is what
Michael: it is.
Sugu: Yeah, yeah, yeah, exactly.
And the other use case is when you reshard, you leave behind
smaller tables, reference tables, we call them.
And they have to live in a different database because they are
too small and even if you shard them, they won't shard well.
Like if you have a billion rows in 1 table and a thousand rows
in a smaller table, you don't want to shard your thousand row
table.
And there's no benefit to sharding that either.
So it's better that that table lives in a separate database.
But if you want to join between these 2, how do you do it, right?
The only way you join is join at the application level, read
1 and then read the other.
And so at high QPS, it's not efficient.
So what we can do is actually materialize this table on all the
shards as reference, and then all joins become local.
Nikolay: Yeah.
And you definitely need logical replication for all of this.
So this is where we started, like challenges with logical replication.
Sugu: Yeah, yeah.
You do have the, so the reason why 2PC is still important, because
there are trade-offs to this solution, which is, there's a lag.
So it takes time for the things to go through the WAL and come
to the other side.
Whereas 2PC is essentially, basically the transaction system
itself trying to complete a transaction, which means that it
will handle cases where there are race conditions, right?
If somebody else tries to change that row elsewhere while this
row is being changed, 2PC will block that from happening, whereas
in the other case, you cannot do that.
Nikolay: Yeah, if it's just user views of some video like on
YouTube, we can say, okay, there will be some lag, probably some
small mistake, it's fine.
But if it's financial data, it should be 2PC, but latency of
write will be high, throughput will be low, right?
This is...
Sugu: I actually want to...
I read the design of...
Which is again, by the way, very elegant API, and I assume...
I can see the implementation on the API and I Don't think we
will see performance problems with 2PC.
Nikolay: We need to benchmark it
Sugu: We will be we will benchmark it, but I will be very surprised.
I think there are some isolation issues that we may not have
time to go through today because it's a long topic.
Like the way 2PC is currently supported in Postgres, I think
it'll perform really well.
Nikolay: Isolation issues when we sit in read committed and use
2PC.
You mean this, right?
Not in repeatable read.
In default.
Sugu: Yeah, read committed I think will be, there will be some
tradeoffs on read committed, but not the kind that will affect
most applications.
MVCC will be the bigger challenge.
But from what I hear is most people don't use, like the most
common use case is read committed.
Nikolay: Of course, as default.
Yeah, it's faster.
Sugu: It's a default.
Yeah.
So people won't even, yeah, I don't, I think this.
Nikolay: They're already on some, they're already in bad state.
It won't be worse.
Sugu: It won't be worse.
Yes.
Nikolay: Yes.
Yeah. So to PC, of course depends on the distance between nodes, right?
A lot, like if they are far, we need to talk like client is somewhere,
2 nodes are somewhere, and if it's different availability zones,
it depends, right?
So this distance is a big contributor to latency, right?
Network.
Because there are 4 communication messages that are needed.
So.
Sugu: Correct, correct.
Actually, you can, I have actually the mathematics for it?
But you're probably right.
It's about double the number of round trips.
Nikolay: Yeah, if we put everything in 1 AZ, client and both
primaries, we are fine.
But in reality, they will be in different places.
And if it's different regions, it's nightmare, of course.
But at
Sugu: least it's- Yeah, the 2PC is not done by the client, by
the way.
The 2PC would be done by the VTgate, which would be- It should
have the nodes nearby.
Nikolay: Ah, should have, In 1 availability
Sugu: zone?
Unless you did some crazy configuration, they should be pretty
close to each other.
Nikolay: Pretty close means still different availability zones
in general case, right?
Sugu: No, no.
The availability zone is only for durability.
Nikolay: Okay.
Sugu: For replica level.
But a 2PC, you're coordinating between 2 primaries, which may
actually be on the same machine for all you care.
It's not...
Well,
Nikolay: Imagine the real practical case.
We have sharded schema, we have...
Every shard has primary and a couple of standbys, right?
Sugu: Correct.
Correct.
Nikolay: So are you saying that we need to keep primaries all
in the same availability zone?
Sugu: That's usually how things are.
Nikolay: Ah, interesting.
I didn't know about this.
By the way, I wanted to rattle a little bit about PlanetScale benchmarks
they published last week.
They compared to everyone.
I wanted just, it's not like, I'm sorry, I will take a little
bit of time.
They compared to everyone, and they just published like PlanetScale
versus something.
And this very topic, they on charts, we have PlanetScale in single
AZ, everything client and server in the same AZ.
And line, which is like normal case, client is different AZ.
And line with same AZ is active, line is normal, not active.
And others like Neon, Supabase, everyone, It's different.
And of course, PlanetScale looks really well, because by default,
they presented numbers for the same availability zone.
And below the chart, everything is explained, but who reads it,
right? So people just see the graphs.
And you can unselect, select proper PlanetScale numbers and
see that they are similar.
But by default, the same as that number is chosen.
And this is like benchmarking, you know, like.
Sugu: Well, I think if you look at the architecture, like, even
fair comparison, PlanetScale should come out ahead, like, like
the performance of a local disk, of course, should.
Nikolay: But this was SELECT 1 disks.
Sugu: Disks, SELECT 1 is not a benchmark.
Nikolay: Well, it was part of benchmark.
It's just checking query path, but it fully depends on where
client and server are located.
So what's the point showing better numbers just putting client
closer?
I don't like that part of that benchmark.
Sugu: Okay, yeah.
I saw the publications, But I didn't go into the details because
I thought, well, it has to be faster because it's on local disks.
So why?
Nikolay: Yeah, for data which is not fully in cache, of course.
Yeah, local disks are amazing versus ABS volumes.
Sugu: You're right.
Yeah, if the data is in cache, then there is, yeah, then all
performance, the performance of everything would be the same.
Nikolay: Yeah, well, I wanted to share this.
I was annoyed about this, but I fully support the idea of local
disks. It's great.
I think we need to use them more in more systems.
Sugu: I think, I wouldn't be surprised if you reached out to
PlanetScale.
They may be willing to, like if you want to run your benchmarks,
they may be willing to give you the...
Nikolay: Yeah, there is source code published and in general
benchmarks look great, the idea is great.
And actually, with local disks, the only concern is usually the
limit, hard limit.
We cannot have more space.
But if you have sharded solution, there is no such thing.
Sugu: Correct.
But speaking about the hard limit, today's SSDs, you can buy
100 plus terabytes size SSD, single SSD, and you can probably
stack them up on
Nikolay: 1 next to the other.
Yeah, yeah.
But in cloud there are limitations, in cloud there are limitations,
for instance.
Sugu: Okay, yeah.
I saw AWS SSD over 100 terabytes.
Nikolay: In Google Cloud, 72 terabytes is hard limit for Z3 metal.
And I didn't see more.
So 72 terabytes, it's a lot, but sometimes it's already notable.
Sugu: Yeah, at that limit, your storage is not the limit.
You will not be able to run a database of that size on a single
machine.
Why not? You will max out your CPU.
Nikolay: We have cases, CPU.
Well, again, the problem will be replication.
If we talk about single node, we can
Sugu: have- Or replication.
Nikolay: 360 cores in AWS, almost 1,000 cores already for Xeon
scalable generation 5 or something.
So hundreds of cores.
The problem is Postgres design.
If replication, physical replication was multi-threaded, we could
scale more.
Sugu: By the way, replication is not the only problem.
Backup recovery.
If your machine goes down, you're down for hours.
Recovery is something of that size.
Nikolay: Yeah.
Not many hours.
Someone in my team recently saw 17 or 19 terabytes per hour for
recovery with pgBackRest or WAL-G.
In AWS.
I was like, my jaw dropped.
On our podcast I usually say 1 terabyte.
Sugu: Can you repeat that?
17 or 19
Nikolay: terabytes per hour?
17 terabytes per hour.
With local disks.
So this is important.
With EBS, it's impossible.
Sugu: It's good to know.
Nikolay: Yeah. With Michael, I was always saying 1 terabyte is like what you
should achieve.
If it's below, it's bad.
Now I'm thinking 1 terabyte is already...
Yeah, yeah.
So with EBS volumes, we managed to achieve, I think, 7 terabytes
per hour to restore with WAL-G.
And that's great.
But there's
Sugu: a danger there.
You could become a noisy neighbor.
So we actually built throttling in our restore just to prevent
noisy neighbors.
Nikolay: With local disks, you lose ability to use BS snapshots,
cloud disk snapshots.
Sugu: Correct, correct, yeah.
Nikolay: That's what you lose, unfortunately.
And they're great, and people enjoy them more and more.
Yeah, so I agree, and as I remember, for 17 terabytes it was
128 threads of 4g or pgBackRest I don't remember local disks I
need to update
Sugu: my technology is changing too fast.
Nikolay: Exactly, yeah.
And hundreds of cores, terabytes of RAM already, right?
Like,
Michael: yeah.
Yeah, yeah. But it does go straight to your point of the smaller they are,
the faster you can recover still.
Sugu: You don't hit some of these limits.
These systems were not designed with these types of limits in
mind.
Some weird data structure, suddenly the limit of this is only
100 items.
And you hit those limits and then you're stuck.
Recently, Metronome had an issue.
They had that, the routed.
MultiXact.
The MultiXact thing, which nobody has ever run before, but
they hit that problem.
Nikolay: Yeah, we saw many problems also when you're on the edge.
And it pushes forward Postgres actually sometimes, but if you
want to be on the safe side, but I really like the, like, it's
kind of resilience characteristics when, even if it's down, it's
only a small part of your system is down.
That's great.
Sugu: Correct, Nikolay: yeah.
That's mature architecture already.
Sugu: That actually makes it easier to achieve 5 nines uptime.
Because that's the way you calculate.
Like if only 1 node is down, you divide it by the number of users
Nikolay: being affected.
Budget.
Sugu: Downtime budget.
Yeah, budget.
Yeah.
Nikolay: That's good.
Cool.
I think it's maybe 1 of the longest episodes we had.
Enjoyed it.
Oh my God.
I enjoyed it.
I hope we will continue a discussion of issues with logical,
for example, and so on, and maybe if things will be improved
and so on.
Looking forward to test POC once you have it.
Thank you so much
for coming. Sugu: I am so excited.
Nikolay: Yeah, thank you.
Michael: Yeah, is there any last things you wanted to add or
anything you wanted help from people on?
Sugu: I would say it feels like nothing is happening on the repository
except me pushing, you know, a few pushes, a few things, changes,
but a huge amount of work is happening in the background.
Like some of these design work about consensus are all like almost
ready to go and there's also hiring going on there are people
coming on board very soon so you will see this snowball It's
a very tiny snowball right now, but it's going to get very big
as momentum builds up.
So pretty excited about that.
We may still have 1 or 2 spots open to add to the team, but it's
filling up fast.
So If any of you are very familiar, this is a very high bar to
contribute to a multi-base.
You have to understand consensus, you have to understand query
processing.
But if there are people who want to contribute, we are still
looking for maybe 1 or 2 people and also on the orchestration
side and the Kubernetes side of things.
Yeah, so that's...
Nikolay: Do you mind a small joke in the end?
Just not to finish on...
Sugu: I do not mind at all.
Let's hear it.
Nikolay: Yeah, so I know what you're doing.
You're writing a lot of markdown right now and then you will
feed it to AI.
Sugu: I wish!
Oh my god.
I almost hope that day never comes but It is so fun working on
this project, creating it.
Why do I want to give it to an AI to do it, you know?
Nikolay: Okay.
Good.
Thank you.
I enjoyed it a lot.
Michael: Yeah. Yeah.
Thank you so much for joining us.
It's great to have you as part of the Postgres community now
and I'm excited to see what you get up to.
Sugu: And me too.
Michael: Wonderful, thanks so much.