Michael: Hello and welcome to Postgres.FM, a weekly show about
all things PostgreSQL.
I'm Michael, founder of pgMustard, I'm joined as usual by
Nikolay, founder of Postgres.AI.
Hey Nikolay.
Nikolay: Hi Michael.
Michael: And today we are delighted to be joined by Alexander Kukushkin,
a Postgres contributor currently working at Microsoft and most
famously, maintainer of Patroni.
We had a listener request to discuss Patroni, so we're delighted
you agreed to join us for an episode.
Alexander.
Alexander: Yeah, hello Michael, hello Nikolay.
Thank you for inviting me.
I'm really excited to talk about my favorite project.
Michael: Us too.
Perhaps as a starting point, could you give us an introduction?
Like for most people, I think will have heard of Patroni and
know what it is, but for anybody that doesn't, could you give
an introduction what it is and why it's important?
Alexander: Yeah, so Patroni, like in the simple words, it's
a failover manager for Postgres.
It solves the problem of availability of a primary.
In Postgres, we don't use some words that are non-inclusive,
like master.
That's why we call it primary, and Patroni actually recently
get rid of this non-inclusive words completely.
And the way how Patroni does, it makes sure that we are running
just a single primary at a time, and at the same time, Patroni
helps you to manage as many read-only replicas as you like to
have, and keeping those replicas ready to become primary in case
the primary has failed.
At the same time, Patroni helps to automate usual DBA tasks like
switchover, configuration management, stuff like that.
Nikolay: Node provisioning also, right?
Alexander: Node provisioning, not really.
Node provisioning is a task for DBA.
DBA has to start Patroni, and Patroni will take care of bootstrapping
this node.
In case it's a totally new cluster, Patroni will start as a primary.
In case the node joins an existing cluster, the replica node
will take a pg_basebackup by default from the running primary
and start the replica.
And the most interesting part, let's say we bring node back which
was previously running as a primary, And Patroni does everything
to convert this failed primary as a new standby to join the cluster
and be prepared for next unforeseen event.
Nikolay: At least you agree that it does part of node provisioning,
because otherwise, we wouldn't have situations when old data
directory, old PGDATA was copied and new 1 is created and we
are suddenly out of disk space.
And if you don't expect Patroni to participate in node provisioning,
then you think, what's happening?
Why am I out of disk space?
Right?
It happens sometimes.
Alexander: It used to happen, I think, with bootstrap mode.
Like, when Patroni, like, I don't remember up until which version,
but Patroni, when it tries to create a new cluster, usually by
using initDB, but in some cases you can configure Patroni to
create a cluster from existing backup, like from base backup.
And if something goes wrong, Patroni does not remove this data
directory, but renames it.
And it used to apply current timestamp to the file name.
And therefore, after the first failure, it gives up, waits a
little bit and does the next attempt.
Nikolay: To directory, right?
Alexander: Yeah, it creates, it uses yet another base backup,
creates a new data directory, fails, and renames.
Now it is not working like this, it just renames PGDATA to PGDATA-old,
something like this, and that's why you will not have an infinite
number of directories.
And having just 1 is enough to investigate the failure.
Nikolay: But maximum we end up like if we if our data directory
we expected to fill 70% of the disk we still might have out of
disk space.
Alexander: Yeah that's unfortunate but the other option really
like you just drop it but at the same time like all the evidences
of what failed, why it failed are also gone.
You have nothing to investigate.
Nikolay: Okay.
To me it still sounds like Patroni participates in node provisioning.
Yes, it doesn't bring you resources like disk and virtual machine
and so on, but it brings data, like the most important part of
Postgres node provisioning, right?
Okay, I just wanted to be right a little bit.
Okay.
Alexander: Okay.
Nikolay: It's a joke.
Okay.
Michael: I think diving deep quickly is great.
It'd be good to discuss complex topics but I think something
simple would also be good.
I would love to hear a little bit almost about the history of
Patroni.
Like the early days, what were you doing before Patroni to solve
this kind of issue and why was it built?
What problems were there with the existing setups?
Alexander: To be honest, while working for my previous company
We didn't have any automatic failover solution in place.
What we relied on was just a good monitoring system that sent
you a message or some on-call engineer just calls you in the
night, the database failed.
There were a lot of false positives, unfortunately, but it still
felt more reliable than using solutions like replication manager,
repmgr.
Nikolay: Yeah, I remember this very well.
Like people constantly saying we don't need autofailover, it's
evil because it can switch over suddenly, failover suddenly,
and it's a mess.
Let's rely on manual actions.
I remember this time very well.
Alexander: Yeah, to our excuse, an amount of databases, like
database clusters that we run wasn't so high, like I think a
few dozens, and it was running on-prem, didn't fail so often,
and therefore it was manageable.
A bit later, we started moving to the cloud, and suddenly, not
suddenly, but luckily for us, we found a project named Governor,
which basically brought an idea of how to implement autofailover
in a very nice manner, without having so much false positives
and without risks of running to a split-brain.
Nikolay: Was it abandoned project already?
Alexander: No, no, so it was not really abandoned, but it wasn't
also very active.
So we started applying it, found some problems, reported problems
to the maintainer of the Governor, got no reaction unfortunately,
started fixing those problems on our own, and at some moment
a number of fixes and some new nice features accumulated and
we decided just to fork it and give a new name to the project.
So this is how Patroni
Nikolay: was born.
Georgian name, right?
Alexander: Right.
Nikolay: What does
Michael: it mean?
Governor in Georgian.
Nikolay: Oh, governor.
I think so.
Michael: Yeah,
Alexander: almost, almost.
Almost.
Very close, but I'm not a good person to explain like or to translate
from Georgian because I don't
Nikolay: Name
Alexander: I know yet another word in Georgian and it's a spillo.
Yeah, it translates from Georgian as elephant.
Nikolay: And the name chose I guess Valentin Gogitseshvili, right?
Alexander: Yes, he was, no, at that time he wasn't my boss anymore
but we still worked close together and I really appreciate his
creativity in inventing good names for projects.
Michael: Yeah, great names.
And is this a good time to bring up Spilo?
Like, what is Spilo and how is that relevant?
Alexander: Spilo, as I said, it translates from Georgian elephant.
When we started playing with Governor, we were already targeting
to deploy everything in the cloud.
We had no other choice but to build a Docker image and provision
Postgres in a container.
And we called this Docker image Spilo basically we packaged for
Governor Postgres if you Postgres extensions and I think it was
WAL-G back then as a backup and point-and-tumble recovery solution.
Michael: And it still exists to this day as Spilo, but now with
Patroni?
Alexander: Yeah, of course.
Now there is a Patroni inside, and now Spilo includes plenty
of Postgres major versions, which may be an anti-pattern, but
it allows you to run major upgrades, like in-place major upgrades.
It also includes WAL-G nowadays, as a modern replacement of WAL-G.
And it's used not only by...
Operator, right?
Not really part of Operator.
Spilo is a product on its own.
I know that some people run Postgres on Kubernetes or even just
on virtual machines with Spilo, without using the Operator.
Nikolay: But using Docker for example?
Alexander: Yeah, of course.
Michael: But that is a good opportunity to discuss Postgres-operator.
Postgres-operator was Zalando's...
Was that one of the first operators of its type?
I know we've got lots these days.
Alexander: Well, maybe it was, but at the same time, the same
name was used by Crunchy for their Operator.
They were developed in parallel and back then Crunchy wasn't
relying on Patroni yet.
As I said, we started moving things to the cloud and at some
point, Vector moved a little bit and started running plenty of
workloads on Kubernetes, including Postgres.
Since deploying everything manually, and more importantly, managing
so many Postgres clusters manually was really a nightmare, we
started building Postgres-Operator.
Back then, I don't think some very nice Go library to implement
the Operator pattern existed and therefore people had to invent
everything from scratch and there is a lot of boilerplate code
that copied over and so on.
Nikolay: Is it only the move to the cloud what mattered here,
but maybe also moving to microservices, splitting everything
to microservices?
Because I remember from Valentin, for example...
Alexander: Microservices, of course, played a big role.
And probably...
Not probably...
Microservices were really driving force to move to the cloud,
Because with the scale of the organization, it wasn't possible
to keep monolith.
And the idea was, let's split everything to microservices, and
every microservice usually requires its own Database.
Nikolay: Right.
Alexander: Sometimes sharded Database, like we used application
sharding.
In certain cases, the same Database is used by multiple microservices,
but it's a different story.
But really, the number of Database clusters that we had to support
exploded.
From dozens to hundreds and then to thousands.
Nikolay: Yeah.
And this is already when you cannot rely on humans to perform
a failover, right?
Alexander: Even when you run a few hundred Database clusters,
better not to rely on humans to do maintenance, in my opinion.
Nikolay: Right, so that's interesting and maybe it's also the
right time to discuss why Postgres doesn't have internal built-in
to failover.
I remember discussions about replication when we relied on Slony
and Londiste and some people resisted to bring replication inside
Postgres, but somehow it was resolved eventually.
And Postgres has good replication, physical, logical, sometimes
not good, but it's a different story.
In general, it's very good and improving, improving every release.
We just last week discussed with Michael what improvements of
logical replication in 17, and maybe it will resonate a little
bit with topic today, Patroni, but it doesn't happen to autofailover
at all, right?
Why so?
Alexander: I can only guess, because to do it correctly, we cannot
just have 2 nodes, which most people run, like primary and standby,
because there are many different factors involved.
1 of the most critical ones is the network between those nodes.
When just having 2 machines, you cannot distinguish between failure
on the networking and failure of the primary.
If you just run health check from a standby and making decision
based on the health check, you may have a false positive.
Basically, the network just experienced some short glitch, which
could last even a few seconds, sometimes a few minutes, but at
the same time the old primary is still there.
If we promote a standby, we get to a split-brain situation.
With 2 primaries and not being clear to which 1 transactions
are running.
In the worst case, you end up in an application connecting to
both of them.
Good luck with assembling all these changes together.
Nikolay: This is what tools like repmgr do.
So I ended up calling repmgr a split-brain solution.
Because I observed it many, many times.
Alexander: Like as a mitigation, what maybe is possible to do,
the primary can also run a health check and in case if standby
is not available, just stop accepting writes by either restarting
in read-only or maybe by implementing some other mechanisms.
But it also means that we lose availability without a good reason.
So with this scenario, when we promote standby, technically if
standby cannot access someone else, it shouldn't be accepting
writes either, like in the network split.
Basically, we closely come to set up with how repmgr
call it, witness node.
Nikolay: Witness node, yes exactly.
Alexander: Witness node, basically you need to have more than
2.
And the witness node should help in making decisions.
Let's say we have a witness node in some third failure domain,
the primary can see the witness node, therefore it can continue
to run as a primary.
And standby shouldn't be allowed to promote if it cannot access
the witness node.
And it already reminds some systems like ETCD that complement
consensus algorithm and write is possible when it is accepted
by majority of nodes.
Nikolay: This wheel already invented, right?
Alexander: Yeah, so this is already invented, and what Patroni
is really relying on to implement after failover reliably.
I can guess that at some moment in Postgres it will be added,
and we already have plenty of such components in Postgres that
exist.
We have write-ahead log with LSN which is always incremented.
We have timelines which is very similar to terms in etcd.
So basically at the end we will just need to have more than 2
nodes, better 3, so that we don't stop writes while 1 node is
temporarily down.
It will give possibility to implement after failover without
even doing pg_rewind, let's say.
Because when primary writes to write-ahead log, it will be first
confirmed by standby nodes, and only after that.
So effectively, this is what we already have, but it's not enough,
unfortunately.
Nikolay: S.
So do you think at some point Patroni will not be needed and
everything will be inside Postgres or no?
A.
Alexander: I hope so, really.
Nikolay: S.
I hope so.
Alexander: A.
No, no, no, no, no, no.
I'm tired of maintaining Patroni, but because this is what people
really want to have.
To deploy highly available Postgres without necessity to research
and learn a lot of external tools like Patroni, solutions for
backup and point...
Nikolay: Upgrade them sometimes because we're always lagging
with these
Alexander: upgrades.
Yeah.
But at the same time, Let's imagine that it happens in a couple
of years, but with a five-year support cycle, there will still
be a lot of setups that are running not recent Postgres versions,
and they still need to use something external, like Patroni.
Nikolay: Yeah, I'm actually looking right now at commits of
repmgr.
It looks like the project is inactive for more than 1 year, almost.
Like a few commits, that's it.
It's like going down.
Alexander: So I have probably some insights about it, not about
repmgr, but I know that EnterpriseDB was contributing
some features and bug fixes to Patroni, so they officially support
Patroni.
Nikolay: So it sounds interesting, right?
So Patroni is a winner, obviously.
It's used by many Kubernetes operators, many of them, and not
only Kubernetes, of course, and winning, of course, some projects
were abandoned, not only repmgr, we know some others,
right?
But you thinking about 1 day everything will be in core and Patroni
will be abandoned maybe, right?
And you think it's maybe for good.
Alexander: So every project has its own life cycle.
At some moment, the project is abandoned and not used by anyone.
We are not there yet.
Nikolay: Right, right.
While we're in this area, I wanted to ask you what you think
about, Kubernetes also has, it also relies on consensus algorithm,
right?
Itself, it has it.
Why some operators choose, why do they choose to use Patroni
while others like CloudNativePG decide to rely on Kubernetes
native mechanisms and avoid using Patroni?
Alexander: To be honest, I don't know what driving people that
build cloud-native Postgres.
Nikolay: But what's better in general?
What are pros and cons?
How to compare?
What would you do?
Alexander: In a sense, CloudNativePG, there is a component
that tries to manage all Postgres clusters and decide whether
some primary is failed and promote 1 of the standbys.
I'm not sure how they implement in the fencing of the failed
primary, because if you don't correctly implement fencing and
promote the standby to the primary, you again end up in a split-brain
situation.
And let's imagine that 1 Kubernetes node is isolated in the network.
Nikolay: Network partition.
Alexander: Yeah.
And it automatically means that you will not be able to stop
pods for containers that are running on this node.
At the same time, applications that are running on this node
will still use Kubernetes services to be able to connect to the
isolated primary.
Nikolay: Right, yeah.
Alexander: So Patroni detects such scenarios very easily, because
Patroni component runs in the same port as Postgres, and in case
it cannot write to Kubernetes API, it just does self-fencing,
It stops Postgres to read only.
Nikolay: It's simple, by the way, right?
Alexander: Yeah, so I don't know if they do something similar.
In case if they don't, it's dangerous.
Michael: We should do a whole separate episode of CloudNativePG
actually I think that would be a good 1
Alexander: yeah I'm not saying that CloudNativePG is better
like does something wrong
Nikolay: I'm just raising questions
Alexander: raising my concerns
Michael: of course right back to Patroni it worked like this
from the beginning, but it feels like
Alexander: in version 10, which is end of life for a couple of
years, by the way.
From the very beginning, we wanted to support this feature, but
what was stopping us was the promise of Patroni with synchronous
replication that we want to promote a node that was synchronous
at the time when primary failed.
If we just have a single name in synchronous standby names, like
single node, it's very easy to say, okay, so this node was synchronous
and therefore we can just promote it.
When there are more than 1 node and we require all of them to
be synchronous, we can promote any of them.
But with quorum-based replication, you can have something like
any 1 from a list of, let's say, 3 nodes.
Which 1 is synchronous when the primary failed?
I'm not demanding to answer this question, So I will just explain
how it works in Patroni, like in the last major release.
This information about current value of synchronized and bynames
is also stored in etcd.
Therefore, those 3 nodes that are listed in synchronized and
bynames know that we are listed as quorum nodes and during the
leader race they need to access each other and get some number
of votes.
If there are 3 nodes, It means that every node, to become a new
primary, like a new candidate, needs to access 2 remaining nodes,
at least.
And get confirmation that they're not ahead of all LSN on the
current node.
Is it clear?
I should elaborate a little bit more.
Michael: So if they were ahead, let me ask the stupid question,
If a node checks that it is ahead of the current candidate to
be leader, that's then a bad decision to promote that leader
because a different 1 would...
Alexander: So just for your understanding, in Patroni there is
no central component that decides on which node to promote.
Every node makes a decision on its own.
Therefore, every standby node, like listed in Synchronous Standby
Names, goes through the cycle of health checks.
It accesses remaining nodes from synchronous to node names and
checks at what LSN are there.
And if they're on the same LSN or behind, we can assume that
this node is the healthiest 1.
And the same procedure happens on remaining nodes.
Basically this way we can find, okay, so this node is eligible
to become a new primary.
In case if we have something like any 2 and 3 nodes, we can make
a decision by asking just a single node.
Because we know that 2 nodes will have the latest commits, like
the latest commits that are reported to the client.
And it will be enough to just ask a single node.
Although it will ask all nodes from synchronous standby names,
but in case if 1 of them, let's say, failed, together with the
primary, it is still enough to make a decision by asking the
remaining 1.
Nice.
And the tricky part comes when we need to change synchronous
standby names and the values that we store in etcd.
Let's say we want to increase the number of synchronous nodes
from 1 to 2.
What should we change first, synchronous standby names, GUK,
or value in etcd?
So that we can correctly make a decision.
If we change first value in etcd, it will assume, okay, so we
need to ask just a single node to make a decision, although there
is just 1 node that has the latest commits, 100%.
And in fact we need to ask 2.
Therefore, when we increase this from 1 to 2, first we need to
update the synchronous standby names, and only after that change
in etcd.
And there are almost a dozen of rules that 1 needs to follow
to do such changes in the correct order.
Because it's not only about changing replication factor, It's
also about adding new nodes to synchronize standby names or removing
nodes that are gone and so on.
And I don't think any other failover solution implements a general
algorithm to do such changes.
Nikolay: How much time did you spend to develop this?
Alexander: Originally this feature was implemented by Ants Aasma,
he's working for CYBERTEC, it happened in 2018.
I did a few attempts to understand this great logic of this algorithm.
And finally, almost 5 years after, I was able to get enough time
to fully focus on the problem.
And even after that I spent, I don't know, a couple of months
implementing and fixing some bugs and corner cases and implementing
all possible unit tests to cover all such transitions.
Nikolay: There is no book which describes this, that you could
follow.
This is something really new that needs to be invented, right?
Alexander: Well, the idea was obvious, like how to do it, like,
or what to do, but like implementing it correctly and proving
that it is really working correctly, like, it's really a challenge.
Nikolay: Finding all the edge cases, right.
There is another thing I would like to discuss a little bit.
It was in Patroni 3, version 3.0, DCS failsafe mode.
So DCS is distributed configuration storage.
And actually we just experienced a couple of outages because
we are in Google Cloud and they're running Salon operator, Patroni
of course.
And I just checked the version of Patroni, and it seems to have
it.
But we...
Alexander: But I don't think it is enabled by default.
Nikolay: Exactly, this is my second question, actually, why it's
not enabled.
So, first question, what is it, like, how do you solve this problem
when etcd or console is temporarily out?
Alexander: Let's start from problem statement.
The promise of Patroni is that it will run as a primary when it
can write to a distributed configuration store, like to etcd.
If it cannot write to etcd, it means that maybe something is
wrong with etcd, or maybe this node is isolated, and therefore
writes are failing.
And when node is isolated, it's apparently working by design,
Patroni cannot write to etcd, it will stop Postgres in read-only
mode, but in case if etcd is totally down, because of some human
mistake, you cannot access any single node of etcd.
And in this case, Patroni also stops primary and starts it in
read-only to protect from the case, let's say, some standby nodes
can access DCS at the same time and promote 1 of the nodes.
So people were really annoyed by this problem, and were asking
why we are demoting primary.
So far the answer was always, alright, so we cannot determine
the state, and therefore we demote to be on the safe side.
The idea how to improve on that came at one of Postgres conferences
after talking with other Patroni users.
Like, how it is improved using the failsafe mode.
The primary, like when it can determine that none of etcd nodes
are accessible, it will try to access all Patroni nodes in the
cluster using the Patroni REST API.
And in case if the Patroni primary can get a response from all
nodes in the Patroni cluster in the failsafe mode, it will continue
to run as a primary.
In this case, it's a much stronger requirement than quorum or
consensus.
So it is not expecting to get responses from, let's say, majority.
It really wants to get responses from all standby nodes to continue
to run as a primary.
This feature was introduced in Patroni version 3, but it is
not enabled by default, because I think there are some side effects
when you enable this mode in certain environments.
Probably it is related to environments where your node may respond
with a different name.
Nikolay: I need to think about it.
Alexander: This behavior is documented.
Nikolay: Yeah, we will explore this.
Thank you so much for it.
But it sounds
Alexander: like...
On Kubernetes it is safe to enable it.
Nikolay: Yeah, we should start using this, this is what I think
as well.
Yeah, definitely we'll explore, thanks.
Alexander: Like pods always have the same name, just different
IP addresses.
Nikolay: I just got help for it.
And as usual, I just wanted to publicly thank you for all the
help you do for me and actually many companies. Many years it's
huge.
Thank you so much So another thing I wanted to discuss is probably
replication slots.
And I remember a few years ago you implemented support for failover
of logical slots.
Now we have it in Postgres, right?
So one more, finally, yeah.
One thing was basically removed, I guess, from Patroni, right?
Or you still keep this functionality?
Alexander: No, We still keep it and we didn't do anything special
in Postgres 17.
Nikolay: It was, I think it was 16 even, no?
Alexander: Failover of, ah.
Nikolay: Or 17.
Well, ability to use a logical slot on physical standbys was
in 16, but fell over in 17, we just discussed it.
Alexander: Yes, exactly, exactly.
I confused you.
That's why I'm saying we didn't do anything special.
Although I did some tweaks to make this feature work with Patroni,
because it requires to have your database name in the primary
coninfo.
Patroni wasn't putting the DB name to primary coninfo, because
for physical replication, it's not useful.
Nikolay: But I wonder...
Alexander: How it does it?
Nikolay: I wonder in my head, like, of course, we create slot
on the primary, it's clear, but Patroni main task is to keep
primary alive, to take care of high availability HA for the primary.
Okay, but if we have multiple replicas, multiple standby nodes,
and 1 of them is used, or maybe a few, but at least 1, 1 of them
is used to logically replicate to some Postgres or Snowflake
or anywhere or Kafka or something in this case if this...
Yeah, from standby because it's good, we like, less risks on
the primary and so on and wall sender is not using CPU and so
on.
And no out of disk risks.
So now we have this standby and it's dead suddenly.
It's not the job of Patroni to take care of it, right?
Because we need some mechanism to failover standby now.
Alexander: Well, you mean to keep logical replication slot on
a new standby where you would like to connect.
In theory, Patroni maybe can take care of it, since it's possible
to do logical replication from standby nodes since Postgres 16.
So how it's implemented currently in Patroni, like logical failover
slots, it creates logical slots on standby nodes and uses
pg_replication_slot_advance() to move the slot to the same LSN as
it's currently on the primary.
So basically the assumption is that logical replication happens
on the primary.
In theory, there is no reason why it cannot be done for standby
nodes.
Let's say we create logical slots on all standby nodes with the
same name, and Patroni can watch which 1 is active and publish
this information to ATCD and remaining standby nodes will again,
like Patroni remaining standby nodes will use pg_replication_slot_advance()
to move LSN on standby nodes.
So in theory it could work, but
Nikolay: I don't
Alexander: know if I would have time to work on it.
Nikolay: I'm just trying to understand, This is a relatively
new feature since 16 to be able to logically replicate from physical
standbys, but...
Alexander: But please keep in mind that it still affects primary.
Nikolay: Right.
Alexander: So, Maybe like pg_wal will not bloat, but pg_catalog
certainly will.
Nikolay: Yeah, this for sure.
I was referring to the need to preserve WAL files on the primary.
This risk has gone if we do this.
But I cannot imagine how we can start using logical slots on
physical standbys in serious projects without HA ideas.
Because right now I don't understand how we solve HA for this.
Alexander: Yeah, and unfortunately, this hack that Patroni implements
with pg_replication_slot_advance() has its downsides.
It literally takes as much time to move the position of the logical
slot as you consume it from the slot.
That's unfortunate.
And how it's solved in Postgres 17, it basically does not need
to parse the whole file and decode it, so it just literally overwrites
some values in the replication slot, because it knows exact locations
and does it safely.
Patroni cannot do it.
Although, probably, pg_failover_slots can also do the same.
For older versions.
Nikolay: Okay, some area, additionally, for me to explore deeper,
because I like understanding many places here.
Good pieces of advice as well, thank you so much.
Anything else, Michael, you wanted to discuss?
Like, obviously, like 1 of the biggest features was Citus support,
right?
But I'm not using Citus actively, so I don't know.
If you want to discuss it, let's discuss.
Alexander: I know that some people certainly do, because from
time to time I get questions about Citus with Patroni on Slack,
or maybe not Citus-specific questions, but according to the output
of the Patroni control list, they are running Citus Cluster.
There is certainly a demand, and I believe with Patroni implementing
Citus support, it improved quality of life of some organizations
and people that want to run sharded setups.
Nikolay: Is there anything specific you needed to solve to support
this or like technical details?
Alexander: To support Citus?
So, Citus, I wouldn't say that it was very hard, but it wasn't
very easy either.
So, Citus has a notion of Citus Coordinator, where you, like
Originally you're supposed to use coordinator for everything,
to do DDL, to run transactional workload and so on.
And on coordinator there is a metadata table where you register
all worker nodes.
And worker nodes, This is where you keep the actual data, like
charts.
And what I had to implement in Patroni is registering automatically
worker nodes inside this metadata.
And in case of failover happens For the worker nodes, we need
to update metadata and put new IPs or host names, whatever.
Basically, when you want to scale out your Citus cluster, you
just start more worker nodes.
Every worker node, in fact, is another small Patroni cluster.
So technically, in Patroni control, it looks like just a single
cluster, but in fact it's 1 cluster for a coordinator, 1 cluster
for every worker node, and on each of them there is its own failover
happening.
If you start worker nodes in a different group, like in the new
1, it joins existing Citus cluster and Patroni, the coordinator,
registers new worker nodes.
But what Patroni will not do, it will not redistribute existing
data to the new workers.
This is something that you will have to do manually afterwards
and it has to be your own decision how to scale your data and
replicate to other nodes.
Although, like nowadays it's possible to do it without downtime
because all enterprise features of Citus are included in Citus
version 10.
So everything that was enterprise now is an open source.
Nikolay: That's cool.
Michael: I saw Alexander has a good demo of this, of Citus and
Patroni working together, including rebalancing.
I think it was Citus Con last year?
Alexander: Yeah, it was Citus Con.
Michael: Nice, I'll include that video in the show notes.
Nikolay: I wish I had all this a few years ago.
Alexander: When I...
Yeah, of course, like, There was a little bit more work under
the hood.
In case if you do write workload via coordinator, it's possible,
like Patroni can do some tricks to avoid client connection termination,
like while switchover of working nodes is happening.
This is what I did during the demo.
There are certain tricks, but unfortunately it works only on
coordinator and only for write workloads.
For read-only workloads, your connection will be broken.
That's unfortunate.
Maybe 1 day it will be fixed.
So in the Citus, maybe 1 day the same stuff will also work on
worker nodes.
And by the way, on Citus, you can run transactional workload
by connecting to every worker node.
Only DDL must happen via coordinator.
Michael: Nice.
Speaking of improvements in the future, do you have anything
lined up that you still want to improve in Patroni?
Alexander: That's a very good question.
Usually some nice improvements are coming out of nothing.
You don't plan anything, but you talk to people and they say,
it would be nice to have this improvement or this feature.
And you start thinking about it, wow, yeah, it's a very nice
idea and it's great to have it.
But I rarely plan some big features from the ground up, let's
say.
So what I had in my mind, for example, it's a failover to a standby
cluster, like in Patroni.
Right now it's possible to run a standby cluster which is not
aware of the source where it replicates from.
It could be replicating from another Patroni cluster.
And what people ask, we have a primary Patroni cluster, we have
standby Patroni clusters, but there is no mechanism to automatically
promote standby cluster, because it's running in a different
region and it is using completely another etcd.
So they simply don't know about each other.
It would be nice to have, but again I cannot promise when I can
start working on it and whether it will happen.
I know that people from CYBERTEC did some experiments and have
some proof-of-concept solutions that seem to work but for some
reason they also they're also not happy with such solution they
implemented.
Michael: Yeah, sounds tricky.
Alexander: Distributed systems are always tricky.
Michael: Yeah,
get that on a t-shirt.
Nikolay: Thank you for coming.
I, as usual, I use podcast and all events, I participate and
organize and so on.
I use just for my personal education and daily work as well.
I just thank you so much for help.
Again.
Alexander: Yes, thank you for inviting me.
Yeah, it's a nice job that you are doing.
I know that many people listening to your podcasts and very happy
about it.
They learn a lot of great stuff and also making a big list of
to-do items like what to check and what to learn.
I cannot say the same about myself that I watch every single
episode but sometimes I do.
Nikolay: Cool, thank you.
Michael: Thanks so much Alexander.
Cheers Nikolay.