Michael: Hello and welcome to Postgres.FM, a weekly show about
all things PostgreSQL.
I am Michael, founder of pgMustard, and I'm joined as usual
by my co-host Nikolay, founder of Postgres.AI.
Hey Nikolay.
Nikolay: Hi Michael, hi.
Michael: And today we are joined by special guest Gwen Shapira,
co-founder and chief product officer at Nile, to talk all things
multi-tenancy.
Hello Gwen, thank you for joining us.
Gwen: Thank you for having me, it's very exciting for me to be
in your show.
Michael: Oh, we're excited to have you, thank you.
So to start perhaps you could give us a little bit of background
and what got you interested in this topic of multi-tenancy
in general?
Gwen: As many things it started with an incident but this time
not actually one of mine, so when my co-founder and I started Nile
we actually started with a very different idea.
After about 9 months, we were like, this idea is not working
very well.
We developed some things, we're not finding the market we hoped
to find.
We were sitting in the Hacker Space over in Mountain View, and
we're like, what did we learn?
We talked to 200 companies all doing SaaS.
What have we found out as a result?
And the things that called to us is that very early on, basically
when the first lines of code are getting written, you have to
choose a multi-tenancy model.
And then about 2, 3 or 4 years later, depending on how fast you're
growing, you have to change it.
And we heard a lot of stories on what caused people to change
it and whether they regret earlier choices or they're like, we
didn't know better and how things went for them.
And then we started looking in different blogs and we found so
many by very famous companies with either incidents where something
that was done to a single tenant caused a whole chain of events
that took down their entire system, sometimes for days.
And also a lot of slightly better stories, how we sharded our
highly multi-tenant database.
And we found story after story after story on how people had
to re-architect their entire database, which is, as you guys
know, extremely painful to do after you're a successful company
3 years in.
And we're like, this is a good problem.
So many people have it.
It is so common.
My past was in databases, not as much Postgres, more Oracle and
MySQL.
But I've seen this problem again and again in all kinds of companies.
My co-founders seen this problem again and again in all kinds
of companies.
This is such a good problem.
Everyone has it and nobody's working on it.
Why is nobody working on it?
And that's how we got into it.
Michael: Yeah, nice.
Should we go back then in terms of how do you like to describe
the different models or the different options that people have
at the in the early stages?
Gwen: Yeah, so everyone basically starts with 1 out of 2.
And I'm using AWS terminology, even though there is other terminology
that people apply to it.
AWS calls it the pooled model versus isolated model.
And in a pooled model, you basically create your tables as normal,
and then add a tenant ID Column to each and every 1 of your tables,
pretty much.
Some, maybe not everyone, some have shared data, but most of
your tables are going to end up with a tenant ID Column that
tells you which tenant this Row belongs to.
Very easy when you start out and all you have to do is sprinkle
some work clauses every now and then and you're pretty much good
to go.
How hard can it possibly be?
The places where it gets you is that you have no solution.
Everyone is in 1 big pool.
And if you have a problem tenant, if someone grows really, really
large and suddenly you need to Query start getting slow for them,
If you need to do an upgrade and 1 Customer absolutely refuses
to accept changes or needs their own time window.
There is a lot of different ways you may discover that by putting
all your Customers in 1 big pool, you save a lot of effort, you
save a lot of money.
This is by far the cheapest option you're going to have.
It's shared resources in 1 Database.
But you are not allowing yourself to do anything specific for
any 1 Customer should they need it.
The other approach is basically the reverse.
It gives every Customer its own Database or sometimes its own
schema.
This still counts as isolated, even though It's not all that
isolated.
You share quite a lot in that scenario.
But the schema is separate.
And it's quite a bit easier to move them out if needed, if the
schema is separated.
And In this scenario, first of all, there is a nice benefit that
you get help from a lot of popular frameworks.
I think Ruby has an, I think it's called apartment plugin for
having this kind of multi-tenancy model.
Django has something.
So a lot of very popular frameworks have something that helps
you in that model.
But if you accidentally grow to a large number of databases,
it starts being very painful.
Obviously, a database with a large number of objects is no longer
as easy to work with.
Suddenly, you start learning how much space and memory the catalog
can really take when you have connections.
Suddenly you start learning that pg_dump can take a very long
time.
If you actually have a database for each tenant, then doing any
kind of maintenance on 100 databases is already not fun.
If you end up going into a thousand databases, it's really not
fun.
And if you think about it, a lot of SaaS have customers in the
hundreds of thousands, not just thousand.
So it becomes very painful exactly when you grow.
Nikolay: Yeah.
For example, upgrade includes dump restore schema.
And we've had cases with 250,000 tables, 2 million indexes.
Well, indexes are there not dumped, but tables.
Gwen: Exactly.
This is worse than no dump, right?
If only we could dump indexes.
Nikolay: Yeah, and then you need to update statistics after upgrade
for all of those tables.
It's a nightmare, honestly.
Gwen: Yeah.
On the other hand, at least you get to do it to a customer at
a time.
Imagine that everyone is in 1 really big database and now you
have to upgrade all of them together.
Nikolay: Yeah.
Yeah.
It's painful, all this.
Gwen: Yeah.
And usually, and then there is the mixed model where I think
pretty much everyone ends up with where you basically have, you
start with the pool model, and as you grow, you shard it, but
you're actually pretty smart, and you realize that not all customers
are of the same size, and you can have some dedicated shards
for your biggest or most sensitive or most demanding customers.
And this is, I think, if you look 5 years into the life of a
company, I would say that this is the dominant model.
Some variation of we have shared databases with pool and then
some dedicated databases with specific customers.
Nikolay: What is the problem we're trying to solve?
Is it security or performance or both?
And if we go back to some customers, they share this pool, it
affects security goal, right?
So we don't achieve it.
Gwen: Absolutely.
So this is 1, First of all, this is 1 driver that people actually
start with the isolated model.
They know that they're going into a sensitive area.
They're focusing on SaaS for healthcare, SaaS for finance.
Those companies definitely start with isolated model and try
to figure out how to manage large number of databases.
A lot of times those companies don't become huge.
There is that many hospitals in the United States, but they still
have to build all the tooling to manage a large number of isolated
databases.
For other companies, it's more complicated.
I would say maybe 70% of the time, the reason for eventually
moving customers out and sharding and isolating would be performance.
It's amazing how many performance problems can be solved by just
having less data in each database.
On the other side, there are the story where 2 years in, suddenly
a very sensitive customer shows up or you want to sell into,
like you thought you were building a normal CRM or some kind
of a rug database, but then a healthcare company shows up, a
bank shows up, or even worse, a government shows up, and they
show up with a list of demands and since they usually have good
amounts of money to back those demands there is a lot of incentive
to figure out a solution for them.
Michael: Nice.
We have 1 kind of ugly duckling in the Postgres world that I'm
not sure quite fits either of these models.
I wonder if it's worth discussing row level security briefly,
because if I was to bucket it based on those definitions, it's
kind of the pooled model in a way because all of the data is
together but there is some isolation
Gwen: between tenants
Michael: absolutely yeah
Gwen: yeah I have a love-hate relationship with RLS I think a
lot of people do.
Because you're right, on 1 hand, it's absolutely a lifesaver
in the pooled model.
Developers make mistakes as joins get, and conditions get more
complicated.
It's very easy to misplace a work clause and actually leak data
that you don't want to.
So RLS will prevent you from doing it if you do it right.
It turns out that a lot of times the rules could get complicated
and then it leads to bugs.
It also turns out that a lot of times the rules get complicated
and it leads to terrible performance.
And 1 thing that developers really don't realize, I'd say almost
no developer realizes it until they run into it.
The work conditions that RLS introduces are not optimized like
the work conditions that you introduce.
Because Postgres, thankfully, is very good about security, It
treats the work conditions in RLS differently.
I think they have, they call it security conditions or something
like that.
And they are very, very conservative on how they optimize it
and how they plan for it.
This has benefits.
You get very strong security guarantees, very few bugs as a result.
This is fantastic.
On the other hand, the plan would be sometimes significantly
worse than what you would come up with if you were to look at
it really hard and do it yourself.
And with RLS, there is basically no way to force the plan you
want.
You cannot set, enable, disable different rules because, again,
the main overriding rule is that we're very conservative on how
we optimize those RLS conditions.
So Some people call RLS a performance killer.
I wouldn't necessarily go this far, but you can definitely run
into gotchas and you need to be aware that it's not a normal
wear conditions that you're looking at.
Michael: Yeah.
Nice.
Nikolay: So what does Nile offer today and what is the ideal
solution to all this in Postgres context of course?
Gwen: Yeah, so basically we wanted to do maybe 3 things.
First of all, give isolation while not degrading the developer
experience.
So for example, we partitioned data by tenant out of the box
for you, completely transparently, because we know that a bit
later on, you're going to want it, and it's going to be a pain
in the ass to edit.
We shard it Transparently.
Basically, your database may be spread across multiple different
charts.
We will route the queries for you and make sure that they are
working as you should.
So, in a way, you get the model you will have anyway in 4 to
5 years, but you're getting it from the get-go and without doing
a lot of the work because we are doing a lot of the management
for you.
The other thing is that we have done some work to basically bypass
RLS and still give you isolation.
So the queries, you kind of do the same RLS set tenant ID equal.
We use that to actually direct queries.
We rewrite the queries immediately to the partitions that we
know has the data for that tenant.
So we have a small extension that kind of replaces table names
with partition names in the query itself.
And this vastly improves performance in the majority of cases.
I mean, we've seen it in a bunch of cases, especially if you
have slightly weird indexes that RLS may conservatively not use.
The improvement is quite stark, depending, obviously, on table
sizes.
You can get a benchmark that proves anything, so I don't want
to throw numbers out there.
But obviously, if you break down a table with a million rows
into thousand tenants with a thousand rows each, then you can
show quite a, you can see where I'm going with that.
The other thing we did, and that was probably the most work,
and this is still work in progress, is allow moving tenants around.
Because 1 of the biggest problems is that the tenant gets large
or noisy.
And you want to give it its own machine, moving it is usually
a long downtime.
If you catch it after the tenant is already large.
By doing the compute storage separation, we can basically make
it transparent.
It's a latency spike while we're holding off some queries, while
we're moving things like setting up sequence ideas, moving, pointing
into a different compute into the same part of the storage.
But it's essentially a no downtime operation.
So we think it's a huge deal because again, it's just a problem
that we keep seeing again and again.
Nikolay: Yeah, I'm curious, is it all open source what you build
or only parts of it?
Gwen: Right now it's mostly hidden.
We have started registering parts of it under a Apache license
veto.
So yeah, the goal is to open source it and we already publicly
declared that it's going to be open source.
We have not a date, but a point of completion where we plan to
open it.
Nikolay: Yeah.
So I've heard about this, several interesting things here.
So 1 is extension for this, I guess it's called, in your documentation,
it's called RLS virtualization or how's it called?
Gwen: We call it tenant virtualization.
The extension itself, I think we called it Karnak.
We call everything after stuff in Egypt and Karnak is a famous
temple.
Nikolay: So data is stored in separate tables and in the same
database, but extension rewrites queries to basically route query
to proper table, right?
Is this based on?
Gwen: It's in separate partitions And we rewrite queries to go
to the correct partition.
We basically bypass RLS, we bypass the planner trying to make
those calls.
We found out that with a large number of partitions, this is
significantly more efficient.
Nikolay: I see.
So it's Postgres partitioning.
I see.
And you mentioned also another thing you mentioned here is sharding,
right?
Gwen: Yeah.
So we use foreign data wrappers to allow...
So we have 2 things in the architecture.
First of all, we have a proxy, it's a routing proxy.
So it keeps track of every connection, which tenant is the current
tenant and it routes it to the shard that has the correct tenant
in it.
And then we also have some cases where some developers want to
write queries that touch multiple tenants.
Those are not going to be as fast, but we do allow them by use
of foreign data wrappers.
And mixing partitions with our own partitioning rules with foreign
data wrappers and still keeping things efficient, we didn't want
the planner on any machine to be aware of all the partitions
in all the other shards, because it just explodes the planning
time in ways that we saw as unacceptable.
So what we did is represent each shard with a table and then
put hierarchical table inheritance on top of it.
And the end result is basically a union all between the table
with the table inheritance that points to all those other shards,
and the table with the partitions.
Now this gives us basically predicate pushdown because the planner
will push the plan to all those different charts.
The other charts know that they have partitions, which the source
planner didn't know.
They will plan correctly with all the partitions, but they only
know about a subset of partitions.
So we see it's a bit hacky and it's a bit weird to explain.
And we think we can do better with some modifications to Postgres,
which we have not done yet, but this does give us predicate pushdown,
fairly fast planning, and the ability to do queries that cross
tenants in situations where this is required, essentially.
Nikolay: Yeah, if you don't involve 2PC, just rely on front-end
wrappers, no two-phase commit.
I'm curious what kind of anomalies can happen there.
Gwen: Yes, and we prevent a lot of things that could cause anomalies.
So we do have a transaction coordinator, but in order to not
overload and also not overcomplicate our architecture, we limit
some things.
So, DDL has to be done on a single tenant, and you cannot mix
cross-tenant queries.
Sorry, DML, INSERTs and UPDATEs have to be done on a single tenant,
and you cannot mix in a Transaction cross, so the moment you
start a Transaction, you have to know what tenant you're working
on.
And then we route it to the correct shard, which has the correct
table, and everything has the absolutely correct guarantees.
If you need to do something cross-tenant, you do not involve
it with any kind of UPDATE.
You could still be exposed to some anomalies, I agree, because
it could be ongoing Transactions from other people in other places.
So you get the basic Read Committed guarantees that Postgres
gives you, I believe, but not anything more than that.
But again, we believe that cross tenant Queries are rare and
mostly done in analytical cases, where you do reporting where
it's slightly less critical to have those.
Nikolay: So you forbid the writing to 2 shards in 1 Transaction?
Gwen: If you want to write to a shard, it's fantastic.
You tell us what tenant you're writing data into, and we will
direct you to the correct shard.
Nikolay: I mean, if there is a Transaction which needs to write
to 2 different shards, this is a big problem, because without
PC.
Gwen: Yes, exactly.
And we don't let you do that, essentially, in order to avoid
anomalies.
Nikolay: I see.
Another question here, have you considered the approach used
in Vitesse?
As I understand it, maybe I'm wrong.
Where mostly for analytical Queries maybe, to avoid distributed
Transactions, data is brought synchronously from 1 shard to another.
And we have it locally, like basically kind of Materialized U
on top of logical replication, for example, or something.
And it has eventual consistency approach, of course, but you
can just join it in 1 Postgres, in 1 shard, right?
Have you considered this approach?
Gwen: Y.K.
We have considered it.
I think maybe CitusDB has something similar, if I remember correctly.
I'm not 100% sure.
But yeah, it's something that we were like, yeah, this is a good
idea that we may examine in the future.
It's definitely, we are trying to build something useful gradually,
and we understand that early on, it's almost safer to have a
bunch of limitations that over time will resolve, rather than
allow people to do something unsafe and also build a kitchen
sink.
Nikolay: It sounds to me like Postgres versus MySQL approaches
because MySQL approach, you remember MyISAM?
Maybe you don't remember, but it was like quite bad.
You need to run a repair table all the time because it's not
ACID and so on.
It allowed too much.
Gwen: Exactly, yeah.
And Marathon had a lot of issues.
I mean, there is a reason why InnoDB became extremely popular.
Nikolay: Right, right.
Michael: But also, with a multi-tenancy use case, I think you're
quite right that, well, we, I mean, you'll find out soon enough,
right, if lots of people want these cross-tenant or cross-shard
queries which are by definition cross-tenant queries and if they
don't, if you don't need to worry about it, you save a bunch
of effort having to even implement that.
So yeah, I like that a lot.
Nikolay: Yeah, last comment here is I'm excited to see that finally
Postgres ecosystem receives some tension in the area of sharding.
I guess it's just time has come and more and more databases became
too large to be handled.
Gwen: I'm almost surprised.
I'm honestly surprised it took that long.
I mean, again, if you look at MySQL.
Nikolay: We just, it's just unfortunate.
First time I touched this topic, it was 2006 immediately when
we started working with Postgres, honestly.
And There was a PL/Proxy from Skype at that time already, but
it required you to write everything in functions.
Gwen: Partitioning didn't exist back then.
Nikolay: Existed.
It was based on inheritance.
It
required much more manual.
It was fun, actually.
You understood it better, you know?
But yeah, but it was not convenient, 100%.
Not super convenient.
It's just, I see that it's just unfortunate how it turned out
in Postgres ecosystem.
And now definitely there is huge pressure.
Many companies need partitioning...
Need sharding.
And it seems
Gwen: like How would history work?
Imagine that YouTube picked Postgres and not MySQL as their first
database.
Nikolay: Yeah Google or Facebook they both chose MySQL somehow
Yeah.
Michael: The test would be for Postgres first, right?
Gwen: Exactly.
It could have turned out so differently.
Michael: Yeah.
Funny, isn't it?
Changing topics slightly or LLMs are quite top of mind at the
moment, are you seeing people's initial choices change as a result
of asking for advice earlier from our robot friends?
Gwen: Oh my god, we're seeing so many weird things, it's just
unbelievable how much things are changing.
First of all, we're having people show up on our Discord and
say things like, I'm using Nile because my LLM thought it's a
good idea.
And I don't really know Postgres, so I need some help, but my
LLM assured me that this is still a good idea.
Like a lot of people are just like people who are very much beginners,
like maybe I can say when I started developing, the first time
I had to use a database, my company sent me to a 3 weeks database
class.
I think it was Oracle about 20 years back.
And I came back a lot more confident that I know how to use Oracle
and not to leave transactions open for too long because people
will yell at me.
These days people don't do the three-week class before they try
using a database.
So you see a lot more, people are using a database earlier on
and they do expect more hand-holding from the vendors.
Like the LLMs give them advice up to a certain point, but eventually,
if things are slow, they will come to you and say, hey, why is
my query slow?
I'm sure you guys have seen your share of that.
I'm also seeing people use Postgres for their LLMs in different
ways.
And this is really exciting to me.
People use Postgres via MCPs, people use Postgres with vectors,
people building AI applications on top of Postgres.
We're seeing a lot of that.
And, I mean, personally, I'm really excited that people can program
with LLMs, not knowing a lot about Postgres, not knowing even
a lot about software engineering at all, and still get reasonable
security guarantees.
You don't need to know to ask your LLM about RLS or about, Are
you sure this query actually properly isolates tenants?
And it's also interesting how much the results differ when people
use different LLMs.
Like I would say syncing models do fairly well iterating in order
to get good code and checking their own results, again, given
via MCP access to a database.
I would say that if you use HRGPT 4.0, you will get a lot of
random hallucinatory stuff still in your code.
Nikolay: Yeah, it's funny.
I already told Michael, we had the cases doing consulting, But
like maybe already almost a year ago.
I started noticing that people send us Like some we are building
this part of database.
Can you review it?
We are reviewing we use different lamps supporting this review
and then we have a call and I'm curious, code looks great, I
mean schema looks great but something is off, right?
And then we have a call and I see they open tabs and charge a
PT, Claude there as well so I realize they used LLM to create
schema and then send us for review and we use a lens to review
it and then There's like a 4 party process, you know It's it
leads to good place, but someone needs to jump with proper expertise
and say, this is not a good approach.
Gwen: This is hilarious.
Do you think at some point you and your customer can just step
out and let the LLMs figure it out between themselves?
Nikolay: Well, there's a problem here because like it's great,
but there is like it does 80% of drop in 1% not not even 20%
of effort very quickly, but there is 20% of problems, which again,
like, I don't know, maybe in next few years, it will change.
I think it will change.
But right now I feel my internal LLM trained much better than
than Charger PT.
Gwen: You trained your own LLM, right?
Nikolay: Yeah, no, I mean my own.
Gwen: Oh yeah, no, but I think you actually trained your own.
Nikolay: Yeah, we experiment.
We do some stuff, we experiment.
We have some things like we have, we start with fine tuning,
moving to our own LLM, but not yet there still.
So yeah, there is a lot of stuff can be done there to properly.
It's hard to compete with Cloud and they have very high pace.
With Cloud 4 release, I see, wow, it's really great.
But it's still missing many things you learn from practice, which
were not discussed.
That's why they don't bring it.
Many problems were not discussed yet.
And you explore them if you have a lot of data and heavy workloads.
Gwen: So you're saying that even training on the Postgres mailing
list doesn't have all the information in it, essentially?
Nikolay: Well, yes.
For example, random problem, we recently touched it.
There is a buffer pool in Postgres.
And there are 128 basically partitions.
So you can have 128 locks.
And if you have a huge buffer pool, this becomes bottleneck.
And some people say, let's maybe make it tunable, configurable.
But when you start researching this topic, you end up finding
some recent, well, recent last August and September last year,
conversation in hackers mailing list, which is open-ended.
It's not complete, because somebody needs to run benchmarks and
prove that this is worth having new setting.
And that's it.
There's a patch proposed, but that's it.
No experiments yet.
Gwen: This is, by the way, 1 of the reasons I was so excited
about your LLM approach, because if you think about what is the
bottleneck for doing a lot of the database improvement things,
and I am feeling it Very personally, running benchmarks is hard.
Properly planning a benchmark is hard.
LLMs can help with that, but again, they sometimes just go off
the rails.
And even if they help plan that, I don't know LLMs that actually
run benchmarks to the point where they provision the machines
in AWS and know that you have to provision a separate machine
as a database and a separate machine, maybe a few of them, to
drive the workload and they both need to have appropriate resources.
Like all those and then don't get me started on analyzing the
results, which is kind of 99.9% of the work.
So the fact that you actually kind of started your LLM for I
have an LLM that can actually do benchmarks, is just, I think
this will be the biggest breakthrough in both people tuning their
own Postgres, and also Postgres as a community being able to
advance the state of the art?
Nikolay: Let me share sad news.
Like, I'm not giving up, but it's a roller coaster.
I spend more than we spend, like the team of maybe 5 engineers
spent more than 1 year trying to achieve that.
We achieved many things, but first of all, we chose Gemini because
they gave us credits.
And I think it was a huge mistake.
Gemini has a lot of problems.
Suddenly you have 500 errors, which are so many problems.
It's just not mature product, Gemini, and it has hallucinations
all the time.
It's good, for example, for JSON, working with JSON, because
when you need to run experiment, we decided to choose JSON as
config format.
It writes much better than GPT-4, 4.0 and so on.
But many things, it just hallucinates.
It invents all the time some things.
It just makes up results all the time.
And we have a system to control it but it bypasses all the time.
Like it's silly hard.
So then yeah, we experimented with additional DeepSeek, Lama
and we've fine-tuned a lot.
All versions of GPT, all modern fresh versions, we also bring
them all the time.
And Claude, Claude is much better, we just added it to this system
we have.
But After 1 year, I decided, you know what, I'm like, benchmarks
is extremely hard topic.
And we cannot trust it anymore.
I mean, we cannot trust LLM to create precise configuration and
process results fully.
So we decided that LLM is just more like connection thing.
When you engineer benchmark, expert needs to engineer.
I don't trust any LLM for now, because any experiment, we plan
to publish maybe 15 to 20 experiments in our blog last year.
And if you open our blog, you see just 1 experiment.
And even there we screwed up and someone on Twitter said this
is not right and we quickly corrected, which is good.
And we had achievements like interesting things, bottlenecks
popped up here and there.
It's really fun to iterate with LLM.
But once you allow to think, to design experiment and to treat
results, in 99% of time we have wrong, wrong results, wrong conclusions
and so on.
So for now we are thinking, okay, this is just, accelerator of
performing experiments, but design and understanding results
should be in human brain for now.
100%.
Gwen: I love the fact that Benchmarks is also hard for robots,
to be honest.
It's so hard for humans, right?
Nikolay: Yeah, and we collect so many artifacts, But somehow
it's super hard still.
So you always think where is the bottleneck?
And simple question, right?
But for now, it's extremely hard to let LLM find bottleneck and
draw proper conclusions.
Gwen: And honestly, if you just had an LM that always said it's
a network, it would be correct about 80% of the time.
Nikolay: What if it's local?
If it's, there's no network, everything through Unix sockets
and we had these cases as well.
So I agree with you in production.
Like in production, yes, but in experiments when we learn Postgres
behavior on single machine running pgbench locally, we don't
care sometimes.
So there's no network there sometimes.
So it's hard.
So I had many moments of frustration.
But it's so good, like I still believe that iterations are great.
So if you say this is great benchmark, just check it on new version.
Just changing 1 thing.
This is good.
We have automation, we have interface, and it repeats the process
of analysis again.
This is where LLM helps a lot.
Because without LLM, you could just, you have some form, and
oh, we don't have this parameter programmed.
It's not exposed in the interface.
It's bad.
With LLM, you have freedom to change things and iterate based
on existing good benchmarks.
So, yeah, I cannot say we are there yet.
This project, like, right now is we are, like, thinking about
next level of it, where I think we will let LLM, we will give
it less freedom, you know, that's the key, and control more by
human brain.
Gwen: Human in the loop kind of thing.
Nikolay: Yeah, yeah, exactly.
Exactly.
So design and first analysis, only human should be there.
But once you have confidence that you're moving in the right
direction and you just need to iterate and expand, for example,
to different versions, platforms, everything, this is where you
can relax.
You already verified results.
You can say just repeat, but on different something.
This is where LLM already can bring you.
It can just speed up everything.
You can throw it to this benchmarking process and have like 10
experiments running in 10 different versions or something.
Gwen: And I think this is also kind of almost the general directions
that agents are taking shape.
I mean, people started to 2025 was supposed to be the year of
the agent.
I think it's almost becoming the year of the human in the loop
with the agent.
Yeah.
Like all the successful products I see are you tell the agent
to do some stuff.
You ask it, please plan something.
You give it feedback.
You then say, okay, now that we have a good plan, go and execute
on it.
You come back an hour later, you had your coffee.
Okay, let's see what you've got.
Here's some feedback.
Go fix some stuff.
I think it's always every successful product is a bit like this.
Nikolay: Right.
But sometimes humans start using different LLM when reviewing
things, right?
Being lazy, right?
Gwen: Yes.
Nikolay: That's interesting.
So I'm not sure how successful it will be, but my gut tells me
that we need to move, move, move in this direction anyway and
have some, I don't know, like more experiments and so on.
I hope we will have more soon to publish, I mean, and start iterating.
But yeah, it was a rollercoaster last year, So now we are rebuilding
stuff.
We will see how it works.
And for example, 1 of the experiments we must do, I think, is
to conduct various benchmarks for RLS.
Because it's obviously...
That would
Gwen: be a fantastic example.
Yes.
And I mean, this is something that humans with experience are
pretty good at finding cases where you're like, is RLS going
to actually be an issue and have the stories that then the LLM
can go implement and test.
Nikolay: Yeah, we had several cases and also Supabase has public
materials, blog posts about this.
Obviously, there is already quite known case when you have like
current_setting function inside RLS expression.
And this is, yeah, if you select count 1000000 rows, it's terrible.
And it's quite easy to fix actually.
But yeah, so these kinds of experiments to collect them and see
how.
Actually, my goal with this experiment I think will be to prove
that they are not a problem if you do it right.
Gwen: Interesting.
I would contribute, I think there was a recent post on the bug
tracker where basically the optimizer, the planner refused to
use, I think it was a GiST index or a GiN index due to the belief
that the RLS optimization is incorrect, is unsafe.
I can look it up and send it to you.
Yeah, it's interesting.
But yeah, that can also be interesting.
Like if you have a fix for Postgres, then you can, it will obviously
be nice to showcase.
Nikolay: Right.
I think also if you, so you don't use RLS, you said bypass it,
right?
Gwen: We bypass it, yeah.
Nikolay: Yeah, that's interesting.
I'm curious if you, in this mixed schema when we have partitions
or shards, and RLS doesn't make sense at all to involve RLS additionally
locally if some of shards have a mixed pool of customers.
Gwen: Okay, 1 of the questions that I have in mind, and this
is something that we're trying to help figure out for our users,
often on top of the tenant, you still have permissions for specific
users in the tenant.
Like you have an admin that can do anything, and then you may
have someone who is not allowed to see some rows at all because
they're too sensitive, all this kind of stuff.
And our users ask us, they can do it with RLS inside those partitions,
or they can do it in their application.
There is a lot of application level tools or like middleware
kind of tools that will do it for them.
Is it better to do it in the app layer or in Postgres with RLS
is a good question that I don't have an immediate answer for.
Nikolay: Right.
So there are several layers of multi-tenancy basically.
Yeah.
Not multi-tenancy, but different layers.
And If your customer being a tenant for you, they might have
tenants for them, but also inside them, they might have additional
clusters or segments of users inside each tenant.
So it's apartments and rooms inside apartments.
Gwen: Nice.
I like that.
Yeah.
This is 1 of the things that make multi-tenancy confusing, right?
Because it's almost like Matryoshka kind of scenario.
Nikolay: Yes.
This term, by the way, used in Postgres ecosystem multiple times,
starting with, you mentioned GiST original paper by Hellerstein
mentions that Matryoshka.., RD-tree, Russian doll tree, so basically
Matryoshka
tree.
Gwen: That's true.
And also in embeddings, they have the Matryoshka type embeddings
where you can make them of any size.
Nikolay: Yeah, I read about this as well.
Yeah, it's funny.
So yeah, great.
And what are your plans for AI?
I saw MCP servers already with some integration.
Gwen: We basically have 3 directions.
1 is MCP server and making it public, giving it authentication.
Right now, it's open-source, you can run it on your own, but
we're not hosting it, so we should start hosting it at some point.
So things that just make people who use LLM, make it easier for
them to use it.
Other thing we've done that we think is very useful for LLMs,
this we already have, just make it 0 time to create new databases.
Because LLMs just 0 time and 0 cost.
Because they love creating a lot of them.
Every time something goes wrong, okay, let's try from scratch
with a new database kind of situation.
So we're making it fast and cheap.
The other thing that we're still working on is really to make
our documentation more LLM friendly.
LLMs.txt is absolutely not enough.
It is actually, it creates a very large file.
The LLMs tend to get lost in it.
We need to figure out how to make it better.
And then also, everyone is kind of thinking, can we have our
own agents?
Can we do something around that?
We're kind of thinking about that.
Something we already have that is useful is just that with the
multi-tenant model, the embedding, the vector indexes are much
smaller.
And this is a huge deal for people building those agents and
LLMs.
Nikolay: Yeah.
And actually, you mentioned, so this zero startup goal, you mentioned
separation of compute and data.
How is that achieved?
Using what approach?
Can you elaborate?
Gwen: Oh, yeah.
Sorry.
This is more than a two-minute answer, but the short one is that
you kind of, you patch Postgres, and then you find a better way
to do your storage and basically wrap every Postgres storage
function with an equivalent with your storage.
And then you also need to apply the WAL continuously to the
storage layer.
Nikolay: Great job doing this like in actually 20 seconds.
I understood very well.
Yeah.
So, and do you have plans to make this open source as well?
Gwen: Yes, absolutely.
I mean, this is a first of all, as you know, WAL readers have
to be registered, ours already is, and with the open-source license.
And then, yeah, we are planning to open
Nikolay: source our
Gwen: storage layer, our extension.
I think at this point it's about seven different patches we've made
on Postgres.
I don't think we'll want to open source it as a Postgres fork,
because the number of patches is quite small and we are maintaining
it.
I think everything from Postgres 12 to 18 at this point, or maybe
13 to 18, something along those lines.
Nikolay: All supported versions, I guess.
Yeah.
That's great.
Yeah.
Well, looking forward to checking it out.
And actually, one more question for me, maybe last one.
Are you open to some benchmarks we probably will do?
We plan to do some benchmarks with various platforms.
It all started after acquisition of Neon and there's a discussion
on LinkedIn.
So I thought about trying some benchmarks for different platforms.
How do you think, like what are your thoughts about it?
Gwen: Oh my God, this is very scary.
We are benchmarking ourselves all the time.
So I'm keenly aware of exactly like what benchmarks make us look
good and what benchmark make us look bad.
And I think that's also how we react to benchmarks in social
media.
Unless it's a benchmark that shows something in Postgres itself
and like clearly attempting to educate and help people.
You can design a benchmark to make anyone in the world look bad.
Nikolay: Benchmarketing, it's called.
Michael: You can design
Nikolay: a benchmark to
Gwen: make everyone look good.
Exactly.
So, I will, at any given time, I can publish my benchmarks that
makes Nile look fantastic, and you can run benchmarks that you
decided how to build them, and may be very realistic and may
even expose a problem that we have that I didn't know about.
But, yeah, in general, I love benchmarks.
I just have opinions on how benchmarks are used by marketing
people.
Michael: Very good answer.
For anybody interested in a bit more about Niall's architecture,
Gwen gave a really good talk at PGConf.dev recently and the video
just went up on YouTube so I will put that in the show notes
for anybody that wants a deeper dive there.
There was also a good talk I saw at a PgDay Paris event by Pierre
Ducroquet, I'm not sure if I'm pronouncing that anywhere near
correctly, all about multi-tenant database design, especially
focusing on something we didn't focus much on today, which was
the downsides of schema per tenant design, including things like
observability and monitoring, which I thought was really fascinating.
So anybody considering going down that route definitely check
out that video and I'll put that in the show notes as well.
So yeah thank you so much Gwen I think we're out of time it's
been a real pleasure.
Gwen: It's been a pleasure Thank you for having me on.