Michael: Hello and welcome to Postgres.FM, a weekly show about all things PostgreSQL.
I am Michael, founder of pgMustard.
I'm joined as always by Nik, founder of Postgres.ai.
Hey, Nik
Nik: Hi, Michael
Michael: And we have two special guests with us today.
We have Qian, who is CEO and co-founder of DBOS, and Peter,
CTO and co-founder of DBOS.
Welcome both
Qian: Thank you
Michael: I wonder if one of you could give us a bit of the backstory of DBOS.
I think it's quite a cool story
Qian: Yeah.
So we started DBOS as a research project.
I think it was started quite in the middle of pandemic in summer 2020, when Professor
Mike Stonebraker heard a talk from Matei Zaharia, the profes- our advisors and
also co-founder of Databricks, where they were talking about how Databricks were u-
was using Postgres to do some scheduling work because, distributed scheduling
and cluster management was a mess.
And they were talking about how to use a database to simplify that.
And then I think during that summer we talked and discussed, like, how we can
u- extend that idea to make database more helpful to build, reliable systems
to even replace the operating system.
So that's how we started the original research project called DBOS.
And then Peter and I were the co-leads PhD students working on that research project.
We published several papers, did a bunch of work.
And then what we found was that, the most interesting thing was actually to use the
database as a foundation for distributed systems, to build distributed systems.
And that's why when we both graduated in 2023 we decided to do, to
commercialize this idea, to do a startup.
And now we are working on DBOS Inc., the company.
Nik: that's cool And I must admit I observed queues in Postgres
for so long, like 20 plus years.
And somehow when DBOS DBOS was announced, of course it it attracted
my attention, but I didn't quite get it like Because like w-w-workflow,
like what i-what is it, right?
But then I started to realize, actually, to realize that in any
business there are workflows, and to keep the state and all the history and
so on in database, it's very natural.
So this is a very good idea.
And I think it's, it's-- it revives the idea that queue in that database
is not only possible, but it's possible in quite efficient way.
But you need to… With Postgres, you need to be very careful, right?
Because of some, some things that might happen if you implement
queues and workflows in naive way.
But I don't quite get it still why operational system.
I understand when Andrei Karpaty talks about like AI as a CPU in
the center, like memory, like every p-peripheral devices, everything.
I understand this analogy, but here I don't quite get the analogy why
is there is some intent or idea to implement whole works operational
system in, in da-database.
Like what's your view?
Peter: Yeah.
So what we're doing as a company is using the database as a foundation
for distributed workflows, and then for things around the distributed
workflows like queues, like messaging, like notifications, and so on.
I think the core idea here is that as your program runs, you regularly
checkpoint the state of that program in a Postgres database so that if the
program ever fails, if anything happens to it, if it crashes, if it's interrupted,
it can reload those checkpoints to recover from exactly where it left off.
And that, durable workflows idea backed by Postgres is the core idea of DBOS,
using Postgres as an orchestrator to make durable workflows that run really
w- that run with high performance and that are really simple because
all they have is, all they have, all they rely on is your database
Nik: Yeah, this makes sense.
I understand that.
I'm just still processing the idea.
It's like whole operational system.
Qian: I think our goal is that when we think about operating system,
we should not be limited to the operating system we're running
our, on our laptop or desktop.
Thinking like anything operational, I think Kubernetes can be called
as a distributed operating system.
Thinking about like anything that can run your application as like a
meta name for like operating system.
So I think nowadays, especially with AI, operating system is like cool again.
Everyone is calling themselves an AI operating system or, like something
AI-powered operating system for engineers, for sales, for whatsoever.
So I think operating system now, like the idea is more about how to properly
run your applications in any environment, like in a distributed environment,
in a highly available environment.
It's about how to provide the guarantee and provide kind of an
illusion that you are ru- like normal, like modern developers no longer
directly write or interact with low-level operating system primitives.
They usually interact with some- something like Kubernetes or cloud resources.
So like you can think of the cloud as an entire operating system.
So like I think the name like DBOS right now really focuses on how to operate
those, applications at scale and reliably.
So it's I would say it's eventually at a grand vision, we hope everything
will be backed by a database,
Nik: Yeah.
Qian: then that will be like a super operating system.
Yeah.
Nik: Yeah this makes sense in, in broader meaning operating system.
Yeah.
Not just like Linux or Windows
or something, Right.
I get this.
And also like there are even broader systems like there, there are companies
like which are building like an operational-- operating system for
businesses in broader meaning, right?
I also remember when Anthropic released I think Ultracode with workflows.
I remember on Hacker News top both topics were present.
One was about their workflows, another was about some blog
post from your blog, actually.
I remember.
It was like-- And for me, it was like this is two different things.
And I remember internal discussion in our team actually, that we use these
workflows in Claude Code, but workflows which are like saving some state
in database is something different.
And then I remember an aha moment when we realized we actually have agentic
workflows ins-internally already working, and we save state in Postgres.
So I realized it's very close topics.
I think you also use this resonating moment when they released it and you like
to discuss workflows in database, right?
So I have one specific technical question though because Peter, you mentioned
that if something crashes, right?
And ch- this checkpoint and database were all good, like we ha- we have stayed safe.
I don't like fully understand why checkpoint is needed because we always
can replay more after checkpoint.
There are WALs and so on, right?
But anyway, we
Michael: checkpoint
Nik: Ah, okay, a checkpoint there.
But what wonders me, if, for example, we se-send an email but we don't
know the result of that yet, right?
I'm like moving us slowly to the topic of two-phase commit or something, right?
How do you handle that?
Peter: Yeah.
So if your step is taking an action in an external system and it fails
in the middle of the step, then we're going to have to retry that step, and
that, that's unavoidable in any kind of workflow or durability system.
So our recommendation then is for individual steps that take-- have
side effects, external systems to make those steps idempotent.
And, we provide help for that.
Every workflow has a unique ID that you can use as an idempotency
key in systems that support that.
Many email systems do support idempotency keys.
Some problems are fundamental to distributed systems.
If you're taking a side effect somewhere else and it fails and you
don't know whether it succeeded, you're gonna have to retry it.
And so you should really try to make sure that, that action's idempotent
Michael: And another thing I didn't fully appreciate till recently, I kept
focusing on the word durable I see durable workflows mentioned quite often.
I'm thinking about reliability and retries, and that all makes sense.
The thing I didn't really appreciate was actions that
quite-- could be quite costly.
If you've got a five-step workflow and you've spent a lot of tokens or a lot
of money, or you've sent physical goods or something like that in steps one,
two, and three, and it fails at step four, you can avoid the c- the cost
of redoing steps one, two, and three.
Like it, the-- that hadn't really clicked for me, that sometimes it's about other
side effects, like might just be cost or it might be safety or it might be
something else that you don't have to risk re-replaying is that a big part of it at
the moment in terms of… or are there parts of that might be helpful to explain?
Qian: Yeah, definitely.
I think especially nowadays, everyone's building AI agents
and those new models are becom- becoming more and more expensive.
So if you have already asked the model about something, you want to
use that output as much as possible.
And if something crashes, you want to resume from where it left off.
Say, like you already asked the model what to do next, then you
should continue from that step instead of asking the models again.
Because otherwise you have to burn all the tokens and this will quickly become too
costly if you're having like processing thousands of documents and then you
want to do it in a more efficient way.
That's where DBOS is used a lot, like building agents and
building AI-powered pipelines.
I think cost saving is definitely… It's like becoming
more and more important, I'd say.
Yeah.
Michael: Makes sense.
Interesting
Qian: And also compute resources, right?
Even if you don't have to pay those, Even for some people who will self-host
their GPUs and - self-host their model inference, then you don't want to
burn all the compute resources again.
It's all about efficiency.
I think yeah, durability itself sounds like not, doesn't sound like
a very useful word, but it indicates a lot about cost savings and also
in- indicates about after you've done something, you don't repeat it again.
So it's indicates about correctness as well.
Yeah
Nik: I have a question about why should it be inside Postgres?
We know there is-- there are systems which do the same outside, right?
Temporal is one of the most popular systems.
I even from my personal experience, I even had a moment when one of our
clients reimplemented our Postgres major upgrade workflow in Temporal.
it was, It was Gadget
Michael: They spoke about it on the episode.
Nik: Yeah, we had an episode about that, and it was like we
always used Ansible and so on.
Of course it's very different, right?
When you have workflow, you save everything, it's great.
But you move further, let's bring the whole thing inside Postgres.
What are pros and cons and, why do you think this is a winning strategy?
Peter: Yeah.
So I'd, what I'd ask is why not Postgres?
The temporal model is that you are taking… is that you're outsourcing
the control flow of your program to an external orchestrator, and that's
a lot of additional infrastructure.
You can either self-host it, in which case to run your workflows, you have to spin
up highly available temporal cluster, a database for it, which is Cassan- and its
database is Cassandra and Elasticsearch.
And now you're spinning up three new distributed systems to run your
workflows, and all three of those systems are totally on your critical path.
If anything happens to them, Nothing will work.
Your workflows will not function.
Your business grinds to a halt.
So you're putting three complex distributed systems on your critical path.
Or - you use their cloud service.
Now you're putting an external SaaS on your critical path.
You don't need to do that.
You just need a database.
And what we're trying to do in DBOS is totally replace all of those distributed
systems, all those external orchestrators with your Postgres database so that you
can use your database as an orchestrator.
It's already on your critical path.
It's already something that you're making sure is highly available.
It's already critical for your application to work.
Why not also use it as your workflow orchestrator so that you have
less infrastructure, so that your operations are simpler, so that
there are fewer things that can break and bring down everything you have
Nik: Two things here.
One is who will-- like Postgres doesn't have scheduler, right?
You can use pg_cron or something, or you need to still need
external help to tick, right?
To move forward.
And another thing is that if it's all running on the primary and primary CPU,
first of all, is not infinite, right?
So you need to avoid heavy things and have them outside still, right?
Peter: Yeah, and that's the reason Postgres is
the orchestrator.
The actual business logic of your workflows is still
running in your application.
The scheduler is running in your application, for example, and
then Postgres is the orchestrator.
So your applications are running a scheduler, and your applications
will enqueue when the sched- when the cron fires, your, one of your
application servers will enqueue a scheduled workflow into Postgres.
And then Postgres there does orchestration.
It coordinates multiple servers to make sure that only one of those
servers actually executes the scheduled workflow when its cron tick fires.
So your applications your regular servers are doing the work, and
Postgres is orchestrating them to make sure that every, all the work
is being done totally reliably, that everything happens exactly once.
Nik: Makes sense.
So this is when you say critical path, you don't include things
that like sending email can be done from multiple servers and they are
replaceable like stateless and so on.
Yeah, I get that.
It feels slightly awkward for me to try to criticize the idea.
I'm a big fan of myself, like maybe Michael should join
and find some other questions
Michael: I was actually gonna ask a question that probably
goes even more positive.
M- or maybe not, but I would expect there to be performance benefits to
using your already h- used database for this rather than an external service.
So I was only gonna say What are, what are the performance trade-offs and what do you
do to minimize the impact you're having on primary CPU and that kind of thing?
Peter: Yeah, absolutely.
So latency in particular gets a lot better because now the latency of executing
a step is just one write to your local Postgres as opposed to a async dispatch
from somebody else's cloud service.
So definitely latency improves.
Now throughput then becomes database bounded.
You can process as many workflows per second as, as your
Postgres allows if you want.
And if you need to scale beyond that, you need to shard across
multiple Postgres databases.
The good news is that these operations are fairly light.
So our rule of thumb is that four Postgres cores can handle 1,000 actions per second.
So for a large database, that means you can vertically scale for a large
Postgres database up to… we've benchmarked around 40,000 actions per
second on a single Postgres database.
And then of course, you need more than that, you can shard or you can
use multiple Postgres databases.
So your throughput does become bounded by your database, but the
good news is that your database can handle a lot of throughput
Nik: Yeah.
Yeah.
Michael: and we're
talking… so let's say we've got a workflow with six steps.
We've got one write at the beginning, one write per step, and one write at the end.
So we're still only talking eight writes total for a si- and this workflow's having
to do things in the meantime, it's not gonna be running through them instantly.
So yeah, great.
Okay, cool.
Peter: And you can handle 40,000 of those actions per second on one database
Nik: Yeah, I'm curious.
I researched a little bit and I noticed y-that you started with repeatable
read isolation l-level and then reduced it to default read committed, right?
I'm very curious what was behind that.
Is it the throughput question or anything else?
Peter: So we use isolation levels there's a lot of different parts
of our system that use isolation levels for different reason.
we've designed the system so that almost everything is a single statement.
So it can just be a, it can just execute a recommitted, and that
makes things far more efficient.
As many things as possible execute as a single Postgres statement,
so they can just be atomic and you don't have to worry about isolation.
You set it to default recommitted.
The one thing that we can't do that for is the main DQ query.
this is probably the single most important s- query in the entire system,
and it's the query that every worker runs periodically on every queue to
dequeue the latest work off the queue And that's a very complex query we've
put a ton of effort in optimizing.
That query has a, actually has a conditional isolation level, s- depending
on what, how you've configured your queue.
So if your queue uses global flow control primitives, if your queue, for example,
has a global concurrency limit or a global rate limit, so you say, for example,
that this queue can only execute one operation at a time across every worker,
then that requires global coordination, which means we have to use a high
isolation level like repeatable read.
And that has coordination costs, but if you have those global limits, you're
probably fine paying the cost because you're only executing one workflow
at a time across all your servers.
th- your throughput was limited anyways in practice though, what we
see is that high throughput queues usually only use local limits.
They might say only execute 10 workflows at a time per worker.
And those local limits don't require global coordination.
And that means that it's perfectly safe to run the queues query at a very low
isolation level at at read committed.
And that way you can have this query running across hundreds of servers at the
same time, or thousands of servers at the same time without them conflicting because
they're all running read committed.
And that lets you get this thirty, forty thousand dequeues, dequeued workflows per
second, actions per second throughput.
And it really caps out at thirty thousand dequeues per second on a single
Postgres across thousands of workers because we've optimized this query
Nik: And in case of sharding solution re-repeat-repeatable read
wouldn't be available anyway, right?
So because like global limits it's… Yeah.
You
Peter: But in practice, global limits tend to go with relatively low throughputs.
Nik: And it's flexible now, right?
So u-u-user can control it and decide what, what's in.
That's great.
And I also noticed listen to file was recently added, right?
Or not recently, maybe I'm wrong.
Qian: We've been having listen notify, but recently we've been optimizing it
Nik: yeah.
We ha-- Yeah, I think it was last year, Recall AI they discovered this awful
global lock on commit for Notifier, right?
What's your deal with this right now?
Peter: Listen/notify is really powerful but has multiple performance issues.
There was that issue that… I forget.
There was that issue that reached top of Hacker News last year.
That was actually not the issue we ran into.
That issue was solved in Postgres 19.
That is not the issue we ran into.
that issue is that notify had terrible perform- notify's broadcast
mode had terrible performance for large numbers of sparse readers.
Uh,
that's actually an unrelated issue.
We do not have that.
We have dense readers.
We don't have sparse readers.
Nik: Can you explain please the sparse readers versus dense readers?
Peter: Oh, sorry.
The problem that everybody was talking about last year was that if you're
broadcasting to a huge number of readers, each of which is only reading
a small fraction of all messages, that had terrible performance.
So that's actually not the issue we ran into.
The way DBOS is set up, pretty much all workers are listening to all
messages and that performs fine.
The issue we ran into was the global lock at notify.
When you commit a transaction that performs a notify, it has to acquire a
global lock for the entire duration of the commit, and that includes the fsync.
And what that means is that these transactions can't be group committed.
They have to commit sequentially, which essentially bounds the throughput of
any operations that perform notify.
You can't perform more than a certain number of them per second
because you can't group commit.
So you're stuck waiting out the entire duration of the fsync.
the optimization that we settled on there to make listen notify, because
we really want to use listen notify.
It provides incredibly low latency, and when it works, it
provides incredibly low latency.
So w- the optimization we settled on was notify batching.
So instead of sending a notify, originally, we were sending notifies
in a trigger every single time you, for example, wrote to a stream.
And the optimization was instead it was removing the trigger and instead
buffering stream writes in a background thread and then sending all the
notifies in the batch and send that batch e-every ten milliseconds or so.
A huge batch of notifies is actually very fast because you
only have to hold the global lock for the duration of that commit.
that's fast.
And what this does is this allows us to use listen notify without re- running
it, worrying about the global lock because we're batching commits and
the global lock's only required for commits, and still getting the benefits
of low latency because we can still emit these every, every few milliseconds
Qian: I think, the overall observation is that we don't really need that global o-
ordering when we're doing the application level streaming, like streaming,
for example, results to the client.
Then what do you-- we're leveraging the listen/notify to do is to notify like,
"Hey, we have some results available."
So we don't really need that strict ordering.
And I think that was the main aha moment for us that, okay, we don't need to
limit ourselves to that exclusive locks.
We can do batching.
Yeah.
Nik: And yeah, and this is… okay.
So and also the problem is happening.
You, Peter mentioned that it was fixed in 19, and it's happening when
commit has a lot to write, right?
So like it's this fsync.
Basically, transaction is significant in terms of how much it's writing, right?
So th-this increases the problem.
But in your case, you have only like brief writes in transaction.
Not a lot, right?
Not like a lot of JSONs to be written and fsynced during commit,
So the problem shouldn't be huge a-already if I'm not mistaken.
and you decided to group it.
This is interesting optimization.
I, I wanted to raise this question, of course.
You use skip locked updates and what's happening if we have some
xmin horizon blockers, so long transactions or lagging logical slots?
Obviously, a lot of dead tuples should be accumulated, right?
Is it a requirement if you use DBOS at, like at high rates, at a lot
of events per second and so on?
Is it a requirement I need to watch closely what is my xmin horizon?
Don't allow long-running transactions, don't allow logical… lagging
logical replication slots long transactions, so on standbys with
hot standby feedback on and so on.
Is it a requirement or you solve this somehow I don't know, like
maybe mitigate-mitigating bloat with repacking, reindexing and so on?
Peter: Got it.
So just to be clear, we're only using skip locked on our own workflow status
table, and that's typically isolated within its own Postgres database.
So our recommendation is Don't take out ultra long-running transactions
inside the DBOS database, and you probably weren't doing that anyways
because DBOS certainly doesn't do that.
And then the rest should be fine.
The DBOS operations are isolated here
Nik: So you don't use skip locked for like events t- tables where
events or workflow data is stored?
Peter: It's only used for deques off of the main workflow status table, and that's
usually pretty isolated from user data
Nik: But still, it's you need a s- single update on every status change, right?
Peter: Yes, and that does lead to dead tuple accumulation.
but that's a separate issue that y-yes, at ma- at massive, at
really high throughputs, like when we were doing these benchmarks at,
40,000 workflows per second, we saw high CPU usage due to dead
tuple cleanup and autovacuum because we're turning on these indexes.
And we've done a lot of work to optimize indexes, so most,
almost every index is partial.
So most indexes are only being maintained either for a small
subset of workflows or only for a small subset of workflow lifetime.
But there's still a fair amount of index churn on the order of… And this, of
course, not as hot a dead tuple, and it's going to create a bunch of dead index.
It's going to create two or three dead index entries that are
going to have to be cleaned up.
At really high throughputs, this can become the bottleneck but that sets in
around 20 or 30,000 actions per second is when we start seeing the tuple
accumulation really becoming a problem, because there really aren't that…
Tuple accumulation become a problem, because we have
optimized this as much as we can.
Nik: Yeah, I see it very differently.
I see it like when you say 20, 30,000 per second, it means for me it
becomes 2,000, 3,000 or 200, 300 per second if we talk about really long
transaction happening in parallel because they block vacuum, uh, work.
We cannot delete the tuples, and at very high throughput with shorter
transactions will be-become the same if you have much lower throughput,
Peter: Yeah, which is why our recommendation is isolate
the DBOS system database
from the database we're using, we're using long-running transactions.
These don't have to be in the same, in the… These don't have to be in the
same Postgres database, and they probably shouldn't be, especially for long-running
transactions that could block Autovacuum
Nik: Yeah, this is what my… Yeah, th-this is exactly what I was asking.
It's not like like requirement, but it definitely if you have long-running
transactions or any other explain horizon blockers performance of
DB-DBOS will, will suffer eventually.
Peter: Yeah, but these don't, these can be isolated from each other.
They don't have to run in the same place, and they probably shouldn't run
in the same place, especially at scale
Nik: This is interesting.
We have a lot of common things.
I agree with putting everything to Postgres open source.
I w- didn't mention that the thing you're building is open source, which
is great, and like this is super cool.
And and like we don't need extra things on critical path.
That's great.
Also I guess you have self-served option as well, right?
If it is open source, you can self-host yourself.
Like freedom, right?
It's great.
And you run also y-you can run DBOS in, on RDS.
Absolute fr-freedom.
But when you say let's split it, it's against the first principle.
Like first principle, let's have it everything in one place.
Because if we have, this thing and we have other tables, we can
organize code right inside database.
Sometimes it's a bad idea if it's very long, if CPU-consuming.
But if it's something simple, I want to update everything inside,
like using triggers, PostgreSQL code functions, and so on.
I don't want to split.
But once I have database, say terabyte in size, and I start some analytical queries
right there because I'm not I'm not grown enough to have ClickHouse or something.
I have only Postgres, right?
Those long-running queries will take time, right?
And in this case updates and data, MVCC in Postgres actually is not a good thing
to deal with, as we all know, right?
This is I'm j- I'm asking because, like you m- you know that I'm trying to revive
this PGQ from Skype, which is bloat-free.
No, almost no updates.
For event table, there are no updates there.
But I don't see how it will… I don't see clearly how it will be
used to implement workflows yet.
But for many tasks, it's great, and we don't have problems with bloat there.
And I'm trying to understand how our operate properly.
Like in, in case of updates, we need to establish some rules.
Like for example, we should stop allowing long-running transactions.
I'm I'm glad I participated in building transaction timeout.
So we set transaction timeout, say like 10 minutes.
That's it.
And I'm very curious, like maybe there is a way to define some
boundaries and rules when it's time to move away and to like to split to
different Postgres cluster, right?
To split these workloads.
M- And there are no good rules yet.
Th-This is a problem
Peter: One is that you don't have to split multiple Postgres clusters or servers.
You can, if I remember how Autovacuum works correctly, it is isolated across
multiple databases on the same cluster.
Nik: Logical databases.
Peter: Yes.
So like the xmin horizon is set per database.
So so you don't have to split your physical infrastructure.
You can share physical infrastructure
Nik: Yeah that's a good point.
I agree.
Still, I feel like it would be great to have some benchmarks or something to
define when it's time, when it's time to split, when it's fine, and which
transactions should be disallowed.
I think it-- you're very right.
If you have 30,000 workflows per second it-- your requirements become
much stricter in terms of what means long-running transaction, right?
How much xmin horizon is lagging.
Anyway, thank you for diving into this topic.
I'm also trying to learn this myself
Qian: Yeah.
I think that's actually a good, really good point because you are right that
if you put… Y- there are benefits of co-locating the workflow tables
with your actual application data.
One thing we were talking about in some of our blog posts is this
transactional outbox pattern.
If you say someone, like we updated the customer data, now we want to
kickstart some of the workflows, then by co-locating them in the same
database, you can do it in a trigger.
Say, whenever you update this customer data, automatically start or enqueue
a DBOS workflow in the same database, and that's really convenient and
that, that's a way that you can guarantee whenever there's updates
in this table, you'll kickstart a workflow that is durable, that will,
that DBOS will guarantee will finish.
And that's a pattern that is really difficult to do with,
say, like external orchestrator because there will always be a gap.
And by co-locating in the same database, there's a way to close this gap.
But everything has a trade-off.
Like you said, if you put in the same database, you have to be really
careful about what you, what queries you put in that database, like how
to avoid head of line blocking, how to avoid bloating in that database.
So I think that's a topic we're continuously, benchmarking and
also by talking to users and see what kind of workflows and what
kind of workloads they're running.
And we're, we are also trying to categorize like a more detailed,
guidelines for people as well.
Nik: Yeah, interesting.
But any-anyway I'm glad that this, this topic is becoming more and more popular
and other companies started to join t- like M-Microsoft recently, right?
With pgDurable.
What do you think about that system?
Qian: Yeah.
So this is great.
I think pgDurable is mostly for pure SQLs, right?
It's, it doesn't really work for if you want to send a email or
interact with external systems.
So we'll definitely, take a look and see if there are any ways to collaborate
or borrow ideas from pgDurable.
But I think in general it's another validation of the Postgres
Nik: yeah.
yeah.
Everything more and more people start looking in the same direction.
I think yeah, the topic is warming up.
I agree with you here.
Also, there was a system called Absurd, which got my attention.
Single SQL file as well.
It… I think it was released last November.
And yeah.
I noticed in pgdurable is that the tendency to have a SQL PL, PostgreSQL
code quite ef-ephemeral way.
And this is w- like you're just passing it as a parameter, like
to lump the functions, right?
And this concerns me because I want my code still to be in Git, right?
To be versionable, fix, reviewable, and so on, right?
So this leads me to question, what are the best use cases, case studies
you ha- you had already, you where you observed debor… I'm sorry,
like I pronounce it DBOS for myself.
I don't know why.
What are the best use case studies, use cases you have, like where
it shines in terms of how like, like it suits really well, right?
Sending emails or like processing orders in e-commerce or something like this.
What are the best cases?
Peter: Yeah.
the thing about workflows is that they're really powerful f- they're
really powerful for a lot of things.
So of course, the, the really hype new use case we see is AI
agents and operations around them.
So people using workflows both to make their actual agents durable.
This is particularly important for long-running interactive agents that
might, talk to a person or do multiple rounds of conversation with a person
or require approvals for a person over the course of hours or days.
Similarly, data pipelines like document processing pipelines for agents often
require durability, and, you know, workflows can really help there.
So those are kind of the, the new use cases.
And then there's this whole host of traditional business workflows that
have always needed durability and where Postgres backed durability
just makes it easier to provide it.
And that's things like logistics.
It's things like, like a- like lab automation.
It's things like new user registration, making sure new users are properly
updated in every single one of your cor- of your corporation's databases.
So we see this really broad spectrum of use cases for workflows because
really any important operation that touches customer data, touches important
data, it probably should be a workflow if you're really operating at scale
and you really care about reliability
Qian: I think in summary it's we really view Postgres as the central
orchestrator to connect things together.
You can connect external systems to your local database, or you
can do event processing to connect multiple systems like your other
databases or your email server, payment system, all sorts of things.
So we re- try to make sure your Postgres is the glue for everything.
Yeah
Nik: And speaking of Postgres maybe last question from me.
So now Postgres 19 beta 2 just released, Postgres 20
development has started already.
What what do you think should be changed in Postgres or brought to Postgres to make
the product you are building even better?
Peter: I think we do have to go… I would like to go back to notify.
Removing that global lock is not easy, and I'm not sure if it's doable with the
way notify is currently implemented, but that area of the code hasn't really been
worked on much since Postgres 9 and, maybe there's something that could be done
there to remove the footguns from notify.
And even if you don't remove them, I think notify's really complicated
performance characteristics should probably be, should probably be actually
documented instead of having to figure this out like through by… I mean,
we figured this out basically by, by reading the Postgres source code.
That, that's one thing.
Qian: Yeah, I think another thing is about, like, how to ho- better
s- even better support for larger larger blobs or larger columns.
when you-- when people are using DBOS, sometimes they want to checkpoint, say, a
PDF or something as part of the output.
And still I know Postgres has some good support for larger blobs right now.
But I think most of the time, we still recommend people to say "Upload your
large documents to S3, then store a pointer in the database." But that still
leaves, a little bit gap in the durable checkpointing, because there could be
the case where you upload something but you don't checkpoint or vice versa.
It'll be great if we can really put more, like, larger data into Postgres.
Michael: There is a nice change, I think, coming in 19 around that
in that the, I think the default TOAST compression is changing from
like a, an older PGLZ, like a Postgres specific compression
algorithm to, I think it's LZ4.
but yeah only minor on, on the grand scheme of things.
nice one
Nik: Yeah.
And speaking of notify, I think, yes, global lock is still there, not removed.
I remember some work last year in PostgreSQL
Peter: There was work in 19 on improving the performance of
broadcast, of the notify broadcast when not all listeners are listening
for all messages, which is great.
That's a real performance improvement.
The global lock is much harder to get rid of, and I'm not sure if it's possible.
But that is…
Nik: work on removing that lock.
I remember that, that work.
I also remember I just founded my email to document, as you said,
the limitations from July 10, 2025.
Nobody answered my email.
So there is a patch proposed.
Let's document it at least.
Yeah,
Peter: Yeah, the I, certainly 19 doesn't include any work on the global lock.
Maybe there's some in development for 20, I'm not
Nik: Yeah.
Yeah.
It's worth checking.
And that documentation should be changed.
I agree with you.
So maybe someone listening to us will go and comment on that proposal, and
we will get it documented at least.
Great.
Thank you so much.
I enjoyed it a lot.
I wish you all the best with this wonderful product.
I'm going to try it in a couple of places, and I encourage
our listeners to try it out.
Especially it's open source, so it's so easy to start, right?
Thank you
Michael: Nice one.
Thanks very much, both of you
Peter: Yeah, thanks a lot
Nik: Yeah, have a great week.
Qian: You
too.
Nik: Bye-bye