Michael: Hello and welcome to Postgres.FM, a weekly show about
all things PostgreSQL.
I am Michael, founder of pgMustard, and this is my co-host Nikolay,
founder of Postgres.AI.
Hey Nikolay, what are we talking about today?
Nikolay: Hi Michael, let's talk about performance cliffs and
1 case particularly, track_planning.
The reason why I brought this to our attention is that I observed
recently several, actually over the last few years, I observed
several strong cases in production systems, various production
systems, where lack of planning phase tracking caused huge amount,
huge effort invested to troubleshooting.
And you know, our best example is, we talked about this many
times, during planning Postgres locks all indexes with access
share lock.
All tables, all indexes involved, even those which will not be
used, it still locks them, right?
And we talked a lot about lightweight lock, LockManager contention
issues, right?
And it happens during planning.
And if we don't have track_planning, pg_stat_statements.track_planning
enabled, it's not visible to our top-down query analysis.
We cannot see which queries spend a lot of time in planning and
a lot of buffers, maybe, right, and so on.
Michael: Yes, and crucially, not only is pg_stat_statements not
on by default although a lot of people have it turned on but
pg_stat_statements track timing is off by default even when you
enable pg_stat_statements so most folks that enable pg_stat_statements
don't enable track_planning including most cloud providers.
Nikolay: So let's zoom out, actually.
This is a good point.
By default, Postgres presents only poor metrics for query analysis.
If you have I/O timing enabled, in the pg_stat_database you see
I/O timing.
You have a number of transactions, you can understand how many
transactions per second not queries per second.
It's like Postgres doesn't track it and that's it so you don't
understand like?
You don't understand throughput.
You don't understand the average latency in general.
It's super hard.
And even at the highest level, like my whole database, how is
it doing?
How many TPS?
How many TPS is fine?
How many QPS?
Queries per second.
And what's about latencies?
But if you want to go down and understand which groups of queries,
parts of workload are responsible for, like, where do we spend
time, For example, or do some work with the buffer pool.
To do that, you don't have it by default.
You need to install the pg_stat_statements extension.
And this is actually what most people I know do.
This lesson is learned and pg_stat_statements is maybe the most popular extension
because of that, right?
Definitely.
And we did have a couple of episodes about pg_stat_statements.
It's hard to overstate that this extension is not good.
It's really valuable.
And there are opinions that it should be already...
It's time for it to go to the core engine, because it sucks that
we install this extension and Postgres needs to parse, analyze
query once again, just for metrics.
So it should be part of core.
But it's already a non-default situation when you have a good
query analysis.
And it has problems, as we discussed many times, it doesn't track
failing queries and so on.
It's another story.
But next step, we install pg_stat_statements, we see all queries,
well, top 5,000 by default, right?
It has pg_stat_statements.max parameter, which controls how many
normalized, aggregated, or official languages, normalized queries,
queries without parameters, are tracked.
But then, at some point, and actually I think many folks don't
understand it.
Honestly, I didn't feel it so deeply as I do it now, like last
couple of years maybe.
We track only part of our, of Postgres work.
We track only execution.
We don't track planning.
And planning can be a huge headache if workload is heavy and
machine is huge and so on.
So in heavily loaded projects, not tracking planning means like
you don't...
You can, like, very roughly, You can be in trouble in 50% of
production cases when you need to investigate what's happening,
what's causing high CPU consumption, for example.
But pg_stat_statements doesn't know, it doesn't see it.
It doesn't see the planning phase at all.
Only execution phase.
So it feels like very weird.
It's like we have, okay, we install pg_stat_statements, we tell
all people who have LTP install it, we know overhead is quite
low, and we will discuss it in depth in a moment.
But then we say, okay, actually, the pg_stat_statements extension
we actually installed, it's like half of it.
Half of the extension, because it's only about execution.
But planning is a super important phase and we don't have it.
So all folks who have pg_stat_statements, most of them, most
of them.
All folks who haven't changed settings of pg_stat_statements,
they have only part of solution.
Michael: Yeah, I would probably make the case that it's a bit
more than half.
Like I think planning in general is like a much lower proportion
of performance issues are planning related than execution related,
in my experience.
But I do take your point that it's not it's not 0 percent planning
related so even if it's 80 20 or 90 10 it's still there's still
a whole category of issues that we're not spotting by having
this off and I think there's a couple of extra things like that
I don't think it's just heavily loaded systems I've seen planning
dominated queries that are analytical in nature where it's just
somebody trying to diagnose why a single query is slow and that
has hidden as well if you're just looking at things like...
Nikolay: This is just-in-time comment actually.
Michael: Well, there's just-in-time compilation as well.
But, yeah, good.
No, but sticking to planning, I think there's a few confusing
things.
And I see this mistake made by all sorts of quite expert people
as well when they're they might be looking at a query a single
query plan and they say this query plan only takes 100 milliseconds
but there's 10 milliseconds of planning as well like they're
only looking at the execution time they don't realize that the
planning is in addition to the execution time, which they should
be saying is 110 or how they should be summing those 2.
And the same is true in pg_stat_statements, we should be summing
the 2.
And we can't sum the 2 if it's off or we can, It's just it mistakenly
tells us that there's 0 planning time where it's actually just
not being tracked
Nikolay: and Just a set once had only metrics Call like total
time to mean time and so on before Postgres 13 and since 13 it
was renamed it was renamed right it never tracked planning right
Michael: there was no planning it was only execution but the
naming made it clearer.
And on 13...
Nikolay: And 13 it was renamed to total_exec_time,
min_exec_time and so on and there is another group of metrics
total_plan_time and so on, min_plan_time and so on.
And it means that, okay, we have now the setting, but it's not
on by default.
And It means we have a second lesson to learn.
Should we enable it?
We say install pg_stat_statements.
Don't go without it.
It's the number 1 recommendation in terms of query analysis.
Always have pg_stat_statements settings.
I remember when it appeared, some rumors said, it adds 7% of
overhead, but it's worth it.
Well, as we can see now, it's actually very small overhead under
normal circumstances, but we had an episode showing, remember,
4 million TPS, showing that on edge case, overhead can be drastical
and drop performance.
It's like performance cliff.
But I doubt normal projects reach that point.
Everyone should be aware of that overhead, but again, it's a
cliff.
It doesn't come immediately.
It comes only when you reach a lot, a lot of queries per second
for a particular query ID, right?
Michael: Yes, and I think this suffers.
Tell me if you think I'm wrong I think the problem here is that
the easiest way to stress test Postgres or the easiest way to
do a quick benchmark is using pgbench yes which has this exact
Nikolay: it's not a lot of secrets immediately let's okay let's
talk about like we are like problem like it just says Edmonds
is number number 1 recommendation for query analysis.
Install it.
Everyone agrees?
Good.
There is some overhead.
We don't see it.
Like, it's not huge.
Okay.
Well, we saw it can be huge, but with some pathological workloads,
right?
And we will discuss it.
Now, question of this episode, basically.
Should everyone start saying, enable track_planning?
Should we, like, first default situation when Postgres doesn't
have anything and then we install extension, okay, everyone learned
this lesson.
It's kind of solved.
Should we consider the second lesson similarly and tell everyone
enable track_planning?
Because this gives you full power of producer statements now
Michael: Yeah my current opinion And I'll be happy to adjust
this as the episode goes on is I would enable this very early
in a project these days and Doing research for this.
I did come across some interesting things.
I didn't realize including why and when the default was made
off.
But yeah, I would say when you've not got high volume, high load,
turn it on while you've not got high load and then reassess if
you end up with a pathological
Nikolay: workload.
Which I
Michael: don't think you will.
Nikolay: I understand this approach And usually it's good.
For example, if we had some overhead which would grow monotonically
with...
Maybe linearly or somehow with our workload.
I would understand this approach.
Because take these weights and go to the gym with additional
weights all the time.
So you got used to it.
But in this case, it's very different.
We don't have any noticeable overhead, as we saw from benchmarks.
We don't have it and it happens only at very, very, very, very
extreme situations.
Let's talk about history.
I just learned about what you found right before our recording
so I was super surprised.
Tell us more about it, very interesting.
Michael: Yeah so 1 thing, like an easy thing I do in preparation
for most episodes is just check when was the feature introduced,
what was the commit, like what was the discussion around it And
I noticed it was in version 13 that it was introduced and I went
to the version 13 release notes to see what was said, kind of
as a high-level summary.
And the release notes now have these awesome little links to
the commits.
Even for old versions?
Well, 13 had it.
I knew they did it for the latest version and I was surprised
to see them in 13 but very pleasantly surprised so thank you
to whoever did that.
Like Nikolay: a portal, that's great.
Yeah.
So every item in RezNodes we can quickly trace discussion, commits
and so on, right?
That's great.
Michael: Yes, exactly.
Or you can go to the commits and from the commits you can go
to the discussions.
In most cases.
Yeah, in most cases.
But often new features especially simpler ones like this like
small like a new parameter they'll only have 1 commit this 1
had 2 which which piqued my interest immediately so I opened
them both and the first 1 was pretty pretty normal it made sense
that this was added.
And then the second 1, it was when I realized, oh it was during
the beta, it was during the beta phase for 13, somebody had reported
a performance issue with having track_planning on and turning
it off made the performance go back to exactly how it was in
12.
So basically they had a regression according to a synthetic benchmark
and then asked could we turn it off and there was it was pretty
unanimous I think it got a few replies and all the replies were
in favor of turning it off for the version 13 release and as
far as I can tell it's not been revisited since.
Nikolay: Yeah, so I now understand much better what's happening
here.
Let me like unwrap it here.
So what happened?
It was obviously, it would be obviously good if it was enabled
for all by default, right?
But then it was learned that there is performance degradation
45%.
Okay, it doesn't matter, actually, it can be 90% in some cases,
it depends on your machine and so on.
And it was related to spin lock contention in the results.
And this is exactly what we have recently observed in our benchmarks
using our AI workflow which we discussed also several times.
I wanted to say how I see our benchmark workflow for synthetic
benchmarks, pgbench and so on.
We built a lot, and we collect 80 artifacts for each run, very
comprehensive configuration, everything is stored, we can iterate.
And I see LLMs are just like kind of oil so eventually we should
have well oiled machine right engine and but the Main part is
not in LLM.
It's just it just makes it easier to iterate So we wanted to
check overhead and we checked it So pgbench which was also used
in like in this Research mentioned in the mailing list pgbench
by default is not limiting TPS But in normal circumstances under
normal circumstances in production, of course we don't, so this
is not just load test, it's a kind of load test which is called
stress test.
So we are on the edge, We are checking what's happening on the
edge.
And in production we don't have it normally.
You don't want to work at 100% CPU usually in OLTP because you
will experience various interesting phenomena like this basically,
right?
So spin lock.
And we checked it and like we took very big machines, I think
192 cores with 5 generation, fifth generation Google Cloud, still
spending Google Cloud credits, right?
Fifth generation of Xeon scalable Intel and almost 200 cores,
a lot of memory.
It doesn't matter because we also took small scale, scale 100,
that means 10 million entries in pgbench accounts only.
And we started, first we started not select only, no difference.
Like if you use read-write workload, you don't see difference.
But once you switch to select only workload, you quickly observe.
So what we did, and like in our show notes, we will have links
to detailed details, all details with all numbers and reproducible
with all pictures and so on.
So we start with just 1 client and 10, 20 and so on, 100 and
so on until 200 clients and we expect that we should basically
grow in terms of TPS, right?
Because more clients, more TPS, and Postgres these days scales
to many cores quite well.
But we quickly saw that we've enabled on this machine, quite
powerful machine, right?
We've enabled pg_stat_statements.track_planning
Peak is reached after 30, between 30 and 40 clients.
Very early.
And then TPS goes down.
And without track_planning, similar picture actually, but it's
later, until like 70-80, like 2 times to the right, 2 times more
clients and TPS also higher.
And once the peak is reached and we go down, so first the lines
are together, but once the peak is reached and we go down, we
see like kind of 2 times difference.
Peak is reached 2 times sooner And TPS is 2 times slower.
Interesting, right?
But looking at this, I was thinking, okay, why do we have peak
without track_planning reached much sooner than 196 cores, I
think, vCPUs?
Because normally it should be there.
How many cores we have?
This is our maximum, most optimal point of load.
There are nuances because we run pgbench on the same machine,
so we limit number of threads by 25% of cores, so we couldn't
have all CPU consumed by clients, basically.
So anyway, Then I recalled our February tests and we had a podcast
episode about it.
pg_stat_statements overhead, 4 million TPS.
We needed to remove pg_stat_statements to reach maximum workload.
I remember when we removed it, peak shifted to normal position,
closer to number of vCPUs we have.
So what does it tell me?
Okay, I'm thinking, oh, it looks like pg_stat_statements has some
significant overhead, but when we enable track_planning, this
overhead doubles.
And then, since our automation collects flame graphs, it collects
pg_wait_sampling analysis and so on, so we quickly identify
there are spin locks indeed, but the same spin locks are present
when you just use previous assessments.
This is the most interesting part, right?
So on flame graphs we see, and like, I think I should publish
a report.
If I have it already, please let's have it in show notes, link
to the report.
So in Flame Graphs we see that without track_planning enabled,
for this particular workload, we have spin lock contention, and
there is a very wide s_lock function inside pgss store.
pgss store is a function which saves metrics.
So what's happening?
Michael: To pg_stat_statements, that's the pass, right?
Nikolay: Yeah, yeah.
And like I would expect, like, I remember in February I expected,
like, I wanted to quickly reach 1 million TPS and go further.
And I know it was achieved by Alexander Korotkov in 2016, like
8 years ago.
So I was very surprised I couldn't reach it easily.
And only when I removed pg_stat_statements, I reached it.
So what's happening here is with pg_stat_statements used, not
track_planning, just pg_stat_statements, if you have such a weird workload,
like it's just a single query ID and it runs hundreds of thousands
of TPS for this, QPS in this case, queries per second, For this
particular machine, it can be lower.
Actually, we reproduce problems on 8 core machines.
This is super interesting point as well.
Yeah, yeah.
So this performance cliff can come to you on small machines as
well.
Michael: But wait, wait, how many concurrent connections?
I guess it comes sooner?
Nikolay: Yeah, well, it comes sooner.
I don't remember from the top of my head, but I know this problem
can happen sooner.
So the idea is, if we have limited resources and transactions,
these queries are so fast, like I think it was like 15 microseconds
or so, it's super fast for primary key lookup.
But they all fight to update metrics in a single pg_stat_statements
record.
And spin lock is required for that.
That's why we see spin lock contention.
So it's observer effect as it is.
Pure observer effect.
And pg_stat_statements can have observer effect, but you need to
reach a lot of queries per second.
Michael: Good point.
Nikolay: Because of limited resources, right?
So, yeah, and this means that in reality, very unlikely you will
see it.
Maybe, maybe, but unlikely.
It should be super fast index-only scan, for example, and the
frequency is so high.
But When you look at flame graphs with track_planning enabled
you see exactly 2 areas similar width Both are as log inside
pgss store and 1 pgss store is in execution phase and other pgss
store in planning phase So metrics are saved separately and if
you enable they are saved 2 times.
That fully explains what's happening here.
If we enable it we just move performance cliff 2 times closer
to us.
Well if we sit in 0, right?
If we shift it already, it's...
Yeah, so...
Michael: Well it explains why in your synthetic benchmarks you
got saturation twice as fast.
Nikolay: Yes, yes.
All pieces of the puzzle are like...
This took quite a lot.
We had 4 sessions of various kinds of research starting in February
with pg_stat_statements and idea, let's squeeze a lot of TPS from
our big machines.
But this is interesting, right?
Let's think about it.
So we have this pgss store.
Obviously, I think it should be possible to find a way to store
it just once.
I mean, to save it once.
Michael: Well, I remember you suggested that when we were discussing
privately, and since then, having read the discussions, it's
a deliberate design decision to save them separately.
And I didn't know about that.
So you mentioned a while back, we don't even track queries that
fail for example.
Well now in a way we are like because the number of times a query
was planned doesn't have to equal the number of times it was
executed anymore in pg_stat_statements.
Once you've got track_planning on you can see the number of times
it was planned versus the number of times it was executed.
Nikolay: Are you saying that if we have a failed quiz but it
was planned, planned time would be saved?
Michael: Yes.
That's my understanding from reading the...
I haven't tested this, but that's my understanding from reading
the discussions around the design.
Nikolay: Still I think it should be possible to optimize.
Yeah, I understand the design decision, but if we think about...
Michael: Either the design would need to change or the...
Yeah exactly but it's a trade-off and that's like I found that
interesting.
Nikolay: I remember cases, terrible cases which were very hard,
extremely hard to diagnose because pg_stat_statements didn't show
like for example merge joins which like merge join itself like
not merge join somehow like maybe hash join was used, but considering
merge join, planar spent many seconds, like 5 to 10 seconds,
just considering it.
Because it needs sometimes to check the table actual data and
see like min or max values for some columns, which is unexpected,
but planner sometimes does it.
And if it takes too long, planning phase, even if merge join
is not chosen eventually.
In the end, we don't see it, right?
Log manager contention, right?
Sometimes with JSONB values, we have something like planning
is long and consuming a lot of CPU, but we should see it.
We should see it, right?
And pg_stat_kcache, which tracks physical metrics and those people
who are, I don't know, like on self-managed Postgres can use
it and it's great.
It also has track_planning actually the same, same parameter,
right?
Michael: Oh no, off by default.
Nikolay: It's all by default.
Yes.
So now we understand the design decision and we understand that
some benchmark, which checked very, very like edge case, I would
say maybe even corner case because you have unlimited TPS and
you have select only single query ID it's edge it's corner case
not just edge we have 2 edges coming here it's a corner case
so likelihood of having it in production extremely low and decision
in my opinion decision was wrong what do you think
Michael: yeah I think I agree I can see why it was made I think
if you see it's quite a startling number if you if you look at
45% drop that was the number reported and shown.
I can see why people were scared of that, especially when it
was in the beta phase.
But in hindsight, especially with what I know now based on what
you've said, it seems like we'd be much better off having it
on for everybody.
Nikolay: 45% is roughly 50, right?
It's like 2x.
Michael: It sounds scary though, right?
Nikolay: It's reasonable.
If you are sitting on the edge, you are suffering from this penalty
pg_stat_statements already gives you.
You just double it, because pgss store is called twice.
That's why it's roughly 50%.
But the conclusion here is not, let's not enable by default.
Let's put a note in documentation that pg_stat_statements is
sensitive to the cases when it's a single query ID and a lot
of high frequent queries happening.
Michael: I think there is that note, yeah.
Nikolay: If there is, it's good.
Then track_planning just doubles this penalty and that's it,
but it's not happening under normal circumstances.
Michael: It's a really good point that it's not track_planning.
track_planning isn't the thing causing this.
The track_planning makes it twice as bad.
But twice as bad is quite a small multiplication factor when
you're talking about such an extreme case.
Nikolay: Yeah.
It would be like
Michael: if we imagine if we had twice as you know that the transaction
ID wraparound is a couple of billion?
Or a couple of billion in the positive.
If it was 4000000000 in the positive, that would make a bit of
difference, but not a huge, huge difference.
In these extreme cases, twice isn't that bad.
Nikolay: So I have 2 separate questions here.
Can we have a single pgss store instead of 2?
And second question, should we recommend enabling it to everyone
but understanding that doubles the problem of pg_stat_statements which
already is present, it's there already, but you just don't feel
it.
You don't feel it, for example, on that machine, almost 200 vCPUs.
You don't feel it unless a single query ID has like 200, 000
calls per second.
200, 000.
It's insane.
This is the edge.
It's super far.
And without track_planning, okay, it's 400, 000.
So not enabling it, you're just saying, okay, instead of 200,
000, we will have 400, 000.
But looking at your statistics, you see, okay, maximum calls
per second is what?
1, 000?
5, 000?
10, 000?
It's already a lot, right?
And usually we have different kinds of problems with such queries
which are exactly present during planning time because locking,
right?
It's already a problem.
Michael: Yeah, such a good point.
Nikolay: And I wonder, if we have a high frequent query, a single
query ID like 10, 000, primary key lookup in many cases, maybe
in most cases.
We say, okay, log manager can come after you, right?
Let's cache it.
I don't know, like prepare statements, or maybe plp.js.calc function,
indirect caching and so on.
Let's avoid planning.
Planning will be okay, right?
Michael: I haven't checked, but that would be an interesting
test.
Actually, no.
Yeah, you could do the same test again, but with prepared statements.
It would be interesting to see if track_planning is updated at
all.
Presumably it's not doing planning so it shouldn't do that.
Nikolay: That's a great point I'm going to check it because like
as I said with our almost well oiled machine I will just ask
to repeat experiment and with hyphen capital M prepared we will
see and basically it should have effect like we disable it but
we keep it enabled right
Michael: with a pool and maybe like with some way of keeping
the sessions Remember last time it was doing the first...
Nikolay: Yeah, I remember.
Michael: Would it need to be that or not?
I actually can't remember.
Maybe that would be fine in this case.
Nikolay: In this case it will be fine because the problem is
like it's vice versa.
pgbench maintains connection here.
What we discussed is that like in reality, if connection is not
maintained, we connect again and then we don't have cache.
So the connection, okay, I will double check this area.
It's good point.
And additional research to make the picture complete.
But my gut tells me it should be enabled in many cases, but we
should be aware of spin-lock contention, which can happen if
we have high frequency.
Honestly, I'm thinking to find some rule maybe to in monitoring,
for example, we know this machine has this number of vCPUs, and
we can quickly check how many calls per second, even if
track_planning is not enabled, we can check it how many calls per second
we have for most frequent,
Michael: Very high frequency.
Order by calls, descending.
Nikolay: Order by calls, exactly.
It's not hard.
Yeah.
So top 10 by calls, and how big is that?
And okay, this number of cores, this number of QPS, this level
of QPS, just roughly estimate how far we are from the performance
cliff.
Maybe it's possible to do this, right?
For modern...
Of course, it depends on many things.
It doesn't depend on the plan, I think.
Because saving happens only once.
It depends on duration, but not on duration, on frequency.
And of course, on the resources, type of processor you're using.
So I think it's possible to have some kind of prediction, how
far we are from this.
And if we are really far, let's enable track_planning.
This is my point.
Meanwhile, let's think maybe it's possible to save it just once
and reconsider decision-making.
I'm really thankful you made this research.
I liked it in my puzzle.
Like I still collecting pieces of it.
Cool.
So enable it, but it will double overhead and be aware of overhead.
This is the bottom line from pg_stat_statements.
Michael: Yeah.
If you ran a managed Postgres service, would you enable it for
everybody by default at the beginning?
Nikolay: Well, yeah, this is another piece of great info you
found.
We almost forgot to discuss it.
You found the...
So you found that many...
Well, it's not surprise to me.
You sent me the list of managed service providers and what's
possible what's not there it's not surprise for me that most
have it off by default surprise for me it was that Google Cloud
even doesn't allow to change it
Michael: not just Google Cloud quite a few of the providers...
Nikolay: Crunchy Bridge right?
Michael: So yeah yeah I is it's definitely not a it's definitely
not a full list, and I'm guilty of testing more of the ones that
make it easy to spin up new instances.
Nikolay: But let's drop some names here.
RDS or by default?
Michael: Should we go the other way around?
Is the other direction is the ones that do make it configurable.
RDS is I think still by far the most popular.
Off by default but configurable.
Bear in mind pg_stat_statements in RDS is on by default.
So they do change some of
Nikolay: the defaults
Michael: around this, for example.
Most of these provide, in fact, I can't remember the last time
I checked a provider that didn't have pg_stat_statements on by
default.
You mean
Nikolay: on by default, you mean it's in shared preload libraries,
or it's created in like template database?
Michael: It's, yeah, it's, it starts tracking queries without
you doing anything.
You don't have to create extension or whatever the normal thing
is to do.
Nikolay: I don't remember should we create extension.
Michael: Almost all providers, because they rely on it for their
monitor, like the monitoring they present you runs off
pg_stat_statements for most of these.
So it's on by default, but track_planning isn't.
That's true for almost all of the ones I tested on.
But I think Timescale...
Nikolay: Timescale is champion here.
Michael: Yeah.
Well, it's configurable but it's also on by default which was,
I think, the only one that I found that had it on by default.
There probably are others I didn't check them all.
It's quite time consuming.
Nikolay: That's great.
I love this decision.
Michael: Yeah, it's great.
Nikolay: But a
Michael: few of the others made it configurable, which is nice.
I just was surprised that at least one of the major ones doesn't
even.
Nikolay: Yeah, Google CloudSQL even doesn't allow to change
it and we...
You gave up trying to check Azure.
Michael: Yeah, I had limited time and the UI and I don't...
I need to talk to somebody there to teach me how to use it or
something because I was really struggling.
Nikolay: Yeah, I understand that.
So, yeah, so who can like, our recommendation is to for those
who are listening from those teams like CloudSQL, Crunchy and
Supabase guys like make it configurable.
It's not normal that it's not.
I would say consider making it on by default, but at least bare
minimum should be let users to decide.
Michael: I think so, and I think normally when these providers
change a default they'll change it for new instances only where
I think there's just so such low risk so I like that approach
of you know if people create new clusters or new instances, have
it on by default.
It will help people when they come across planning time related
issues, diagnose those issues much quicker, much easier, whether
that's on their own or once they pull people in.
Nikolay: Yeah yeah yeah that's that's that's for sure yeah so
good I hope we brought some food for thoughts to guys who are
listening to us.
So yeah, thank you.
Michael: Nice one Nikolay.
Nikolay: Yeah, thank you.
Michael: Take care, have a good week, bye.