Madelyn: I think AWS has made a lot of
progress and they really, I think they
have a much more nuanced opinion about
how they help open source projects.
And I, I, for one, appreciate it a lot.
Corey: Welcome to Screaming in the Cloud.
I'm Corey Quinn, and I figure it's
time to kick an open source beehive.
My guest today is Madelyn Olson, who
is one of the maintainers of Valkey,
which you may know as being a more
popularized historically through
its proprietary fork known as Redis.
Madelyn,
Madelyn: thank you for joining me.
Thank you for inviting me.
It's exciting to be here and talk
about my favorite topic, open source.
Corey: This episode's been sponsored by
our friends at Panoptica, part of Cisco.
This is one of those real rarities
where it's a security product that
you can get started with for free.
but also scale to enterprise grade.
Take a look.
In fact, you sign up for
an enterprise account.
They'll even throw you one of the
limited heavily discounted AWS skill
builder licenses they got because believe
it or not, unlike so many companies
out there, they do understand AWS.
To learn more, please visit panoptica.
app slash last week in AWS.
That's panoptica.
app slash last week in AWS.
Oh yes, I do want to give a little
backstory because it turns out that
there are people in this industry who
don't spend their entire time on the
internet reading about open source project
drama and license changes and the rest.
Uh, the high level, uh, For those
who, for the recap at the beginning
of the episode, for those who
remember, who don't remember what last
season held for us, uh, effectively
there was an open source project.
It was called Redis.
It got very popular.
A company called Garantia Labs was a
service provider around it, wound up
becoming Redis Labs, hired the creator
of Redis for a few years, bought the
copyrights to it, and last year, this
year, recently, wound up doing a license
pivot so that it is source available.
Citing that Amazon was going to
launch a Redis clone and destroy
them so they're not going to
be allowed to do that anymore.
Have I nailed the salient points?
Madelyn: Uh, yeah, that was the high
level points that are important to hit.
Corey: My observation on this
is it's a bunch of lunacy.
I think Amazon makes a
convenient boogeyman.
We should probably point out as well
that by day you are a principal engineer
at AWS, where you work on What is it?
You work half the time, I think.
How does it actually work?
I know that you're historically
ElastiCache, which needs to be
rebranded for ElastiCache for Valkey.
Like, I don't care what proprietary
fork it was named after.
Make it Valkey.
Madelyn: Yeah, so my day job historically
was working on Amazon ElastiCache,
which used to be the managed in
memory cache provider for, with
compatibility with Redis and Memcache.
I got started on working on Redis Open
Source about six years ago in 2018.
So since then, about half my time
has been on Open Source, the other
half has been on the managed service.
I want to point out as
Corey: well that there, first, you
have ElastiCache, which does, which is
a for sale implementation of Valkey.
Great.
You folks also sell MemoryDB, which is
less beloved for a variety of reasons, but
it's the less in memory, more persistent
datastore version of ElastiCache.
And I don't know what these
people are worried about.
You're going to launch a third service
that's going to try to compete with them?
Oh, good heavens!
The community has overwhelmingly
gone in the direction of Valkey,
when the fork was announced.
The Linux Foundation has given you
folks a home, as I understand it.
And, uh, It is now something that,
that this is effectively a schism in
the community, except the only people
really on their side of this seem to
work for the company called Redis.
Madelyn: Right.
So, you know, most of the, so before
the schism happens, there was five
maintainers of the Redis project,
two worked for non Redis companies.
So there was me from ADBS and
a guy named Xiao from Alibaba.
And then there were several
other very major contributors
from companies like Ericsson.
From Google and from Huawei, there's also
a very important contributor from Tencent.
And all of these folks all
just kind of went with Valkey.
None of them wanted to continue
working on a source available database.
So, you know, everyone's sort
of on this Valkey train now.
Corey: I have to back up here because
in 2024, I think the question has
to be asked, Does open source still
matter the way that it once did?
Madelyn: Open source definitely matters.
Open source is the place to solve problems
that, you know, it doesn't make sense for
all the different managed, you know, large
software companies to solve independently.
The reason that Redis was working so well
as an open source group is, you know,
AWS was willing to come to the table
and say, Hey, we have this big problem.
We have this feature we'd like to build in
the open source and everyone can have it.
And then we can have it
in our managed service.
You see this play out all over the place.
I take a lot of inspiration from
the Postgres community, which
is a lot of the same thing.
A lot of big tech companies all
come in and contribute their
features and it benefits everyone.
Like there's still a lot of value in this.
It's pretty hard for a company to
come in and say, Hey, I, you know,
we're going to build this thing
proprietary better than the rest of
the world can build it as a community.
I really think these like open groups are.
Historically, pretty good at building
certain types of open source of
software products, of which I
think databases is one of them.
Corey: I think that there have
been a few things that have
been conflated across the board.
The first is that open source
has been mistaken for a business
model by a bunch of folks.
And two, it was assumed originally, which
made sense at the time, that, oh, well,
we're the folks that built this software.
Clearly, we're the ones best positioned
to operate and run it for folks.
Well, cloud providers have turned that
a bit on its head, and I can understand,
on some level, it feels galling when work
you have done, effectively volunteering
for, now is being turned around and sold
for a profit, and you see absolutely
none of it other than representatives of
that company showing up in the various
community areas to make borderline demands
about feature requests and whatnot.
That can be irksome.
I get it.
But I don't think that Amazon
did anything wrong in this case.
I have a litany of things I think Amazon
does wrong, in case anyone is thinking
I'm somehow shilling for the company here.
But this isn't one.
The licenses have been very clear.
Open source means that any person
or entity can take, use, and in some
cases have to contribute the code back.
There's no carve out for, except
for these types of companies,
because that's not how it works.
That's no longer open.
Madelyn: Yeah, and that's really
in the spirit of open source.
Like, you know, sometimes we just
talk about open source licensing,
but there's also like free,
as in freedoms, open source.
That's the thing.
Anyone can go take the software
and do whatever they want with it.
And we see a lot of good come out of
these very free open source projects.
And Redis had originally launched as
not a very liberal open source license.
A lot of the reasons it's become
so popular is people can take it,
modify it a bit, to solve their
specific use cases, and, you know,
build it as, you know, build it as
a product, build it as a project,
like, build it as whatever they want.
So, you know, to your point, like,
that's why open source works so well.
It's like, people can do
whatever they want with it.
So, it makes sense that people would
kind of feel bad about, you know,
people doing not exactly what you
want with it, but You're right.
That's what the license is for.
Corey: An open source project
is not a for profit entity.
It is not built with the idea of how
do we monetize this thing directly.
If you want to start something like
that, it's usually called a company.
And the fact that, oh, well,
we've now done this thing.
How do we turn it into money is not really
a problem for the open source world.
Though it's increasingly becoming one
with a lot of these license forks.
I do not believe that these companies
are doing anything, uh, illegal.
I am not a lawyer.
Talk to one, please.
I, this is not legal advice, my god.
But they're not doing anything
even morally wrong, necessarily.
By, by, by switching these licenses over,
but they are violating an unspoken social
contract that has historically existed.
And I know that when a company does
something like that, but they have
other stuff that is open, I am a
lot less likely to contribute to
any of those things going forward.
If it has a contributor, a
contribute, a CAL, one of those
contributor authorization licenses,
I forget the exact acronym.
Madelyn: CLA.
Contributor License Agreement.
The
Corey: CLA, the
Madelyn: CLA's.
Corey: Yes, the contributor license
agreement, you sign copyright over
and the rest, and it's, okay, at this
point this feels too much like work
and you can start paying me to do it.
Now, I am not a good developer.
Frankly, that's a feature, not a bug.
Oh, Corey won't contribute
to the project if we do this?
Where do we sign up?
Not really the direction I'm trying to
take it in, but it's, uh, That's something
that I think is going to have a chilling
effect on the next generation of things.
I don't volunteer for for profit entities
and I don't let people volunteer for mine.
I am curious about Valkey and the
governance model going forward.
You work at AWS, obviously.
And I, I am sympathetic to the
viewpoint that, well, Amazon is
the worst possible spokesperson
for open source good behavior.
But, and I think a lot of the criticisms
that were levied at Amazon, well, yeah,
it doesn't feel great when the multi
trillion dollar company is monetizing
everyone else's work as services.
They have started doing a lot of
the work over the last few years.
To the point where those
accusations feel increasingly dated.
What I find curious is when I start
talking to random folks who are working
on open source projects of varying
sizes, they'll say, oh yeah, we're
getting sponsored by AWS, they're just
giving us a bunch of credits to run
the infrastructure for the project, and
they're asked whether they're having
developers contribute stuff to it.
Amazon has never been particularly
good at telling its own story.
They're Their marketing mumbles.
So, and I don't know how to fix
that other than every time we give
1, 000 to a project, we're going to
spend 10, 000 more telling the world
that we did it, which is sort of
the wrong behavior pattern as well.
So I am sympathetic to this, but I want
it to be clear since I've, I've done some
digging into this that I assure you I
would not be saying this were it not true.
Valkey is not an AWS project.
They are sponsoring the project in
the form of I don't know, paying
your salary to work on this thing,
but they have no ownership rights.
The, the, those have been given
to the Linux Foundation, which
is a separate kettle of debate.
We can have a different time.
They're doing the right thing
here, as best I can tell.
Madelyn: Honestly, I'm really
happy with the way AWS has been
enabling the Valkey project.
As you said, they're primarily
sponsoring it through, you know,
supporting me so that I can spend
my entire time on the project.
I mentioned earlier, I spend
about half my time on Redis,
I spend all my time on Valkey.
And they've also been doing a lot
of the right things with like, you
know, helping us get to, we went, we
visited the Open Source Summit, uh, in
Seattle recently, they helped provide
the mechanics that we could show up.
And they were very good at taking a
step back and being like, hey, like,
this isn't an AWS project, we're
just helping you get into the room.
As you said, like, This is
a Linux Foundation project.
This is not an AWS fork.
AWS does not own the IP
or any of the licensing.
This is all in the Linux Foundation,
which is a non for profit company or
organization to help open source projects.
Balky is an open source project and
it is very much community driven.
There is a six member technical
steering committee, of which only
one person is in from AWS, myself.
And they're doing a really good
job just, you know, being there
to help the community, but not
trying to step in and direct the
community and telling it what to do.
And you're definitely right that I
think AWS has made a lot of progress
and they really, I think they have a
much more nuanced opinion about how
they help open source projects and
I, I for one appreciate it a lot.
Like I, I'm only working at AWS because
of the way they've helped me, you know,
You know, work on an open source project.
Corey: I have to ask this question
because I, I feel like it almost
becomes counterproductive at
some point when companies do it.
But every open source project that,
that goes through one of these
license switches, there's been a
smell around the project for years
beforehand, before that shift happens.
And what that manifests as to me is
when there are a number of features That
are not brought into the open source
project because they would explicitly
compete with one of the sponsoring
company's financial, uh, goals.
And I think there's an analytics
offering for, uh, for Valkey,
which, for Redis, technically,
which has never been open sourced
as a result as one example of this.
Madelyn: Yeah, so there's actually a
lot of features that people have asked
for from Redis that we got a lot of
pushback for from, you know, because
there was a technical steering committee.
We got a lot of pushback.
Because they, you know, officially
the comment was, Oh, it's not, it's
too complicated, we don't want it.
But it's possible that that was part
of the reason, this sort of like
pushback from the corporate competition.
And I can say that now with Valkey, it
will become much easier to contribute
a lot of these big, exciting features.
One of the big things we're working on
is, you know, active active replication,
which was a big feature that people asked
for Redis that got pushed back from.
There's also a lot of interest in a lot
of the extensibility stuff that Rez has,
like JSON support, like search support.
And we are, you know, engaging with
the community to figure out, you know,
what's the best way to build that?
How do people want that?
And we're trying to, you know,
make the Valkey project more of
what We think the community really
wants it in a way that, you know,
the res community didn't want to.
Corey: I am personally excited about
the active active replication piece.
In my last real job, uh, seven years
ago now, uh, the one time that I got
paged on a weekend because I care very
much about not having my family time
disrupted was when it was due to an
ElastiCache challenge where I don't
believe there was anything that AWS
themselves had done, but we were pointing
at one of the replica members instead
of the virtual IP post maintenance.
Then there was a changeover where, oh,
time to take the, you folks took the
thing down for maintenance, which you're
entitled to do, and suddenly you could
no longer send writes to that thing.
Which meant that a whole bunch
of stuff that we had broke.
Now there were a number of contributing
factors to this, but on some level, if we
just had the one endpoint we could have
pointed to that always would have been
the, this is where writes always go and
reads and it's fine, it feels like that
would have solved some of the problem.
We would have been pointing to a cluster
member if everything can take writes.
That would have been glorious.
Uh, past me would have appreciated it.
Madelyn: I will say, ValkeyCluster, the
topology discovery should enable you to
figure out what the correct writer is,
even if you're talking to the wrong node.
So it sounds like you were using the old
Valkey standalone type of configuration.
Corey: This was 2015.
So it was one of the, 2015 2016, and
it was on a, I think a Older version
that was running in ElastiCache.
I, I, the details have escaped me.
Again, there were many of misconfiguration
that were done internally.
This was not something
I'm blaming you folks for.
Like any reasonable issue,
there are going to be a number
of contributing factors to it.
It's not, Oh, Ted was dumb.
Ted is always going to be dumb.
How do you make sure that one person being
dumb doesn't cause larger challenges?
Madelyn: Oh, I fully agree.
And I will say, so at 2015, that was
about the time that Redis released
cluster mode, which does solve this
problem, but you As you said, at the
time, you probably weren't using it.
So today, hindsight, well, even
hindsight, it didn't really, it
wasn't really production ready yet.
Corey: No, I, I would argue most of my
code has never become production ready,
but that's a separate argument and, you
know, personal attack stuff on myself.
Do you find it strange to be effectively
spending all of your time doing open
source work for, that is done in the, in
the public eye, uh, while Having a full
time job at a company, because every
time I've done stuff like that, uh,
when I've been at more normal jobs than
this, and I spent significant amounts
of time working on open source projects,
it felt like I was slacking off somehow.
Like I may as well have been playing
World of Warcraft or something
as far as my boss was concerned.
Now, these are different times, and
AWS is a more enlightened company
than that, and clearly you're
not doing something wrong here.
But does it ever feel weird to you?
Madelyn: I think it used to feel weird.
It took me a lot, like I was
a big, I got into the Redis
community trying to contribute TLS.
And so at the time there was alignment
between what AWS wanted and what
the open source community wanted.
And so I've always found that to
be like the secret sauce to being
able to work a lot in open source.
You just have to convince, you
know, the leadership, like, hey,
this contributing to open source is
aligned with your business interests.
And then it becomes so much easier to
just kind of go work in open source,
because, you know, the better the original
thinking was, the better that Redis was,
the better Amazon Redis was, and the same
is true are going to be about Valkeyrie.
You know, we want Valkeyrie to be as
good of a product as possible, because
Atomis has indicated their intent to
support it, and we would like to make
it as good of a product as possible.
So, you know, I also do have
a, I do tend, I work a lot.
I'm a workaholic.
I work like 50 hours a week, but I
try to, during nine to five, I try to
make sure I'm working on the important
parts, like the real job stuff.
And then the other 10 hours I spend
a bunch of time, you know, making
off brand Valkey stickers, which,
you know, is part of the important
for the community, but maybe not
exactly the top priority for Amazon.
Corey: Few things are better for
your career and your company than
achieving more expertise in the cloud.
Security improves, compensation goes
up, employee retention skyrockets.
Panoptica, a cloud security platform
from Cisco, has created an academy
of free courses just for you.
Head on over to academy.
panoptica.
app to get started.
The wisdom that I heard from someone
somewhat recently, which I wish I'd
heard decades ago, was that if you're
one of those people that can't walk
away from work, make sure that the
things you do outside of the core hours
are things that give you energy and
that you really enjoy doing, because
otherwise it just becomes miserable.
Madelyn: I follow that to a T.
Like I, like outside, as soon as like
five o'clock rolls around, well a little
bit earlier, I'm a morninging person.
But like, as soon as like 4 o'clock
rolls around, I kind of like go read
esoteric database papers about improving
hash table performance by like 2%.
That's what makes me happy.
Corey: That would be significant at scale.
It would be.
Yeah, I mean, you're also, I'm speaking
to someone who, honestly, one of the most
valuable things that I do for the business
from time to time is fire up my Twitter
client, which is insane, but also true.
I have to ask, though, my
negative reaction to Redis going
source available instead of its
historical open source roots.
has been negative, but that negative
reaction has been largely emotional,
which is usually not a reason to go
ahead and do various business things.
So, you know, from a more objective
perspective, other than offending
my personal sense of rightness
here, why did you create Valkey?
Madelyn: As I said, alluded to a little
bit earlier, the res community had
been very technologically conservative.
They've been very slow moving, they
didn't want to implement a lot of
features, they were very hesitant to
like increase the number of maintainers
to like allow for more contributions.
And there's actually a very big
backlog of features, like the active
active replication we talked about,
that no one was really contributing.
And now that we have Valkey,
like, I'm very intent to sort of
grow the number of maintainers.
We have a lot of contributors.
We've had, you know, 52 ish
total contributors so far to
Valkey in the last two months.
And we're hoping that we're going to have
a much higher velocity of development.
And, you know, as I said, a
lot of the major contributors
moved from Redis to Valkeyrie.
And like, that's where we really see our
differentiation kind of coming along.
We have a lot of features planned.
We're planning on building significant
performance improvements, things
that Redis has long said they don't
really want to mess too much with.
Uh, we also have like big improvements
to stability and reliability.
We also want to build all those
extensions we talked about.
We think there'll be a
lot of functionality.
And the other thing is just, it's
going to be, it's like sort of, We're
hoping it, we're hoping for it to
become basically the standard, right?
We're looking, there's a huge amount
of support from other companies.
We have so many supporters of Valkey, like
Ivan, and Percona, and GCP, and Oracle,
and Heroku, and Ericsson, like there's
all these companies coming behind it.
Corey: And these are
companies that normally do not
collaborate much on things.
They are, uh, I don't know what the
corporate equivalent of sworn blood
enemy is, but, but they're that.
Madelyn: Correct, yeah, like,
they're not typical companies
that like, like to get together.
And so we're really hoping,
like, all that will come together
just to, to be a better product.
And at the end of the day,
that's what people want.
They want something that works
well, that's highly available,
that's highly performant.
Like, that's what, that's
what Valkeyrie's gonna be.
Corey: Yeah, one of the ways I've always
viewed it has been that if, if I'm
working on something that other people
can take advantage of in their workplaces,
terrific, great, awesome, I have no, I
have nothing against open source, the,
at some point, though, if I'm starting
to implement things specifically for
you and your, your use case, okay,
great, now it's time to probably pay
me to do those things, because, you
know, At that point, it is no longer
something that benefits everyone, it's
something that benefits you individually.
I periodically find developers behind
open source projects that I'm using
if I need something specific done for
me, and I'll pay them for that thing,
but invariably, as that happens, there
will be pull requests made to the,
uh, made to the upstream project, and
sure enough, a lot of those things get
baked in, and this is a great thing.
Like, I feel like sometimes they're
surprised that I take that perspective,
it's, If I'm sponsoring upstream
development from this, terrific.
Or if you're doing it separately,
great, that's fine too.
Please, keep going.
That's the whole reason that
we have an open source culture.
We all stand on the shoulders of giants.
In fact, you're one of those
giants for a lot of people now,
whether you like it or not.
Madelyn: Yeah, I fully agree with that.
Like, I'm like one of the reasons
that I'm, you know, I'm an engineer.
I'm an introvert.
I don't, you know, I like talking
with people, but if I could have my
way, I'd be hiding most of the time.
But we're trying to find people
like that people that, you know,
want to contribute to Valkeyrie.
And like, if you have these things
that, you know, if someone's
proposing, you know, a minor
improvement, that's very specific
to a niche use case, we'll probably
still accept it, we still want it.
Someone else somewhere in the
world probably, you know, would
also like that improvement.
So that's what I really love about
open source is there's all these,
you know, individual niche things
that sort of come together to just be
better than the sum of their parts.
Corey: One of the challenges I've had
with open source and one of the reasons
I stepped away from a lot of it has been
the first, the untitled, the unsettled
expectation that everyone has on it.
For example of, great, well, I'm using
this in production for my business
and there's a problem with it.
So you wake up and fix it immediately.
That's not a great attitude to
take toward these things, for one.
And for another, it really does
become this obnoxious thing where
you have so many folks who are, I
guess, caught up in a variety of
different challenges and going on.
It always seems like it's a PTA meeting.
When it comes to open source governance,
where everyone's arguing with each
other constantly, it becomes more
work to keep the project functioning
than it feels like value is getting
derived from it from time to time.
And that's usually my cue
that when I start feeling like
that, it's time to walk away.
How do you avoid that problem?
Because clearly, you're still way into
it, and I have backed so far away,
people are often surprised to learn
I have a history with open source.
Madelyn: I am a very tenacious person.
So the history with me and TLS
that I mentioned earlier, like that
was my big, big, big contribution.
It took me two and a half years.
to get TLS into open source Redis.
And that was just because
I'm a very stubborn person.
Like, I am gonna keep trying
to make something work.
And you know, I've had a lot of
frustrations with getting stuff
into the Redis open source after the
transition to the open governance.
And, you know, Valkeyrie's only been
going along two months and we're still a
little bit in the honey, honeymoon phase,
but like everything is great because we
have all these features we want to build.
Everyone's sort of like, You know, we
have all these things that everyone kind
of all agrees these are good things.
But we're trying to build a system so
that it works well in the long term.
We have a contributor summit that's
going to be next week, June 6th and the
7th, where we're going to get all of
the kind of core maintainers together, a
lot of the major contributors together.
And we're trying to, you know, get all
together to figure out like, how do we
maintain, how do we make this a project
that's maintainable for the long term?
And I don't have a lot
of answers right now.
I mentioned, you know, before we started
chatting that I'm actually at pgconf.
dev to sort of figure out how
Postgres is sort of, you know,
not tearing itself apart.
And I'm still not entirely sure how
we're gonna not tear ourselves apart.
But I'm, I'm hopeful.
Like, I believe that if you are
willing to lean in and invest in
an open community that you will,
you will get returns back from it.
It's just sort of the thing you have to,
you have to kind of have some faith for.
And, um, I guess I'm just an
internal optimist, and I'll keep
hoping that it'll work out well.
And it's worked out well so far,
so I'm gonna just keep hoping.
Corey: One last question I have for
you on this, and if you can't answer
it, I completely understand that.
But I have to imagine that there
is some sort of internal process
that winds up taking whatever the
latest released version of Valkey
is and turning it into ElastiCache.
There's a whole bunch of stuff that
has to happen to Amazonify the service.
There's probably a security
review, et cetera, et cetera.
It's not just a press the
deploy button, go to lunch and
assume everything is great.
How do you separate out whatever,
whatever hooks into the upstream
project would make a lot of that
stuff easier for your employer?
from, from what you're doing
on behalf of the project.
Not necessarily that there are
inherent misalignments there, but
there's always going to be a choice
at some level of do we do this in a
way that's easier for the community
versus easier for Amazon themselves.
How do you, do you have
a framework for that?
Madelyn: Yeah, so I mean that's normally,
when people say they're like wearing
two hats, so like I have my Amazon hat
and I have my open source hat, and I
try to just make it very clear that
when I'm stating an opinion, it's for
one of those two, like, positions.
When I'm accepting something
as a maintainer for Valkeyrie,
I'm wearing my open source hat.
Like, is this a good
contribution for the project?
And I'll make it clear that when I'm
advocating for a feature in open source
to the other maintainers, that I'm
saying, hey, on behalf of AWS, this
is a feature I think would be useful.
And that's it.
You know, in an abstract sense, that's
this pipeline you talked about, you know,
makes Amazon a customer, in a sense.
We like to talk about customer obsession.
We're a customer of the
open source project.
So we do have a valid claim to be
like, hey, we would like this feature.
And so, and then so do all the other
IT vendors that have Redis services,
or hopefully soon Docky services,
and they can go say the same thing.
We were talking a bit recently in
open source how there's, you know,
a certain type of, uh, manageability
that would be nice to have.
And all the managed services
were like, yes, we'd all kind
of like this functionality.
Um, and even though it's, you know, pretty
much a managed service specific feature,
it's still, since everyone's still
willing to sort of like work and it's
like the community buys into the feature,
it still, you know, works out well.
So as long as you're just making that
distinction very clear, I haven't had
too much issue, you know, with people
being concerned that I'm like pushing
something on behalf of AWS or not.
Corey: I do confess to having been
surprised to have learned throughout
this drama unfolding that ElastiCache
is based on the open source code base.
What I, I guess what I had assumed is
that on some level when you operate at
a point of scale that Amazon does, where
you no longer really need to have all
of the affordances for all the disparate
operating environments that an open
source project might get used within
because you can control all of that.
It's similar to my criticism historically
of Amazon crowing about Graviton for RDS.
It's like, it's a SQL endpoint.
As long as you respect the SQL API, And
return the results that are expected
within a price performance window.
I don't care if it's Intel,
Graviton, Magic Elves, or
whatever else is under the hood.
Just make it work.
I just want the behavior to be fine.
It could be a black box beyond that.
I don't know whether it's the actual
open source upstream database engine
or a complete re implementation of
it in ways that result, create the
same results that take advantage of
existing substrates within AWS itself.
I was surprised to learn that it is
largely an open source, the open source
thing that you would expect running
with relatively minor modification.
Madelyn: I don't know if I
have much to say about that.
Corey: It's true.
No, sorry, I I, I don't mean to
put you in a position where you
can't, but I was, I was very
surprised just from the perspective
of why would AWS care about this?
Was, was where it led, where it
led me to, and it's no, because we,
we do use the open source thing.
This is, that is important to us.
And I've talked to a number of
engineers, not just you, who, for
whom the open source aspect is
incredibly important, and it's one
of the reasons they work at Amazon.
Madelyn: I, I mean, you know, the open
source Redis product was, was pretty good.
It solved a lot of our use cases, and
so, you know, we, we found it beneficial
to use within our managed service.
You know, the more you basically diverge
away, like if you have to re implement
an open API with a proprietary system,
it takes a lot of effort to maintain.
Like every time there's a new
piece of functionality, it might
kind of break the way that you're
thinking about the service.
And so there's a lot of benefit to just
staying very current to open source.
Um, if you're basically providing
like an API compatible endpoint.
as a service.
Corey: Yeah, and API compatible
always needs an asterisk next to it.
Every time I talk to a company
that says, Oh yeah, we are a
100 percent S3 API compatible.
It's, are you really?
I'm betting you're not.
And they're like, yes, we are.
And then a quick run of a test suite
later shows, No, no, you are not.
And that's, you need to
be bug for bug compatible.
You need to be able to support really
weird things that no one knows exist.
And the fact that you're not in most
of those cases is fine, but be upfront
about where those divergences are.
Yeah.
I honestly should have figured this out
years ago just because either you are
using a lot of these open source things
directly or you have a preternaturally
good re implementation team.
Just, yeah, let's re imagine this and
make sure all these weird behaviors
work exactly the same way at scale.
I wish that people were that dark
wizardry there, but in practice, nope.
I don't think so.
I think it's just what you see
is pretty much what you'd expect.
Madelyn: Yeah, it's a lot
of work to reimplement it.
When there's an open source piece
of software right there, you can
just take off the shelf and use.
The great part about open source.
Corey: So I have to ask this.
I'm sure people are going
to wonder about this.
By people, I of course mean me.
How different is Valkeyrie from Redis?
Is it effectively a one to one clone
of the exact existing source code at
moment of fork, and then effectively
reconstituting the exact same governance
process, release cadence, etc, etc, etc?
Or are you taking this opportunity
to change things up about it?
Madelyn: We actually are trying
to keep as much as possible.
There was a couple of major forks in
the community that were doing a bunch
of little changes, and we sort of took
a very strong stance that we want to
keep everything as similar as possible.
So the official stable
release of Valkeyrie is 7.
2.
5, which is basically a
complete fork of Redis 7.
2, with a couple of
extra compatibility APIs.
So Redis, the development tree was
actually a good bit ahead of that.
There was about another six
ish months of development.
So we also forked all of that, and that's
going to go into our Valkey 8 release.
So that's sort of the compatibility
that we're sort of seeing.
And besides that, we kept the
governance pretty much the same.
In Redis, there's a five member core team.
We now have a six member
technical steering committee.
We are keeping a lot of the
nuances of Redis's design.
The same as well, we plan on continuing
to do the even number of releases.
Redis was sort of known for doing like 6.
We'll also do, you know, if
we were going to do another 7
release, we would have done 7.
4, 8.
2.
But we are changing some things.
Like, you know, one thing that we added
unit tests, which is a kind of a minor
unimportant thing, but it's something that
we never had before in open source Redis.
But that was one of the things we added.
We added a formatter because we finally
got sick of people nitpicking, being
like, you should put a space here.
But we could never get
consensus before to add one.
So there's some of that type stuff.
And then, you know, the, the
culture of pretty technologically
conservative, um, being
technologically conservative as well.
Like we're, we're, we're pushing
a lot more into the performance
realm than we were kind of before.
And we're willing to rethink a lot of
the core assumptions of that REST, which
I think we weren't willing to do before.
Corey: It's always a tricky balance.
I'm extraordinarily conservative when
it comes to file systems, databases,
security, fire safety, you know, the
things that are going to have massive
repercussions if they go wrong.
I'm willing to play around in my
dev environment with all kinds of
ridiculous nonsense, but when it
comes time to, and now, by day job,
back when I worked at a giant, Add
the world's largest asset manager.
It's okay.
Mistakes on this stuff.
We'll absolutely show why
don't we act like the grownups?
We, at least in my case, pretend to be
so there's always a tricky balancing
act, but you can't hold still so long
that everything becomes ossified.
There are no right answers.
Just a question.
How do you want to be wrong?
Madelyn: Right.
I mean, we're still struggling to figure
out kind of where is the balance of that
is a lot of people put Redis is like,
that's like their main data endpoint for
like, real time like end user applications
where if there's any degradation like
it's very obvious that something's wrong
so we're taking that very seriously but
at the same time like other databases are
you know you know there's like 10 years of
innovation like hash tables and tcp pack
like processing and network processing
like all that stuff we really haven't
added much to reddit and you know we
can't We don't want to just sit around.
There's other forks of rice that are also
starting to invest in all this technology
and like they're going to do it too.
So we, we need to sort of
be on top of all of that.
So it's hard to balance that.
Um, but we're still
trying to figure it out.
Like the best defense we have
is, you know, lots of testing.
We have a lot of really smart
people that we have on the project.
Um, and we're really hoping we'll be able
to, you know, build a really robust, you
know, test suite to make sure everything
is still working exactly as it should be.
You know, bug for bug compatible.
We're really good at not breaking APIs.
Like, one of the big things Redis does,
like, basically never deprecated APIs.
They never removed APIs.
And we kind of intend
to do the same thing.
Corey: That's very Amazonian of you.
Madelyn: I know.
Yeah.
Never release a new version.
Keep maintaining stuff forever.
Corey: Ugh.
No, it's I'm looking forward
to seeing how it plays out.
It's definitely one of those things
that is going to be of significant
interest to an awful lot of people.
If people want to keep an eye on it
or participate themselves, where's
the best place for them to find you?
Madelyn: So yeah, we have, you know,
Twitter, you can follow me on Twitter.
Uh, you can follow, there's
a Valkey I O Twitter.
We also, I'm very active
on LinkedIn at the moment.
There's a lot of updates.
Uh, you, you should expect an
update in a couple, couple of weeks.
I mentioned a contributor summit.
We're hoping to basically say, Hey, here's
the six to 12 month roadmap for Valkey.
It's still a little bit
tentative right now.
I think in a couple of weeks, we'll
have like, Hey, this is the thing.
And it'll be obvious.
You'll find a bunch of posts about,
there'll be a blog post about it.
There's officially a blog at falki.
io.
You can also go check out that.
That should include a lot of the
relevant up to date information.
Um, other than that, GitHub is
a great place if you're looking
for, you know, the latest release.
Yeah, if you're interested
in contributing, that's a
good place to go as well.
Corey: And we will, of course, put
links to all of that in the show notes.
Thank you so much for taking
the time to speak with me today.
I appreciate it.
Thanks for spending the time.
It was a great conversation.
Madelyn Olson, Valkey Maintainer and
Technical Steering Committee member.
I'm Cloud Economist Corey Quinn,
and this is Screaming in the Cloud.
If you enjoyed this podcast,
please leave a five star review on
your podcast platform of choice.
Whereas if you hated this podcast,
please leave a 5 star review on your
podcast platform of choice, along with
an angry comment in which you replace
the title of the episode with your name
and claim that you created the episode.