There's another kind of interesting decision here on Dropbox by
design was always like a sidecar.
It's always something that just sits and it looks at your files.
Your files are just regular files on the file system.
And if Dropbox, the app isn't running, your files are there and they're safe,
and it's something that you know, regular apps can just read and write
to, and in some sense like Dropbox was unintentionally local-first
from that perspective, right?
Because it's saying that no matter what happens, your data
is just there and you own it.
Welcome to the localfirst.fm podcast.
I'm your host, Johannes Schickling, and I'm a web developer, a
startup founder, and love the craft of software engineering.
For the past few years, I've been on a journey to build a modern high quality
music app using web technologies, and in doing so, I've been following down
the rabbit hole of local-first software.
This podcast is your invitation to join me on that journey.
In this episode, I'm speaking to Sujay Jayakar.
Co-founder of Convex and Early Engineer at Dropbox.
In this conversation, Sujay shares the story on how the Sync Engine
powering Dropbox was built initially and later redesigned to address all
sorts of distributed systems problems.
Before getting started, also a big thank you to Jazz for supporting this podcast.
And now my interview with Sujay.
Hey, Sujay.
So nice to have you on the show.
How are you doing?
Doing great.
Great.
Really happy to be here.
I'm super excited to have you on the show.
I've been using your work really since over a decade at this point
when I was really getting into using computers productively.
And we just the other time had another really interesting guest, Seph Gentle on
the podcast, who has worked on a really fascinating tool, called Google Wave
back then that had a big impact on me.
And you've been working on another technology that had a big impact
on me, which is Dropbox and still has a very positive impact on me.
That was all the way back then over 10 years ago in 2014.
I don't think I need to explain to the audience what Dropbox is, but, I
want to hear it from you, like what led you to join Dropbox, I think very
early on and just hearing a little bit just embedded in your personal
context when you joined it, and then we're gonna go dive really deep into
all things syncing related, et cetera.
How does that sound?
Yeah, that sounds great.
It's actually a really funny story.
my career here in technology started in 2012.
I was actually studying mathematics.
I was going to go work at the NSA doing cryptography, and I was born in India.
but I'm a naturalized citizen for the United States, and, you have to
be, have security clearance to go do these types of cryptography things.
And you know, my clearance kept on dragging on and on and on and
they like interviewed my roommates and apparently just a very sketchy
guy so I had an offer to go work there, but it kept on dragging on.
And then my roommate, at the time was a computer science major who wanted some
like someone to go with him to the career fair and, just started chatting with the
Dropbox people and you know, it's about like a hundred people around that time.
And, chatting turned into hanging out at dinner, turned into interviewing
and being a math person, I did my interviews all in Haskell and
didn't know any real programming.
and then yeah, that turned into doing an internship, dropping out of undergrad and.
just following the dream.
And so I worked on, at Dropbox, I worked on a bunch of things.
I started off working on our, like, growth team.
So I did a lot of like email system.
Like I did this, I worked on this thing called the space raise, like a promotion.
Oh, I remember that.
Yes.
I think I've, I've earned quite a lot of like free storage, which I think
over the time has like gone down.
But that was a very smart and effective mechanism.
I surely invited all my friends back then.
I couldn't afford a premium plan being a broke student that worked.
And then from there worked on the sync engine for some time.
And then right now I'm the co-founder and chief scientist of a startup called Convex
and my three co-founders and I met working on this project called Magic Pocket,
where Dropbox stores hundreds of petabytes now exabytes of files, for users.
And we used to do that in S3.
And so the three of us worked together on a team to build Amazon S3, but in-house
and migrate all of the data over.
so we did that for a few years and then, Worked on rewriting the entirety of
Dropbox, the sync engine, the thing that runs on all of our desktop computers.
we rewrote it to be really correct and scalable and very flexible.
and shipped that.
after that left Dropbox in 2020 I was trying to decide if I wanted
to get back to academics or not.
So I did some research in networking and then decided to start Convex in 2021.
Certainly curious, which sort of research has had your interest the
most in this sort of transitionary per period, but maybe we stash that
for a moment and go back to the beginning when you joined Dropbox.
you mentioned there were around a hundred people working there currently.
how do I need to imagine the technology behind Dropbox at this point?
it clearly started all out with like, desktop focused like daemon project,
like daemon process that's running on your machine somehow keeps track of the files
on your system and then applies the magic.
So explain to me how things worked back then and what was it like to
work at Dropbox when there were around about a hundred people.
Yeah, I mean, it was pretty magical, right?
Because the company had, I think gotten so many things right on the product side
and then those showed up in technology.
But just this feeling of like Dropbox being this product that just worked right?
It was for everyone.
It was not just for technologists, but anyone should be able, anyone who's
comfortable using a computer should be able to install Dropbox and have
a folder of theirs become magical.
And without understanding anything about how it works, they should
just think of it as like an extension of what they know already.
yeah.
And so like the ways that that showed up I think were really interesting.
At the time there was a very strong culture of like reverse engineering.
So to have this daemon that runs locally.
You know, there was one of the amazing early moments in Dropbox was that,
if like you open up finder or explore and you have the overlays on it.
Like that used to be done by like attaching to the finder
process and injecting code into it
and to the point where, uh, when some folks had gone to talk to Apple at the
time and about like working with the file system and everything like the,
there were teams at Apple that asked Dropbox, how did you do that in Finder?
So you wanted to offer the most native experience.
There weren't the necessary APIs for that.
And so you just made it happen.
That's amazing.
Yeah.
And so that that idea of like, how do you create the best user experience,
something that you know, for the purpose of making non-technical users feel very
confident and feel very safe using it.
That was another, I think, really deep like company value of like
being worthy of trust and taking people's files very seriously.
You know, I like remember having a friend who was in residency at the
time and he was telling me that he keeps all of his, like some of his non
HIPAA stuff, but like his things that he looks at on Dropbox and you know,
pulls him up and he's consulting 'em.
And there's a part of me which is terrified by that, right?
Like we think of software as something where like throwing a 500
error is fine every once in a while.
And a Dropbox that was there was a culture of making users feel
like they could really trust us.
And then that showed up for things like making sure that, like when
we give feedback to users, if we put that green overlay in finder.
They know that no matter what happens, they could throw their laptop in a pool.
They could like they, anything could happen and their files are safe.
Like if their house burns down, they don't have to worry about that thing.
And that's like all of that reverse engineering and all of the emphasis
on correctness and durability.
It was all in service of that feeling, which I think was really cool.
so on the engineering side, at the time it was like in hyper growth mode.
So they had a Python desktop client.
Almost all of Dropbox was in Python at the time.
And so there's a pre my py, like big rapidly changing desktop client that
needed to support Mac, windows and Linux and all these different file systems.
and then on the server, it was like we had one big server called Meta Server,
meta, I think was from metadata.
and that like ran almost all of Dropbox.
We stored the metadata in MySQL.
The files were stored in S3, and then we had a separate notification server
for managing pushes and things like that.
And so it was like kind of classic architecture and like reach was
starting to reach the limits of its scaling even at that time.
And, those were a lot of the things we worked on over the next 10 years.
Wow.
So was the server also written in Python?
So it was all one big python shop.
Yeah.
And the server was all written in Python.
we, had some pretty funny bugs that were due to it's kind of
crazy to think about it now.
You know, we, you working in TypeScript and full time and to think of, like
back in the day we just had these like hundreds of thousands, millions of lines
of code with no type safety and with all types of crazy meta programming and
decorators and meta classes and stuff.
And yeah, so there was a, it was all in Python when I showed up.
it was not all in Python and not all in one big monolithic service when I left.
So you mentioning joining when there were around a hundred people and you
probably already at this point had like multitudes more in terms of users.
Being in hypergrowth, it is sort of this race against time where you only have
so much time to work on something, but growth may be outrun you already and
things are already starting to break.
Or You know like, okay, if things gonna grow like this, this system will
break and it's gonna be pretty bad.
So tell me more about how you were dealing with like the constant r
race against time to rebuild systems, redesign systems, putting out fires.
What was that like?
Yeah, and I think there's like kind of an interesting place to take this on.
I think like the normal things were on scale right there.
Those were like.
One, kinda class of problems of being able to handle the load.
But I think one kind of really interesting, dimension of this that led
to our decision to start rewriting all of the sync engine in 2016 was actually
just like customer debugging load.
You know, we have we had hundreds of millions of active users and they were
using Dropbox in all types of crazy ways.
Like one of the stories is someone was using Dropbox with like, I think
it was running on some, I don't know if it was like a raspberry pie or
something, something on his tractor.
Like the guy ran a farm and he was using Dropbox to sink like
pads in text files to his tractor.
And I might be getting some of the details wrong, but
it's something like that.
And so people would just use Dropbox in all types of crazy ways on crazy
file systems with kernel modules running that are messing things around
or so I think, You know, in terms of getting ahead of scale, I think we found
ourselves around 2015, 2016, in the place where for the syn engine on the
desktop client, the entire team just spent all of its time debugging issues.
We had this principle of like anything that's possible, anything that a
protocol allows, anything that some threading race condition that's
theoretically possible will be possible.
And then we would see it, right?
Like users would write in saying, my files aren't sinking.
And then we would look at it and we would spend months debugging each one of these
issues and trying to read the tea leaves from traces and reports and reproductions.
And it'll be like, oh they mounted this file system over here
and then this one and this one are in a different file system.
So moving the file actually did a copy, but then the X adders
were in, preserved this and that.
You know, in terms of that theme of like getting ahead of scale, like I think there
was first this realization that like the set of possible things that can happen in
the system is just astronomically large.
And all of them will happen if they're allowed to.
And we do not have, no matter how much like incremental time
we put into debugging things, we will never be able to keep up.
And the cost of doing that is that the entire team is working
on maintenance like this.
We couldn't build any new features.
So I think that was a motivation then for the rewrite to is can we find like points
of leverage where if we just invest a little bit in technology upfront, like by
architecting things a particular way, can we just eliminate a much bigger set of
potential work from debugging and working with customers and stuff like that.
So maybe this is a good time to take a step back and try to
better understand what was Dropbox sync Engine actually back then?
So from just thinking about it through like a user's perspective, I have maybe
two computers, and I have files over here.
I. I want to make sure that I have the files synced over from here to here.
So I could now think about this as sort of like a Git style, approach.
Maybe there's other ways as well.
walk me through sort of like through the solution space, how this could have been
approached and how was it approached?
is there some sort of like diffing involved between different file states
over time, those are being synced around.
Do you sync around the actual file content itself?
Help me to understand.
Building a mental model, what does it mean back then for the sync engine to work?
Yeah.
It's a super interesting question, right?
Because I think like you're saying, there's so many different paths one
can take and it's, I think one of those things where like if someone
asks, like design Dropbox in an interview question, there's like
definitely not one right answer, right?
It's like there are so many trade-offs and like different forks in the decision tree.
I think one of the first things is that, so you have your desktop A and you
have your, maybe you have your desktop and your laptop, and one of the first
decisions for Dropbox is that we would have a central server in the middle,
that there would be a Dropbox file system in the middle that Dropbox, the company
ran, and we did that from this trust perspective, we wanted to say that we
will run this infallibly when you get that green check mark when it's there.
You know, even if an asteroid destroys the eastern side of the United
States, like we will have things replicated in multiple data centers.
And that you know, and then also it's accessible anywhere
on the internet, right?
You can go to the library.
This is not so common these days but I remember when I was a student, like,
go to the library, log into Dropbox and read all your things right?
rather than having to bring a USB stick around.
And so I think that is the first decision, but it's not necessary, right?
Like there were plenty of distributed, entirely peer to
peer file syncing, designs, right?
And so that was the first decision.
And I think the kind of second decision was that if we imagine our desktop and
our laptop and you have the server in the middle, the desktop might be on
Windows, the laptop might be on Mac OS.
So I think that decision to support multiple platforms.
Is like another really interesting one.
This is like where I think Git and Dropbox can be a little bit different.
And that Git is at the end of the day quite Linux centric.
It's case sensitive for its file system.
It deals with directories and it makes particular assumptions about
how directories should behave.
And that was something with Dropbox.
We wanted to be consumer, we wanted to support everything and we wanted
it to feel very automatic, right?
That like, someone shouldn't have to understand like what a, like unicode,
normalization disagreement means.
Right?
Where in Git like in really bad settings, like you might have to understand
that, that you're right, you with an accent differently on Mac and Windows.
so I think that's the kind of like next, side.
So then Dropbox has its design for a file system and it's a
central, it's like the hub and all those folks are your phone, your.
desktop, your laptop and whatnot.
and then so to kind of get down to the details a bit more.
So then, yeah, we have a process that runs on your computer, that's the
Dropbox app, and that watches all of the files on your file system, and then
it looks at what's happened and then syncs them up to the Dropbox server.
And then whenever changes happen on the Dropbox server, it syncs them down.
there's another kind of interesting decision here on Dropbox by
design was always like a sidecar.
It's always something that just sits and it looks at your files.
Your files are just regular files on the file system.
And if Dropbox, the app isn't running, your files are there and they're safe,
and it's something that you know, regular apps can just read and write
to, and in some sense like Dropbox was unintentionally local-first
from that perspective, right?
Because it's saying that no matter what happens, your data
is just there and you own it.
and you know, there are other systems, right?
Like if you use NFS a like a network file system, then if you unmount it or
if you lose connection to the server.
You might not be able to actually open any files that you have the metadata for.
Right.
And I remember from a user perspective, the local-first aspect, I really went
through like all the stages where I had a computer that wasn't connected
to the internet yet, and that at some point I had an internet connection.
But, files were always where like everything depended on files.
Like if I didn't have a file, things wouldn't work.
Everything depended on files.
There were barely websites that where you could do meaningful things.
certainly web apps weren't very common yet.
And then Dropbox made everything seamlessly work together.
And then when web apps and SaaS software more came along, I was a
bit confused because I felt Okay.
I t gives me some collaboration, but seems to be a different kind of collaboration
since I had collaboration before.
But I also understood the limitations of, when I'm working on the same doc
file, through Dropbox, which gets sort of like the first copy, second
copy, third copy, and now I need to somehow manually reconcile that.
And when I saw Google Docs for the first time.
That was really like a revelation because, oh, now we can do this at the same time.
But at the same while I saw that, I still remember the feeling
where, but where are my files?
This is my stuff now.
Where, where is it?
And that trust that you've mentioned with Dropbox, I felt like I lost some,
some control here and it required a lot of trust, in those tools that I
started now step by step, embracing.
And frankly, I think a lot of those tools didn't deserve my trust in hindsight.
I still feel like we've lost something by no longer being able to like call
the foundation our own in a way.
And I'm still hoping that we kind of find the best of both worlds where
we get that seamless collaboration that we now take for granted.
Something like that Figma gives us.
but also the control and just being ready for whatever happens, that's
something Dropbox gave us out of the box.
I just wanna share this sort of like anecdote and like almost
emotional confusion as I walk through those different stages
of how we work with software.
Totally.
And we've ended up in a place that's not great in a lot of ways.
Right.
And I think you know, I think part of the sad thing, and maybe from
even like an operating systems design perspective is that I feel like files
have lots of design decisions that are.
Packaged up together.
You know, like one of the amazing things about files is that
they're self-contained, right?
Like on Google, I don't know what Google's backend looks like for Google
Docs, but they probably have like all of the metadata and pieces of the data
spread and different rows in a database and different things in an object store.
And just even thinking about like the physical implementation of that
data, it's like scattered around probably a bunch of servers, right?
Maybe in different data centers.
And there's something really nice about a file where a file is just
like a piece of state, right?
That is just self-contained.
And I think the thing that I think is one of the things I think is very
unfortunate is like from a operating systems perspective is that that decision
then has also been coupled with a very anemic API like with files, they're just
sequences of bytes that can be read and written to and impended and there's
no additional structure beyond that.
And I think like.
Folks the way that things have evolved is that we've given up on, too
have more structure, too make things like Google Docs, too be able to
reconcile and have collaboration and interpret things more than just bites.
We've also given up this ability to package things together.
Mac os had like a very kind of baby step in this direction with I
think they're called bundles.
Like the things where like if you have like your.app, they're
actually a zip file, right?
And there's all types of ways, all types of brain damage for how this
like, doesn't actually work well.
You know?
But the idea is kind of interesting, right?
It's like what if files had some more structure and what if you still
considered something, an atomic unit, but then it had pieces of it that
weren't just uninterpretable bites.
And I think that's like, the path dependent, way that we've
ended up where we are today.
That makes sense.
So going back to the sync engine implementation did the Python process
back in the day did that mostly index all of the files and then actually
send across the actual bites probably in some chunks, across the wire?
Or was there some more intelligent and diffing happening client side
that you would only send kind of the changes across the wire and how do I
need to think about what is a change when I'm dealing with like a ton of
bites before and a ton of bites after?
Yeah.
It's really, really good questions.
I think maybe like the first starting point is that like files
in Dropbox were stored, just broken up into four megabyte chunks.
And that was just a decision at the very beginning to pick some size.
And on the server, the way that those chunks were stored is that they,
each four megabyte chunk was stored by key to by its shot to 56 hash.
So we would assume that those are globally unique.
So then if you had the same copy of a bunch of file, or you had
a file copied many times in your Dropbox, we would only store it once.
And that would just happen organically because we would say
like, okay, I looked at this file, it has three chunks A, B, and C.
And then the client would ask the server, do you have A, B, and C?
Like the server would say, yes, I have B and C already, please send A, then we
would upload A. so there was already like at the file level there was this like
kind of very coarse grained Delta sync.
at the four megabyte chunk layer.
and then the kind of, it's funny, these things evolve, right?
Like then the next thing we layered on up top was that in that setting where
you decided B and C were there already and you needed to upload a then with
a, the desktop client could use rsync to know that there was previously a
prime and do a patch between the two and then send just those contents.
the kind of thing that was pretty interesting is that a lot of the content
on Dropbox was very incompressible stuff like video, images, so the
benefits of deduplication both across users or even within a user.
And the benefit of like rsync was not actually as much as one might think,
at least from the like, terms of bandwidth going through the system.
It wasn't that reductive because a lot of this content was just kind of unique and
not getting updated in small patches.
And on your server side, blob store, now that you had those hashes for those four
megabyte chunks, that also means that you could probably deduplicate some content
across users, which makes me think of all sorts of other implications of that.
When do you know it's safe to let go of a junk?
do you also now know that, you could kind of go backwards and
say like, oh, from this hash, we know this is sensitive content.
And have some further implications for, whatever we don't need to go too
much into depth on that now, but, yeah.
I'm curious like how you thought of those design decisions and
the possible implications.
Yeah.
Yeah, for the first one yeah, like distributed garbage collection
was a very hard problem for us.
We called it vacuuming and in terms of making Dropbox economics work out
of, like, when we couldn't afford to keep a lot of content that was deleted
that we couldn't charge users for.
So that was you know, there's all additional complexity where different
users would have like the ability to restore for different periods of time.
So we would say like, anything that's deleted, it doesn't actually
get deleted for 30 days or a year or whatnot based on their plan.
so then, yeah, like having to do this like big distributed mark and
sweep garbage collection algorithm across hundreds of petabytes,
exabytes of content that was something that we had to get pretty good at.
And when we designed Magic Pocket, where we, implemented S3 in-house, we
had specific primitives for making it a little bit easier to avoid race conditions
where like, if a file was deleted.
And we decided that no one needed it anymore.
But then just at that point in time, someone uploads it again, making sure
that we don't accidentally delete it.
So that was like, yeah, definitely a very tricky problem.
And I think in retrospect this is like an interesting design exercise, right?
And that if deduplication wasn't actually that valuable for us, we could have
eliminated a lot of complexity for this garbage collection by not doing it right.
I think for the second thing, yeah.
So at the beginning when Dropbox started, if you had a file with A, B and C and you
uploaded it, it would just check, does A, B and C exist anywhere in Dropbox?
And, that got changed over time to be does do you as your user
have access to A, B, and C?
And you know, 'cause otherwise you could use this for all types of purposes, right?
To see if there exists some content anywhere in Dropbox.
And, that was something where we would in the case where the user was
uploading A, B, and C, say none of them were present in their account, we would
actually force them to upload it, incur the bandwidth for doing so, and then
discard it if B and C existed elsewhere.
Yeah.
Very interesting.
I mean, this would be an interesting rabbit hole just to go down just the
kind of second order effects of that design decision, particularly at
the scale and importance of Dropbox.
But maybe we save that for another time.
So going back to the sync engine, now that we have a better understanding of, how it
worked in that shape and form back then.
You've been already mentioning before, like as things as usage went through
the roof, all sorts of different usage scenarios also expanded.
you had all sorts of more esoteric ways, how you didn't kind of even think
before that it would be used this way.
Now all of that came to light.
I'm curious which sort of, helper systems you put in place that you could
even have a grasp of what's going on since a part of the trust that Dropbox
owned or that earned over time, was probably also related to privacy.
So you, you couldn't just like read everything that's going on in someone's
system, so you're probably also relying to some degree on the help of a user
that they like send something over.
Yeah.
Walk me through like the evolution of that and that you, like as
an engineer, if there's a bug reproducing that bug is everything.
So walk me through that process.
Yeah, and you know, like we had a very strict rule, right, where it just,
we do not look at content, right?
and so that was the thing when debugging issues, the saving grace is
that for most of the issues we saw.
They were more metadata issues around like sync, not converging or sync, getting
to the client thinking it's in sync with the server, but them disagreeing.
so we had a few pretty, yeah, like pretty interesting
supporting algorithms for this.
So one of them was just simple like hang detection, like making sure, like
if, when should a client reasonably expect that they are in sync?
And if they're online and if they've downloaded all the recent
versions and things are getting stuck, why are they getting stuck?
So are they getting stuck because they can't read stuff from the
server, either metadata or data?
Are they getting stuck because they can't write to the file system and
there's some permission errors?
So I think having very fine-grained classification of that and having the
client do that in a way that's like not including any private information and
sending that up for reports and then aggregating that over all of the clients
and being able to classify was a big part of us being able to get a handle on it.
And I think this is just generally very useful for these sync engines.
the biggest return on investment we got was from consistency checkers.
So part of sync is that there's the same data duplicated in many places, right?
Like, so we had the data that's on the user's local file system.
We had all of the metadata that we stored in SQLite or we would store like what
we think should be on the file system.
We would store what the latest view from the server was.
We would store things that were in progress, and then we have
what's stored on the server.
And for each one of those like hops, we would have a consistency checker that
would go and see if those two matched.
And those would, that was like the highest return on investment we got.
Because before we had that, people would write in and they would
complain that Dropbox wasn't working.
And until we had these consistency checkers, we had no idea the
order of magnitude of how many issues were happening.
And when we started doing it, we're like, wow.
There's actually a lot.
So a consistency check in this regard was mostly like a hash over some
packets that you're sending around.
And with that you could verify, okay, up until like from A to B to C to D, we're
all seeing the same hash, but suddenly on the hop from D to E, the hash changes.
Ah-huh.
Let's investigate.
Exactly.
And so, and to do that in a way that's respectful of the users,
even like resources on their system.
Like we wouldn't just go and blast their CPU and their disc and their network to go
and like turn through a bunch of things.
So we would have like a sampling process where we like sample a random
path in the tree and the client and do it the same on the server.
we would have stuff with like Merkle trees and then when things would diverge,
we would try to see like, is there a way we can compare on the client and see like
for example one of the kind of really important, goals for us as an operational
team was to have like the power of zero.
I think it might be from AWS or something.
My co-founder James, has a really good talk on it.
but we would want to have a metric of saying that the number of unexplained
inconsistencies is zero and one 'cause.
Then the nice thing right, is that if it's a zero and it regresses,
you know that it's a regression.
If it's at like fluctuating at like 15 or like a hundred thousand and it kind
of goes up by 5%, it's very hard to know when evaluating a new release, right?
That like that's actually safe or not.
so then that would mean that whenever we would have an inconsistency due to a bit
flip, which we would see all the time on client devices, then we would have to
categorize that and then bucket that out.
So we would have a baseline.
Expectation of how many bit flips there are across all of the devices on Dropbox.
And we would see that that's staying consistent or increasing or
decreasing, and that the number of unexplained things was still at zero.
now let's take those detours since you got me curious.
Uh, what would cause bit flips on a local device?
I think a few, few causes, one of them is just that in the data center, most
memory uses error correction and you have to pay more for it, usually have to pay
more for a motherboard that supports it.
at least back then.
now like on client devices we don't have that.
So this is a little bit above my pay grade for hardware cosmic
rays or thermal noise or whatever.
But memory is much more resilient in the data center.
I think another is just that, storage devices are very greatly in quality.
Like your SSDs and your hard drives are much higher quality inside the data
center than they are on local devices.
And so.
You know, there's that.
it also could be like I had mentioned that people have all
types of weird configurations.
Like on Mac there are all these kernel extensions on Windows, there's
all of these mini filter drivers.
There are all these things that are interposing between
Dropbox, the user space process and writing to the file system.
And if those have any memory safety issues where they're corrupting memory
'cause of the written in archaic C you know, or something that that's
the way things can get corrupted.
I mean, we've seen all types of things.
We've seen network routers get having corrupting data, but usually
that fails some checksum, right?
Or we've seen even registers on CPUs being bad where the memory gets replaced
and the memory seems like it's fine, but then it just turns out the CPU has its
own registers on CHIP that are busted.
And so all of that stuff I think just can happen at scale.
Right.
that makes sense.
And I'm happy to say that I've hadn't had yet to worry about flip bits, whether
it's being for storage or other things, but huge respect to whoever had already
to, tame those parts of the system.
So, you mentioning the consistency check as probably the biggest lever that you
had to understand which health stage your sync engine is in the first place.
was this the only kind of metric and proxy for understanding with how well
the syn system is working or were there some other aspects that gave
you visibility both macro and micro?
Yeah, I mean, I think this yeah, the kind of hangs, so like knowing
that something gets to a sync state and knowing the duration, right?
So the kind of performance of that was one of our top line metrics.
And the other one was this consistency check.
And then first specific like operations, right?
Like uploading a file, like how much bandwidth are people able to use
because for like, people wanted to use Dropbox, but, and upload lots,
like huge data, like huge number of files where each file is really large.
And then they might do it on in Australia or Japan where they're
far away from a data center.
So latency is high, but bandwidth is very high too, right?
So making sure that we could fully saturate their pipes and all
types of stuff with debugging.
Things in the internet, right?
People having really bad routes to AWS and all that.
so we would track things like that.
I think other than that it was mostly just the usual quality stuff,
like just exceptions and making sure that features all work.
I think when we rewrote this system and we, designed it to be very correct.
We moved a lot of these things into testing before we would release.
So we this is I think one of the, to jump ahead a little bit, we designed,
decided to rewrite Dropbox's sync engine from this big Python code base into Rust.
And one of the specific design decisions was to make things extremely testable.
So we would have everything be deterministic on a single thread,
have all of the reads and rights to the network and file system,
be, through a virtualized API.
So then we could run all of these simulations of exploring what would
happen if you uploaded a file here and deleted it concurrently and then had a
network issue that forced you to retry.
And so by simulating all of those in ci, we would be able to then have very
strong in variance about them that knowing that like a file should never
get deleted in this case, or that it should always converge, or things
like the sharing that this file should never get exposed to this other viewer.
I think like the, having much, like having stronger guarantees was something
that we only could really do effectively once we designed the system to make
it easy to test those guarantees.
Right.
That makes a lot of sense.
And I think we're seeing more and more systems, also in the
database world, embrace this.
I think TigerBeetle is, is quite popular for that.
I think the folks at Torso are now also embracing this approach.
I think it goes under the umbrella of simulation testing.
that sounds very interesting.
Can you explain a little bit more how maybe in a much smaller program would
this basically be Just that every assumption and any potential branch,
any sort of side effect thing that might impact the execution of my program.
Now I need to make explicit and it's almost like a parameter that I put into
the arguments of my functions and now I call it under these circumstances, and I
can therefore simulate, oh, if that file suddenly gives me an unexpected error.
Then this is how we're gonna handle it.
Yeah, exactly.
So it's like and there's techniques that like the TigerBeetle folks, like
we, we do this at Convex in rust with the right, like abstractions, there's like
techniques to make it not so awkward.
But yeah, it is like this idea of like, can you pin all of the non-determinism in
the system can, whether it's like reading from a random number generator, whether
it's looking at time, whether it's reading and writing to files or the network.
Can that all be like pulled out so that in, production it's just using the
random AP or the regular APIs for it.
so there's like for any of these sync engines, there's a core
of the system which represents all the sync rules, right?
Like when I get a new file from the server, what do I do?
You know, if there's a concurrent edit to this, what do I do?
and that I. Core of the code is often the part that has the most bugs, right?
It has the, it doesn't think about some of the corner cases or if
there are errors or needs retries or doesn't handle concurrency.
It might have race conditions.
So the kind of, I think the core idea for determination, determin deterministic
simulation testing is to take that core and just kind of like pull out all of the
non-determinism from it into an interface.
So time randomness, reading and writing to the network, reading
and writing to the file system, and making it so that in production,
those are just using the regular APIs.
But in a testing situation, those can be using mocks.
Like they could be using things that for a particular test
and wants to test a scenario or setting it up in a specific way.
Or it could be randomized, right?
Where it might be that reading from Like time, the test framework might
decide pseudo randomly to advance it or to keep it at the current time or
might serialize things differently.
And that type of ability to have random search explore the state space of
all the things that are possible is just one of those like unreasonably
effective ideas, I think for testing.
And then that like getting a system to pass that type of
deterministic simulation testing.
It's not at the threshold of having formal verification, but in our
experience it's pretty close and with a much, much, smaller amount of work.
And you mentioning Haskell at the beginning?
I still remember when I, after a a lot of time having spent writing unit tests in
JavaScript and I, back then, in the other order, I first had JavaScript and then I
learned Haskell, and then I found quick test and was quick test, Quick Check.
which one was it?
I think it was Quick check, right?
Well, right.
So I found Quick Check and I could express sort of like, Hey, this is this type.
It has sort of those aspects to it, those invariants and then would just
go along and test all of those things.
Like, wait, I never thought of that, but of course, yes.
And then you combine those and you would get way too lazy to write unit
tests for the combinatorial explosion of like all of your different things.
And then you can say, sample it like that, and like, focus on this.
and so I actually also, started embracing this practice a lot more in the
TypeScript work that I'm doing through a great project called Prop Check.
and that is, picking up the same ideas and for particularly those
sort of scenarios where, okay, Murphy's Law will come and haunt you.
this is in distributed systems.
That is typically the case.
Building things in such a way where all the aspects can be, specifically
injected and the, the sweet spot.
If you can do so still in an ergonomic way, I think that's the way to go.
It's so, so valuable, right?
And yeah.
And yeah, the ability to, for prop tasks, for quick check for all of these to
also minimize is just magical, right?
Like it comes up with this crazy counter example and it might be
like a list with 700 elements, but then is able to shrink it down to
the, like, real core of the bug.
It's magic, right?
And you know, I mean, I think this is something like, you know.
A totally different theme, right?
Like one thing at Convex we're exploring a lot is like coding has changed a lot
in the past year with AI coding tools.
And one of the things we've observed for getting coding tools to work very
well with Convex is that these types of like very succinct tests that can
be generated easily and have like a really high strength to weight or power
to weight ratio are just really good for like autonomous coding, right?
Like, if you are gonna take like cursor agent and let it go wild,
like what does it take to just let it operate without you doing anything?
It takes something like a prop test because then it can just continuously
make changes, run the test, and not know that it's done until that test passes.
Yeah, that makes a lot of sense.
So let's go back for a moment to the point where you were just transitioning
from the previous Python based sync engine to the Rust based sync engine.
So you're embracing simulation testing to have a better sense of
like all the different aspects that might influence the outcome here.
walk me through like how you, went about.
Deploying that new system.
Were there any sort of big headaches associated with migrating from the
previous system to the new system?
since you, for everything, you had sort of a defacto source
of truth, which are the files.
So could you maybe just forget everything the old system has done and you just
treat it as like, oh, the, user would've just installed this fresh, walk me
through like how you thought about that since migrating systems on such
a big scale is typically, quite dread
Yeah, dreadsome is, yeah.
appropriate word.
I think one of the biggest challenges was that by design we had a very different
data model for the old sync engine.
We called it sync engine Classic.
Affectionately.
And then we had for Nucleus was a new one.
Nucleus had a very different data model, and the motivation for that was that
sync engine Classic just had a ton of possible states that were illegitimate.
It could, if you had like a, the server update a file and the client update
a file, but then a shared folder gets mounted above it, things could get
into all of these really weird states that were legal but would cause bugs.
And then I think that was like one of the big guiding principles more
than even just like Rust or Python, was just like designing what states
should the system be allowed to be in and design away everything else,
make illegal states unrepresentable.
And so that, what that then meant is once we had that.
When we needed to migrate, we had a long tail of really weird starting positions.
So where you basically realized, okay, this system is in this state A, how the
heck did it ever get into that state?
And B, what are we gonna do about it now where we can basically,
it's like from a mapping function, this is like invalid input.
So can you explain a little bit of like, how you constrained the space of, and how
you designed the space of, legitimate, valid states and what were some of the,
if you think about this as like a big matrix of combinations, what are some
of the more intuitive ones that were, not allowed that you saw quite a bit?
Yeah, so I think part of the difficulty for Dropbox, like as syncing things
from the file system is that file system APIs are really anemic.
File system aPIs don't have transactions.
They don't things can get reordered in all types of ways.
So we would just read and write to files from the local file system, and
we would use file system events on Mac, we would use the equivalent on
Windows and Linux to get, updates.
But everything can be reordered and racy and everything.
So one, like common invariant would be that if you have a
directory you know, like files have to exist within directories.
If a file exists, then it's parent directory exists.
And like simultaneously, if you delete a directory, it shouldn't
have any files within it.
And that invariant guarantees and that the file system is a tree.
Right?
And then we, it's very easy to come up with settings, with reads from the
local file system where if you just naively take that and write it into
your SQLite database, you will end up with data that does not form a tree.
and then especially even with like I know it's being unique, right?
Like if I move a file from A to B, then I might observe the add for it at B
way before the delete at B or I might observe it vice versa, where the file
is transiently gone and disappeared and we definitely don't wanna sync that.
and then with directories, if I have like a, as a directory and then B as
a directory, and then I move it's, I could observe a state where A moves into
B, which then without doing the right bookkeeping, might introduce a cycle in
the graph and a cycle for directories would be really bad news, right?
so all of these invariants were things that the file system APIs, they don't
respect, even though the file system internally has these invariants, right?
You cannot create a direct recycle on any file system.
Definitely.
I mean certainly without root And all of these invariants exist but
are not observable through the APIs.
And so then we sync Engine Classic would get into the state where it's
like local SQLite file would have all types of violations like that.
So then how do we read the tea leaves of like the database is in
a really weird state we can't lose.
And to go back to, I think what you had talked about at the beginning of this was
that we always had the nuclear option of dropping all of our local state and doing
a full resync from the files themselves.
But then the problem is that we would entirely lose user intent.
So if, for example, I was offline for a month and I had a bunch of files,
and then during that month other people in my team deleted those files.
If I came back online and didn't have my local database, we would have to
recreate those files and people would complain about this all the time because.
They would delete something and wanna delete it, and then Dropbox would
just randomly decide to resurrect it.
So those types of decisions we, we tried to avoid that as much as possible, but
then that meant having to look at a potentially really confusing database and
read what the user intent might have been.
Right.
I wanna dig a little bit more into the topic of user intent.
Since with Dropbox you've built a sync engine very specifically for the use
case of file management, et cetera, where user intent has a particular meaning that
might be very different from moving a cursor around in a Google Docs document.
So can you explain a little bit, what are some of the, common scenarios of, and
maybe subtle scenarios of user intent, when it comes to the Dropbox design space?
Yeah, totally.
and I think the for regular things like say editing files.
I think we saw that like people just generally did not, maybe because
of the way the system was even its capabilities, people did not
edit the same files all too often.
So maintaining user intent when file, when everyone is online, just kind of
taking last writer wins Where I think user intent became very interesting is
if someone went offline, like they're on an airplane before wifi and airplanes
And they worked on their document and someone else worked on the same time.
In that case, we observed that users always wanted to see the conflicted
copy and that they wanted to get the opportunity to say, like, I did.
I put in a lot of effort into working on this when I was on the plane.
Someone else, put in probably a similar amount of effort when they were online and
you know, so last writer wins policies.
There violated user expectations quite a lot because either a person
had to win and then the person who lost would be really upset.
so I think those were pretty interesting.
I think with Moose, like with more metadata operations I think people
were a little bit more permissive.
Like if I moved something from one folder to another, another person
moved it to a different folder.
having it just converged on something as long as it converges.
We observed it being like people didn't worry about it too much.
I think the place where user intent is really interesting
with moves is with sharing.
So I think thinking about this from like the distributed systems
perspective on causality, there would be like someone might have like,
I dunno, their HR folder, right?
And I don't know, like, let's say that someone is transferring to the HR team is
they're getting added to the HR folder.
But then say before they were on the team, they were on a
performance improvement plan.
So then the administrator for HR would delete that file, make sure it's
deleted, and then add them to the folder.
And so their user intent is express in a very specific
sequencing of operations, right?
That like this causally depended on this.
I would not have invited 'em to the folder unless the delete was stably synced.
And that making sure that gets preserved throughout the system,
even when people are going online and offline and everything is a very
hard distributed systems problem.
Right.
and it was intimately related with the details of the product.
Right.
yeah.
How did you capture that causality chain of events since you probably also
couldn't quite trust the system clock?
How did you go about that?
Yeah, this became even more difficult, right?
Where file system metadata was partitioned across many shards in the database.
So then we ended up using something like Lamport timestamp, where every single
operation would get assigned a timestamp.
And those timestamps were usually only reading and writing to their
particular shard and for whatever timestamp the client had observed.
But then in these cases where there were potentially cross shard, they
weren't transactions, but like causal dependencies, we would be able to say
like, the operation to mount this or to add someone to the shared folder
and there them mounting it within their file system has to have a higher
timestamp than any right within that or.
Rights including deletes.
so then that way when the client is syncing it would be able to know that when
I am merging operation logs across all of the different shards, I need to assemble
them in a causally consistent order.
And that would then respect all of these particular invariants.
Right.
So you having thought through those different scenarios for Dropbox and
made very intentional design decisions that, for example, in one scenario
last writer wins is not desirable.
Since that might lead to a very sad person stepping off the plane because
all of your data is suddenly gone, or the other person's data is gone.
so you make very specific design trade-offs here when it
comes to somehow squaring the circle of distributed systems.
Which sort of advice would you have for application developers or people even
who are sitting inside of a company and are now thinking about, oh, maybe
we should have our own Dropbox style, linear style sync engine internally.
Which sort of advice would you give them when they Yeah.
Start thinking this through to the detail.
Yeah, I'll talk through kind of how we structured things at Dropbox to be able
to navigate these types of problems.
And I think the patterns here, can be quite general.
I think what we ended up with was that like thinking like distributed
systems syncing is hard, right?
So we would have the kind of base layer of the sync protocol and how state
gets moved around between the clients and the servers and all the shards.
We would have very strong consistency guarantees there.
So we would not use any of the knowledge of the product at that layer.
So from a, like thinking of Dropbox in the file system as a CRDT.
Dropbox allows, like moves to happen concurrently.
It ha allows you to add something while another thing is happening.
But at the protocol level, we kept things very strict.
We kept them very close to being serializable that every view of the
system was identified by a very small amount of state, like a timestamp.
And that would fully determine the state of the system and like the
amount of entropy in that was very low.
And then whenever you are modifying it, you would say, here's what I expect
the data to be, and if it doesn't match exactly, it will reject the operation.
And then by doing it, structuring things in that way, then we made it very easy
for product teams and for even us working on sync to embed all of these like
looser more product focused requirements.
They also may wanna change over time into the end points, like layered on top.
So every time we wanted to change a policy on how like a delete reconciles with an.
You know, add for a folder or something.
We didn't have to solve any distributed systems problems to do that.
So I think that like pattern of saying that, like is there a good abstraction?
Is there something that is like very powerful that could solve a large
class of problems, doing that well at the lowest layer and then potentially
weakening the consistency above it.
I actually really like the Rocicorp folks have a really great description of
their consistency model for Replicache of it being like session plus consistency.
And it's like a very similar idea where like when we build things on
a platform, we may as our with our product hats on, like want users to
not have to think about conflicts and merging and all that in a lot of cases.
But those decisions might be very particular to our app.
And that's something that holds for everything on the platform.
And then there's always a way to embed those decisions onto, say.
Session consistency and Replicache or serializability and other systems.
And so I think that's like that separation of concerns I
think is something that can apply to a lot of systems.
Right.
So maybe we use this also as a transition to talk a bit more about what you're
now designing and working on Convex.
What were some of the key insights that you've taken with you from Dropbox that
ultimately led to you co-founding Convex?
Yeah, when we first were starting Convex we were looking at how apps
are getting built today, right?
Like web apps are easier to build than ever.
Even in 2021, it's incredible how much, like more productive
that compared to 10 years before.
Right.
It was, and I think we noticed that the hard part for so many discussions
was managing state and like how state propagates I think it was from
the Riffle paper right, on how like so many issues in app development
are kind of database problems in disguise and that how techniques
from databases might be able to help.
So with Convex we were saying like, well if we start with the idea of designing
a database from first principles, can we apply some of those database solutions
to things across the whole stack?
So say for example, when I'm reading data from it within in my app, I have
all of these React components that are all reading different pieces of data.
It'd be really nice if all of them just executed at the same timestamp
and I never had to handle consistency issues where one component knows
about a user or the other one doesn't.
Similarly, like why isn't it possible to be that I just use query across
all my components and they just all live update whenever I read anything,
it's a automatically reactive.
So those were some of the like the initial kind of thought
experiments for what led to Convex.
I think the other one that was really motivated from our time at
Dropbox and I think is like kind of a both a blessing and a curse.
It's kind of like one of the key design decisions for Convex is
that Convex is very opinionated about there being a separation
between the client and the server.
So we saw this at Dropbox where they were just different teams, right?
And you know, as we've seen with like even the origin of GraphQL, right?
Like that ability to decouple development between.
teams working on user facing features and the way that the data fetching
is implemented on the backend, it's gonna be really powerful.
And so kind of the kind of thought experiment with Convex is, can we
maintain a very strong separation while still getting like live updating, while
still getting a really good ergonomics for both consuming data on the client
and like fetching it on the server.
Right.
So yeah, walk me through a little bit more through the evolution of Convex then.
And so, in, in terms of all the other options that are out there in terms
of state management and I think most what applications are using is probably
something that at least to some degree is somewhat customized and hand rolled and
comes with its own huge set of trade-offs.
Help me better understand sort of the, where you mentioned the,
opinionated nature of Convex.
What are the, benefits of that?
What are the downsides of that and other implications?
Yeah, so when you write an app on Convex we can use maybe
like a basic to do app, right?
The linear clone, everyone does.
you write endpoints like you might be used to, right?
Where it's like list all the to-dos in a project like update a to-do in a project.
and those get pushed as your API to your Convex server.
the implementations of that API can then read and write to the database
and Convex has like a, kinda like Mongo or Firebase, like API for doing so.
I think the main benefit then of Convex relative to more traditional
architectures is that if you're on the client, the only thing you need to do
is call the, like the use query hook.
You're saying like, I am looking at a project I just do use like use query
list tasks and project that will then talk to the server, run that query, but
then also set up the subscription and then whenever any data that that query
looked at changes, it will efficiently determine that and then push the update.
So part of what is like been nice with Convex is that you are getting
a client that has a web socket protocol, it has a sync engine built in.
You're getting infrastructure for running JavaScript at scale and for
handling sandboxing and all of that.
And then you're also getting a database, which is, you know.
One, supporting transactions or reading and writing to it.
But then it also supports this efficient like being able to subscribe
on, I ran this query, this query just ran a bunch of JavaScript.
It looked at different rows and it ran some queries.
the system will automatically efficiently determine if any right overlaps with that.
So the combination of all of those things is like part of the benefit of
Convex, you just write TypeScript and you write it in a way that's, feels
very natural and everything just works.
And I think some of the like downsides is that it's it is a different set of APIs.
it's not using sql, it's doing things a little bit differently
than they've been done before.
yeah, it's like kind of interesting even today to see like what you know.
Talking about AI code gen, right?
Like models have been trained, pre-trained on this huge corpus
of stuff on the internet.
And when are they good at adopting new technologies?
Technologies that might be after their knowledge cutoff.
And when are they like it's better just to stick to things that they know already.
Right.
So what you've mentioned before where you say, Convex is rather opinionated for me.
in let's say five years ago, I might've been much more of
like, oh, but maybe there's a technology that's less opinionated
and I can use it for everything.
But the more experience I got, the more I realized no, actually.
I want something that's very opinionated, but opinionated
and I share those opinions.
Those are exactly for my use case.
So I think that is much better.
This is why we have different technologies and they are great for different
scenarios, and I think the more a technology tries to say, no, we're,
we're best for everything, I think the, less it's actually good at anything.
And so I greatly appreciate you standing your ground and saying
like, Hey, those are, our design, decisions that we've made.
And those are the use cases where, you'd be really well served building
on top of something like Convex.
And, I particularly like for now where TypeScript is really the, default
language to build full stack applications.
And it's also increasingly becoming the default for.
ai, based applications as well.
And AI based systems speak type script, just as well as English.
And given that Convex makes that full stack super easy.
And also I think you can, when you build local-first apps, it can
sometimes get really tricky because you empower the client so much.
You give the client so much responsibility and therefore there's
many, many things that can go wrong.
And I think Convex therefore, takes a more conservative approach and says
like, Hey, everything that happens on the server is like highly privileged
and this is your safe environment.
And the client will try to give you the best user experience and
developer experience out of the box.
But the client could be in a more adversarial environment.
And I think those are great design trade offs.
So, I think that is a fantastic foundation for tons of different applications.
Yeah.
talking about some of these strong opinions being both
blessings and curses, right?
Like over the past few months, one thing we've been working on is trying
to bridge the gap between those two points in the spectrum, right?
we wrote a blog post on it a few months ago of like working on what we're calling
our like Object sync engine, trying to take a lot of the principles from more of
a local-first type approach of having a data model that it is synced to the client
and the only interaction between the server and the client is through the sync.
And the client then can always render its UI just looking at the local
database and it can be offline.
It's also fully describes the app stage so it can be exported
and rehydrated or whatever.
it's very interesting design exercise we've been on to say like, can
you structure a protocol on a sync engine in a way such that the UI
is still reading and writing to a local store that is authoritative.
But then that local store is like to kind of use like an electric SQL terminology is
like that is a shape that is some mapping of a strongly separated server data model.
So we still have a client data model and server data model, which might be
owned by different teams and evolve independently and, we also have that
strong separation where the implementation of the shape is privileged and running
on the server and has authorization rules built in and get the best of both worlds.
And we've kind of, we have a like beta that we've not released publicly thought
open, sourced out there, but kind of a thing where we, I think they're
still figuring out like the DX for it.
And I think we have something that like algorithmically works
and it's like the protocol works, but it's like, it's kind of hard.
Right.
It kind of reminds me a lot of writing GraphQL resolvers of like saying How do I
take the messages table from my chat app?
Then under the hood that might be joining stuff from many different
tables and filtering rows, or might even be doing a full tech search
query in another view or something.
and coming up with the right ergonomics to make that feel
great for a day one experience.
I think something that's like still we're working on, still
kinda like a research project,
right?
Well, when it comes to data, there is no free lunch, but I'd much rather to have
it be done in the order and sequencing that you're going through, which is
having a solid foundation that I can trust and then figuring out the right
ergonomics afterwards, since I think there's many, many tools that start with
great ergonomics, but later realize that it's on a built, on a unsound foundation.
So when it comes to data, I want a trustworthy foundation, and I think
you're going about in the right order.
Hey, Sujay, I've been learning so much about one of my favorite
products of all time, Dropbox.
I've learned so much of like how the sausage was actually made, how it evolved
over time and I'm really excited that you got to share the story today and
many me included, got to, learn from it.
Thank you so much for taking the time and sharing all of this.
Thanks for having me.
This is super, super fun.
Thank you for listening to the localfirst.fm podcast.
If you've enjoyed this episode and haven't done so already, please
subscribe and leave a review.
Please also share this episode with your friends and colleagues.
Spreading the word about the podcast is a great way to support
it and to help me keep it going.
A special thanks again to Jazz for supporting this podcast.
I'll see you next time.