What if I'm on my phone, I type a thing and then I go back to
my computer and I try to see it?
Is it gonna merge it together?
Is it gonna break?
I have no idea because.
Client side stores don't help you with those problems.
They're not designed for those problems.
They're designed for like ephemeral state that can disappear at a moment's
notice if it needs to for some reason.
so that led me down exploring local-first technologies and I found pretty quickly
how capable SQLite is for these use cases, including in the browser.
Welcome to the localfirst.fm podcast.
I'm your host, Johannes Schickling, and I'm a web developer, a
startup founder, and I love the craft of software engineering.
For the past few years, I've been on a journey to build a modern, high quality
music app using web technologies, and in doing so, I've been falling down the
rabbit hole of local-first software.
This podcast is your invitation to join me on that journey.
In this episode, I'm speaking to Ben Holmes.
A senior web developer and educator known for his whiteboard videos
after having spent most of his career building server centric applications.
Ben recently explored local-first software by building a simple sync engine
from scratch before getting started.
Also, a big thank you to Jazz for supporting this podcast,
and now my interview with Ben.
Hey Ben, so nice to have you on the show.
How are you doing?
Hey, I'm doing great.
Yeah, how are you doing?
I'm doing fantastic.
Super excited to have you on the show.
I'm very certain that most of the audience already are familiar with who
you are since you have quite a reach on, Twitter, x, other platforms, et
cetera, where I think you're doing an excellent job of taking like novel
concepts and breaking them down in a very simple and approachable way.
And I think you've done the same for some local-first related topics recently.
So maybe some of the audience are already familiar with your work in that regard.
But, for those who are new to you, would you mind giving a background who you are?
Yeah, totally.
I'd be shocked if everyone knows, but if you've seen a guy with a whiteboard
around the internet, it might be me, especially if it's a vertical video.
so I've been doing a lot of work in just the engineering space for a long time.
So been at Astro for a few years and we've been building the
framework for content sites.
And I know actually you have some experience working with Contentlayer
and other things, but content sites are a nice place to get started
because they're a nice, well-defined use case for web technologies and
Astro was trying to spearhead being like the simplest way to do that.
so was able to contribute to that for a long time and also produce a lot of
videos on learnings in the process.
So it started with taking a rock band microphone and a whiteboard outta my
closet and just recording stuff and just seeing where it went and been doing it for
years now and kind of honing the craft.
And we've covered all sorts of topics, including, different
web frameworks, Tailwind tips, database providers using SQLite.
Most recently putting SQLite in the browser, which is gonna be
very relevant today, I'm sure.
and most recently I've made the jump to Warp Terminal, so I'm gonna
be, well, I have joined their team as a product engineer and we're
sort of spearheading the future of bringing AI and agent workflows to
everything you do in the terminal.
So if you forget a command, need to kill a port, need to go through
a full get workflow or even edit files in your projects, you can just
ask Warp to do that for you or fall back to all the features that make
Warp just a really nice terminal.
It's a really great fit.
Been working with them for a while and now get to do that full time.
We'll continue to do all the videos that you see around the internet.
That is awesome.
I'm certainly coming back to Warp since I wanna learn more about that as well,
but taking things one step at a time.
you've been digging into local-first related, you've been mentioning
SQLite running SQLite in the browser.
Looking into that over the course of the last year or so.
I'm very curious, like what led you to explore that since with Astro, et
cetera, you're probably, that's like a different part of building websites,
really where you maybe like Astro is famous for introducing the island
architecture where you go like very light on bringing JavaScript into the, website.
And if it's almost like the other extreme where you want to bring a lot of
JavaScript into the web app, deemphasize the server part, and also more on
the spectrum from website to web app.
Certainly much heavier on the web app spectrum.
So yeah.
Can you share more what led you to, to this path?
well the past year has been kind of like an existential crisis of how are you
supposed to build websites, and I think we all went through that as an industry,
as things kind of shifted back from client side, heavy react apps towards things
that are more server rendered and people are slowly trying to question that idea
again and see what we can learn from storing more information in the client
and sinking it back to your servers.
So I also noticed a wave that's completely opposite.
Of local-first apps, which would be something like HTMX, where everything is
purely server driven, state stateless.
Everything is from like the REST API protocol.
And there's a really nice simplicity to it where a state lives in one
place, the server's a source of truth for everything, and you use different
return values for each HTML sheet to decide what is going to render next.
And you accept that there's going to be network latency for most interactions
on the site, except for very small dropdown toggles and the like.
But anything that involves state is always going to go back to
the server and map it back.
That obviously has trade-offs that people have tried to get away from
with client side architectures.
But the reason it's so nice is you don't have to think about, you have
this server state, you have this client state, and you're constantly trying
to keep them sort of melded together.
And I think that's something that a lot of server side rendered applications have
run into most recently react trying to add on, like use optimistic hooks and
libraries like TRPC, letting you show data optimistically and then replace it
with the server value when it comes in.
And these are important principles, but it's a lot of manual effort to first send
a request to the server and also keep the client in some optimistic version
of that to cut down on network latency.
You're having to pull levers on both sides and you don't really
know where the truth lives.
So, from my experience that's led to a lot of manual work, writing and
rewriting local stores, maybe with Redux back in like the mid 2010s now
using, query hooks, even GraphQL if you're racing for those kinds of tools.
So it's, very messy using those things that are isomorphic, AKA, things that run
both on the server and on the client, and trying to think of it in the same way.
So one response to that is just, we don't need client side JavaScript.
We're gonna do everything on the server.
And that's very easy to understand because it goes back to like how
rest was designed in the eighties.
And then there's the other reaction, which is the, to other side, the
client is the source of truth for pretty much everything that's going on.
And the server's just a broker to keep different clients in sync
and to push changes so everyone can stay up to date or even doing
decentralized servers if you go further.
But it's that other side of the coin.
Of we want one source of truth.
We don't want to think about, we have this server and this client and we constantly
have to write logic to glue them together.
No, you either store all the state on the server, kind of like an HTMX style,
or you store all the state on the client using local-first technologies.
And having explored the first one for a long time, since Astro is meant to be
like a static website or server driven website, I was excited to explore the
other side of storing everything on the client, keeping it up to date, and then
figuring out how synchronization happens.
That's also been around for a long time.
If you look back to just like the first calendar app on your phone or on your
computer, this challenge has been around since probably SQLlite was created.
but it's only now that web devs are starting to get a lot of footholds
to also apply this to websites if that's something that you need.
I think you've summarized it really well.
And, I would go even as far as saying there is an elephant in the room that
most people are aware that there is an elephant, but they really don't
have yet the, the right terminology.
And I would say the term here of the elephant is distributed systems.
We have distributed states and distributed systems are really, like,
that's a core discipline of computer science that doesn't really have
just like the good answer, but it's a really, really hard problem alongside
of like naming things and so on.
But, that part, like everyone who's building a web app,
anything that has state.
On the server, like even in the server, typically you have by definition
also already a distributed system where you have states in your API
throughout the request lifecycle, but then you also have it in the database.
You might have like concurrent requests.
So there you already have a distributed system.
Typically there it's much less bad because they're like, you can just
trust the server already less.
So you just trust the database.
So you basically push it all down to the single choke point.
But if you wanna trust the client even more now all the distributed
parts are get the distance grows and therefore the divergence.
And I have in university, when I studied computer science, I don't
think I've actually taken a class on distributed systems and I've missed a
memo where someone would've instilled it in me is like, Hey, everything
that are you gonna build will suffer from distributed systems, make sure to
understand this problem properly and then design the architecture around it.
And I think most web devs are not aware of that.
And what you've just laid out, I think is exactly suffering from this problem.
And this is what I think where some framework creators are making
the very smart decision to empower the server as much as possible.
Because if you live by that sort of maxim, at that point, the implied
damage that you can cause by building it in a certain way is minimized.
And, my journey over the last five years or so has been almost like
intentionally letting the pendulum swing to the most other extreme where all the
state is governed by the client, since I'm pretty convinced that the middle
ground is just pain and suffering.
So I'm pretty convinced that you should really analyze the use case that you have.
And if it's a daily driver productivity app, you probably wanna move as much
state to the client as possible.
Where it happens, like you produce most of your state in your calendar app or in
your Notion, or in your other notes app or whatever you want to have it declined.
And, if it's something like New York Times, then you don't
produce any of that date.
So it should be on the server.
And, I, feel like more people should, start with that assumption, then
build an architecture around it.
But, yeah, I think you summarized it super well.
Yeah.
And it is tough.
To prescribe either side because there are neat buckets like content sites
and client side apps like Notion.
But the classic example is, well what about e-commerce where things are
server driven until you're in the add to cart flow and now it's client driven.
So what do you do then?
How do you architect it?
We actually met this challenge building the Astro storefront front template,
which was mostly server driven with some client side components sprinkled in.
I think the answer there was, it's still fine to leave things server
side and not rely on optimistic updates too much, except for like
small examples like increase and decrease quantity in your cart.
Do you want that to feel instant and happen on a delay or a debounce but
everything else is server driven.
There is no perfect architecture, I guess is what I'm saying.
It's more painful when you really try to blend it 50 50 and nail every use case.
But if you can keep it 80 20, where 80% is in one realm and 20% is business logic in
the other realm, like 80% server driven, 20% client side complexity or less, then
you're kind of minimizing the footprint of pain that you could run into since I've
certainly noticed that with some of the newer patterns and older patterns with.
Client side apps.
but there was a second thing you mentioned about distributed
systems that I, totally agree with.
I was reading the book Designing Data Intensive Applications, the big
O'Reilly book, probably the first one you find looking up like CS principles.
And I think there was one section about distributed systems where it walked you
from like single leader replication.
You got one database and if you want to have some caches to make reads
faster, you just put all your rights to one database and then it'll replicate
the reads out to everyone else.
So you can't write to a bunch of different regions.
You can write data to one region and then it'll sort of.
Push out all of the clones and allow eventual consistency to work.
but there are cases where that doesn't work anymore.
Like in Notion, I cannot wait for the server to update the document.
It's just gonna update when I'm typing and when I add blocks and when people
are invited to join the document.
All that stuff matters.
So you need some way to be able to write to maybe the nearest node.
Like if you go fancy with CloudFlare, you could have really localized,
durable objects that are two feet away from your computer and that reduces
latency in some sort of magical way.
but then it kind of explains well if you just put a data replica just on the
client device, that's another version of multi leader replication, where
instead of replicating in some really edge node, you're just replicating
on the person's computer and everyone is like a mini server unto itself,
where you just read and write data and they'll all report back to some
centralized source of truth later on.
So when you're working with, like, is it a distributed system?
It is.
Even when it's running on your device, if there's some sort of synchronization
layer, you're just moving it closer and closer and closer to the computer
until it iSQLiterally in the computer.
It's inside the house.
I've, heard a, very interesting framing of sort of like that
elephant in the room and like that problem that we just talked about.
I think it's by Carl who worked on SQLSync and recently, released Graft,
which is another fascinating project.
And he has written a blog post about, don't recall the exact name we're gonna
put it in the, show notes, but it was basically along the lines of like, your
application is a database and that's bad.
I gotta look it up.
what the, exactly the title was.
But it was basically, he's making the point that every app that is sufficiently
getting more complicated will basically build its own version of a ad hoc database
and if you're just using React useState or something else enough, and then
you try to add persistence, et cetera.
What your app is basically becoming is a poor implementation of a database
where like all the things that a database does a lot of like R&D great
work to make it fast, to make it correct, to make it like nicely, like
transactionally correct, et cetera.
All of those things you're now trying to handle through, useState and useEffect
and like when this thing changes, also change that thing, et cetera.
And, I just thought that framing was so elegant and, his conclusion, which
I agree with is like, hey, if let's try to make the app about what the app tries
to do and let's leverage a database so we can focus on the actual like
features, et cetera we want to implement.
And I think that's another kinda articulation of the elephant in the room
that we're accidentally and without being aware of it Building databases as that
are sort of camouflaged as apps and, we should embrace it more if we're starting
to see those signs of those problems.
So you've mentioned that you've gone down this path, a lot out of curiosity and
just because you've, pretty exhaustively explored the more server centric
paths, can you share more about like what were your frustrations and pain
points that you felt where you wanted to go more, like embrace the client
more, but still trying to do that with the more server centric architecture.
So you've mentioned optimistic state, et cetera, maybe can motivate
this through a concrete app you wanted to build where you just felt
that that pain and that impedance.
Yeah, I mean I've played with all sorts of side projects as we all have.
and I was working on a few different things.
One was like a localized note taking app, and I was also playing with local
LLMs and other pieces to add vector search into a local environment because
I was running up on the limits of using Notion and being very frustrated with
loading spinners and offline warnings when I was trying to use the app.
I hear that's being changed and rectified as they build out their
own, like SQLite replication.
I know that's in there.
There's some fascinating videos about that on the internet.
but I still do see the value of just open up Apple Notes, you type things and it's
just kind of there in a SQLlite store.
You can find it on your file system.
All your Apple Notes are just in a SQLlite store and it's great.
so I thought, well, it seems like that's the answer.
If I were to go the traditional server route, I would sit here
waiting for all these updates to persist, which just wouldn't work.
Or I'd be sort of gluing together a bunch of useState calls and figuring
out how do I update this server and what if someone else updates it?
What if I'm on my phone, I type a thing and then I go back to
my computer and I try to see it?
Is it gonna merge it together?
Is it gonna break?
I have no idea because.
Client side stores don't help you with those problems.
They're not designed for those problems.
They're designed for like ephemeral state that can disappear at a moment's
notice if it needs to for some reason.
so that led me down exploring local-first technologies and I found pretty quickly
how capable SQLite is for these use cases, including in the browser.
you can load up SQLite with like a wasm build as you're
very aware and use it even with.
Like very nice SQLite libraries.
If you wanted to use Drizzle for example, which is a common SQL querying library
and JavaScript, it matches onto the browser version of SQLite perfectly.
So you can actually make declarative, find many notes and
it'll just do the little SQL query.
It'll join it up with all the authors of the post and it'll
just give it back to you.
Kind of like you're on the server, but you're on the client.
And you can use client side ORMs or query builders, whatever
your flavor of preference.
So that was very empowering to see.
Yeah, you can bring all of these niceties you get from server side data
querying and bring it into the client.
And I tried to stretch it a little bit further by asking,
what about SQL extensions?
Could I add a vector search plugin, for example?
The answer is yes.
There actually is a vector search plugin that I think is developed
by someone on the Mozilla team.
So it is pretty battle tested in different languages.
I think they're sponsored, yeah I've had the chance to meet them in October
when I was in LA super lovely person.
And, they have some sponsorship from the Mozilla team currently.
Nice.
Yeah, and I was playing with the, rust flavor of that since, well now
I'm working at Warp, and Warp is a rust powered terminal, so naturally I
need to get up on the Rust knowledge.
And also if you build apps with Tori, which is a native desktop application
tool that uses Rust as well, so.
Side tangent, but it is nice to use tools that could work in JavaScript as well
as rust in very efficient languages.
so I reached for that.
I put vector search in.
I was also able to run an entire LLM across the data in the
browser and just load up like the entire vector search setup thing.
and I just said, all right, I'm gonna backport all of my
markdown files into this thing.
I'm gonna vectorize all of them in the browser.
I'm gonna search them in the browser.
And I was able to get it working with like next key search.
And it was absolutely mind blowing.
Like this is like an actual AI powered search tool.
And you can get like with every keystroke new results
And is isn't that wild like that this machine that we have like sitting
on our laps or like this machine here in my hands that is capable
of all of those things and just the way how we kind of build web apps.
Over the last decade or so, we've kind of forgotten or denied the
capabilities of our client devices.
And we only just like trust the server and we've almost like, why did no
one tell us that this is possible?
And that's so magical when you see that this is working and just the
stuff that's currently, in the works with like web, LLM, et cetera, where
it can run like a full blown like Llama model locally in your browser.
running on web GPU is absolutely wild to the capabilities that we have.
But I think what's holding us back is where does our data
live and everything else.
Kinda like, it's almost like a second order effect from where the data lives.
And this is what, your anecdote really nicely highlights of like, you go with
SQLite with your data and then you like bring in another superpower of like SQLite
Vec with the vector embedding, et cetera.
And I think it all starts with where the data is, how your
application is being shaped.
Yeah.
And it is good to find that.
Just common data layer.
SQLite is the easy answer.
PG light is a more robust exploration of bringing Postgres to local devices.
That's a bit earlier on.
but I think we're entering a world where software is just so easy to spin
up that you will very quickly have a web client, a mobile app client,
a desktop client, and they're all talking to the same sync server.
And when you're in that world, it's nice to just reach for SQLite.
'cause I can run SQLlite on my iPhone.
I can run SQLlite on my Android device, I can run it in the browser and I
could run it in a desktop application.
So as long as you just have this concept of clients write to SQL servers and
those SQL servers have some way to talk to each other, then you can build
these multi-platform applications very quickly, even across different languages.
And just figure out what that sync layer looks like, which
we can probably talk about.
that would've been my next question since, I think what you started with
was probably without the sync part yet, where you can just locally in
your browser web app, you successfully ran SQLite using the wasm build.
Then you brought in SQLite Vec, you could, get the AI magic to work.
You saw like how insanely fast everything feels.
You write something, you reload the browser.
Even if you're offline, it's all still there.
Great.
But now you're thinking, okay, it works on like local host 3000.
how do I get it deployed and how do I get it so that, if I accidentally
open this in a cognitive tab and I close it, poof, everything is gone.
and how do I get it show up on my, phone so this is kind of
collaborating with yourself, but on also collaborating with others, which
we maybe punt on that for a moment.
But yeah.
How did you go from, it works locally almost like if you use like just
local storage to trying to share the goodness across your devices.
Well, I think we both have the privilege of just ask Andrew on the Rocicorp team
and he'll point you to some resources.
if you don't have that, I do recommend that.
Well, the one approach that I used was just reading through the Replicache docs.
That explain how their sync engine works on a very high level.
And Replicache is a batteries included library, well some batteries included
library that sets up a simple key value store for you where you can write and read
keys on the client and you can author code that will post those changes to a server
that you own and pull changes back from the server whenever changes are detected.
And you can implement that on a SQLite or anything else if you just have
their simplified mental model, which is very inspired by just how Git works.
So in Git, if you had to change locally and you wanted to push it up to the
main line, you would run Git push.
And the same kind of thing happens with their sync engine service, where
anytime you make a change locally and you can decide what that means,
maybe it's a debounce as you're editing a document, maybe it's when
you click on a status toggle in Linear and you wanna change that status.
Whenever the trigger is, you can use that to call push, which would probably call
an endpoint written on your own server.
That's simply a push endpoint and that can receive whatever event you ran
on the client so that the server can replay it, back it up into its own
logs and tell other clients about it whenever they try to pull those changes
back down so you can run your own little push whenever you make a change.
And then other clients on an interval or a web socket or some
other connection can pull for data whenever they want to get the most
recent changes made by other people.
Now the question would be, what am I pushing and what am I pulling?
Like what data?
Needs to be sent across.
And there are a couple different methods.
I know Replicache uses a data snapshot mechanism.
I don't fully know the intricacies of it, but I know that because they use
like a key value storage, which is much simpler than like a full-blown database.
They can take a snapshot of what the server looks like right now and then the
client has its own copy of that server snapshot and it can just run a little
diff the same way you'd run a get diff to see what code changes you have made.
It can run that diff and you can see, all right, I added this key.
All right, updated this status flag, and then tell the server,
this is the diff, this is the change that was made to the data.
And the server can receive that request and say, okay, this is the change
that I need to make against my copy.
And then I will tell other clients to also apply that diff whenever they pull.
it's a very Git inspired model, and it works if you're able to diff data easily.
If you are working with like a full blown SQLlite table, you run table
migrations, tables, change shapes, that's a very hard thing to keep track of.
So another option that I implemented for the, learning resource I
created called Simple Sync Engine.
If you find that on my GitHub, probably in the show notes, you can see that.
But it was meant to be a very basic implementation of this pattern
that uses event sourcing instead.
So rather than sending a diff of how the data should change, instead sends
an event that describes the change that is made like a function call or
an RPC, however you wanna think about it, where you would tell the server
I updated the status to this value.
And in our implementation we create these little like function helpers that
can describe what those events are.
So you might have like an add status event.
So like the type of the event is add status and that accepts a few arguments.
It accepts the status you wanna change it to, and the ID of the
record that has that status currently.
So the server receives that event, it sees, okay, the type was set status.
I see two arguments here.
The ID of the record.
And the new status that should be applied.
I'm gonna go ahead and run that event as a database update.
So it has that mapping understanding of, I received this event, I know that
maps to this SQL query, so I'm gonna go ahead and make that change on my copy.
And then whenever people poll, you can send that event log out, or you
can send whatever events the client hasn't received up until that point.
You can kind of think of those like Git commits where you're pulling
the latest commits on the branch and the server's able to tell you, here
are the latest events that were run.
Go ahead and run those on your replicas so both the server
and client know what events.
Actually mean, like this event means I need to make this SQL query.
And it's able to do that mapping.
and you can have a bit of freedom there.
If you had like a very specific kind of data store on the server, like
MongoDB, you could customize it to say, whenever I receive this event, it
means this MongoDB query and this call to the Century error logging system,
or whatever middleware you wanna do.
As long as server and client agree on what events exist and what changes they
make in the data stores, then everyone can be on the same page whenever
they're syncing things up and down.
you're very familiar with event sourcing as well.
I'm curious if there's things that I've missed or important edge cases
that we should probably talk about.
I think you very elegantly, described how simple the foundation of this can be and
hence the name of like Simple Sync Engine.
I think this has served as a great learning resource for you and
I'm sure for many, many others.
And once you like, start pulling more on that thread, you realize, oh shit like,
okay, that thing I didn't think about.
Oh, what about this?
So you've mentioned already the reason why you preferred, event sourcing over
the snapshot approach because like with SQLlite, What would you actually compare?
This is where I would give a shout out again, to Carl's work with Graft.
this is what he's been working on.
We should have him on the show highlighting this as well.
But, where he's built a new kinda sync engine that, is all about that
diffing of like blocks of storage.
I think all focused around SQLite and that gives, that does the heavy work
of like, diffing and then sending out the minimal amount of changes
to make all of that efficient.
Since there's this, famous saying of like make it work, make it right
or correct and then make it fast.
And working on all of this has really given me a deep appreciation for all
of this since like, and I'm sure you probably also went through those stages
with the Simple Sync Engine, like with making something work in the first place
was already quite the accomplishment.
But then you also realize, ah, okay, in those cases this is not quite yet correct.
And then you could go back and like try to iterate and then you also realize,
okay, so now it has worked for my little to-do app, but if I, now, depending
on the architecture, if I roll this out and put in hack news and suddenly
have like 5,000 people there on the same time, this thing will break apart.
Will A not be correct in some ways you didn't anticipate and
will also not be fast enough.
So now making this fast.
It is at the end of the day is like really reinventing.
It's like your app is becoming a little database and you want
to like move as much of that database burden to the sync engine.
This is why folks like Rocicorp, ElectricSQL, et cetera, they're
doing a fantastic job trying to absorb as much of that complexity.
But building something like this by yourself really gives you an understanding
and an appreciation for what is going on.
I love the Git analogy that you've used, but just a, a couple of
points just similarly to how, your sync engine works is actually very
analogous to how the Livestore architecture on a high level works.
But I've had to, before I arrived at that, I really wanted to think through a lot
of like the more further down the road.
Like, what if situations since, one that I'm curious whether you've
already run into, whether you resolved in some way or left for the
future, is how would you impose a total order of your change events?
So this is where When you have, like, let's say this to-do app
or like a mini Linear, app.
Let's say you create an issue and then you say you, complete the issue or
you toggle the issue state, et cetera.
It can mean something very different, if one happens first and then the
other, or the other way around.
And for that, where you have your events of like, hey, the issue was created,
the issue status was changed, this, the issue status was changed, that,
the order really matters in which way in which order everything happened.
And, this might be not so bad if you're the absolutely only person using your
app and typically only on one device.
But when you then do stuff between multiple devices or multiple
users, then it's no longer in your single user's control.
That stuff happens concurrently.
And then it really matters that everyone has the same order of the events.
And this is where you need to impose what's called a total order.
And I'm very curious whether you've already, hit that point, where you thought
that through and whether you found a mechanism to impose it since there's many,
many downstream consequences of that.
Right.
And I definitely did hit it and.
You will find in the resource, it's not addressing that issue right now.
It's in a very naive state of when change is made, send, fetch, call to server.
And there are a few problems with that, even if you're the only client.
Because first off, if you send one event per request, it's very possible that you
send a lot of very quick secession events in like an order, and then they reach the
server in a different order because maybe you pushed the first change on low network
latency, the next one on high latency, or actually the reverse of that where
the first change hits after the second change because that's just the speed of
the network and that's what happened.
And also you need to think about offline capabilities.
If you push changes when they happen, how do you queue them
when you're not connected to the internet and then run through that
queue once you're back online?
That's another consideration you kind of have to think about.
Could be solved with just like an in-memory event log and
just kind of work with that.
But you still have the order issue.
I'm familiar with atomic clocks as a method to do this.
There are even SQLite extensions that'll sort of enforce that, having
not implemented atomic clocks.
Is it kind of this silver bullet to that problem or are there more
considerations to think about than just reaching for something like that?
Right.
I suppose you're referring to vector clocks or logical
clocks on a more higher level?
Yeah.
since the atomic clocks, at least my understanding is like that's
actually what's, at least in some super high-end hardware is like
an atomic clock that is, like that actually gives us like the wall clock.
So Right, right now is like.
Uh, 6:30 PM on my time, but this clock might drift, and this
is what makes it so difficult.
So what you were referring to with logical clocks, this is where it basically,
instead of saying like, Hey, it's 6:30 with this time zone, which makes
everything even more complicated, I'm keeping track of my time is like 1, 2, 3.
It like might just be a logical counter, like much simpler
actually than wall clock time.
but this is easier to reason about and there might be no weird issues of
like, Daylight saving where certainly like the, the clock is going backwards
or someone tinkers with the time, this is why you need logical clocks.
And, there, at least the mechanism that I've also landed on to
implement, to impose a total order.
But then it's also tricky, how do you exchange that?
how does your client know what like three means in my client, et cetera?
And the answer that I found to this is to like that we all trust.
A single, authority in the system.
So this is where, and I think this is also what you're going for, and with the Git
analogy, what we are trusting as authority in that system is GitHub or GitLab.
And this is where we are basically, we could theoretically, you could
send me your IP address and I could try to like pull directly from you.
It would work, and that would also work with the system that you've built.
However, there might still be, they're called network petitions,
where like the two of us have like, synced up, but some others haven't.
So as long as we're all connected to the same, like main upstream node, that
is the easiest way to, to model this.
An alternative would be to go full on peer to peer, which makes everything
a lot, lot, lot more complicated.
And this is where like something, like an extension of logical clocks called
vector clocks, can come in handy.
you've mentioned the, the book, designing dataset intensive application by Martin
Kleppman had him on the show before.
he's actually working on the version two of that book right now, but he's also done
a fantastic free course about distributed systems where he is walking through all of
that, with a whiteboard, I actually think so, I think does what, what the two of
you have very much like you've both nailed the, craft of like showing with simple
strokes, some very complicated matters.
so highly recommend to anyone who wants to learn more there.
Like, learn it from, from Martin.
He's, like an absolute master of explaining those difficult
concepts in a simple way.
But, yeah, a lot of things go kind of downstream from that total order.
So just to, go together on like one little journey to understand like a downstream
problem of this, let's say we have implemented the queuing of those events.
So let's say you're currently on a plane ride and, you're like.
Writing your blog post, you're very happy with it.
You have now like a thousand of events of like change
events that captures your work.
Your SQLite database is up to date.
but you didn't just create this new blog post, but you maybe while you're still at
the airport, like you created the initial version with it with like TBD in the body.
And your coworker thought like, oh, actually I have a lot of thoughts on this.
And they also started writing down some notes in there.
And now, the worlds have like, kind of drifted apart.
Your coworker.
Has written down some important things they don't want to lose,
and you've written down some things you are not aware of the other ones
neither are they, and at some point the semantic merge needs to happen.
But how do you even make that happen in this sync engine thing here?
And this is where you need the total order, where you basically, in the worst
case, this is what decides, like who, gets a say in this, who gets the last say, in
which order those events have happened.
The model that I've landed on, and I think that's similar to what Git
does with rebasing, is basically that before you get to push your own stuff,
you need to pull down the events first, and then you need to reconcile
your kind of stash local changes.
On top of the work that whoever has gotten the, who got lucky enough to push
first without being told to pull first.
So in that case, it might have been your coworker because they've
stayed online and kept pushing.
And now it sort of like falls on you to reconcile that.
And I've implemented a, like an actual rebase mechanism for this,
where you now have this set of new events that your coworker has
produced and you still have your set of events that, reflect your changes.
And now you need to reconcile this.
So that is purely on the.
Event log level, but given that we both, want to use SQLite now, we don't
need to just think about going forward with SQLite, but we also now need to
think about like, Hey, how do we go?
Like in Git you have like, you have this stack of events, right?
So you have like a commit, which has a parent of another commit, which
has a parent of another commit.
It's very similar to how your events and this event log look like, except it's now
no longer just one event log, but you also get this little branch from your coworker.
So now you need to go to the last common ancestor.
And from there you need to figure out like.
How do I linearize this?
I've opted for a model where everything that was pushed once cannot be
overwritten, so there's no force push.
So you basically just get to append stuff at the end.
But, in order to get there, you need to first roll back your own stuff, then
play forward what you've gotten first.
and then on top add those.
And the rolling back with SQLite is a, thing that I've like put a lot of
time into where I've been using another SQLite extension, called the SQLite
Sessions extension, which allows you, per SQLite write, to basically, record
what has the thing actually done.
So instead of storing, insert.
Into issues, blah, blah, blah.
when running that, you get a blob of let's say 30 bytes, and that has
recorded on SQLite level, what has happened to the SQLite database.
And I store that alongside of each change event, that sits in the event log.
And the very cool thing about this is, I can use that to replay it
on top of another database, but to kind of catch it up more quickly.
But I can also invert it.
So now I have basically this like, let's say 20 events.
And for each, I've recorded what has happened on SQLite level,
and now I can basically say.
When I need to roll back, I can revisit each of those, invert each of those
change sets, apply them again on the SQLite database, and then I'll end up
where I was before and that's how I've implemented rollback on top of SQL Lite.
So this is as mentioned when you're going, down the, rabbit hole
of like imposing a total order.
There's a lot of downstream things you need to do that makes
this even more complicated.
But, from what I can see, you're, on the right track if
you wanna pursue this further.
Yeah.
And I do have a rebasing mechanism in place in mind that's more,
just kind of a sledgehammer.
I got two SQLite databases in mind.
in the same way that on Git you have like your local copy of the main line and your
local copy of your work, there's always this local copy of Main, that's just
whatever events have come from the server.
So this is the source of truth that the server has told me about and that was
something I forgot to mention earlier.
Explaining all of this is the server is the source of truth.
It has that main line of the order of all of the events, and that is
what all the clients use to trust.
But yeah, it has like that local copy, and then when it pulls from
the server, it'll update that copy.
It'll look at all the events that are kind of ahead in the client,
and then it'll say, okay, I'm gonna roll back my client copy of my
branch to whatever the server is.
And it's literally just a file right call.
So it just overwrites.
Your like client SQLlite file with a copy of the server one.
And then we look at the events that the server didn't acknowledge yet
and then we replay those on top as a very basic way to pull and make
sure, because it's very possible that you made some changes locally that
the server hasn't acknowledged yet.
Like you've pushed them up still in process and you pull down the
latest changes and you don't see all of that stuff that you pushed
up yet because of network latency.
So this sort of avoids that problem where you pull down from the server
and now you need to replay whatever you did on the client that the
server hasn't acknowledged yet.
It hasn't received that network request.
So that was a very basic need to have some rebasing, but it does
get a lot more complicated when you have collaborators on a document.
I've seen a few different versions of this.
CRDTs is the fun, like magic wand.
It does everything.
but there are also solutions from Figma, for example, where they
say everything in Figma is kind of its own little data structure.
Like you can put some text and that's its own little data field.
You have rectangles.
Those are a data field.
And whenever you update a rectangle, like you update the pixel width of
a rectangle, that's like an update event on some SQL table that stores
all the rectangles for this document.
So whenever you make that update, it'll update the pixel value of whatever
that row entry is, and then it'll push it up for other people to receive.
And when you pull it down, it's last right wins.
In other words, whoever the last person is in that order that the
server decided on that total order.
That's a new word I know about now.
Didn't know it was called total order, but yeah, that, once you pull it down,
whatever the server said was the order of events, that's gonna be the final
state of that rectangle on your device.
The only time it becomes a problem, and you may have experienced this, if you're
ever working on like a fig jam together with a bunch of people, if you're all
typing in the same text box, everyone's just like overriding each other and a
text box glitches out and changes to whatever's on the other person's screen.
You can't see people's cursors because you're fighting to update
the exact same entry in the database and it can't reconcile those changes.
so it only works up to, like you're editing different things
in the file and you're not really stepping on each other too much.
As soon as you're stepping on each other trying to edit like the same text field,
then you wanna reach for something that's very, very fancy, like CRDTs.
Which will try to merge elegantly all of the changes that you're
typing into the same database field.
It's maybe over-prescribed because of how powerful it is, but for those specific
scenarios, it's really nice to reach for, and we can talk about them if you want.
I only have a high level understanding of what CRDTs do, but it would be
something to apply that kind of problem.
my takeaway from where to apply, CRDTs versus where I would apply event sourcing
is, CR DTs great for in two scenarios.
One, if you don't quite know yet where you want to go.
And where in the past you might've reached for, let's say, Firebase to
just like have a backend of service.
You know, you might want to change it later, but you just, for now,
you just want to get going and, you can, particularly if you
don't have like a strict schema across your entire application.
So you just try to like, not go off the rails too much, but at least the
data is like, mostly, like across the applications in a good spot.
But as you roll this out in production, and, we are shipping
an iOS app as well, that someone is, running an old version on.
Now you don't quite know, oh, this document, this data document that has
been synced around here, this might not yet have this field that the
newer application version depends on.
So now you have, like, this is where time drifts in a more significant
way and in the more traditional application architecture approach
you would, this way you don't trust the client in the first place.
Then you have like your API endpoint and the APIs, versioned, et cetera, and
everything is governed through the, API.
But now you also need to tame that problem somehow.
So at this point you're already, going a little bit beyond where I
think CRDTs shine right now, which brings me to my next kind of more
evergreen scenario for CRDTs, which are like very specific, tasks.
And so text editing, particularly rich text editing.
Is such a scenario where I think CRDTs are just like a very, very good, approach.
There's also like, you can also use ot, like operational transform, which
is, somewhat related under the covers, works a bit differently, but the way how
you would use it is pretty similarly.
And, related to rich text editing is also when you have like complex
list structures where you wanna move things within the list.
So if you want to go for the, Figma scenario, let's say you change the
order of like multiple rectangles, like where do they sit in that layer order?
how do you convey how you wanna change that?
You could always, have like maybe an array of all the IDs that give
you this perfect order, but if this kind of happens concurrently,
then you need to reconcile that.
So that's not great.
And this is where CRDTs are also like a very, special purpose
tool, which works super well.
And so what I've landed on is use event sourcing for everything except
where I need those special purpose tools, and this is where them reach
for CRDTs or for something else.
That's kind of the conclusion I, took away if you like the event sourcing approach.
But, I think ultimately it really comes down to what is the application
that you're building and what are, like, what is the domain of what
you're building and which sort of trade-offs does this require?
So I think in Figma.
The real timeness is really important and it is recognized that those different
pieces that are floating around, they're like pretty, independent from each other.
So, and if they're independent, then you don't need that total order
between that, which makes everything a lot easier in terms of scalability,
in terms of correctness, and then you don't need to rebase as much.
distributed systems is the ultimate case of it depends.
and I think trying to build one like you did, I think is a very good way
to like build a better understanding.
And also I think that opens your eyes of like, ah, now I understand why Figma
has this shortcoming or Notion if we are trying to change the same line, change the
same block as where last writers, applies.
Whereas in Google Docs, for example, we could easily change the, same word even.
And it would reconcile that in a, in a better way.
But, maybe you have some advice for people like yourself when you're
just getting started on that journey.
What would you tell people what they should do maybe shouldn't yet do?
today 2025?
There's more technologies out there now.
What would you recommend to someone who's curious?
Depends on the type of learner you are.
Sometimes some are very.
outcome driven, like I need to see an app running in production for me to
really get excited about this Tech.
Other people are very first principles driven.
Like I want to like screw in every nut and bolt myself to
get excited about this thing.
I tend to fall into the first camp where I think it is very useful to just
look at the docs for something like Replicache and see how would you implement
this kind of protocol step by step.
Like how would you set up the event sourcing?
How would you put the SQLlite store in the browser in the first place?
Like what capabilities are there?
And then try to think through those edge cases.
As you run into them trying to build something, I use Linear as my sort
of learning example, but you could use pretty much anything you want.
so that's definitely one approach.
Now there's just so many resources for how these things work under
the hood that you can easily learn about the intricacies yourself.
Another, resource is the talks given by who's the engineer at Linear.
I think you've had him on the show.
Tuomas?
Yes.
Yeah.
He gave a few really helpful talks about how the Linear sync engine works on a high
level, and that one's more opinionated.
It reaches for technologies like MobX, which is a react specific state
store, and also MongoDB for documents.
but you still get a high level of how they think about the
problem, which is really nice.
the other option, if you're really results driven, you wanna
see a local-first step running.
You can reach for all sorts of frameworks and libraries at this point.
Zero is the one that I've played with most recently, and it is Alpha Software you'll
run into, it holds your hands, plugging in every battery and setting up everything.
But error codes could be very confusing.
but luckily their Discord is very welcoming and will answer any question
that you have since their only goal is for everyone to get excited about
the tech and use it in production.
So I think Zero is a really great starting point as just, I wanna build an app.
I'm gonna reach for a library.
It will give you a query builder.
So instead of writing raw SQL, it'll help you write SQL queries
with some JavaScript functions.
And it also works you through very common problems that you do hit at some point.
And the big one is data migrations and, well, not data migrations, schema
migrations, because when you have a data store on the client and you have a source
of truth on the server everyone has to agree on how that data is shaped if
you're using a SQL model and not something that's Firebasey as you were mentioning.
So in those cases, you have to know like the four or five step
process of update the server schema to add the new field, then update
the client to add that new field.
And then if you're trying to delete an old field for some reason, you would
need to execute those on client, then server in the correct order, and then
manage a database version so that if a client tries to connect with really,
really old application code, the server can say, sorry, I only accept people who
are on version five of this SQL schema.
You're on version three, so I'm just gonna hard refresh your webpage and
get you up to the latest version.
all of these challenges are really interesting to think about and Zero
helps you think through them out of the box and presents docs on all of these
problems before you run into them.
but I happen to be the type that wants to run into as many brick
walls as possible without someone telling me what to worry about.
I just wanna worry about it.
so I think the Simple Sync Engine resource is great just because it
doesn't do very much and there's a lot left up to the reader to go off
and try to run into those challenges.
I'm sure splunking through like the LiveStore implementation, I would
find 50 ways that I could improve what I'm doing to get to that next step of
like resilience, schema, migrations.
I literally didn't even touch schema migrations.
there's so much that you need to think about that just crawling
through open source libraries is really, really helpful with, so
that's my preferred learning approach.
I just like going that way.
I completely agree.
And I also like, it's, it's sort of a bit of convincing yourself,
is this entire thing worth it?
And what I always appreciate if someone knows a little thing about
me and then tells me, you know what?
I don't think this is for you.
I wouldn't hold anything back for someone who wants to look into this.
to say like, this might not be what you're looking for.
If someone is very happy with like building web apps with Vite Astro NextJS,
et cetera, and they're productive, they're building this, e-commerce
platform, or they're building a more static website, I don't think there's
anything really where local-first would change their work situation.
But if they're frustrated with like actual apps that they use day to day, when you're
frustrated like yourself, when you're frustrated with Notion being too slow,
et cetera, and you're building those more productivity daily driver apps yourself.
For me that was like a music app.
I got frustrated with Spotify and other music apps.
I think this is the, right scenario where like local-first has something to offer,
but, and I think it has also the potential to become a lot simpler and easier over
has already become a lot simpler and easier or the past couple of years, and
it's gonna be even more so in the future.
And there will be use cases where it's actually simpler.
To use local-first to build something, then using Next for something.
but that won't apply to all scenarios.
And so it is not a silver bullet.
the closest thing you'll get to a silver bullet is the right architecture for
the right application scenario, but by default there is no silver bullet.
Neither is local-first.
And I think someone should evaluate, Hey, is this even for me?
that's, I think should be the starting point.
Yeah, and a meta comment just because now I'm in the agent coding space, Warp
is getting more capable by the day of actually editing files and scaffolding new
applications for you, from the terminal.
I've found it's less valuable to know the syntax of how all of these libraries work
and a lot more valuable to just know high level, what are they doing, what's the
architecture and how would I debug it?
because these agents are very good at spitting out the syntax,
if you draw a very clear picture.
So if you go off and read designing data intensive applications, and you
start diagramming to yourself how all of these systems are distributed, you
could bring that diagram to Warp or just the cloud website if you want, and
say, I wanna build this kind of app.
Here's how the architecture's gonna work.
This is gonna talk to this, and I know about this library.
I know LiveStore uses event sourcing, so I would like you to implement that
and use React, but follow the handrails because I understand the architecture.
It'll give you a way better application than if you were to just say, give
me a local-first app with React.
It would probably maybe not struggle in the beginning, but definitely
struggle as you try to figure out what it is doing or debug whatever sort
of system level issues you're having.
I fully agree.
And given that the both of us are not just application developers but also
tool creators, we spend a lot of time thinking about like, how do I leverage
the degree of freedom that I have here in the API, the way how I design the API
that is intuitive for someone that they like, ideally that this becomes like a
pit of success where they intuitively use it the right way, but also if they
use it the wrong way, how do they notice?
Do they notice like as early on as possible through type safety or only
if they're already in production and they felt like, wait, no,
this, like, this was a path that I've wrongly taken six months ago.
so you want to design all of this in a way that you like learn as early as possible
whether you're in the right track or not.
And I think you can't get better than simplicity than going for simplicity.
And this is why I love the path that you've taken with the Simple Sync Engine.
Through the push pull model because that's already, that is deeply familiar for
developers and that is how we're using Git and that has really been proven.
And there you can't really get much simpler than that.
And I think simple is great for everyone.
and once we have a simple foundation, we have a reliable foundation.
We can build fast and nice things on top of it.
But particularly mentioning AI systems, I make a lot of design trade offs
now differently, where I care less about how much effort it will be
to write or to discover that thing.
Since we have now LLMs, do TXT, et cetera, I care a lot more about like,
how does, how will you even spot where like this doesn't seem right.
The robot has given me something weird and just doesn't match my,
like, primitive understanding of how this entire thing fits together.
And that should also help the robot to like not go in the wrong
direction in, in the first place.
So, yeah, I love that.
Like, and going for a simple design decision, the simple like overall
system architecture that's gonna help you as a future programmer, observing
little robots, building things, a lot more to know what's going on.
So maybe we use that as a last segue to hear a little bit more
about what you're doing now at Warp in regards to agents, et cetera.
I've been using Warp for, a little bit.
I still use it, for my standalone, terminal, but most of my terminal
work is also happening within Cursor, which is integrated in like the
agent thing and Cursor, et cetera.
Maybe can, yeah, help me a little bit of like how I bring those two together
and use them for what they're best at.
Yeah, and it's an interesting world of sort of agentic coding solutions.
It feels like there's a new approach to it every other day.
before joining Warp, I was also using Kline a lot, which is a VS Code
extension that's fully open source that will, from my experience, give
you a more quality agent output.
Since it has these two phases, you can flip on a plan switch and
it'll use a reasoning model to take whatever you tell it and turn
it into like a step-by-step plan.
And you could walk through like, no, that architecture doesn't make sense,
or Let me upload this six cal draw.
I actually want to work like this.
And you can go back and forth on like a design doc and then you can
flip it to act mode and that engages Claude to go build that for you.
And it's very hit or miss actually doing like the code edits.
We're all struggling with that.
but it was like this really nice mental model of, oh yeah, we're
gonna plan it out together.
We're gonna design jam, how this thing's gonna work and then we're
gonna go build it and I can just let it run and see how it ends up working.
and Warp is doing something similar but not within the confines of VS code.
And also with the addition of a voice button that I've been using a lot more
recently because you can talk faster than you can type is generally what I found.
So I can just speak into my terminal, here's how I want the app to
work, this, that, and the other.
And then it will, depending on how complicated the question is, it
will reach for the planning step.
Otherwise it'll just give you an answer right away.
So if it, see that's kind of complicated, let me plan it out.
It'll give you that same kind of like document, here's how it's going to work.
And then you can say, okay, do it.
And then it will go off and tell Claude to make file edits and do other
things on your machine, which is much further than Warp has gone in the past.
and I've really enjoyed using this to build Swift apps recently.
Since I was just fascinated with like, how could I build a really
slick desktop client like ChatGPT there's this desktop shortcut to
like pull up a little chat bar.
Like I want something that integrated.
I don't wanna be confined to Google Chrome anymore.
I wanna break out of it.
but if you open up XCode, you're just met with this like decade old auto
complete that doesn't have anything that you want in order to get stuff done.
but Warp is just a terminal.
So I'm like, okay, I'll just open the Swift project in Warp
and say implement this feature.
And it doesn't it, it can literally just enter any directory that you
have and just start doing things.
I've also used it to like migrate my open source projects from a mono repo to a
set of micro repos on my system and says, oh yeah, I'll just make a new directory.
I'll move all those files over and I'll make the necessary file edits with Cloud.
Very like hit or miss quality.
We're dialing it in.
But this idea of you're not constrained to the IDE anymore, you can kind
of just pull up your terminal and ask it to modify anything from the
simplest request of, help me get revert, whatever the heck I just did.
And it'll help you get to something more complicated, like, why
isn't my Postgres server running?
And then it'll check your Homebrew installation.
And then you can take it one step further to, I actually want to fix this
error I see in my dev server right now.
Because you're running the dev server in your terminal, it can say, all
right, pause the server, debug, debug, debug, restart the server.
And then if something fails again, I'll go back to debugging.
So it's like watching your terminal session and figuring
out how to help you do something.
It feels like it's this natural next step of let's go from you're in an
editor typing code quickly to like, this is a general purpose tool on
your machine to edit all the software that you're writing in any setting.
so I'm just very excited about that kind of future.
And we've been moving very, very quickly towards it.
Just in the past month, it's gone from barely usable to, I'm actually using
this a lot for projects in a language that I don't even know how to speak.
Swift.
it's been kind of crazy how far you can get from zero to one
without a lot of field knowledge.
And intuitively this makes a lot of sense since like Eternal is kind of like the
OG chat up in a way where like all the way back to like IRC, et cetera, but
also now with, using ChatGPT a lot or, or other LLM chat products, like yes,
you're, chatting with the thing, like you write a command, the command happens
to be like plain English or another language, and you get something back.
But the, kinda like back and forth.
And the interplay is very similar and it makes so much more, so much sense that you
now bring that into the terminal as well, where you get the best of both worlds.
You can like ride out things in a fuzzy way.
The terminal helps you to like put that into proper computer speak.
but then you also get the efficiency and the correctness from what you can do in
a terminal and with all of like just a top-notch craft that you get within Warp
as a terminal with like blocks, et cetera.
So yeah, highly recommend everyone to, give it a try.
Yeah.
And it's free to reach for all those things.
And anyone who is just bothered by AI in their terminal, you can turn it
off and just use Warp as a really good terminal, which is how I started using
it way back, probably like, 2021, I think is when it said I created my account.
I just used it because I wanted something that looked nice and
now it's going a lot deeper.
And yeah, the, chat app analogy is perfect.
There's literally a toggle between typing out commands and asking it a question,
and it'll even flip back and forth based on like natural language, which is fancy.
I mean, I'll just hit the keyboard shortcut, but why
not make it a little flashier?
That is awesome.
Well, I've already seen, a bunch of your recent videos,
about content related to that.
I'm looking forward to many more of those.
Is there anything else that you wanna share related to your local-first,
explorations or otherwise?
Yeah.
so my profile is bholmesdev everywhere.
So any learning resources I've put out like videos on local-first and
conference talks, bholmesdev on YouTube and Twitter and Bluesky.
And also on GitHub, so these Simple Sync Engine project I mentioned,
that's on my personal GitHub.
Also under bholmesdev.
You should see it as one of the star repos if you look up the profile.
That's it.
Perfect.
I've also put it in the show notes, so everything, you'll find it there as well.
But, I'm really, really excited you have put in the effort to create this project,
because I think there's no better way to learn something than trying to do it.
And you've done that and you've allowed other people to follow
your footsteps, and I think that's a fantastic learning resource.
So thank you so much for doing that and for, yeah, helping others learn
and, sharing what you've learned and for coming on the show today.
Thank you.
Yeah, thanks so much.
This was like a really far reaching conversation.
I hope it turns out good.
Thank you for listening to the localfirst.fm podcast.
If you've enjoyed this episode and haven't done so already, please
subscribe and leave a review.
Please also share this episode with your friends and colleagues.
Spreading the word about the podcast is a great way to support
it and to help me keep it going.
A special thanks again to Jazz for supporting this podcast.
I'll see you next time.