1 00:00:00,000 --> 00:00:02,460 What if I'm on my phone, I type a thing and then I go back to 2 00:00:02,460 --> 00:00:04,170 my computer and I try to see it? 3 00:00:04,440 --> 00:00:06,300 Is it gonna merge it together? 4 00:00:06,300 --> 00:00:07,320 Is it gonna break? 5 00:00:07,350 --> 00:00:09,000 I have no idea because. 6 00:00:10,035 --> 00:00:12,225 Client side stores don't help you with those problems. 7 00:00:12,225 --> 00:00:13,875 They're not designed for those problems. 8 00:00:14,144 --> 00:00:18,884 They're designed for like ephemeral state that can disappear at a moment's 9 00:00:18,884 --> 00:00:21,045 notice if it needs to for some reason. 10 00:00:21,518 --> 00:00:26,198 so that led me down exploring local-first technologies and I found pretty quickly 11 00:00:26,198 --> 00:00:31,568 how capable SQLite is for these use cases, including in the browser. 12 00:00:32,096 --> 00:00:34,376 Welcome to the localfirst.fm podcast. 13 00:00:34,976 --> 00:00:37,946 I'm your host, Johannes Schickling, and I'm a web developer, a 14 00:00:37,946 --> 00:00:41,036 startup founder, and I love the craft of software engineering. 15 00:00:41,696 --> 00:00:45,416 For the past few years, I've been on a journey to build a modern, high quality 16 00:00:45,416 --> 00:00:49,766 music app using web technologies, and in doing so, I've been falling down the 17 00:00:49,766 --> 00:00:51,626 rabbit hole of local-first software. 18 00:00:52,226 --> 00:00:55,106 This podcast is your invitation to join me on that journey. 19 00:00:55,643 --> 00:00:58,403 In this episode, I'm speaking to Ben Holmes. 20 00:00:58,748 --> 00:01:02,828 A senior web developer and educator known for his whiteboard videos 21 00:01:03,368 --> 00:01:07,148 after having spent most of his career building server centric applications. 22 00:01:07,448 --> 00:01:11,918 Ben recently explored local-first software by building a simple sync engine 23 00:01:11,918 --> 00:01:14,678 from scratch before getting started. 24 00:01:14,888 --> 00:01:18,278 Also, a big thank you to Jazz for supporting this podcast, 25 00:01:18,638 --> 00:01:20,288 and now my interview with Ben. 26 00:01:20,817 --> 00:01:22,707 Hey Ben, so nice to have you on the show. 27 00:01:22,707 --> 00:01:23,367 How are you doing? 28 00:01:24,447 --> 00:01:25,977 Hey, I'm doing great. 29 00:01:26,217 --> 00:01:27,027 Yeah, how are you doing? 30 00:01:27,957 --> 00:01:29,037 I'm doing fantastic. 31 00:01:29,037 --> 00:01:31,017 Super excited to have you on the show. 32 00:01:31,159 --> 00:01:34,939 I'm very certain that most of the audience already are familiar with who 33 00:01:34,939 --> 00:01:40,651 you are since you have quite a reach on, Twitter, x, other platforms, et 34 00:01:40,651 --> 00:01:45,181 cetera, where I think you're doing an excellent job of taking like novel 35 00:01:45,181 --> 00:01:49,690 concepts and breaking them down in a very simple and approachable way. 36 00:01:49,960 --> 00:01:54,310 And I think you've done the same for some local-first related topics recently. 37 00:01:54,310 --> 00:01:58,690 So maybe some of the audience are already familiar with your work in that regard. 38 00:01:59,140 --> 00:02:03,013 But, for those who are new to you, would you mind giving a background who you are? 39 00:02:04,168 --> 00:02:05,158 Yeah, totally. 40 00:02:05,388 --> 00:02:09,378 I'd be shocked if everyone knows, but if you've seen a guy with a whiteboard 41 00:02:09,498 --> 00:02:13,188 around the internet, it might be me, especially if it's a vertical video. 42 00:02:13,542 --> 00:02:19,932 so I've been doing a lot of work in just the engineering space for a long time. 43 00:02:20,142 --> 00:02:26,142 So been at Astro for a few years and we've been building the 44 00:02:26,142 --> 00:02:27,792 framework for content sites. 45 00:02:28,032 --> 00:02:31,402 And I know actually you have some experience working with Contentlayer 46 00:02:31,422 --> 00:02:34,632 and other things, but content sites are a nice place to get started 47 00:02:34,632 --> 00:02:39,372 because they're a nice, well-defined use case for web technologies and 48 00:02:39,372 --> 00:02:42,762 Astro was trying to spearhead being like the simplest way to do that. 49 00:02:43,465 --> 00:02:48,655 so was able to contribute to that for a long time and also produce a lot of 50 00:02:48,655 --> 00:02:50,995 videos on learnings in the process. 51 00:02:51,085 --> 00:02:55,405 So it started with taking a rock band microphone and a whiteboard outta my 52 00:02:55,405 --> 00:03:00,865 closet and just recording stuff and just seeing where it went and been doing it for 53 00:03:00,865 --> 00:03:03,235 years now and kind of honing the craft. 54 00:03:03,625 --> 00:03:07,655 And we've covered all sorts of topics, including, different 55 00:03:07,655 --> 00:03:12,800 web frameworks, Tailwind tips, database providers using SQLite. 56 00:03:12,815 --> 00:03:16,295 Most recently putting SQLite in the browser, which is gonna be 57 00:03:16,295 --> 00:03:18,095 very relevant today, I'm sure. 58 00:03:18,588 --> 00:03:23,418 and most recently I've made the jump to Warp Terminal, so I'm gonna 59 00:03:23,418 --> 00:03:28,458 be, well, I have joined their team as a product engineer and we're 60 00:03:28,548 --> 00:03:33,888 sort of spearheading the future of bringing AI and agent workflows to 61 00:03:33,918 --> 00:03:35,328 everything you do in the terminal. 62 00:03:35,418 --> 00:03:38,748 So if you forget a command, need to kill a port, need to go through 63 00:03:38,748 --> 00:03:43,188 a full get workflow or even edit files in your projects, you can just 64 00:03:43,188 --> 00:03:46,818 ask Warp to do that for you or fall back to all the features that make 65 00:03:46,818 --> 00:03:48,348 Warp just a really nice terminal. 66 00:03:48,528 --> 00:03:49,578 It's a really great fit. 67 00:03:49,698 --> 00:03:53,238 Been working with them for a while and now get to do that full time. 68 00:03:53,298 --> 00:03:56,118 We'll continue to do all the videos that you see around the internet. 69 00:03:56,515 --> 00:03:57,325 That is awesome. 70 00:03:57,325 --> 00:04:01,225 I'm certainly coming back to Warp since I wanna learn more about that as well, 71 00:04:01,495 --> 00:04:03,925 but taking things one step at a time. 72 00:04:04,271 --> 00:04:08,411 you've been digging into local-first related, you've been mentioning 73 00:04:08,411 --> 00:04:10,481 SQLite running SQLite in the browser. 74 00:04:10,923 --> 00:04:13,923 Looking into that over the course of the last year or so. 75 00:04:14,286 --> 00:04:19,626 I'm very curious, like what led you to explore that since with Astro, et 76 00:04:19,626 --> 00:04:24,833 cetera, you're probably, that's like a different part of building websites, 77 00:04:24,833 --> 00:04:28,973 really where you maybe like Astro is famous for introducing the island 78 00:04:28,973 --> 00:04:35,180 architecture where you go like very light on bringing JavaScript into the, website. 79 00:04:35,630 --> 00:04:40,280 And if it's almost like the other extreme where you want to bring a lot of 80 00:04:40,280 --> 00:04:45,716 JavaScript into the web app, deemphasize the server part, and also more on 81 00:04:45,716 --> 00:04:48,086 the spectrum from website to web app. 82 00:04:48,086 --> 00:04:50,756 Certainly much heavier on the web app spectrum. 83 00:04:51,056 --> 00:04:51,926 So yeah. 84 00:04:51,926 --> 00:04:54,626 Can you share more what led you to, to this path? 85 00:04:55,053 --> 00:04:59,476 well the past year has been kind of like an existential crisis of how are you 86 00:04:59,476 --> 00:05:03,046 supposed to build websites, and I think we all went through that as an industry, 87 00:05:03,286 --> 00:05:09,766 as things kind of shifted back from client side, heavy react apps towards things 88 00:05:09,766 --> 00:05:14,386 that are more server rendered and people are slowly trying to question that idea 89 00:05:14,386 --> 00:05:19,006 again and see what we can learn from storing more information in the client 90 00:05:19,006 --> 00:05:20,836 and sinking it back to your servers. 91 00:05:21,136 --> 00:05:26,476 So I also noticed a wave that's completely opposite. 92 00:05:26,986 --> 00:05:32,866 Of local-first apps, which would be something like HTMX, where everything is 93 00:05:32,866 --> 00:05:36,346 purely server driven, state stateless. 94 00:05:36,466 --> 00:05:39,706 Everything is from like the REST API protocol. 95 00:05:40,366 --> 00:05:45,166 And there's a really nice simplicity to it where a state lives in one 96 00:05:45,166 --> 00:05:50,656 place, the server's a source of truth for everything, and you use different 97 00:05:50,656 --> 00:05:55,576 return values for each HTML sheet to decide what is going to render next. 98 00:05:55,876 --> 00:06:00,346 And you accept that there's going to be network latency for most interactions 99 00:06:00,346 --> 00:06:04,246 on the site, except for very small dropdown toggles and the like. 100 00:06:04,306 --> 00:06:07,156 But anything that involves state is always going to go back to 101 00:06:07,156 --> 00:06:08,746 the server and map it back. 102 00:06:09,166 --> 00:06:12,766 That obviously has trade-offs that people have tried to get away from 103 00:06:12,766 --> 00:06:14,326 with client side architectures. 104 00:06:14,716 --> 00:06:18,556 But the reason it's so nice is you don't have to think about, you have 105 00:06:18,556 --> 00:06:22,006 this server state, you have this client state, and you're constantly trying 106 00:06:22,006 --> 00:06:24,646 to keep them sort of melded together. 107 00:06:24,766 --> 00:06:30,616 And I think that's something that a lot of server side rendered applications have 108 00:06:30,616 --> 00:06:37,126 run into most recently react trying to add on, like use optimistic hooks and 109 00:06:37,126 --> 00:06:41,566 libraries like TRPC, letting you show data optimistically and then replace it 110 00:06:41,566 --> 00:06:43,366 with the server value when it comes in. 111 00:06:43,786 --> 00:06:49,246 And these are important principles, but it's a lot of manual effort to first send 112 00:06:49,246 --> 00:06:53,986 a request to the server and also keep the client in some optimistic version 113 00:06:53,986 --> 00:06:56,146 of that to cut down on network latency. 114 00:06:56,176 --> 00:06:58,786 You're having to pull levers on both sides and you don't really 115 00:06:58,786 --> 00:07:00,466 know where the truth lives. 116 00:07:00,886 --> 00:07:06,451 So, from my experience that's led to a lot of manual work, writing and 117 00:07:06,451 --> 00:07:11,851 rewriting local stores, maybe with Redux back in like the mid 2010s now 118 00:07:11,851 --> 00:07:16,478 using, query hooks, even GraphQL if you're racing for those kinds of tools. 119 00:07:16,868 --> 00:07:23,635 So it's, very messy using those things that are isomorphic, AKA, things that run 120 00:07:23,635 --> 00:07:27,715 both on the server and on the client, and trying to think of it in the same way. 121 00:07:28,255 --> 00:07:32,905 So one response to that is just, we don't need client side JavaScript. 122 00:07:32,905 --> 00:07:34,585 We're gonna do everything on the server. 123 00:07:34,885 --> 00:07:38,575 And that's very easy to understand because it goes back to like how 124 00:07:38,575 --> 00:07:40,285 rest was designed in the eighties. 125 00:07:40,705 --> 00:07:43,375 And then there's the other reaction, which is the, to other side, the 126 00:07:43,375 --> 00:07:46,765 client is the source of truth for pretty much everything that's going on. 127 00:07:46,915 --> 00:07:50,605 And the server's just a broker to keep different clients in sync 128 00:07:50,635 --> 00:07:54,955 and to push changes so everyone can stay up to date or even doing 129 00:07:54,955 --> 00:07:56,995 decentralized servers if you go further. 130 00:07:57,415 --> 00:07:59,575 But it's that other side of the coin. 131 00:08:00,160 --> 00:08:02,500 Of we want one source of truth. 132 00:08:02,500 --> 00:08:05,590 We don't want to think about, we have this server and this client and we constantly 133 00:08:05,590 --> 00:08:07,630 have to write logic to glue them together. 134 00:08:07,780 --> 00:08:11,800 No, you either store all the state on the server, kind of like an HTMX style, 135 00:08:12,010 --> 00:08:16,600 or you store all the state on the client using local-first technologies. 136 00:08:16,960 --> 00:08:21,490 And having explored the first one for a long time, since Astro is meant to be 137 00:08:21,520 --> 00:08:26,680 like a static website or server driven website, I was excited to explore the 138 00:08:26,680 --> 00:08:30,820 other side of storing everything on the client, keeping it up to date, and then 139 00:08:30,820 --> 00:08:33,580 figuring out how synchronization happens. 140 00:08:33,910 --> 00:08:35,650 That's also been around for a long time. 141 00:08:35,650 --> 00:08:40,000 If you look back to just like the first calendar app on your phone or on your 142 00:08:40,000 --> 00:08:44,440 computer, this challenge has been around since probably SQLlite was created. 143 00:08:44,723 --> 00:08:48,953 but it's only now that web devs are starting to get a lot of footholds 144 00:08:48,953 --> 00:08:53,453 to also apply this to websites if that's something that you need. 145 00:08:53,873 --> 00:08:55,623 I think you've summarized it really well. 146 00:08:55,863 --> 00:09:00,963 And, I would go even as far as saying there is an elephant in the room that 147 00:09:00,963 --> 00:09:04,353 most people are aware that there is an elephant, but they really don't 148 00:09:04,353 --> 00:09:06,273 have yet the, the right terminology. 149 00:09:06,273 --> 00:09:10,806 And I would say the term here of the elephant is distributed systems. 150 00:09:10,926 --> 00:09:14,616 We have distributed states and distributed systems are really, like, 151 00:09:14,766 --> 00:09:19,296 that's a core discipline of computer science that doesn't really have 152 00:09:19,296 --> 00:09:24,609 just like the good answer, but it's a really, really hard problem alongside 153 00:09:24,609 --> 00:09:26,469 of like naming things and so on. 154 00:09:26,829 --> 00:09:30,219 But, that part, like everyone who's building a web app, 155 00:09:30,279 --> 00:09:32,469 anything that has state. 156 00:09:32,949 --> 00:09:37,659 On the server, like even in the server, typically you have by definition 157 00:09:37,659 --> 00:09:41,979 also already a distributed system where you have states in your API 158 00:09:41,979 --> 00:09:45,729 throughout the request lifecycle, but then you also have it in the database. 159 00:09:45,939 --> 00:09:48,249 You might have like concurrent requests. 160 00:09:48,429 --> 00:09:50,739 So there you already have a distributed system. 161 00:09:50,739 --> 00:09:55,239 Typically there it's much less bad because they're like, you can just 162 00:09:55,269 --> 00:09:56,979 trust the server already less. 163 00:09:57,189 --> 00:09:58,879 So you just trust the database. 164 00:09:59,289 --> 00:10:02,349 So you basically push it all down to the single choke point. 165 00:10:02,709 --> 00:10:07,599 But if you wanna trust the client even more now all the distributed 166 00:10:07,599 --> 00:10:11,799 parts are get the distance grows and therefore the divergence. 167 00:10:12,189 --> 00:10:15,776 And I have in university, when I studied computer science, I don't 168 00:10:15,776 --> 00:10:20,306 think I've actually taken a class on distributed systems and I've missed a 169 00:10:20,306 --> 00:10:24,236 memo where someone would've instilled it in me is like, Hey, everything 170 00:10:24,236 --> 00:10:28,181 that are you gonna build will suffer from distributed systems, make sure to 171 00:10:28,181 --> 00:10:31,961 understand this problem properly and then design the architecture around it. 172 00:10:31,991 --> 00:10:34,841 And I think most web devs are not aware of that. 173 00:10:35,621 --> 00:10:40,651 And what you've just laid out, I think is exactly suffering from this problem. 174 00:10:41,131 --> 00:10:45,691 And this is what I think where some framework creators are making 175 00:10:45,691 --> 00:10:50,011 the very smart decision to empower the server as much as possible. 176 00:10:50,191 --> 00:10:56,722 Because if you live by that sort of maxim, at that point, the implied 177 00:10:57,022 --> 00:11:01,762 damage that you can cause by building it in a certain way is minimized. 178 00:11:02,302 --> 00:11:06,339 And, my journey over the last five years or so has been almost like 179 00:11:06,369 --> 00:11:11,979 intentionally letting the pendulum swing to the most other extreme where all the 180 00:11:11,979 --> 00:11:16,449 state is governed by the client, since I'm pretty convinced that the middle 181 00:11:16,449 --> 00:11:18,639 ground is just pain and suffering. 182 00:11:19,222 --> 00:11:24,112 So I'm pretty convinced that you should really analyze the use case that you have. 183 00:11:24,532 --> 00:11:29,062 And if it's a daily driver productivity app, you probably wanna move as much 184 00:11:29,062 --> 00:11:30,862 state to the client as possible. 185 00:11:31,012 --> 00:11:35,992 Where it happens, like you produce most of your state in your calendar app or in 186 00:11:35,992 --> 00:11:41,092 your Notion, or in your other notes app or whatever you want to have it declined. 187 00:11:41,572 --> 00:11:45,429 And, if it's something like New York Times, then you don't 188 00:11:45,429 --> 00:11:46,599 produce any of that date. 189 00:11:46,599 --> 00:11:47,799 So it should be on the server. 190 00:11:48,249 --> 00:11:52,862 And, I, feel like more people should, start with that assumption, then 191 00:11:52,862 --> 00:11:54,422 build an architecture around it. 192 00:11:54,962 --> 00:11:57,806 But, yeah, I think you summarized it super well. 193 00:11:58,105 --> 00:11:58,555 Yeah. 194 00:11:58,855 --> 00:12:00,175 And it is tough. 195 00:12:00,864 --> 00:12:06,534 To prescribe either side because there are neat buckets like content sites 196 00:12:06,564 --> 00:12:09,414 and client side apps like Notion. 197 00:12:09,924 --> 00:12:15,054 But the classic example is, well what about e-commerce where things are 198 00:12:15,054 --> 00:12:18,594 server driven until you're in the add to cart flow and now it's client driven. 199 00:12:18,594 --> 00:12:19,764 So what do you do then? 200 00:12:19,794 --> 00:12:21,024 How do you architect it? 201 00:12:21,444 --> 00:12:26,454 We actually met this challenge building the Astro storefront front template, 202 00:12:26,964 --> 00:12:31,554 which was mostly server driven with some client side components sprinkled in. 203 00:12:32,004 --> 00:12:36,924 I think the answer there was, it's still fine to leave things server 204 00:12:36,924 --> 00:12:42,819 side and not rely on optimistic updates too much, except for like 205 00:12:42,819 --> 00:12:47,109 small examples like increase and decrease quantity in your cart. 206 00:12:47,109 --> 00:12:51,459 Do you want that to feel instant and happen on a delay or a debounce but 207 00:12:51,459 --> 00:12:52,989 everything else is server driven. 208 00:12:53,289 --> 00:12:56,019 There is no perfect architecture, I guess is what I'm saying. 209 00:12:56,229 --> 00:13:00,849 It's more painful when you really try to blend it 50 50 and nail every use case. 210 00:13:01,179 --> 00:13:07,899 But if you can keep it 80 20, where 80% is in one realm and 20% is business logic in 211 00:13:07,899 --> 00:13:13,599 the other realm, like 80% server driven, 20% client side complexity or less, then 212 00:13:13,749 --> 00:13:18,369 you're kind of minimizing the footprint of pain that you could run into since I've 213 00:13:18,369 --> 00:13:22,419 certainly noticed that with some of the newer patterns and older patterns with. 214 00:13:23,064 --> 00:13:24,174 Client side apps. 215 00:13:24,627 --> 00:13:26,577 but there was a second thing you mentioned about distributed 216 00:13:26,577 --> 00:13:28,917 systems that I, totally agree with. 217 00:13:29,157 --> 00:13:35,487 I was reading the book Designing Data Intensive Applications, the big 218 00:13:35,487 --> 00:13:39,987 O'Reilly book, probably the first one you find looking up like CS principles. 219 00:13:40,467 --> 00:13:44,757 And I think there was one section about distributed systems where it walked you 220 00:13:44,757 --> 00:13:47,607 from like single leader replication. 221 00:13:47,667 --> 00:13:52,347 You got one database and if you want to have some caches to make reads 222 00:13:52,347 --> 00:13:56,757 faster, you just put all your rights to one database and then it'll replicate 223 00:13:56,757 --> 00:13:58,257 the reads out to everyone else. 224 00:13:58,287 --> 00:14:00,837 So you can't write to a bunch of different regions. 225 00:14:00,837 --> 00:14:03,867 You can write data to one region and then it'll sort of. 226 00:14:04,437 --> 00:14:08,907 Push out all of the clones and allow eventual consistency to work. 227 00:14:09,374 --> 00:14:12,014 but there are cases where that doesn't work anymore. 228 00:14:12,224 --> 00:14:16,184 Like in Notion, I cannot wait for the server to update the document. 229 00:14:16,364 --> 00:14:19,934 It's just gonna update when I'm typing and when I add blocks and when people 230 00:14:19,934 --> 00:14:21,824 are invited to join the document. 231 00:14:21,824 --> 00:14:22,889 All that stuff matters. 232 00:14:22,889 --> 00:14:28,214 So you need some way to be able to write to maybe the nearest node. 233 00:14:28,244 --> 00:14:31,814 Like if you go fancy with CloudFlare, you could have really localized, 234 00:14:31,814 --> 00:14:35,864 durable objects that are two feet away from your computer and that reduces 235 00:14:35,864 --> 00:14:38,114 latency in some sort of magical way. 236 00:14:38,601 --> 00:14:43,991 but then it kind of explains well if you just put a data replica just on the 237 00:14:43,991 --> 00:14:49,901 client device, that's another version of multi leader replication, where 238 00:14:49,901 --> 00:14:53,711 instead of replicating in some really edge node, you're just replicating 239 00:14:53,711 --> 00:14:57,941 on the person's computer and everyone is like a mini server unto itself, 240 00:14:57,941 --> 00:15:01,541 where you just read and write data and they'll all report back to some 241 00:15:01,541 --> 00:15:04,001 centralized source of truth later on. 242 00:15:04,611 --> 00:15:07,641 So when you're working with, like, is it a distributed system? 243 00:15:07,701 --> 00:15:08,241 It is. 244 00:15:08,241 --> 00:15:12,531 Even when it's running on your device, if there's some sort of synchronization 245 00:15:12,531 --> 00:15:15,951 layer, you're just moving it closer and closer and closer to the computer 246 00:15:15,951 --> 00:15:18,171 until it iSQLiterally in the computer. 247 00:15:18,494 --> 00:15:19,844 It's inside the house. 248 00:15:20,294 --> 00:15:24,234 I've, heard a, very interesting framing of sort of like that 249 00:15:24,234 --> 00:15:27,504 elephant in the room and like that problem that we just talked about. 250 00:15:27,874 --> 00:15:32,957 I think it's by Carl who worked on SQLSync and recently, released Graft, 251 00:15:32,987 --> 00:15:34,847 which is another fascinating project. 252 00:15:35,297 --> 00:15:39,664 And he has written a blog post about, don't recall the exact name we're gonna 253 00:15:39,664 --> 00:15:43,640 put it in the, show notes, but it was basically along the lines of like, your 254 00:15:43,790 --> 00:15:46,803 application is a database and that's bad. 255 00:15:47,117 --> 00:15:47,977 I gotta look it up. 256 00:15:48,051 --> 00:15:49,520 what the, exactly the title was. 257 00:15:49,820 --> 00:15:56,030 But it was basically, he's making the point that every app that is sufficiently 258 00:15:56,030 --> 00:16:03,355 getting more complicated will basically build its own version of a ad hoc database 259 00:16:03,652 --> 00:16:08,272 and if you're just using React useState or something else enough, and then 260 00:16:08,272 --> 00:16:10,552 you try to add persistence, et cetera. 261 00:16:10,792 --> 00:16:15,502 What your app is basically becoming is a poor implementation of a database 262 00:16:15,892 --> 00:16:21,542 where like all the things that a database does a lot of like R&D great 263 00:16:21,542 --> 00:16:26,918 work to make it fast, to make it correct, to make it like nicely, like 264 00:16:27,158 --> 00:16:29,198 transactionally correct, et cetera. 265 00:16:29,648 --> 00:16:35,975 All of those things you're now trying to handle through, useState and useEffect 266 00:16:35,995 --> 00:16:39,775 and like when this thing changes, also change that thing, et cetera. 267 00:16:40,195 --> 00:16:45,935 And, I just thought that framing was so elegant and, his conclusion, which 268 00:16:45,935 --> 00:16:50,855 I agree with is like, hey, if let's try to make the app about what the app tries 269 00:16:50,855 --> 00:16:55,835 to do and let's leverage a database so we can focus on the actual like 270 00:16:55,835 --> 00:16:57,905 features, et cetera we want to implement. 271 00:16:58,445 --> 00:17:03,335 And I think that's another kinda articulation of the elephant in the room 272 00:17:03,752 --> 00:17:09,275 that we're accidentally and without being aware of it Building databases as that 273 00:17:09,275 --> 00:17:14,498 are sort of camouflaged as apps and, we should embrace it more if we're starting 274 00:17:14,498 --> 00:17:16,628 to see those signs of those problems. 275 00:17:17,018 --> 00:17:22,438 So you've mentioned that you've gone down this path, a lot out of curiosity and 276 00:17:22,438 --> 00:17:27,175 just because you've, pretty exhaustively explored the more server centric 277 00:17:27,175 --> 00:17:31,745 paths, can you share more about like what were your frustrations and pain 278 00:17:31,745 --> 00:17:35,825 points that you felt where you wanted to go more, like embrace the client 279 00:17:35,825 --> 00:17:41,515 more, but still trying to do that with the more server centric architecture. 280 00:17:41,725 --> 00:17:45,295 So you've mentioned optimistic state, et cetera, maybe can motivate 281 00:17:45,295 --> 00:17:49,255 this through a concrete app you wanted to build where you just felt 282 00:17:49,255 --> 00:17:51,265 that that pain and that impedance. 283 00:17:51,738 --> 00:17:56,688 Yeah, I mean I've played with all sorts of side projects as we all have. 284 00:17:56,883 --> 00:17:59,853 and I was working on a few different things. 285 00:18:00,363 --> 00:18:06,723 One was like a localized note taking app, and I was also playing with local 286 00:18:06,753 --> 00:18:12,663 LLMs and other pieces to add vector search into a local environment because 287 00:18:12,693 --> 00:18:17,553 I was running up on the limits of using Notion and being very frustrated with 288 00:18:17,733 --> 00:18:21,633 loading spinners and offline warnings when I was trying to use the app. 289 00:18:21,948 --> 00:18:26,238 I hear that's being changed and rectified as they build out their 290 00:18:26,238 --> 00:18:28,818 own, like SQLite replication. 291 00:18:29,058 --> 00:18:29,958 I know that's in there. 292 00:18:29,958 --> 00:18:32,628 There's some fascinating videos about that on the internet. 293 00:18:33,175 --> 00:18:39,115 but I still do see the value of just open up Apple Notes, you type things and it's 294 00:18:39,115 --> 00:18:40,975 just kind of there in a SQLlite store. 295 00:18:40,975 --> 00:18:42,385 You can find it on your file system. 296 00:18:42,385 --> 00:18:45,265 All your Apple Notes are just in a SQLlite store and it's great. 297 00:18:45,828 --> 00:18:48,888 so I thought, well, it seems like that's the answer. 298 00:18:48,888 --> 00:18:51,528 If I were to go the traditional server route, I would sit here 299 00:18:51,528 --> 00:18:54,858 waiting for all these updates to persist, which just wouldn't work. 300 00:18:55,188 --> 00:18:59,118 Or I'd be sort of gluing together a bunch of useState calls and figuring 301 00:18:59,118 --> 00:19:02,328 out how do I update this server and what if someone else updates it? 302 00:19:02,328 --> 00:19:04,788 What if I'm on my phone, I type a thing and then I go back to 303 00:19:04,788 --> 00:19:06,498 my computer and I try to see it? 304 00:19:06,768 --> 00:19:08,628 Is it gonna merge it together? 305 00:19:08,628 --> 00:19:09,648 Is it gonna break? 306 00:19:09,678 --> 00:19:11,328 I have no idea because. 307 00:19:12,363 --> 00:19:14,553 Client side stores don't help you with those problems. 308 00:19:14,553 --> 00:19:16,203 They're not designed for those problems. 309 00:19:16,473 --> 00:19:21,213 They're designed for like ephemeral state that can disappear at a moment's 310 00:19:21,213 --> 00:19:23,373 notice if it needs to for some reason. 311 00:19:23,847 --> 00:19:28,527 so that led me down exploring local-first technologies and I found pretty quickly 312 00:19:28,527 --> 00:19:33,897 how capable SQLite is for these use cases, including in the browser. 313 00:19:34,210 --> 00:19:38,050 you can load up SQLite with like a wasm build as you're 314 00:19:38,050 --> 00:19:41,410 very aware and use it even with. 315 00:19:41,815 --> 00:19:45,595 Like very nice SQLite libraries. 316 00:19:45,655 --> 00:19:50,318 If you wanted to use Drizzle for example, which is a common SQL querying library 317 00:19:50,318 --> 00:19:55,838 and JavaScript, it matches onto the browser version of SQLite perfectly. 318 00:19:55,928 --> 00:19:59,978 So you can actually make declarative, find many notes and 319 00:19:59,978 --> 00:20:01,808 it'll just do the little SQL query. 320 00:20:01,808 --> 00:20:04,178 It'll join it up with all the authors of the post and it'll 321 00:20:04,178 --> 00:20:05,198 just give it back to you. 322 00:20:05,318 --> 00:20:07,958 Kind of like you're on the server, but you're on the client. 323 00:20:08,108 --> 00:20:12,188 And you can use client side ORMs or query builders, whatever 324 00:20:12,188 --> 00:20:14,048 your flavor of preference. 325 00:20:14,378 --> 00:20:16,448 So that was very empowering to see. 326 00:20:16,448 --> 00:20:20,348 Yeah, you can bring all of these niceties you get from server side data 327 00:20:20,348 --> 00:20:22,748 querying and bring it into the client. 328 00:20:23,378 --> 00:20:26,258 And I tried to stretch it a little bit further by asking, 329 00:20:26,618 --> 00:20:28,538 what about SQL extensions? 330 00:20:28,808 --> 00:20:31,658 Could I add a vector search plugin, for example? 331 00:20:32,288 --> 00:20:33,158 The answer is yes. 332 00:20:33,158 --> 00:20:36,398 There actually is a vector search plugin that I think is developed 333 00:20:36,398 --> 00:20:38,078 by someone on the Mozilla team. 334 00:20:38,168 --> 00:20:40,658 So it is pretty battle tested in different languages. 335 00:20:41,258 --> 00:20:47,992 I think they're sponsored, yeah I've had the chance to meet them in October 336 00:20:47,992 --> 00:20:50,498 when I was in LA super lovely person. 337 00:20:50,528 --> 00:20:54,285 And, they have some sponsorship from the Mozilla team currently. 338 00:20:55,635 --> 00:20:56,295 Nice. 339 00:20:56,625 --> 00:21:01,355 Yeah, and I was playing with the, rust flavor of that since, well now 340 00:21:01,355 --> 00:21:05,465 I'm working at Warp, and Warp is a rust powered terminal, so naturally I 341 00:21:05,465 --> 00:21:08,315 need to get up on the Rust knowledge. 342 00:21:08,375 --> 00:21:13,085 And also if you build apps with Tori, which is a native desktop application 343 00:21:13,085 --> 00:21:15,425 tool that uses Rust as well, so. 344 00:21:16,430 --> 00:21:19,850 Side tangent, but it is nice to use tools that could work in JavaScript as well 345 00:21:19,850 --> 00:21:21,770 as rust in very efficient languages. 346 00:21:22,219 --> 00:21:23,107 so I reached for that. 347 00:21:23,317 --> 00:21:24,607 I put vector search in. 348 00:21:24,967 --> 00:21:29,467 I was also able to run an entire LLM across the data in the 349 00:21:29,467 --> 00:21:34,657 browser and just load up like the entire vector search setup thing. 350 00:21:35,233 --> 00:21:38,173 and I just said, all right, I'm gonna backport all of my 351 00:21:38,203 --> 00:21:40,363 markdown files into this thing. 352 00:21:40,543 --> 00:21:42,553 I'm gonna vectorize all of them in the browser. 353 00:21:42,793 --> 00:21:44,203 I'm gonna search them in the browser. 354 00:21:44,203 --> 00:21:47,983 And I was able to get it working with like next key search. 355 00:21:48,073 --> 00:21:49,783 And it was absolutely mind blowing. 356 00:21:49,843 --> 00:21:53,683 Like this is like an actual AI powered search tool. 357 00:21:53,953 --> 00:21:57,073 And you can get like with every keystroke new results 358 00:21:57,198 --> 00:22:01,938 And is isn't that wild like that this machine that we have like sitting 359 00:22:01,938 --> 00:22:07,218 on our laps or like this machine here in my hands that is capable 360 00:22:07,218 --> 00:22:11,995 of all of those things and just the way how we kind of build web apps. 361 00:22:11,995 --> 00:22:16,045 Over the last decade or so, we've kind of forgotten or denied the 362 00:22:16,045 --> 00:22:18,355 capabilities of our client devices. 363 00:22:18,685 --> 00:22:22,975 And we only just like trust the server and we've almost like, why did no 364 00:22:22,975 --> 00:22:24,835 one tell us that this is possible? 365 00:22:25,105 --> 00:22:28,682 And that's so magical when you see that this is working and just the 366 00:22:28,682 --> 00:22:32,958 stuff that's currently, in the works with like web, LLM, et cetera, where 367 00:22:32,958 --> 00:22:37,522 it can run like a full blown like Llama model locally in your browser. 368 00:22:37,815 --> 00:22:43,125 running on web GPU is absolutely wild to the capabilities that we have. 369 00:22:43,455 --> 00:22:47,395 But I think what's holding us back is where does our data 370 00:22:47,395 --> 00:22:48,745 live and everything else. 371 00:22:48,745 --> 00:22:52,495 Kinda like, it's almost like a second order effect from where the data lives. 372 00:22:52,945 --> 00:22:57,482 And this is what, your anecdote really nicely highlights of like, you go with 373 00:22:57,482 --> 00:23:02,802 SQLite with your data and then you like bring in another superpower of like SQLite 374 00:23:02,802 --> 00:23:05,582 Vec with the vector embedding, et cetera. 375 00:23:05,972 --> 00:23:08,822 And I think it all starts with where the data is, how your 376 00:23:08,882 --> 00:23:10,382 application is being shaped. 377 00:23:10,905 --> 00:23:11,625 Yeah. 378 00:23:11,685 --> 00:23:13,515 And it is good to find that. 379 00:23:13,830 --> 00:23:15,690 Just common data layer. 380 00:23:15,740 --> 00:23:17,730 SQLite is the easy answer. 381 00:23:17,760 --> 00:23:22,380 PG light is a more robust exploration of bringing Postgres to local devices. 382 00:23:22,380 --> 00:23:23,610 That's a bit earlier on. 383 00:23:24,110 --> 00:23:28,334 but I think we're entering a world where software is just so easy to spin 384 00:23:28,334 --> 00:23:33,794 up that you will very quickly have a web client, a mobile app client, 385 00:23:33,914 --> 00:23:37,994 a desktop client, and they're all talking to the same sync server. 386 00:23:38,324 --> 00:23:40,894 And when you're in that world, it's nice to just reach for SQLite. 387 00:23:40,964 --> 00:23:42,724 'cause I can run SQLlite on my iPhone. 388 00:23:42,914 --> 00:23:46,694 I can run SQLlite on my Android device, I can run it in the browser and I 389 00:23:46,694 --> 00:23:48,674 could run it in a desktop application. 390 00:23:48,974 --> 00:23:54,164 So as long as you just have this concept of clients write to SQL servers and 391 00:23:54,164 --> 00:23:58,334 those SQL servers have some way to talk to each other, then you can build 392 00:23:58,334 --> 00:24:04,124 these multi-platform applications very quickly, even across different languages. 393 00:24:04,214 --> 00:24:07,574 And just figure out what that sync layer looks like, which 394 00:24:07,634 --> 00:24:09,374 we can probably talk about. 395 00:24:09,472 --> 00:24:12,839 that would've been my next question since, I think what you started with 396 00:24:12,899 --> 00:24:17,319 was probably without the sync part yet, where you can just locally in 397 00:24:17,319 --> 00:24:23,019 your browser web app, you successfully ran SQLite using the wasm build. 398 00:24:23,109 --> 00:24:27,435 Then you brought in SQLite Vec, you could, get the AI magic to work. 399 00:24:27,675 --> 00:24:30,795 You saw like how insanely fast everything feels. 400 00:24:30,795 --> 00:24:33,435 You write something, you reload the browser. 401 00:24:33,615 --> 00:24:35,775 Even if you're offline, it's all still there. 402 00:24:35,965 --> 00:24:36,475 Great. 403 00:24:36,805 --> 00:24:41,005 But now you're thinking, okay, it works on like local host 3000. 404 00:24:41,552 --> 00:24:47,965 how do I get it deployed and how do I get it so that, if I accidentally 405 00:24:47,965 --> 00:24:52,525 open this in a cognitive tab and I close it, poof, everything is gone. 406 00:24:52,734 --> 00:24:56,395 and how do I get it show up on my, phone so this is kind of 407 00:24:56,395 --> 00:25:00,325 collaborating with yourself, but on also collaborating with others, which 408 00:25:00,325 --> 00:25:02,695 we maybe punt on that for a moment. 409 00:25:03,055 --> 00:25:03,595 But yeah. 410 00:25:03,595 --> 00:25:09,565 How did you go from, it works locally almost like if you use like just 411 00:25:09,595 --> 00:25:14,755 local storage to trying to share the goodness across your devices. 412 00:25:15,224 --> 00:25:19,717 Well, I think we both have the privilege of just ask Andrew on the Rocicorp team 413 00:25:19,717 --> 00:25:21,307 and he'll point you to some resources. 414 00:25:21,710 --> 00:25:25,144 if you don't have that, I do recommend that. 415 00:25:25,174 --> 00:25:31,204 Well, the one approach that I used was just reading through the Replicache docs. 416 00:25:31,469 --> 00:25:34,889 That explain how their sync engine works on a very high level. 417 00:25:35,279 --> 00:25:41,969 And Replicache is a batteries included library, well some batteries included 418 00:25:42,119 --> 00:25:48,509 library that sets up a simple key value store for you where you can write and read 419 00:25:48,509 --> 00:25:54,599 keys on the client and you can author code that will post those changes to a server 420 00:25:54,599 --> 00:25:58,949 that you own and pull changes back from the server whenever changes are detected. 421 00:25:59,549 --> 00:26:04,619 And you can implement that on a SQLite or anything else if you just have 422 00:26:04,969 --> 00:26:09,779 their simplified mental model, which is very inspired by just how Git works. 423 00:26:09,839 --> 00:26:14,189 So in Git, if you had to change locally and you wanted to push it up to the 424 00:26:14,189 --> 00:26:16,859 main line, you would run Git push. 425 00:26:17,309 --> 00:26:21,719 And the same kind of thing happens with their sync engine service, where 426 00:26:21,719 --> 00:26:25,889 anytime you make a change locally and you can decide what that means, 427 00:26:25,889 --> 00:26:30,059 maybe it's a debounce as you're editing a document, maybe it's when 428 00:26:30,059 --> 00:26:33,479 you click on a status toggle in Linear and you wanna change that status. 429 00:26:34,189 --> 00:26:40,219 Whenever the trigger is, you can use that to call push, which would probably call 430 00:26:40,219 --> 00:26:43,219 an endpoint written on your own server. 431 00:26:43,459 --> 00:26:49,069 That's simply a push endpoint and that can receive whatever event you ran 432 00:26:49,069 --> 00:26:53,719 on the client so that the server can replay it, back it up into its own 433 00:26:53,719 --> 00:26:58,669 logs and tell other clients about it whenever they try to pull those changes 434 00:26:58,669 --> 00:27:03,439 back down so you can run your own little push whenever you make a change. 435 00:27:03,559 --> 00:27:07,399 And then other clients on an interval or a web socket or some 436 00:27:07,399 --> 00:27:11,449 other connection can pull for data whenever they want to get the most 437 00:27:11,449 --> 00:27:13,879 recent changes made by other people. 438 00:27:14,479 --> 00:27:18,649 Now the question would be, what am I pushing and what am I pulling? 439 00:27:18,859 --> 00:27:20,059 Like what data? 440 00:27:20,674 --> 00:27:22,414 Needs to be sent across. 441 00:27:22,684 --> 00:27:25,264 And there are a couple different methods. 442 00:27:25,384 --> 00:27:30,244 I know Replicache uses a data snapshot mechanism. 443 00:27:30,364 --> 00:27:35,464 I don't fully know the intricacies of it, but I know that because they use 444 00:27:35,464 --> 00:27:39,874 like a key value storage, which is much simpler than like a full-blown database. 445 00:27:39,949 --> 00:27:44,104 They can take a snapshot of what the server looks like right now and then the 446 00:27:44,104 --> 00:27:47,974 client has its own copy of that server snapshot and it can just run a little 447 00:27:47,974 --> 00:27:51,694 diff the same way you'd run a get diff to see what code changes you have made. 448 00:27:52,024 --> 00:27:55,204 It can run that diff and you can see, all right, I added this key. 449 00:27:55,204 --> 00:27:58,684 All right, updated this status flag, and then tell the server, 450 00:27:58,684 --> 00:28:01,684 this is the diff, this is the change that was made to the data. 451 00:28:02,044 --> 00:28:06,574 And the server can receive that request and say, okay, this is the change 452 00:28:06,574 --> 00:28:08,584 that I need to make against my copy. 453 00:28:08,974 --> 00:28:13,024 And then I will tell other clients to also apply that diff whenever they pull. 454 00:28:13,320 --> 00:28:18,990 it's a very Git inspired model, and it works if you're able to diff data easily. 455 00:28:19,410 --> 00:28:23,670 If you are working with like a full blown SQLlite table, you run table 456 00:28:23,670 --> 00:28:27,600 migrations, tables, change shapes, that's a very hard thing to keep track of. 457 00:28:27,930 --> 00:28:33,484 So another option that I implemented for the, learning resource I 458 00:28:33,484 --> 00:28:35,464 created called Simple Sync Engine. 459 00:28:35,554 --> 00:28:39,514 If you find that on my GitHub, probably in the show notes, you can see that. 460 00:28:40,024 --> 00:28:43,744 But it was meant to be a very basic implementation of this pattern 461 00:28:43,984 --> 00:28:46,474 that uses event sourcing instead. 462 00:28:46,594 --> 00:28:52,714 So rather than sending a diff of how the data should change, instead sends 463 00:28:52,774 --> 00:28:57,034 an event that describes the change that is made like a function call or 464 00:28:57,034 --> 00:29:01,114 an RPC, however you wanna think about it, where you would tell the server 465 00:29:01,324 --> 00:29:04,414 I updated the status to this value. 466 00:29:05,179 --> 00:29:09,559 And in our implementation we create these little like function helpers that 467 00:29:09,559 --> 00:29:11,449 can describe what those events are. 468 00:29:11,509 --> 00:29:13,999 So you might have like an add status event. 469 00:29:13,999 --> 00:29:19,069 So like the type of the event is add status and that accepts a few arguments. 470 00:29:19,129 --> 00:29:22,789 It accepts the status you wanna change it to, and the ID of the 471 00:29:22,849 --> 00:29:25,189 record that has that status currently. 472 00:29:25,369 --> 00:29:30,259 So the server receives that event, it sees, okay, the type was set status. 473 00:29:30,499 --> 00:29:31,669 I see two arguments here. 474 00:29:31,669 --> 00:29:32,869 The ID of the record. 475 00:29:33,149 --> 00:29:35,189 And the new status that should be applied. 476 00:29:35,489 --> 00:29:38,969 I'm gonna go ahead and run that event as a database update. 477 00:29:39,239 --> 00:29:43,679 So it has that mapping understanding of, I received this event, I know that 478 00:29:43,679 --> 00:29:48,299 maps to this SQL query, so I'm gonna go ahead and make that change on my copy. 479 00:29:48,569 --> 00:29:52,949 And then whenever people poll, you can send that event log out, or you 480 00:29:52,949 --> 00:29:57,029 can send whatever events the client hasn't received up until that point. 481 00:29:57,419 --> 00:30:00,179 You can kind of think of those like Git commits where you're pulling 482 00:30:00,179 --> 00:30:04,259 the latest commits on the branch and the server's able to tell you, here 483 00:30:04,259 --> 00:30:05,909 are the latest events that were run. 484 00:30:06,149 --> 00:30:10,109 Go ahead and run those on your replicas so both the server 485 00:30:10,109 --> 00:30:11,819 and client know what events. 486 00:30:12,189 --> 00:30:16,119 Actually mean, like this event means I need to make this SQL query. 487 00:30:16,119 --> 00:30:17,439 And it's able to do that mapping. 488 00:30:17,939 --> 00:30:19,979 and you can have a bit of freedom there. 489 00:30:19,979 --> 00:30:24,539 If you had like a very specific kind of data store on the server, like 490 00:30:24,539 --> 00:30:28,499 MongoDB, you could customize it to say, whenever I receive this event, it 491 00:30:28,499 --> 00:30:33,089 means this MongoDB query and this call to the Century error logging system, 492 00:30:33,089 --> 00:30:35,189 or whatever middleware you wanna do. 493 00:30:35,609 --> 00:30:40,469 As long as server and client agree on what events exist and what changes they 494 00:30:40,469 --> 00:30:44,489 make in the data stores, then everyone can be on the same page whenever 495 00:30:44,489 --> 00:30:46,229 they're syncing things up and down. 496 00:30:46,714 --> 00:30:49,264 you're very familiar with event sourcing as well. 497 00:30:49,414 --> 00:30:53,404 I'm curious if there's things that I've missed or important edge cases 498 00:30:53,404 --> 00:30:54,754 that we should probably talk about. 499 00:30:55,146 --> 00:31:00,684 I think you very elegantly, described how simple the foundation of this can be and 500 00:31:00,684 --> 00:31:02,994 hence the name of like Simple Sync Engine. 501 00:31:03,350 --> 00:31:06,814 I think this has served as a great learning resource for you and 502 00:31:06,814 --> 00:31:08,404 I'm sure for many, many others. 503 00:31:08,974 --> 00:31:13,634 And once you like, start pulling more on that thread, you realize, oh shit like, 504 00:31:13,664 --> 00:31:15,404 okay, that thing I didn't think about. 505 00:31:15,644 --> 00:31:16,634 Oh, what about this? 506 00:31:16,844 --> 00:31:22,194 So you've mentioned already the reason why you preferred, event sourcing over 507 00:31:22,194 --> 00:31:26,898 the snapshot approach because like with SQLlite, What would you actually compare? 508 00:31:27,408 --> 00:31:31,914 This is where I would give a shout out again, to Carl's work with Graft. 509 00:31:32,331 --> 00:31:33,801 this is what he's been working on. 510 00:31:33,801 --> 00:31:36,321 We should have him on the show highlighting this as well. 511 00:31:36,591 --> 00:31:41,881 But, where he's built a new kinda sync engine that, is all about that 512 00:31:41,881 --> 00:31:44,551 diffing of like blocks of storage. 513 00:31:45,164 --> 00:31:50,654 I think all focused around SQLite and that gives, that does the heavy work 514 00:31:50,684 --> 00:31:54,944 of like, diffing and then sending out the minimal amount of changes 515 00:31:55,154 --> 00:31:56,564 to make all of that efficient. 516 00:31:56,564 --> 00:32:02,566 Since there's this, famous saying of like make it work, make it right 517 00:32:02,566 --> 00:32:04,486 or correct and then make it fast. 518 00:32:04,966 --> 00:32:09,976 And working on all of this has really given me a deep appreciation for all 519 00:32:09,976 --> 00:32:14,806 of this since like, and I'm sure you probably also went through those stages 520 00:32:15,046 --> 00:32:19,516 with the Simple Sync Engine, like with making something work in the first place 521 00:32:19,546 --> 00:32:21,346 was already quite the accomplishment. 522 00:32:21,616 --> 00:32:25,576 But then you also realize, ah, okay, in those cases this is not quite yet correct. 523 00:32:25,756 --> 00:32:30,583 And then you could go back and like try to iterate and then you also realize, 524 00:32:30,613 --> 00:32:36,763 okay, so now it has worked for my little to-do app, but if I, now, depending 525 00:32:36,763 --> 00:32:40,663 on the architecture, if I roll this out and put in hack news and suddenly 526 00:32:40,663 --> 00:32:45,073 have like 5,000 people there on the same time, this thing will break apart. 527 00:32:45,073 --> 00:32:48,973 Will A not be correct in some ways you didn't anticipate and 528 00:32:48,973 --> 00:32:50,503 will also not be fast enough. 529 00:32:50,773 --> 00:32:52,513 So now making this fast. 530 00:32:52,768 --> 00:32:55,828 It is at the end of the day is like really reinventing. 531 00:32:55,828 --> 00:33:00,478 It's like your app is becoming a little database and you want 532 00:33:00,478 --> 00:33:04,828 to like move as much of that database burden to the sync engine. 533 00:33:05,038 --> 00:33:09,358 This is why folks like Rocicorp, ElectricSQL, et cetera, they're 534 00:33:09,358 --> 00:33:13,048 doing a fantastic job trying to absorb as much of that complexity. 535 00:33:13,468 --> 00:33:18,238 But building something like this by yourself really gives you an understanding 536 00:33:18,238 --> 00:33:21,508 and an appreciation for what is going on. 537 00:33:21,568 --> 00:33:25,431 I love the Git analogy that you've used, but just a, a couple of 538 00:33:25,431 --> 00:33:30,714 points just similarly to how, your sync engine works is actually very 539 00:33:30,714 --> 00:33:35,634 analogous to how the Livestore architecture on a high level works. 540 00:33:36,054 --> 00:33:42,024 But I've had to, before I arrived at that, I really wanted to think through a lot 541 00:33:42,024 --> 00:33:45,029 of like the more further down the road. 542 00:33:45,029 --> 00:33:50,094 Like, what if situations since, one that I'm curious whether you've 543 00:33:50,094 --> 00:33:53,814 already run into, whether you resolved in some way or left for the 544 00:33:53,814 --> 00:33:59,304 future, is how would you impose a total order of your change events? 545 00:33:59,664 --> 00:34:04,239 So this is where When you have, like, let's say this to-do app 546 00:34:04,239 --> 00:34:06,505 or like a mini Linear, app. 547 00:34:06,685 --> 00:34:12,062 Let's say you create an issue and then you say you, complete the issue or 548 00:34:12,062 --> 00:34:14,152 you toggle the issue state, et cetera. 549 00:34:14,532 --> 00:34:19,745 It can mean something very different, if one happens first and then the 550 00:34:19,745 --> 00:34:21,275 other, or the other way around. 551 00:34:21,905 --> 00:34:26,135 And for that, where you have your events of like, hey, the issue was created, 552 00:34:26,435 --> 00:34:30,425 the issue status was changed, this, the issue status was changed, that, 553 00:34:30,979 --> 00:34:35,584 the order really matters in which way in which order everything happened. 554 00:34:36,004 --> 00:34:40,910 And, this might be not so bad if you're the absolutely only person using your 555 00:34:40,910 --> 00:34:42,800 app and typically only on one device. 556 00:34:43,070 --> 00:34:46,580 But when you then do stuff between multiple devices or multiple 557 00:34:46,580 --> 00:34:50,900 users, then it's no longer in your single user's control. 558 00:34:50,960 --> 00:34:52,670 That stuff happens concurrently. 559 00:34:53,120 --> 00:34:57,994 And then it really matters that everyone has the same order of the events. 560 00:34:58,024 --> 00:35:01,384 And this is where you need to impose what's called a total order. 561 00:35:02,044 --> 00:35:06,794 And I'm very curious whether you've already, hit that point, where you thought 562 00:35:06,794 --> 00:35:10,844 that through and whether you found a mechanism to impose it since there's many, 563 00:35:10,844 --> 00:35:12,584 many downstream consequences of that. 564 00:35:13,040 --> 00:35:13,670 Right. 565 00:35:13,670 --> 00:35:16,190 And I definitely did hit it and. 566 00:35:16,520 --> 00:35:20,840 You will find in the resource, it's not addressing that issue right now. 567 00:35:20,900 --> 00:35:25,730 It's in a very naive state of when change is made, send, fetch, call to server. 568 00:35:26,240 --> 00:35:29,870 And there are a few problems with that, even if you're the only client. 569 00:35:30,020 --> 00:35:35,150 Because first off, if you send one event per request, it's very possible that you 570 00:35:35,150 --> 00:35:41,480 send a lot of very quick secession events in like an order, and then they reach the 571 00:35:41,480 --> 00:35:46,940 server in a different order because maybe you pushed the first change on low network 572 00:35:47,000 --> 00:35:50,870 latency, the next one on high latency, or actually the reverse of that where 573 00:35:50,870 --> 00:35:54,890 the first change hits after the second change because that's just the speed of 574 00:35:54,890 --> 00:35:56,390 the network and that's what happened. 575 00:35:56,851 --> 00:35:59,960 And also you need to think about offline capabilities. 576 00:36:00,080 --> 00:36:04,730 If you push changes when they happen, how do you queue them 577 00:36:04,730 --> 00:36:07,760 when you're not connected to the internet and then run through that 578 00:36:07,760 --> 00:36:09,170 queue once you're back online? 579 00:36:09,170 --> 00:36:11,750 That's another consideration you kind of have to think about. 580 00:36:12,080 --> 00:36:15,050 Could be solved with just like an in-memory event log and 581 00:36:15,050 --> 00:36:16,310 just kind of work with that. 582 00:36:16,520 --> 00:36:19,130 But you still have the order issue. 583 00:36:19,520 --> 00:36:23,660 I'm familiar with atomic clocks as a method to do this. 584 00:36:23,660 --> 00:36:28,270 There are even SQLite extensions that'll sort of enforce that, having 585 00:36:28,270 --> 00:36:30,100 not implemented atomic clocks. 586 00:36:30,130 --> 00:36:34,630 Is it kind of this silver bullet to that problem or are there more 587 00:36:34,630 --> 00:36:37,930 considerations to think about than just reaching for something like that? 588 00:36:38,695 --> 00:36:39,115 Right. 589 00:36:39,115 --> 00:36:42,055 I suppose you're referring to vector clocks or logical 590 00:36:42,055 --> 00:36:43,855 clocks on a more higher level? 591 00:36:43,855 --> 00:36:43,915 Yeah. 592 00:36:44,265 --> 00:36:47,685 since the atomic clocks, at least my understanding is like that's 593 00:36:47,685 --> 00:36:51,679 actually what's, at least in some super high-end hardware is like 594 00:36:51,679 --> 00:36:56,035 an atomic clock that is, like that actually gives us like the wall clock. 595 00:36:56,065 --> 00:36:58,135 So Right, right now is like. 596 00:36:58,545 --> 00:37:03,585 Uh, 6:30 PM on my time, but this clock might drift, and this 597 00:37:03,585 --> 00:37:04,995 is what makes it so difficult. 598 00:37:04,995 --> 00:37:09,645 So what you were referring to with logical clocks, this is where it basically, 599 00:37:09,645 --> 00:37:14,115 instead of saying like, Hey, it's 6:30 with this time zone, which makes 600 00:37:14,115 --> 00:37:20,445 everything even more complicated, I'm keeping track of my time is like 1, 2, 3. 601 00:37:20,685 --> 00:37:24,525 It like might just be a logical counter, like much simpler 602 00:37:24,525 --> 00:37:26,085 actually than wall clock time. 603 00:37:26,423 --> 00:37:31,572 but this is easier to reason about and there might be no weird issues of 604 00:37:31,572 --> 00:37:36,079 like, Daylight saving where certainly like the, the clock is going backwards 605 00:37:36,349 --> 00:37:40,159 or someone tinkers with the time, this is why you need logical clocks. 606 00:37:40,519 --> 00:37:44,829 And, there, at least the mechanism that I've also landed on to 607 00:37:44,859 --> 00:37:46,959 implement, to impose a total order. 608 00:37:47,319 --> 00:37:50,379 But then it's also tricky, how do you exchange that? 609 00:37:50,503 --> 00:37:54,883 how does your client know what like three means in my client, et cetera? 610 00:37:54,883 --> 00:37:59,915 And the answer that I found to this is to like that we all trust. 611 00:38:00,328 --> 00:38:02,674 A single, authority in the system. 612 00:38:02,914 --> 00:38:07,421 So this is where, and I think this is also what you're going for, and with the Git 613 00:38:07,421 --> 00:38:12,971 analogy, what we are trusting as authority in that system is GitHub or GitLab. 614 00:38:13,421 --> 00:38:17,048 And this is where we are basically, we could theoretically, you could 615 00:38:17,048 --> 00:38:20,588 send me your IP address and I could try to like pull directly from you. 616 00:38:20,818 --> 00:38:24,598 It would work, and that would also work with the system that you've built. 617 00:38:25,018 --> 00:38:29,108 However, there might still be, they're called network petitions, 618 00:38:29,408 --> 00:38:33,428 where like the two of us have like, synced up, but some others haven't. 619 00:38:33,728 --> 00:38:39,745 So as long as we're all connected to the same, like main upstream node, that 620 00:38:39,745 --> 00:38:41,935 is the easiest way to, to model this. 621 00:38:41,995 --> 00:38:46,525 An alternative would be to go full on peer to peer, which makes everything 622 00:38:46,585 --> 00:38:48,685 a lot, lot, lot more complicated. 623 00:38:49,225 --> 00:38:53,011 And this is where like something, like an extension of logical clocks called 624 00:38:53,011 --> 00:38:55,461 vector clocks, can come in handy. 625 00:38:55,685 --> 00:39:00,628 you've mentioned the, the book, designing dataset intensive application by Martin 626 00:39:00,628 --> 00:39:02,458 Kleppman had him on the show before. 627 00:39:02,688 --> 00:39:06,968 he's actually working on the version two of that book right now, but he's also done 628 00:39:06,968 --> 00:39:12,671 a fantastic free course about distributed systems where he is walking through all of 629 00:39:12,671 --> 00:39:18,571 that, with a whiteboard, I actually think so, I think does what, what the two of 630 00:39:18,571 --> 00:39:24,555 you have very much like you've both nailed the, craft of like showing with simple 631 00:39:24,555 --> 00:39:27,231 strokes, some very complicated matters. 632 00:39:27,725 --> 00:39:30,935 so highly recommend to anyone who wants to learn more there. 633 00:39:31,815 --> 00:39:33,705 Like, learn it from, from Martin. 634 00:39:33,735 --> 00:39:37,575 He's, like an absolute master of explaining those difficult 635 00:39:37,575 --> 00:39:39,385 concepts in a simple way. 636 00:39:40,125 --> 00:39:45,521 But, yeah, a lot of things go kind of downstream from that total order. 637 00:39:45,551 --> 00:39:51,898 So just to, go together on like one little journey to understand like a downstream 638 00:39:51,898 --> 00:39:56,698 problem of this, let's say we have implemented the queuing of those events. 639 00:39:56,908 --> 00:40:00,751 So let's say you're currently on a plane ride and, you're like. 640 00:40:00,981 --> 00:40:03,601 Writing your blog post, you're very happy with it. 641 00:40:03,841 --> 00:40:07,231 You have now like a thousand of events of like change 642 00:40:07,231 --> 00:40:09,571 events that captures your work. 643 00:40:09,661 --> 00:40:11,641 Your SQLite database is up to date. 644 00:40:12,175 --> 00:40:16,525 but you didn't just create this new blog post, but you maybe while you're still at 645 00:40:16,525 --> 00:40:21,181 the airport, like you created the initial version with it with like TBD in the body. 646 00:40:21,721 --> 00:40:26,185 And your coworker thought like, oh, actually I have a lot of thoughts on this. 647 00:40:26,485 --> 00:40:29,365 And they also started writing down some notes in there. 648 00:40:29,935 --> 00:40:33,488 And now, the worlds have like, kind of drifted apart. 649 00:40:33,848 --> 00:40:35,108 Your coworker. 650 00:40:35,288 --> 00:40:38,618 Has written down some important things they don't want to lose, 651 00:40:38,768 --> 00:40:42,358 and you've written down some things you are not aware of the other ones 652 00:40:42,408 --> 00:40:48,698 neither are they, and at some point the semantic merge needs to happen. 653 00:40:48,938 --> 00:40:52,538 But how do you even make that happen in this sync engine thing here? 654 00:40:52,838 --> 00:40:57,458 And this is where you need the total order, where you basically, in the worst 655 00:40:57,458 --> 00:41:04,329 case, this is what decides, like who, gets a say in this, who gets the last say, in 656 00:41:04,629 --> 00:41:07,029 which order those events have happened. 657 00:41:07,539 --> 00:41:12,159 The model that I've landed on, and I think that's similar to what Git 658 00:41:12,159 --> 00:41:17,619 does with rebasing, is basically that before you get to push your own stuff, 659 00:41:18,099 --> 00:41:22,899 you need to pull down the events first, and then you need to reconcile 660 00:41:22,929 --> 00:41:25,779 your kind of stash local changes. 661 00:41:26,394 --> 00:41:32,374 On top of the work that whoever has gotten the, who got lucky enough to push 662 00:41:32,374 --> 00:41:35,584 first without being told to pull first. 663 00:41:35,974 --> 00:41:39,124 So in that case, it might have been your coworker because they've 664 00:41:39,274 --> 00:41:41,284 stayed online and kept pushing. 665 00:41:41,884 --> 00:41:45,917 And now it sort of like falls on you to reconcile that. 666 00:41:46,067 --> 00:41:50,807 And I've implemented a, like an actual rebase mechanism for this, 667 00:41:51,107 --> 00:41:56,057 where you now have this set of new events that your coworker has 668 00:41:56,057 --> 00:42:01,401 produced and you still have your set of events that, reflect your changes. 669 00:42:01,491 --> 00:42:03,441 And now you need to reconcile this. 670 00:42:03,501 --> 00:42:05,181 So that is purely on the. 671 00:42:05,471 --> 00:42:12,284 Event log level, but given that we both, want to use SQLite now, we don't 672 00:42:12,284 --> 00:42:17,654 need to just think about going forward with SQLite, but we also now need to 673 00:42:17,654 --> 00:42:19,664 think about like, Hey, how do we go? 674 00:42:19,784 --> 00:42:23,774 Like in Git you have like, you have this stack of events, right? 675 00:42:24,014 --> 00:42:27,494 So you have like a commit, which has a parent of another commit, which 676 00:42:27,494 --> 00:42:28,934 has a parent of another commit. 677 00:42:29,264 --> 00:42:36,219 It's very similar to how your events and this event log look like, except it's now 678 00:42:36,249 --> 00:42:41,379 no longer just one event log, but you also get this little branch from your coworker. 679 00:42:41,379 --> 00:42:44,239 So now you need to go to the last common ancestor. 680 00:42:44,569 --> 00:42:46,586 And from there you need to figure out like. 681 00:42:46,891 --> 00:42:48,691 How do I linearize this? 682 00:42:49,304 --> 00:42:53,294 I've opted for a model where everything that was pushed once cannot be 683 00:42:53,294 --> 00:42:55,214 overwritten, so there's no force push. 684 00:42:55,484 --> 00:42:58,904 So you basically just get to append stuff at the end. 685 00:42:59,444 --> 00:43:04,887 But, in order to get there, you need to first roll back your own stuff, then 686 00:43:05,277 --> 00:43:07,497 play forward what you've gotten first. 687 00:43:08,021 --> 00:43:10,151 and then on top add those. 688 00:43:10,481 --> 00:43:15,301 And the rolling back with SQLite is a, thing that I've like put a lot of 689 00:43:15,301 --> 00:43:21,314 time into where I've been using another SQLite extension, called the SQLite 690 00:43:21,334 --> 00:43:27,577 Sessions extension, which allows you, per SQLite write, to basically, record 691 00:43:27,877 --> 00:43:29,677 what has the thing actually done. 692 00:43:30,037 --> 00:43:32,677 So instead of storing, insert. 693 00:43:33,144 --> 00:43:34,764 Into issues, blah, blah, blah. 694 00:43:35,214 --> 00:43:40,614 when running that, you get a blob of let's say 30 bytes, and that has 695 00:43:40,614 --> 00:43:45,891 recorded on SQLite level, what has happened to the SQLite database. 696 00:43:46,311 --> 00:43:52,511 And I store that alongside of each change event, that sits in the event log. 697 00:43:53,081 --> 00:43:57,481 And the very cool thing about this is, I can use that to replay it 698 00:43:57,481 --> 00:44:01,201 on top of another database, but to kind of catch it up more quickly. 699 00:44:01,501 --> 00:44:03,001 But I can also invert it. 700 00:44:03,421 --> 00:44:07,381 So now I have basically this like, let's say 20 events. 701 00:44:07,681 --> 00:44:10,981 And for each, I've recorded what has happened on SQLite level, 702 00:44:11,341 --> 00:44:12,811 and now I can basically say. 703 00:44:13,101 --> 00:44:17,901 When I need to roll back, I can revisit each of those, invert each of those 704 00:44:17,901 --> 00:44:22,614 change sets, apply them again on the SQLite database, and then I'll end up 705 00:44:22,944 --> 00:44:27,114 where I was before and that's how I've implemented rollback on top of SQL Lite. 706 00:44:27,504 --> 00:44:32,331 So this is as mentioned when you're going, down the, rabbit hole 707 00:44:32,331 --> 00:44:34,551 of like imposing a total order. 708 00:44:34,821 --> 00:44:37,881 There's a lot of downstream things you need to do that makes 709 00:44:37,881 --> 00:44:39,501 this even more complicated. 710 00:44:39,891 --> 00:44:43,029 But, from what I can see, you're, on the right track if 711 00:44:43,029 --> 00:44:44,289 you wanna pursue this further. 712 00:44:45,373 --> 00:44:45,583 Yeah. 713 00:44:45,703 --> 00:44:52,333 And I do have a rebasing mechanism in place in mind that's more, 714 00:44:52,569 --> 00:44:53,739 just kind of a sledgehammer. 715 00:44:53,739 --> 00:44:56,259 I got two SQLite databases in mind. 716 00:44:56,566 --> 00:45:00,796 in the same way that on Git you have like your local copy of the main line and your 717 00:45:00,796 --> 00:45:05,536 local copy of your work, there's always this local copy of Main, that's just 718 00:45:05,596 --> 00:45:07,726 whatever events have come from the server. 719 00:45:07,846 --> 00:45:12,286 So this is the source of truth that the server has told me about and that was 720 00:45:12,286 --> 00:45:13,696 something I forgot to mention earlier. 721 00:45:13,696 --> 00:45:16,906 Explaining all of this is the server is the source of truth. 722 00:45:16,906 --> 00:45:21,166 It has that main line of the order of all of the events, and that is 723 00:45:21,166 --> 00:45:22,966 what all the clients use to trust. 724 00:45:23,308 --> 00:45:27,058 But yeah, it has like that local copy, and then when it pulls from 725 00:45:27,058 --> 00:45:29,248 the server, it'll update that copy. 726 00:45:29,638 --> 00:45:33,688 It'll look at all the events that are kind of ahead in the client, 727 00:45:33,988 --> 00:45:39,028 and then it'll say, okay, I'm gonna roll back my client copy of my 728 00:45:39,028 --> 00:45:41,518 branch to whatever the server is. 729 00:45:41,578 --> 00:45:43,768 And it's literally just a file right call. 730 00:45:43,768 --> 00:45:45,898 So it just overwrites. 731 00:45:46,218 --> 00:45:50,448 Your like client SQLlite file with a copy of the server one. 732 00:45:50,898 --> 00:45:53,928 And then we look at the events that the server didn't acknowledge yet 733 00:45:53,928 --> 00:45:58,338 and then we replay those on top as a very basic way to pull and make 734 00:45:58,338 --> 00:46:02,238 sure, because it's very possible that you made some changes locally that 735 00:46:02,238 --> 00:46:04,188 the server hasn't acknowledged yet. 736 00:46:04,218 --> 00:46:08,298 Like you've pushed them up still in process and you pull down the 737 00:46:08,298 --> 00:46:11,748 latest changes and you don't see all of that stuff that you pushed 738 00:46:11,748 --> 00:46:13,938 up yet because of network latency. 739 00:46:14,328 --> 00:46:18,618 So this sort of avoids that problem where you pull down from the server 740 00:46:18,768 --> 00:46:21,948 and now you need to replay whatever you did on the client that the 741 00:46:21,948 --> 00:46:23,538 server hasn't acknowledged yet. 742 00:46:23,538 --> 00:46:25,548 It hasn't received that network request. 743 00:46:25,908 --> 00:46:30,198 So that was a very basic need to have some rebasing, but it does 744 00:46:30,198 --> 00:46:34,518 get a lot more complicated when you have collaborators on a document. 745 00:46:34,938 --> 00:46:37,698 I've seen a few different versions of this. 746 00:46:37,772 --> 00:46:40,784 CRDTs is the fun, like magic wand. 747 00:46:40,784 --> 00:46:41,834 It does everything. 748 00:46:42,258 --> 00:46:47,063 but there are also solutions from Figma, for example, where they 749 00:46:47,063 --> 00:46:50,693 say everything in Figma is kind of its own little data structure. 750 00:46:50,693 --> 00:46:53,813 Like you can put some text and that's its own little data field. 751 00:46:54,053 --> 00:46:54,983 You have rectangles. 752 00:46:54,983 --> 00:46:56,243 Those are a data field. 753 00:46:56,603 --> 00:47:01,253 And whenever you update a rectangle, like you update the pixel width of 754 00:47:01,253 --> 00:47:05,543 a rectangle, that's like an update event on some SQL table that stores 755 00:47:05,543 --> 00:47:07,433 all the rectangles for this document. 756 00:47:07,823 --> 00:47:12,233 So whenever you make that update, it'll update the pixel value of whatever 757 00:47:12,233 --> 00:47:17,183 that row entry is, and then it'll push it up for other people to receive. 758 00:47:17,543 --> 00:47:20,453 And when you pull it down, it's last right wins. 759 00:47:20,843 --> 00:47:24,623 In other words, whoever the last person is in that order that the 760 00:47:24,623 --> 00:47:26,513 server decided on that total order. 761 00:47:26,693 --> 00:47:28,373 That's a new word I know about now. 762 00:47:28,433 --> 00:47:31,799 Didn't know it was called total order, but yeah, that, once you pull it down, 763 00:47:31,799 --> 00:47:35,159 whatever the server said was the order of events, that's gonna be the final 764 00:47:35,159 --> 00:47:37,169 state of that rectangle on your device. 765 00:47:38,059 --> 00:47:41,539 The only time it becomes a problem, and you may have experienced this, if you're 766 00:47:41,539 --> 00:47:45,139 ever working on like a fig jam together with a bunch of people, if you're all 767 00:47:45,139 --> 00:47:48,709 typing in the same text box, everyone's just like overriding each other and a 768 00:47:48,709 --> 00:47:52,099 text box glitches out and changes to whatever's on the other person's screen. 769 00:47:52,309 --> 00:47:55,309 You can't see people's cursors because you're fighting to update 770 00:47:55,309 --> 00:47:59,779 the exact same entry in the database and it can't reconcile those changes. 771 00:48:00,236 --> 00:48:04,946 so it only works up to, like you're editing different things 772 00:48:04,946 --> 00:48:08,486 in the file and you're not really stepping on each other too much. 773 00:48:08,816 --> 00:48:12,146 As soon as you're stepping on each other trying to edit like the same text field, 774 00:48:12,596 --> 00:48:17,326 then you wanna reach for something that's very, very fancy, like CRDTs. 775 00:48:17,531 --> 00:48:20,861 Which will try to merge elegantly all of the changes that you're 776 00:48:20,861 --> 00:48:23,081 typing into the same database field. 777 00:48:23,531 --> 00:48:28,301 It's maybe over-prescribed because of how powerful it is, but for those specific 778 00:48:28,301 --> 00:48:32,531 scenarios, it's really nice to reach for, and we can talk about them if you want. 779 00:48:32,561 --> 00:48:36,551 I only have a high level understanding of what CRDTs do, but it would be 780 00:48:36,551 --> 00:48:38,681 something to apply that kind of problem. 781 00:48:39,198 --> 00:48:45,444 my takeaway from where to apply, CRDTs versus where I would apply event sourcing 782 00:48:45,774 --> 00:48:50,778 is, CR DTs great for in two scenarios. 783 00:48:51,108 --> 00:48:54,151 One, if you don't quite know yet where you want to go. 784 00:48:54,706 --> 00:48:59,026 And where in the past you might've reached for, let's say, Firebase to 785 00:48:59,026 --> 00:49:00,736 just like have a backend of service. 786 00:49:00,916 --> 00:49:04,156 You know, you might want to change it later, but you just, for now, 787 00:49:04,156 --> 00:49:08,466 you just want to get going and, you can, particularly if you 788 00:49:08,466 --> 00:49:12,156 don't have like a strict schema across your entire application. 789 00:49:12,156 --> 00:49:17,196 So you just try to like, not go off the rails too much, but at least the 790 00:49:17,196 --> 00:49:21,846 data is like, mostly, like across the applications in a good spot. 791 00:49:22,326 --> 00:49:26,389 But as you roll this out in production, and, we are shipping 792 00:49:26,449 --> 00:49:31,096 an iOS app as well, that someone is, running an old version on. 793 00:49:31,786 --> 00:49:35,836 Now you don't quite know, oh, this document, this data document that has 794 00:49:35,836 --> 00:49:39,826 been synced around here, this might not yet have this field that the 795 00:49:39,826 --> 00:49:41,806 newer application version depends on. 796 00:49:42,196 --> 00:49:47,416 So now you have, like, this is where time drifts in a more significant 797 00:49:47,416 --> 00:49:52,066 way and in the more traditional application architecture approach 798 00:49:52,066 --> 00:49:54,976 you would, this way you don't trust the client in the first place. 799 00:49:54,976 --> 00:49:58,876 Then you have like your API endpoint and the APIs, versioned, et cetera, and 800 00:49:58,876 --> 00:50:00,876 everything is governed through the, API. 801 00:50:01,056 --> 00:50:03,906 But now you also need to tame that problem somehow. 802 00:50:03,906 --> 00:50:07,736 So at this point you're already, going a little bit beyond where I 803 00:50:07,736 --> 00:50:12,626 think CRDTs shine right now, which brings me to my next kind of more 804 00:50:12,656 --> 00:50:18,953 evergreen scenario for CRDTs, which are like very specific, tasks. 805 00:50:19,253 --> 00:50:22,403 And so text editing, particularly rich text editing. 806 00:50:22,648 --> 00:50:28,391 Is such a scenario where I think CRDTs are just like a very, very good, approach. 807 00:50:28,391 --> 00:50:32,691 There's also like, you can also use ot, like operational transform, which 808 00:50:32,691 --> 00:50:37,808 is, somewhat related under the covers, works a bit differently, but the way how 809 00:50:37,808 --> 00:50:39,688 you would use it is pretty similarly. 810 00:50:40,328 --> 00:50:45,624 And, related to rich text editing is also when you have like complex 811 00:50:45,624 --> 00:50:49,434 list structures where you wanna move things within the list. 812 00:50:49,434 --> 00:50:55,031 So if you want to go for the, Figma scenario, let's say you change the 813 00:50:55,031 --> 00:51:01,261 order of like multiple rectangles, like where do they sit in that layer order? 814 00:51:01,681 --> 00:51:04,111 how do you convey how you wanna change that? 815 00:51:04,111 --> 00:51:08,734 You could always, have like maybe an array of all the IDs that give 816 00:51:08,734 --> 00:51:13,064 you this perfect order, but if this kind of happens concurrently, 817 00:51:13,064 --> 00:51:14,504 then you need to reconcile that. 818 00:51:14,774 --> 00:51:15,914 So that's not great. 819 00:51:16,184 --> 00:51:20,271 And this is where CRDTs are also like a very, special purpose 820 00:51:20,301 --> 00:51:22,371 tool, which works super well. 821 00:51:23,076 --> 00:51:28,236 And so what I've landed on is use event sourcing for everything except 822 00:51:28,236 --> 00:51:33,306 where I need those special purpose tools, and this is where them reach 823 00:51:33,306 --> 00:51:35,676 for CRDTs or for something else. 824 00:51:35,856 --> 00:51:40,863 That's kind of the conclusion I, took away if you like the event sourcing approach. 825 00:51:41,403 --> 00:51:46,576 But, I think ultimately it really comes down to what is the application 826 00:51:46,576 --> 00:51:51,136 that you're building and what are, like, what is the domain of what 827 00:51:51,136 --> 00:51:54,376 you're building and which sort of trade-offs does this require? 828 00:51:54,646 --> 00:51:56,056 So I think in Figma. 829 00:51:56,116 --> 00:52:02,056 The real timeness is really important and it is recognized that those different 830 00:52:02,056 --> 00:52:06,969 pieces that are floating around, they're like pretty, independent from each other. 831 00:52:07,239 --> 00:52:10,389 So, and if they're independent, then you don't need that total order 832 00:52:10,389 --> 00:52:14,993 between that, which makes everything a lot easier in terms of scalability, 833 00:52:14,993 --> 00:52:18,143 in terms of correctness, and then you don't need to rebase as much. 834 00:52:18,743 --> 00:52:21,713 distributed systems is the ultimate case of it depends. 835 00:52:22,336 --> 00:52:27,823 and I think trying to build one like you did, I think is a very good way 836 00:52:28,113 --> 00:52:30,093 to like build a better understanding. 837 00:52:30,153 --> 00:52:35,493 And also I think that opens your eyes of like, ah, now I understand why Figma 838 00:52:35,493 --> 00:52:40,563 has this shortcoming or Notion if we are trying to change the same line, change the 839 00:52:40,563 --> 00:52:43,583 same block as where last writers, applies. 840 00:52:43,973 --> 00:52:48,776 Whereas in Google Docs, for example, we could easily change the, same word even. 841 00:52:49,076 --> 00:52:51,786 And it would reconcile that in a, in a better way. 842 00:52:52,656 --> 00:52:57,636 But, maybe you have some advice for people like yourself when you're 843 00:52:57,636 --> 00:52:59,616 just getting started on that journey. 844 00:53:00,006 --> 00:53:05,226 What would you tell people what they should do maybe shouldn't yet do? 845 00:53:05,609 --> 00:53:07,139 today 2025? 846 00:53:07,139 --> 00:53:08,909 There's more technologies out there now. 847 00:53:09,149 --> 00:53:11,459 What would you recommend to someone who's curious? 848 00:53:12,113 --> 00:53:13,883 Depends on the type of learner you are. 849 00:53:13,883 --> 00:53:16,013 Sometimes some are very. 850 00:53:16,774 --> 00:53:21,634 outcome driven, like I need to see an app running in production for me to 851 00:53:21,754 --> 00:53:24,064 really get excited about this Tech. 852 00:53:24,484 --> 00:53:26,464 Other people are very first principles driven. 853 00:53:26,464 --> 00:53:30,034 Like I want to like screw in every nut and bolt myself to 854 00:53:30,034 --> 00:53:31,564 get excited about this thing. 855 00:53:32,068 --> 00:53:35,698 I tend to fall into the first camp where I think it is very useful to just 856 00:53:35,758 --> 00:53:39,928 look at the docs for something like Replicache and see how would you implement 857 00:53:39,928 --> 00:53:42,478 this kind of protocol step by step. 858 00:53:42,838 --> 00:53:44,788 Like how would you set up the event sourcing? 859 00:53:45,358 --> 00:53:48,838 How would you put the SQLlite store in the browser in the first place? 860 00:53:48,838 --> 00:53:50,608 Like what capabilities are there? 861 00:53:50,998 --> 00:53:53,218 And then try to think through those edge cases. 862 00:53:53,578 --> 00:53:58,168 As you run into them trying to build something, I use Linear as my sort 863 00:53:58,168 --> 00:54:01,468 of learning example, but you could use pretty much anything you want. 864 00:54:02,098 --> 00:54:03,688 so that's definitely one approach. 865 00:54:03,688 --> 00:54:07,258 Now there's just so many resources for how these things work under 866 00:54:07,258 --> 00:54:11,308 the hood that you can easily learn about the intricacies yourself. 867 00:54:11,748 --> 00:54:17,601 Another, resource is the talks given by who's the engineer at Linear. 868 00:54:18,621 --> 00:54:19,611 I think you've had him on the show. 869 00:54:19,611 --> 00:54:20,021 Tuomas? 870 00:54:20,696 --> 00:54:21,326 Yes. 871 00:54:21,386 --> 00:54:21,686 Yeah. 872 00:54:21,686 --> 00:54:27,896 He gave a few really helpful talks about how the Linear sync engine works on a high 873 00:54:28,076 --> 00:54:30,836 level, and that one's more opinionated. 874 00:54:30,896 --> 00:54:36,416 It reaches for technologies like MobX, which is a react specific state 875 00:54:36,446 --> 00:54:39,596 store, and also MongoDB for documents. 876 00:54:39,713 --> 00:54:41,713 but you still get a high level of how they think about the 877 00:54:41,713 --> 00:54:43,183 problem, which is really nice. 878 00:54:43,826 --> 00:54:46,076 the other option, if you're really results driven, you wanna 879 00:54:46,076 --> 00:54:47,726 see a local-first step running. 880 00:54:48,086 --> 00:54:52,346 You can reach for all sorts of frameworks and libraries at this point. 881 00:54:52,573 --> 00:54:58,333 Zero is the one that I've played with most recently, and it is Alpha Software you'll 882 00:54:58,333 --> 00:55:03,620 run into, it holds your hands, plugging in every battery and setting up everything. 883 00:55:03,710 --> 00:55:06,530 But error codes could be very confusing. 884 00:55:06,936 --> 00:55:10,536 but luckily their Discord is very welcoming and will answer any question 885 00:55:10,536 --> 00:55:13,836 that you have since their only goal is for everyone to get excited about 886 00:55:13,836 --> 00:55:15,216 the tech and use it in production. 887 00:55:15,726 --> 00:55:20,496 So I think Zero is a really great starting point as just, I wanna build an app. 888 00:55:20,586 --> 00:55:22,026 I'm gonna reach for a library. 889 00:55:22,416 --> 00:55:24,726 It will give you a query builder. 890 00:55:24,786 --> 00:55:28,326 So instead of writing raw SQL, it'll help you write SQL queries 891 00:55:28,326 --> 00:55:29,856 with some JavaScript functions. 892 00:55:30,276 --> 00:55:34,536 And it also works you through very common problems that you do hit at some point. 893 00:55:35,076 --> 00:55:38,826 And the big one is data migrations and, well, not data migrations, schema 894 00:55:38,826 --> 00:55:44,706 migrations, because when you have a data store on the client and you have a source 895 00:55:44,706 --> 00:55:49,266 of truth on the server everyone has to agree on how that data is shaped if 896 00:55:49,266 --> 00:55:54,096 you're using a SQL model and not something that's Firebasey as you were mentioning. 897 00:55:54,636 --> 00:55:57,876 So in those cases, you have to know like the four or five step 898 00:55:57,876 --> 00:56:02,916 process of update the server schema to add the new field, then update 899 00:56:02,916 --> 00:56:04,566 the client to add that new field. 900 00:56:04,836 --> 00:56:08,706 And then if you're trying to delete an old field for some reason, you would 901 00:56:08,706 --> 00:56:13,896 need to execute those on client, then server in the correct order, and then 902 00:56:13,896 --> 00:56:17,856 manage a database version so that if a client tries to connect with really, 903 00:56:17,856 --> 00:56:22,986 really old application code, the server can say, sorry, I only accept people who 904 00:56:22,986 --> 00:56:26,076 are on version five of this SQL schema. 905 00:56:26,256 --> 00:56:29,826 You're on version three, so I'm just gonna hard refresh your webpage and 906 00:56:29,826 --> 00:56:31,356 get you up to the latest version. 907 00:56:31,993 --> 00:56:35,443 all of these challenges are really interesting to think about and Zero 908 00:56:35,743 --> 00:56:40,123 helps you think through them out of the box and presents docs on all of these 909 00:56:40,123 --> 00:56:41,833 problems before you run into them. 910 00:56:42,440 --> 00:56:45,740 but I happen to be the type that wants to run into as many brick 911 00:56:45,740 --> 00:56:49,280 walls as possible without someone telling me what to worry about. 912 00:56:49,430 --> 00:56:50,690 I just wanna worry about it. 913 00:56:51,050 --> 00:56:57,020 so I think the Simple Sync Engine resource is great just because it 914 00:56:57,020 --> 00:57:01,250 doesn't do very much and there's a lot left up to the reader to go off 915 00:57:01,250 --> 00:57:03,140 and try to run into those challenges. 916 00:57:03,440 --> 00:57:07,460 I'm sure splunking through like the LiveStore implementation, I would 917 00:57:07,460 --> 00:57:12,620 find 50 ways that I could improve what I'm doing to get to that next step of 918 00:57:12,650 --> 00:57:14,690 like resilience, schema, migrations. 919 00:57:14,690 --> 00:57:16,910 I literally didn't even touch schema migrations. 920 00:57:17,312 --> 00:57:20,130 there's so much that you need to think about that just crawling 921 00:57:20,130 --> 00:57:23,423 through open source libraries is really, really helpful with, so 922 00:57:23,423 --> 00:57:25,223 that's my preferred learning approach. 923 00:57:25,223 --> 00:57:26,303 I just like going that way. 924 00:57:26,814 --> 00:57:27,894 I completely agree. 925 00:57:27,894 --> 00:57:31,794 And I also like, it's, it's sort of a bit of convincing yourself, 926 00:57:31,824 --> 00:57:33,504 is this entire thing worth it? 927 00:57:33,954 --> 00:57:38,754 And what I always appreciate if someone knows a little thing about 928 00:57:38,964 --> 00:57:41,064 me and then tells me, you know what? 929 00:57:41,304 --> 00:57:42,804 I don't think this is for you. 930 00:57:43,075 --> 00:57:46,555 I wouldn't hold anything back for someone who wants to look into this. 931 00:57:47,260 --> 00:57:49,990 to say like, this might not be what you're looking for. 932 00:57:49,990 --> 00:57:57,800 If someone is very happy with like building web apps with Vite Astro NextJS, 933 00:57:57,820 --> 00:58:02,600 et cetera, and they're productive, they're building this, e-commerce 934 00:58:02,600 --> 00:58:08,930 platform, or they're building a more static website, I don't think there's 935 00:58:08,930 --> 00:58:14,524 anything really where local-first would change their work situation. 936 00:58:14,914 --> 00:58:21,154 But if they're frustrated with like actual apps that they use day to day, when you're 937 00:58:21,154 --> 00:58:25,054 frustrated like yourself, when you're frustrated with Notion being too slow, 938 00:58:25,054 --> 00:58:30,034 et cetera, and you're building those more productivity daily driver apps yourself. 939 00:58:30,304 --> 00:58:32,134 For me that was like a music app. 940 00:58:32,134 --> 00:58:34,834 I got frustrated with Spotify and other music apps. 941 00:58:35,164 --> 00:58:40,517 I think this is the, right scenario where like local-first has something to offer, 942 00:58:40,907 --> 00:58:47,177 but, and I think it has also the potential to become a lot simpler and easier over 943 00:58:47,177 --> 00:58:51,227 has already become a lot simpler and easier or the past couple of years, and 944 00:58:51,287 --> 00:58:53,627 it's gonna be even more so in the future. 945 00:58:53,897 --> 00:58:56,987 And there will be use cases where it's actually simpler. 946 00:58:57,442 --> 00:59:02,152 To use local-first to build something, then using Next for something. 947 00:59:02,675 --> 00:59:05,255 but that won't apply to all scenarios. 948 00:59:05,645 --> 00:59:07,475 And so it is not a silver bullet. 949 00:59:07,922 --> 00:59:12,152 the closest thing you'll get to a silver bullet is the right architecture for 950 00:59:12,152 --> 00:59:17,612 the right application scenario, but by default there is no silver bullet. 951 00:59:17,882 --> 00:59:19,272 Neither is local-first. 952 00:59:19,292 --> 00:59:22,742 And I think someone should evaluate, Hey, is this even for me? 953 00:59:23,139 --> 00:59:24,729 that's, I think should be the starting point. 954 00:59:25,282 --> 00:59:34,072 Yeah, and a meta comment just because now I'm in the agent coding space, Warp 955 00:59:34,072 --> 00:59:38,392 is getting more capable by the day of actually editing files and scaffolding new 956 00:59:38,392 --> 00:59:41,032 applications for you, from the terminal. 957 00:59:41,685 --> 00:59:47,205 I've found it's less valuable to know the syntax of how all of these libraries work 958 00:59:47,235 --> 00:59:51,045 and a lot more valuable to just know high level, what are they doing, what's the 959 00:59:51,045 --> 00:59:52,875 architecture and how would I debug it? 960 00:59:53,486 --> 00:59:56,892 because these agents are very good at spitting out the syntax, 961 00:59:56,892 --> 00:59:58,422 if you draw a very clear picture. 962 00:59:59,007 --> 01:00:03,057 So if you go off and read designing data intensive applications, and you 963 01:00:03,057 --> 01:00:06,987 start diagramming to yourself how all of these systems are distributed, you 964 01:00:06,987 --> 01:00:12,387 could bring that diagram to Warp or just the cloud website if you want, and 965 01:00:12,387 --> 01:00:13,857 say, I wanna build this kind of app. 966 01:00:13,857 --> 01:00:15,717 Here's how the architecture's gonna work. 967 01:00:15,777 --> 01:00:19,647 This is gonna talk to this, and I know about this library. 968 01:00:19,647 --> 01:00:23,727 I know LiveStore uses event sourcing, so I would like you to implement that 969 01:00:24,147 --> 01:00:28,767 and use React, but follow the handrails because I understand the architecture. 970 01:00:29,097 --> 01:00:32,727 It'll give you a way better application than if you were to just say, give 971 01:00:32,727 --> 01:00:35,067 me a local-first app with React. 972 01:00:35,127 --> 01:00:38,547 It would probably maybe not struggle in the beginning, but definitely 973 01:00:38,547 --> 01:00:43,137 struggle as you try to figure out what it is doing or debug whatever sort 974 01:00:43,137 --> 01:00:44,757 of system level issues you're having. 975 01:00:45,357 --> 01:00:46,497 I fully agree. 976 01:00:46,497 --> 01:00:50,817 And given that the both of us are not just application developers but also 977 01:00:50,847 --> 01:00:55,707 tool creators, we spend a lot of time thinking about like, how do I leverage 978 01:00:55,707 --> 01:01:00,597 the degree of freedom that I have here in the API, the way how I design the API 979 01:01:00,837 --> 01:01:05,007 that is intuitive for someone that they like, ideally that this becomes like a 980 01:01:05,007 --> 01:01:09,597 pit of success where they intuitively use it the right way, but also if they 981 01:01:09,597 --> 01:01:11,667 use it the wrong way, how do they notice? 982 01:01:11,817 --> 01:01:17,187 Do they notice like as early on as possible through type safety or only 983 01:01:17,187 --> 01:01:20,397 if they're already in production and they felt like, wait, no, 984 01:01:20,397 --> 01:01:24,387 this, like, this was a path that I've wrongly taken six months ago. 985 01:01:24,744 --> 01:01:30,424 so you want to design all of this in a way that you like learn as early as possible 986 01:01:30,424 --> 01:01:31,864 whether you're in the right track or not. 987 01:01:31,864 --> 01:01:36,544 And I think you can't get better than simplicity than going for simplicity. 988 01:01:36,544 --> 01:01:41,684 And this is why I love the path that you've taken with the Simple Sync Engine. 989 01:01:42,134 --> 01:01:46,484 Through the push pull model because that's already, that is deeply familiar for 990 01:01:46,484 --> 01:01:51,044 developers and that is how we're using Git and that has really been proven. 991 01:01:51,074 --> 01:01:53,804 And there you can't really get much simpler than that. 992 01:01:54,314 --> 01:01:57,224 And I think simple is great for everyone. 993 01:01:57,621 --> 01:02:00,981 and once we have a simple foundation, we have a reliable foundation. 994 01:02:00,981 --> 01:02:03,591 We can build fast and nice things on top of it. 995 01:02:03,921 --> 01:02:08,931 But particularly mentioning AI systems, I make a lot of design trade offs 996 01:02:08,931 --> 01:02:12,861 now differently, where I care less about how much effort it will be 997 01:02:12,861 --> 01:02:15,591 to write or to discover that thing. 998 01:02:15,591 --> 01:02:20,571 Since we have now LLMs, do TXT, et cetera, I care a lot more about like, 999 01:02:20,571 --> 01:02:24,821 how does, how will you even spot where like this doesn't seem right. 1000 01:02:24,821 --> 01:02:29,321 The robot has given me something weird and just doesn't match my, 1001 01:02:29,321 --> 01:02:32,981 like, primitive understanding of how this entire thing fits together. 1002 01:02:33,281 --> 01:02:37,091 And that should also help the robot to like not go in the wrong 1003 01:02:37,151 --> 01:02:39,101 direction in, in the first place. 1004 01:02:39,461 --> 01:02:41,504 So, yeah, I love that. 1005 01:02:41,714 --> 01:02:47,071 Like, and going for a simple design decision, the simple like overall 1006 01:02:47,101 --> 01:02:52,906 system architecture that's gonna help you as a future programmer, observing 1007 01:02:52,966 --> 01:02:57,609 little robots, building things, a lot more to know what's going on. 1008 01:02:57,639 --> 01:03:01,659 So maybe we use that as a last segue to hear a little bit more 1009 01:03:01,659 --> 01:03:06,219 about what you're doing now at Warp in regards to agents, et cetera. 1010 01:03:06,219 --> 01:03:08,856 I've been using Warp for, a little bit. 1011 01:03:09,306 --> 01:03:15,356 I still use it, for my standalone, terminal, but most of my terminal 1012 01:03:15,356 --> 01:03:20,366 work is also happening within Cursor, which is integrated in like the 1013 01:03:20,666 --> 01:03:22,676 agent thing and Cursor, et cetera. 1014 01:03:22,706 --> 01:03:27,176 Maybe can, yeah, help me a little bit of like how I bring those two together 1015 01:03:27,386 --> 01:03:29,041 and use them for what they're best at. 1016 01:03:29,509 --> 01:03:35,719 Yeah, and it's an interesting world of sort of agentic coding solutions. 1017 01:03:35,719 --> 01:03:38,269 It feels like there's a new approach to it every other day. 1018 01:03:38,611 --> 01:03:42,563 before joining Warp, I was also using Kline a lot, which is a VS Code 1019 01:03:42,563 --> 01:03:47,573 extension that's fully open source that will, from my experience, give 1020 01:03:47,573 --> 01:03:49,643 you a more quality agent output. 1021 01:03:49,793 --> 01:03:54,443 Since it has these two phases, you can flip on a plan switch and 1022 01:03:54,443 --> 01:03:59,693 it'll use a reasoning model to take whatever you tell it and turn 1023 01:03:59,693 --> 01:04:01,853 it into like a step-by-step plan. 1024 01:04:02,003 --> 01:04:04,493 And you could walk through like, no, that architecture doesn't make sense, 1025 01:04:04,493 --> 01:04:06,653 or Let me upload this six cal draw. 1026 01:04:06,653 --> 01:04:07,883 I actually want to work like this. 1027 01:04:07,883 --> 01:04:11,003 And you can go back and forth on like a design doc and then you can 1028 01:04:11,003 --> 01:04:15,383 flip it to act mode and that engages Claude to go build that for you. 1029 01:04:15,863 --> 01:04:19,613 And it's very hit or miss actually doing like the code edits. 1030 01:04:19,643 --> 01:04:20,933 We're all struggling with that. 1031 01:04:21,563 --> 01:04:24,983 but it was like this really nice mental model of, oh yeah, we're 1032 01:04:24,983 --> 01:04:26,363 gonna plan it out together. 1033 01:04:26,648 --> 01:04:29,798 We're gonna design jam, how this thing's gonna work and then we're 1034 01:04:29,798 --> 01:04:33,908 gonna go build it and I can just let it run and see how it ends up working. 1035 01:04:34,338 --> 01:04:39,588 and Warp is doing something similar but not within the confines of VS code. 1036 01:04:39,888 --> 01:04:43,788 And also with the addition of a voice button that I've been using a lot more 1037 01:04:43,788 --> 01:04:48,948 recently because you can talk faster than you can type is generally what I found. 1038 01:04:48,978 --> 01:04:52,608 So I can just speak into my terminal, here's how I want the app to 1039 01:04:52,608 --> 01:04:53,868 work, this, that, and the other. 1040 01:04:54,138 --> 01:04:58,008 And then it will, depending on how complicated the question is, it 1041 01:04:58,008 --> 01:04:59,778 will reach for the planning step. 1042 01:05:00,018 --> 01:05:01,878 Otherwise it'll just give you an answer right away. 1043 01:05:01,998 --> 01:05:04,488 So if it, see that's kind of complicated, let me plan it out. 1044 01:05:04,788 --> 01:05:08,568 It'll give you that same kind of like document, here's how it's going to work. 1045 01:05:08,928 --> 01:05:10,878 And then you can say, okay, do it. 1046 01:05:10,908 --> 01:05:15,408 And then it will go off and tell Claude to make file edits and do other 1047 01:05:15,408 --> 01:05:19,068 things on your machine, which is much further than Warp has gone in the past. 1048 01:05:19,614 --> 01:05:23,214 and I've really enjoyed using this to build Swift apps recently. 1049 01:05:23,544 --> 01:05:27,084 Since I was just fascinated with like, how could I build a really 1050 01:05:27,084 --> 01:05:31,764 slick desktop client like ChatGPT there's this desktop shortcut to 1051 01:05:31,764 --> 01:05:33,264 like pull up a little chat bar. 1052 01:05:33,744 --> 01:05:36,174 Like I want something that integrated. 1053 01:05:36,234 --> 01:05:38,424 I don't wanna be confined to Google Chrome anymore. 1054 01:05:38,424 --> 01:05:39,564 I wanna break out of it. 1055 01:05:40,021 --> 01:05:45,451 but if you open up XCode, you're just met with this like decade old auto 1056 01:05:45,451 --> 01:05:49,441 complete that doesn't have anything that you want in order to get stuff done. 1057 01:05:50,028 --> 01:05:51,618 but Warp is just a terminal. 1058 01:05:51,618 --> 01:05:54,378 So I'm like, okay, I'll just open the Swift project in Warp 1059 01:05:54,378 --> 01:05:55,668 and say implement this feature. 1060 01:05:56,058 --> 01:06:00,318 And it doesn't it, it can literally just enter any directory that you 1061 01:06:00,318 --> 01:06:02,088 have and just start doing things. 1062 01:06:02,508 --> 01:06:07,428 I've also used it to like migrate my open source projects from a mono repo to a 1063 01:06:07,428 --> 01:06:12,348 set of micro repos on my system and says, oh yeah, I'll just make a new directory. 1064 01:06:12,348 --> 01:06:16,538 I'll move all those files over and I'll make the necessary file edits with Cloud. 1065 01:06:17,058 --> 01:06:19,158 Very like hit or miss quality. 1066 01:06:19,248 --> 01:06:20,268 We're dialing it in. 1067 01:06:20,628 --> 01:06:25,608 But this idea of you're not constrained to the IDE anymore, you can kind 1068 01:06:25,608 --> 01:06:30,528 of just pull up your terminal and ask it to modify anything from the 1069 01:06:30,528 --> 01:06:34,698 simplest request of, help me get revert, whatever the heck I just did. 1070 01:06:34,968 --> 01:06:38,838 And it'll help you get to something more complicated, like, why 1071 01:06:38,838 --> 01:06:40,488 isn't my Postgres server running? 1072 01:06:40,488 --> 01:06:42,618 And then it'll check your Homebrew installation. 1073 01:06:43,158 --> 01:06:47,448 And then you can take it one step further to, I actually want to fix this 1074 01:06:47,448 --> 01:06:49,728 error I see in my dev server right now. 1075 01:06:50,058 --> 01:06:53,118 Because you're running the dev server in your terminal, it can say, all 1076 01:06:53,118 --> 01:06:57,198 right, pause the server, debug, debug, debug, restart the server. 1077 01:06:57,618 --> 01:07:01,128 And then if something fails again, I'll go back to debugging. 1078 01:07:01,338 --> 01:07:04,278 So it's like watching your terminal session and figuring 1079 01:07:04,278 --> 01:07:05,868 out how to help you do something. 1080 01:07:06,288 --> 01:07:10,518 It feels like it's this natural next step of let's go from you're in an 1081 01:07:10,518 --> 01:07:14,868 editor typing code quickly to like, this is a general purpose tool on 1082 01:07:14,868 --> 01:07:18,918 your machine to edit all the software that you're writing in any setting. 1083 01:07:19,588 --> 01:07:22,138 so I'm just very excited about that kind of future. 1084 01:07:22,378 --> 01:07:25,078 And we've been moving very, very quickly towards it. 1085 01:07:25,108 --> 01:07:30,448 Just in the past month, it's gone from barely usable to, I'm actually using 1086 01:07:30,448 --> 01:07:34,408 this a lot for projects in a language that I don't even know how to speak. 1087 01:07:34,468 --> 01:07:35,068 Swift. 1088 01:07:35,568 --> 01:07:38,988 it's been kind of crazy how far you can get from zero to one 1089 01:07:39,108 --> 01:07:40,638 without a lot of field knowledge. 1090 01:07:41,313 --> 01:07:45,603 And intuitively this makes a lot of sense since like Eternal is kind of like the 1091 01:07:46,083 --> 01:07:51,333 OG chat up in a way where like all the way back to like IRC, et cetera, but 1092 01:07:51,333 --> 01:07:57,569 also now with, using ChatGPT a lot or, or other LLM chat products, like yes, 1093 01:07:57,569 --> 01:08:01,986 you're, chatting with the thing, like you write a command, the command happens 1094 01:08:01,986 --> 01:08:05,899 to be like plain English or another language, and you get something back. 1095 01:08:06,169 --> 01:08:08,073 But the, kinda like back and forth. 1096 01:08:08,463 --> 01:08:14,708 And the interplay is very similar and it makes so much more, so much sense that you 1097 01:08:14,708 --> 01:08:19,418 now bring that into the terminal as well, where you get the best of both worlds. 1098 01:08:19,418 --> 01:08:22,628 You can like ride out things in a fuzzy way. 1099 01:08:22,658 --> 01:08:27,278 The terminal helps you to like put that into proper computer speak. 1100 01:08:27,641 --> 01:08:32,441 but then you also get the efficiency and the correctness from what you can do in 1101 01:08:32,441 --> 01:08:38,411 a terminal and with all of like just a top-notch craft that you get within Warp 1102 01:08:38,501 --> 01:08:41,021 as a terminal with like blocks, et cetera. 1103 01:08:41,261 --> 01:08:44,191 So yeah, highly recommend everyone to, give it a try. 1104 01:08:44,654 --> 01:08:44,924 Yeah. 1105 01:08:44,924 --> 01:08:47,384 And it's free to reach for all those things. 1106 01:08:47,444 --> 01:08:50,744 And anyone who is just bothered by AI in their terminal, you can turn it 1107 01:08:50,744 --> 01:08:55,484 off and just use Warp as a really good terminal, which is how I started using 1108 01:08:55,484 --> 01:09:01,044 it way back, probably like, 2021, I think is when it said I created my account. 1109 01:09:01,466 --> 01:09:04,844 I just used it because I wanted something that looked nice and 1110 01:09:04,904 --> 01:09:06,854 now it's going a lot deeper. 1111 01:09:06,944 --> 01:09:09,201 And yeah, the, chat app analogy is perfect. 1112 01:09:09,201 --> 01:09:14,331 There's literally a toggle between typing out commands and asking it a question, 1113 01:09:14,811 --> 01:09:19,791 and it'll even flip back and forth based on like natural language, which is fancy. 1114 01:09:19,791 --> 01:09:22,251 I mean, I'll just hit the keyboard shortcut, but why 1115 01:09:22,251 --> 01:09:23,691 not make it a little flashier? 1116 01:09:24,351 --> 01:09:25,071 That is awesome. 1117 01:09:25,071 --> 01:09:28,324 Well, I've already seen, a bunch of your recent videos, 1118 01:09:28,366 --> 01:09:30,121 about content related to that. 1119 01:09:30,121 --> 01:09:32,041 I'm looking forward to many more of those. 1120 01:09:32,431 --> 01:09:37,821 Is there anything else that you wanna share related to your local-first, 1121 01:09:38,193 --> 01:09:39,924 explorations or otherwise? 1122 01:09:40,573 --> 01:09:41,053 Yeah. 1123 01:09:41,259 --> 01:09:44,739 so my profile is bholmesdev everywhere. 1124 01:09:44,799 --> 01:09:50,439 So any learning resources I've put out like videos on local-first and 1125 01:09:50,439 --> 01:09:54,909 conference talks, bholmesdev on YouTube and Twitter and Bluesky. 1126 01:09:55,376 --> 01:09:59,036 And also on GitHub, so these Simple Sync Engine project I mentioned, 1127 01:09:59,096 --> 01:10:00,506 that's on my personal GitHub. 1128 01:10:00,536 --> 01:10:02,276 Also under bholmesdev. 1129 01:10:02,486 --> 01:10:05,606 You should see it as one of the star repos if you look up the profile. 1130 01:10:06,086 --> 01:10:06,416 That's it. 1131 01:10:06,416 --> 01:10:06,806 Perfect. 1132 01:10:07,676 --> 01:10:11,503 I've also put it in the show notes, so everything, you'll find it there as well. 1133 01:10:11,503 --> 01:10:17,203 But, I'm really, really excited you have put in the effort to create this project, 1134 01:10:17,439 --> 01:10:21,946 because I think there's no better way to learn something than trying to do it. 1135 01:10:22,276 --> 01:10:25,096 And you've done that and you've allowed other people to follow 1136 01:10:25,096 --> 01:10:29,206 your footsteps, and I think that's a fantastic learning resource. 1137 01:10:29,206 --> 01:10:34,426 So thank you so much for doing that and for, yeah, helping others learn 1138 01:10:34,456 --> 01:10:37,899 and, sharing what you've learned and for coming on the show today. 1139 01:10:37,929 --> 01:10:38,349 Thank you. 1140 01:10:38,856 --> 01:10:39,636 Yeah, thanks so much. 1141 01:10:39,696 --> 01:10:41,976 This was like a really far reaching conversation. 1142 01:10:42,186 --> 01:10:42,906 I hope it turns out good. 1143 01:10:43,637 --> 01:10:46,217 Thank you for listening to the localfirst.fm podcast. 1144 01:10:46,397 --> 01:10:49,487 If you've enjoyed this episode and haven't done so already, please 1145 01:10:49,487 --> 01:10:50,777 subscribe and leave a review. 1146 01:10:51,167 --> 01:10:53,687 Please also share this episode with your friends and colleagues. 1147 01:10:54,077 --> 01:10:57,077 Spreading the word about the podcast is a great way to support 1148 01:10:57,077 --> 01:10:58,787 it and to help me keep it going. 1149 01:10:59,447 --> 01:11:02,867 A special thanks again to Jazz for supporting this podcast. 1150 01:11:03,167 --> 01:11:04,127 I'll see you next time.