1 00:00:00,078 --> 00:00:02,818 I mean, another thing is like the operational characteristics of the 2 00:00:02,818 --> 00:00:05,338 system, for this type of sync technology. 3 00:00:05,408 --> 00:00:09,648 So comparing HTTP with WebSockets, like WebSockets are stateful, and 4 00:00:09,648 --> 00:00:11,428 you do just keep things in memory. 5 00:00:11,708 --> 00:00:16,073 If you look across most real time systems, They have scalability limits because 6 00:00:16,073 --> 00:00:19,103 you will come to the point where if you have, say, 10, 000 concurrent users, 7 00:00:19,393 --> 00:00:22,353 it's almost like the thing of don't have too many open Postgres connections. 8 00:00:22,393 --> 00:00:26,093 But if you're holding open 10, 000 WebSockets, you may be able to do the 9 00:00:26,093 --> 00:00:30,173 IO efficiently, but you will ultimately be sort of growing that kind of memory 10 00:00:30,173 --> 00:00:31,443 and you'll hit some sort of barrier. 11 00:00:31,771 --> 00:00:34,875 Whereas, with this approach, you can basically offload that 12 00:00:34,875 --> 00:00:36,405 concurrency to the CDN layer. 13 00:00:37,305 --> 00:00:39,585 Welcome to the localfirst.fm podcast. 14 00:00:39,875 --> 00:00:42,755 I'm your host, Johannes Schickling, and I'm a web developer, a 15 00:00:42,755 --> 00:00:45,885 startup founder, and love the craft of software engineering. 16 00:00:46,345 --> 00:00:50,205 For the past few years, I've been on a journey to build a modern, high quality 17 00:00:50,205 --> 00:00:52,175 music app using web technologies. 18 00:00:52,495 --> 00:00:56,315 And in doing so, I've been falling down the rabbit hole of local-first software. 19 00:00:56,835 --> 00:00:59,865 This podcast is your invitation to join me on that journey. 20 00:01:00,255 --> 00:01:03,075 In this episode, I'm speaking to James Arthur. 21 00:01:03,565 --> 00:01:07,835 Founder and CEO of Electric SQL, a Postgres centric sync 22 00:01:07,835 --> 00:01:09,245 engine for local-first apps. 23 00:01:09,945 --> 00:01:14,395 In this conversation, we dive deep into how Electric works and explore 24 00:01:14,395 --> 00:01:18,955 its design decisions, such as read path syncing and using HTTP as a 25 00:01:18,955 --> 00:01:21,185 network layer to improve scalability. 26 00:01:21,745 --> 00:01:25,985 Towards the end, we're also covering PGLite, a new project by Electric 27 00:01:26,225 --> 00:01:27,895 that brings Postgres to Wasm. 28 00:01:28,705 --> 00:01:32,025 Before getting started, a big thank you to Rocicorp and PowerSync 29 00:01:32,905 --> 00:01:34,265 for supporting this podcast. 30 00:01:34,875 --> 00:01:36,535 And now, my interview with James. 31 00:01:37,115 --> 00:01:37,915 Welcome James. 32 00:01:37,995 --> 00:01:39,275 So good to have you on the podcast. 33 00:01:39,325 --> 00:01:40,905 How are you doing? 34 00:01:40,905 --> 00:01:41,225 Great. 35 00:01:41,485 --> 00:01:42,455 Yeah, really good to be here. 36 00:01:42,465 --> 00:01:43,375 Thank you for having me on. 37 00:01:43,525 --> 00:01:47,225 So the two of us know each other for quite a while already. 38 00:01:47,255 --> 00:01:51,565 And to be transparent, the two of us have actually already had quite 39 00:01:51,565 --> 00:01:52,985 a couple of projects together. 40 00:01:53,035 --> 00:01:57,525 The one big one among them is the first Local-First Conference that we 41 00:01:57,525 --> 00:01:59,445 organized together this year in Berlin. 42 00:01:59,695 --> 00:02:00,955 That was a lot of fun. 43 00:02:00,995 --> 00:02:05,002 But for those in the audience who don't know who you are, would 44 00:02:05,002 --> 00:02:06,212 you mind introducing yourself? 45 00:02:07,172 --> 00:02:09,122 So, my name is James Arthur. 46 00:02:09,192 --> 00:02:14,292 I am the CEO and one of the co-founder of Electric SQL. 47 00:02:14,872 --> 00:02:17,782 So, Electric is a Postgres sync engine. 48 00:02:18,202 --> 00:02:24,112 We sync little subsets of data out of Postgres into wherever you 49 00:02:24,112 --> 00:02:26,182 want, like local apps and services. 50 00:02:26,498 --> 00:02:30,008 and we do also have another, project which we developed called PGlite, 51 00:02:30,028 --> 00:02:32,998 which is a lightweight WASM Postgres. 52 00:02:33,368 --> 00:02:38,838 So we can sync out of Postgres in the cloud, into Postgres in the web browser, 53 00:02:39,258 --> 00:02:40,308 or kind of into whatever you want. 54 00:02:40,602 --> 00:02:41,082 Awesome. 55 00:02:41,092 --> 00:02:45,502 So yeah, I want to learn a lot more about Electric as well as PGlite. 56 00:02:45,532 --> 00:02:49,562 Maybe PGlite a little bit towards the end of this conversation. 57 00:02:49,872 --> 00:02:53,482 So Electric, I've seen it a bunch of times. 58 00:02:53,542 --> 00:02:58,652 I've been playing around with it, I think quite a bit last year, but 59 00:02:59,092 --> 00:03:01,422 things seem to also change quite a bit. 60 00:03:01,832 --> 00:03:03,482 Can you walk me through? 61 00:03:03,982 --> 00:03:08,552 What was the history of like the last couple of years as you've been 62 00:03:08,552 --> 00:03:12,982 working on Electric and help me inform the right mental model about 63 00:03:12,992 --> 00:03:15,282 Electric as it is going forward? 64 00:03:15,575 --> 00:03:16,645 Yeah, absolutely. 65 00:03:16,818 --> 00:03:24,053 I think like Electric as a project, it started, in a way, building on a bunch 66 00:03:24,053 --> 00:03:29,913 of research advances in distributed systems, CRDTs, transactional calls of 67 00:03:29,913 --> 00:03:33,983 consistency, a bunch of these primitives that a lot of people are building 68 00:03:33,983 --> 00:03:38,563 off in the local-first space, which actually a bunch of people on our team 69 00:03:38,673 --> 00:03:40,913 developed in the kind of research stage. 70 00:03:41,513 --> 00:03:48,653 And we wanted to create a developer tooling and a platform that allowed people 71 00:03:48,673 --> 00:03:53,613 who weren't experts in distributed systems and didn't have PhDs in CRDTs to be able 72 00:03:53,933 --> 00:03:58,353 to harness the same advances and build systems on the same types of guarantees. 73 00:03:58,363 --> 00:04:00,723 So in a way, that's where we started from. 74 00:04:00,763 --> 00:04:05,460 And we started building out on this research base into stronger consistency 75 00:04:05,470 --> 00:04:11,787 models for distributed databases and doing sync, from like a central 76 00:04:11,787 --> 00:04:15,527 cloud database out into whether it's to the edge or to the client. 77 00:04:16,024 --> 00:04:17,594 And then we're a startup. 78 00:04:17,594 --> 00:04:21,734 So like we built a small team and you go through this journey, building a 79 00:04:21,734 --> 00:04:26,030 company of, you have ideas for what's going to be useful and valuable for 80 00:04:26,030 --> 00:04:29,907 people, and you have a sense of sort of where the state of the art is and, what 81 00:04:29,927 --> 00:04:33,867 doesn't exist yet, but as you then go and experiment, you just learn more and more. 82 00:04:33,867 --> 00:04:36,810 And so you work out actually what people need and what 83 00:04:36,810 --> 00:04:38,430 problems you can solve with it. 84 00:04:38,814 --> 00:04:42,347 and so through that journey, we went from starting off thinking we were building 85 00:04:42,357 --> 00:04:48,557 a next generation distributed database to using the replication technology 86 00:04:48,597 --> 00:04:53,307 for that system behind existing open source databases like Postgres, SQLite, 87 00:04:53,627 --> 00:04:58,242 into finding, local-first software as a pattern is really the killer app for 88 00:04:58,242 --> 00:05:00,462 that type of replication technology. 89 00:05:00,722 --> 00:05:04,642 So people looking to build local-first applications because of all of the 90 00:05:04,642 --> 00:05:09,052 benefits around UX, DX, resilience, et cetera, but to do that, you 91 00:05:09,052 --> 00:05:10,782 need this type of sync layer. 92 00:05:11,059 --> 00:05:15,279 and then when we first focused on that, then we tried to build a 93 00:05:15,279 --> 00:05:19,399 very optimal end to end integrated local-first software platform. 94 00:05:19,829 --> 00:05:23,109 So for instance, if people saw Electric as a project, like this time last 95 00:05:23,109 --> 00:05:24,962 year, that's what we were building. 96 00:05:25,192 --> 00:05:30,017 And in a way we just found that we were having to solve too many problems and 97 00:05:30,017 --> 00:05:34,937 there was too much complexity making a kind of optimal one-size-fits-all sort of 98 00:05:34,947 --> 00:05:37,007 magic active active replication system. 99 00:05:37,017 --> 00:05:40,704 We were doing things like, managing the way you did the database migrations 100 00:05:40,704 --> 00:05:44,264 and schema revolution and generating a type safe client and doing the 101 00:05:44,264 --> 00:05:47,474 client side reactivity as well as all this sort of core sync stuff. 102 00:05:47,474 --> 00:05:51,094 So, as you know, there's a lot to that kind of end to end stack. 103 00:05:51,594 --> 00:05:55,654 Because we had wanted to build a system that integrated with people's 104 00:05:55,684 --> 00:05:59,884 existing software, like if you already had software built on Postgres or if 105 00:05:59,884 --> 00:06:04,717 you already had a working stack, like building that sort of full system was 106 00:06:05,177 --> 00:06:10,297 in a way sort of too complex and was difficult to adopt from existing software. 107 00:06:10,837 --> 00:06:16,737 So more recently we have consolidated down on building a much simpler sync engine, 108 00:06:17,067 --> 00:06:20,177 which is more like a composable tool that. 109 00:06:20,712 --> 00:06:23,772 You can run in front of Postgres, any Postgres. 110 00:06:23,962 --> 00:06:28,242 It works with any standard Postgres, any managed Postgres, any data model, any 111 00:06:28,242 --> 00:06:30,252 data types, any extensions that you have. 112 00:06:30,802 --> 00:06:35,412 And it just does this work of basically consuming the logical 113 00:06:35,412 --> 00:06:37,062 replication stream from Postgres. 114 00:06:37,265 --> 00:06:40,475 and then managing the way that the data is fanned out to clients, 115 00:06:41,065 --> 00:06:42,535 doing partial replication. 116 00:06:42,705 --> 00:06:44,555 So, because when you're syncing out, say, if you have 117 00:06:44,555 --> 00:06:46,465 a larger database in the cloud. 118 00:06:47,255 --> 00:06:49,795 And you're syncing out to like an app or a kind of edge service. 119 00:06:49,795 --> 00:06:51,165 You don't want to sync all the data. 120 00:06:51,425 --> 00:06:53,765 We have this sort of model of partial replication. 121 00:06:54,195 --> 00:06:58,469 And basically what we're aiming to do with the sync engine is just make that, as 122 00:06:58,479 --> 00:07:01,539 simple to use as bulletproof as possible. 123 00:07:01,959 --> 00:07:06,164 And we're making it with standard web technologies that make it easy 124 00:07:06,164 --> 00:07:09,404 to use with your existing systems and with your existing stack. 125 00:07:09,684 --> 00:07:13,954 And so we went in a way from this sort of quite ambitious, tightly integrated 126 00:07:13,954 --> 00:07:18,804 end to end local-first software platform to now building more like composable 127 00:07:18,834 --> 00:07:22,844 tools that can be part of a local-first stack that you would assemble yourself 128 00:07:22,844 --> 00:07:25,304 as a developer, that's designed to be. 129 00:07:25,464 --> 00:07:28,014 Easier to adopt for production applications that work 130 00:07:28,014 --> 00:07:29,194 with your existing code. 131 00:07:29,804 --> 00:07:31,014 That makes a lot of sense. 132 00:07:31,044 --> 00:07:34,644 And that definitely resonates with me personally as well, since maybe, 133 00:07:34,704 --> 00:07:40,130 as you know, before I founded Prisma, Prisma actually came as a pivot out of 134 00:07:40,130 --> 00:07:44,690 like a focusing effort from a previous product that was called GraphQL, 135 00:07:44,700 --> 00:07:48,720 which was meant as a more ambitious next generation backend as a service. 136 00:07:48,740 --> 00:07:52,765 Back then there was like Firebase and Parse and so we wanted to build the 137 00:07:52,765 --> 00:07:57,802 next generation of that, but what we found back then in 2016, that, while 138 00:07:57,802 --> 00:08:02,012 we've been making a lot of progress towards that very ambitious, holistic 139 00:08:02,022 --> 00:08:06,499 vision, we had to basically oil, like, multiple oceans all at the same time. 140 00:08:06,539 --> 00:08:10,814 And that takes a lot of time to fully get to all the different 141 00:08:10,814 --> 00:08:12,384 ambitious things that we wanted to. 142 00:08:12,454 --> 00:08:16,384 So the only way forward for us where we felt like, okay, we can actually 143 00:08:16,384 --> 00:08:21,324 serve the kind of use cases that we want to serve in a realistic timeline 144 00:08:21,344 --> 00:08:26,064 was to focus on a particular problem, which is what Prisma eventually became. 145 00:08:26,464 --> 00:08:30,874 And by focusing just on the database tooling part and leaving the other 146 00:08:31,044 --> 00:08:32,384 back-endy things to other people. 147 00:08:32,384 --> 00:08:36,094 And it sounds like what you've been going through with Electric is a very comparable 148 00:08:36,334 --> 00:08:41,474 exercise, like focusing exercise to trying to, from a starting point of 149 00:08:41,474 --> 00:08:46,654 like, let's build the most ambitious, the best local-first stack, like end to 150 00:08:46,654 --> 00:08:52,264 end by focusing more on like, okay, what we figured out where our expertise is, 151 00:08:52,274 --> 00:08:58,577 is around Postgres, is about, existing applications wanting to adopt local-first 152 00:08:58,602 --> 00:09:01,202 ideas, syncing approaches, et cetera. 153 00:09:01,472 --> 00:09:04,342 And that is what now led to the new version of Electric. 154 00:09:04,569 --> 00:09:05,949 did I summarize that correctly? 155 00:09:06,499 --> 00:09:07,289 Yeah, exactly. 156 00:09:07,319 --> 00:09:07,529 Right. 157 00:09:07,539 --> 00:09:09,319 It sounds like a very similar journey. 158 00:09:09,319 --> 00:09:13,712 And I think it's interesting as well that as you focus in and you learn 159 00:09:13,712 --> 00:09:17,605 more about a problem space, you both discover in a way, more of the 160 00:09:17,605 --> 00:09:19,845 complexity in the sort of aspects of it. 161 00:09:19,855 --> 00:09:23,655 So you realize there's actually more challenges to solve in a smaller sort 162 00:09:23,725 --> 00:09:25,815 of part of it or a smaller scope. 163 00:09:26,169 --> 00:09:29,979 And also it's interesting that I think for instance, when we started the 164 00:09:29,979 --> 00:09:32,559 project, I would have thought coming into this as a software developer, 165 00:09:32,559 --> 00:09:34,770 I'd go, Is a read path sync solved? 166 00:09:34,770 --> 00:09:37,090 I'd be like, well, there's quite a lot of read path kind of sync stuff. 167 00:09:37,110 --> 00:09:38,090 You can kind of do this. 168 00:09:38,090 --> 00:09:42,090 There's various real time solutions, but actually as you dig into it, you find 169 00:09:42,090 --> 00:09:45,390 that there's a whole bunch of weaknesses of those solutions and they're actually 170 00:09:45,620 --> 00:09:48,810 hard to adopt or they have silos or they can't handle the data throughput. 171 00:09:48,940 --> 00:09:53,374 And so you realize that actually you don't necessarily need to bite 172 00:09:53,374 --> 00:09:57,794 off all of the more ambitious scope because actually you can deliver 173 00:09:57,794 --> 00:09:59,877 value by doing something simpler. 174 00:10:00,227 --> 00:10:03,560 And I think also for me personally, learning about stewarding this 175 00:10:03,560 --> 00:10:08,110 type of product, understanding that you can build out still towards 176 00:10:08,110 --> 00:10:09,620 that more ambitious objective. 177 00:10:09,620 --> 00:10:12,680 So in the long run, you know, we want to sort of build back a whole bunch 178 00:10:12,690 --> 00:10:14,410 of capabilities into this platform. 179 00:10:14,742 --> 00:10:17,902 probably a sort of loosely coupled kind of composable tools. 180 00:10:18,202 --> 00:10:21,532 So you mentioned the term read path syncing. 181 00:10:21,752 --> 00:10:24,182 Can you elaborate a little bit what that means? 182 00:10:24,252 --> 00:10:26,872 So let's say I have an existing application. 183 00:10:26,882 --> 00:10:29,472 Let's say I've built an API layer at some point. 184 00:10:29,492 --> 00:10:33,732 I have a React front end and I have all of my data sitting in Postgres. 185 00:10:34,147 --> 00:10:38,347 I've been inspired by products such as Linear, et cetera, who seem to 186 00:10:38,347 --> 00:10:40,247 wield a superpower called syncing. 187 00:10:40,497 --> 00:10:45,297 And now I found ElectricSQL, which seems to connect the ingredients 188 00:10:45,647 --> 00:10:50,597 that I already have, such as Postgres and a front end with my 189 00:10:50,597 --> 00:10:52,707 desirable approach, which is syncing. 190 00:10:52,937 --> 00:10:55,707 So how does Electric fit into that? 191 00:10:55,737 --> 00:10:57,517 And what do you mean by. 192 00:10:57,537 --> 00:10:59,054 Read path syncing. 193 00:10:59,250 --> 00:10:59,670 Yeah. 194 00:10:59,670 --> 00:11:02,400 I mean, the sort of read path and write path when it comes to 195 00:11:02,400 --> 00:11:06,700 sync, the read path is syncing data, like onto the local device. 196 00:11:06,710 --> 00:11:09,290 So it's a bit like kind of data fetching from the server. 197 00:11:09,650 --> 00:11:12,780 And then the write path would be when like a user makes a write, and then 198 00:11:12,780 --> 00:11:16,630 you want to sync that data typically back to the cloud so that's sort 199 00:11:16,630 --> 00:11:18,510 of how we talk about them there. 200 00:11:19,600 --> 00:11:25,449 I think there's something unique about local-first software compared to 201 00:11:25,509 --> 00:11:30,909 more sort of traditional web service systems where you explicitly have 202 00:11:31,159 --> 00:11:33,969 a local copy of the data on device. 203 00:11:34,269 --> 00:11:39,369 And one of the challenges with that is because of course you can just like load 204 00:11:39,369 --> 00:11:44,454 some data from the server and keep it in a cache, but if you do that Then you 205 00:11:44,454 --> 00:11:48,574 immediately actually lose, any information about whether that data is stale. 206 00:11:49,084 --> 00:11:54,410 So say a user goes to a route on your application and then clicks 207 00:11:54,420 --> 00:11:57,450 to go to another route and then comes back to the original one. 208 00:11:57,750 --> 00:12:01,410 So to load that original route, say you did a data fetch, but 209 00:12:01,410 --> 00:12:02,660 now you've navigated back to it. 210 00:12:02,660 --> 00:12:04,750 Can you display that data? 211 00:12:04,770 --> 00:12:07,380 Can you render the route or is the data stale? 212 00:12:08,180 --> 00:12:12,130 And so you have this sort of thing where I don't really know, and you tend to sort 213 00:12:12,130 --> 00:12:15,610 of build systems with like REST APIs and data fetching where you might show the 214 00:12:15,610 --> 00:12:17,757 data and go and try and fetch new data. 215 00:12:17,950 --> 00:12:23,250 but in a way it's that problem of you want the data locally so that your application 216 00:12:23,250 --> 00:12:26,410 code can just talk to it locally and you're not having to code across the 217 00:12:26,420 --> 00:12:27,930 network with local-first software. 218 00:12:28,304 --> 00:12:32,574 But that means that you need a solution to keep the data that is local fresh. 219 00:12:33,094 --> 00:12:34,704 Like you don't want stale data. 220 00:12:35,414 --> 00:12:38,004 And if you build a sort of ad-hoc system. 221 00:12:38,234 --> 00:12:41,410 As we've all done across like many generations of software applications, 222 00:12:41,467 --> 00:12:44,384 it's one of these things where you always end up kind of building some sort 223 00:12:44,384 --> 00:12:46,034 of system to keep the data up to date. 224 00:12:46,417 --> 00:12:49,827 But what you really want is a kind of properly engineered system 225 00:12:49,867 --> 00:12:51,637 that does it systemically for you. 226 00:12:51,997 --> 00:12:55,937 It is really a sort of an aspect of your applications architecture that kind of 227 00:12:56,057 --> 00:12:58,437 can be abstracted away by a sync engine. 228 00:12:58,917 --> 00:13:02,487 And so for us, for this focusing on the read path sync is about saying, 229 00:13:02,487 --> 00:13:06,327 okay, what data should be on the device and let's just keep it. 230 00:13:06,459 --> 00:13:07,219 fresh for you. 231 00:13:07,739 --> 00:13:11,229 And then with the write path, one of the things that we learned through 232 00:13:11,229 --> 00:13:17,019 the project is that there are a lot of valid patterns for handling how, when 233 00:13:17,019 --> 00:13:21,549 you do local writes on the device, how you would get those back to the cloud. 234 00:13:22,894 --> 00:13:26,184 You can do through the database sync, you can do optimistic writes. 235 00:13:26,204 --> 00:13:30,454 You could be happy with online writes and you have different models of 236 00:13:30,454 --> 00:13:32,160 like, can your writes be rejected? 237 00:13:32,350 --> 00:13:34,040 Are they local writes with finality? 238 00:13:34,040 --> 00:13:37,080 Or do you have a server authoritative system where when the write 239 00:13:37,100 --> 00:13:39,730 somehow syncs, it can be rejected and how do you handle that? 240 00:13:40,180 --> 00:13:43,230 And so there's actually a lot of different patterns for those writes, 241 00:13:43,250 --> 00:13:48,310 which are often relatively simple because different applications can 242 00:13:48,310 --> 00:13:51,020 be happy with certain trade offs and you could pick a model like. 243 00:13:51,455 --> 00:13:51,725 Okay. 244 00:13:51,725 --> 00:13:55,865 I'm going to show some optimistic state and make a request to an API server. 245 00:13:56,515 --> 00:13:57,405 And it's fine. 246 00:13:57,435 --> 00:14:00,952 And you get a kind of, you get a local-first, experience with just a 247 00:14:00,952 --> 00:14:03,772 sort of simple model that says, okay, if the write is rejected when it 248 00:14:03,772 --> 00:14:07,299 syncs, then, I'll just sort of roll it back and the user loses that work. 249 00:14:07,299 --> 00:14:08,979 And for many applications, that's fine. 250 00:14:09,269 --> 00:14:13,539 For other applications, you might have a much more complex conflict resolution or 251 00:14:13,605 --> 00:14:16,935 you're trying not to lose local writes and there's different collaborative workloads. 252 00:14:16,935 --> 00:14:17,355 And so. 253 00:14:17,552 --> 00:14:21,802 Building a generic system that can give you a write path that gives you 254 00:14:21,802 --> 00:14:25,642 the best developer experience and user experience for all of those variety of 255 00:14:25,642 --> 00:14:28,892 scenarios is very, very hard, whereas building it on an application by 256 00:14:28,892 --> 00:14:32,952 application basis on the write path is actually often fairly straightforward. 257 00:14:32,952 --> 00:14:36,722 It can be like post your API and use the React use optimistic hook. 258 00:14:37,252 --> 00:14:40,949 And so, with building local-first applications that have both read and 259 00:14:40,949 --> 00:14:45,439 write path with Electric, the idea is that we do this core read path 260 00:14:45,449 --> 00:14:49,059 with partial replication, but then as you're building your application, you 261 00:14:49,059 --> 00:14:53,289 can choose out of a variety, whichever pattern fits your, what you need the 262 00:14:53,289 --> 00:14:56,979 most for sort of how you would choose to get the writes back into the server. 263 00:14:57,239 --> 00:14:58,379 That makes a lot of sense. 264 00:14:58,459 --> 00:15:00,949 So basically the more general purpose. 265 00:15:01,399 --> 00:15:05,459 building block that can be used across a wide range of different applications. 266 00:15:05,459 --> 00:15:09,639 It's actually how you read data, how you distribute the data that you 267 00:15:09,639 --> 00:15:13,359 want to have locally available in your applications that would kind of 268 00:15:13,379 --> 00:15:16,999 replace the API get requests before. 269 00:15:17,239 --> 00:15:21,479 But now what needs to happen in those Put, post, delete requests, 270 00:15:21,719 --> 00:15:24,059 this is where it depends a lot more. 271 00:15:24,079 --> 00:15:28,299 And this is where you basically, what you're arguing is there are different 272 00:15:28,299 --> 00:15:32,169 sort of write patterns that heavily depends on the kind of application. 273 00:15:32,399 --> 00:15:34,479 So that is where you're kind of leaning out. 274 00:15:34,739 --> 00:15:39,284 And previously with Electric, you tried to provide the silver bullet there. 275 00:15:39,424 --> 00:15:43,294 But actually, it's really hard, maybe impossible to find the silver 276 00:15:43,294 --> 00:15:45,624 bullet that applies to all use cases. 277 00:15:45,624 --> 00:15:50,054 However, for the read path, it is very possible to provide a great building 278 00:15:50,054 --> 00:15:51,934 block that works for many use cases. 279 00:15:52,434 --> 00:15:56,784 So, can you provide a bit of a better spectrum of the different write 280 00:15:56,784 --> 00:15:58,584 patterns that you've seen so far? 281 00:15:58,804 --> 00:16:02,164 Maybe map them to canonical applications? 282 00:16:02,404 --> 00:16:04,094 that illustrate those use cases. 283 00:16:04,124 --> 00:16:08,414 And maybe if you know, maybe you can also compare analogies to something 284 00:16:08,424 --> 00:16:12,044 like Automerge, et cetera, which sort of write patterns that would 285 00:16:12,054 --> 00:16:14,384 be a good fit for, or not as much. 286 00:16:14,794 --> 00:16:15,044 Yeah. 287 00:16:15,267 --> 00:16:19,437 So I think the simplest pattern for writes with an application would be to 288 00:16:19,437 --> 00:16:24,152 just, for instance, send a write to a server and require you to be online. 289 00:16:24,612 --> 00:16:27,442 So, because there's many applications that are happy, for instance, with read 290 00:16:27,442 --> 00:16:31,420 only, like there's a lot of people who are building, data analytics applications, 291 00:16:31,450 --> 00:16:33,700 data visualization, dashboards, et cetera. 292 00:16:33,700 --> 00:16:37,380 And so if you have a sort of read heavy application, then in some cases 293 00:16:37,380 --> 00:16:40,440 it may just be a perfectly valid trade off, not to really deal with the 294 00:16:40,440 --> 00:16:42,770 complexity of say offline writes at all. 295 00:16:42,950 --> 00:16:46,350 But you still have a lot of benefits by having local data on device for the read 296 00:16:46,370 --> 00:16:50,590 path, because all the way you can kind of explore the application and the data is 297 00:16:50,770 --> 00:16:56,060 all just instant and local and resilient, then the sort of simplest pattern to 298 00:16:56,270 --> 00:16:59,047 layer on, support for offline writes. 299 00:16:59,797 --> 00:17:03,097 On top of that as a sort of starting point where imagine that you have like a 300 00:17:03,097 --> 00:17:08,124 standard REST API and you're just doing put and post requests to it as normal is 301 00:17:08,124 --> 00:17:10,354 to add this concept of optimistic state. 302 00:17:10,434 --> 00:17:14,254 So optimistic state is just basically you're saying, okay, I'm going to go and 303 00:17:14,264 --> 00:17:16,524 try and send this write to the API server. 304 00:17:16,774 --> 00:17:20,954 And whilst I do so, I'm going to be optimistic and imagine that 305 00:17:20,954 --> 00:17:22,314 that write is going to succeed. 306 00:17:22,334 --> 00:17:25,604 And in two seconds later, it's going to sync back into the state that I have here. 307 00:17:25,954 --> 00:17:30,169 But in the meantime, I'm going to Add this bit of local optimistic state to 308 00:17:30,169 --> 00:17:34,819 display it immediately to the user, and because in most cases that of happy path 309 00:17:34,829 --> 00:17:39,319 is what happens, then you end up with what just feels like a perfect local-first 310 00:17:39,339 --> 00:17:43,319 experience because it's an instantly displayed local write, and that sort 311 00:17:43,319 --> 00:17:45,159 of data is resolved in the background. 312 00:17:45,392 --> 00:17:49,719 Now, You know, immediately with that, you do then just introduce like a layer 313 00:17:49,719 --> 00:17:53,809 of complexity with like, well, what happens when the write is rejected? 314 00:17:54,349 --> 00:18:00,799 And so you have both the challenge of, for instance, say you stacked up three writes. 315 00:18:01,739 --> 00:18:02,929 Did they depend on each other? 316 00:18:03,309 --> 00:18:05,949 So if one of them is rejected, should you reject all of them? 317 00:18:06,075 --> 00:18:09,085 and different applications and different parts of the application would have 318 00:18:09,085 --> 00:18:10,060 different answers to that question. 319 00:18:11,160 --> 00:18:14,420 In some cases, like it's very simple to just go, if there's any problem with 320 00:18:14,420 --> 00:18:16,100 this optimistic state, just wipe it. 321 00:18:16,640 --> 00:18:20,340 And for instance, like the React use optimistic hook, like its approach is just 322 00:18:20,340 --> 00:18:22,620 like, it waits for a promise to resolve. 323 00:18:22,680 --> 00:18:25,180 And when the promise resolves, it wipes the optimistic state. 324 00:18:25,630 --> 00:18:28,800 And so it's very much just like, if anything happens at all, 325 00:18:28,800 --> 00:18:30,780 it's like, And so it's only. 326 00:18:30,780 --> 00:18:35,380 Interestingly enough, there's also a lot of people coming from React Query and so 327 00:18:35,380 --> 00:18:40,280 on, from those sort of more traditional front end state management things. 328 00:18:40,490 --> 00:18:44,740 and that brings them to local-first in the first place, because they're like 329 00:18:44,980 --> 00:18:49,610 layering optimistic, one optimistic state handler on top of the next one. 330 00:18:49,840 --> 00:18:53,520 And if there's a little flaw inside of there, everything collapses 331 00:18:53,530 --> 00:18:57,200 since you don't really know have principled way to reason about things. 332 00:18:57,490 --> 00:18:58,840 So that makes a lot of sense. 333 00:18:59,710 --> 00:19:00,360 Exactly right. 334 00:19:00,400 --> 00:19:05,280 And so like a framework like TanStack, for instance, with TanStack query, it has like 335 00:19:05,320 --> 00:19:10,200 slightly more sophisticated optimistic state primitives than just say the kind 336 00:19:10,200 --> 00:19:12,260 of a primitive use of optimistic hook. 337 00:19:12,537 --> 00:19:15,857 And one of the thing, one of the challenges that you have is that for 338 00:19:15,857 --> 00:19:20,354 say, a simple approach to, to just using optimistic state to display an immediate 339 00:19:20,354 --> 00:19:24,264 write is like, is that optimistic state global to your application? 340 00:19:24,314 --> 00:19:25,604 Shared between components? 341 00:19:25,624 --> 00:19:27,334 Is it scoped within the component? 342 00:19:27,884 --> 00:19:30,924 And so, as you say, like there's an approach where you could come along 343 00:19:30,924 --> 00:19:33,844 and say, okay, I've got three or four different components and so far I've 344 00:19:33,844 --> 00:19:36,824 just been able to sort of render the optimistic state within the component. 345 00:19:37,214 --> 00:19:40,154 But now I've got two components that are actually displaying the same information. 346 00:19:40,519 --> 00:19:42,269 And suddenly I've got like stale data. 347 00:19:42,349 --> 00:19:45,845 It's like the old days of manual DOM manipulation and you forgot 348 00:19:45,865 --> 00:19:47,235 to update a state variable. 349 00:19:47,685 --> 00:19:48,015 And so. 350 00:19:48,565 --> 00:19:53,225 Yeah, in a way that's where you come to a more proper local-first solution 351 00:19:53,235 --> 00:19:58,365 where your optimistic state would be, stored in some sort of shared store. 352 00:19:58,615 --> 00:20:01,515 So it could just be like a JavaScript object store, or it 353 00:20:01,515 --> 00:20:02,895 could be an embedded database. 354 00:20:03,415 --> 00:20:06,975 And so you get a slightly more sophisticated models of 355 00:20:07,015 --> 00:20:08,415 managing optimistic state. 356 00:20:08,915 --> 00:20:11,695 And the great thing is there are, like TanStack Query and others, there's 357 00:20:11,695 --> 00:20:14,665 like, there's a bunch of existing client side frameworks that can handle 358 00:20:14,665 --> 00:20:16,215 that kind of management for you. 359 00:20:17,135 --> 00:20:21,475 Once you go, for instance, like to an embedded database for the state. 360 00:20:21,485 --> 00:20:27,810 So one of the kind of really nice, points in the design space for this is to have a 361 00:20:27,820 --> 00:20:32,210 model where you sync data onto the device and you treat that data as immutable. 362 00:20:32,825 --> 00:20:37,565 And then you can have, for instance, so, so say, for instance, you're syncing a 363 00:20:37,565 --> 00:20:41,995 database table, say it's like a log viewer application, and you're just syncing the 364 00:20:41,995 --> 00:20:43,945 logs in, and it goes into a logs table. 365 00:20:44,365 --> 00:20:47,325 Now, say the user can interact with the logs and delete them, 366 00:20:47,325 --> 00:20:48,765 or change the categorization. 367 00:20:49,095 --> 00:20:52,295 And so you can have a shadow logs table, which is where you would 368 00:20:52,405 --> 00:20:54,465 save the local optimistic state. 369 00:20:54,945 --> 00:20:55,455 And then. 370 00:20:55,675 --> 00:20:59,415 You can do a bunch of different techniques to, for example, create a view or a live 371 00:20:59,415 --> 00:21:02,095 query where you combine those two on read. 372 00:21:02,095 --> 00:21:05,095 So the application just sort of feels like it's interacting with the table, 373 00:21:05,395 --> 00:21:09,235 but actually it's split in the storage layer into a mutable table for the sync 374 00:21:09,235 --> 00:21:11,835 state and a kind of local mutable table. 375 00:21:12,175 --> 00:21:15,305 And the great thing about that is you can have persistence for the, both the 376 00:21:15,305 --> 00:21:18,462 sync state and the, local mutable state. 377 00:21:18,502 --> 00:21:19,812 And of course it can be shared. 378 00:21:19,812 --> 00:21:22,302 So you can have multiple components, which are all sorts of just going 379 00:21:22,302 --> 00:21:24,097 through that unified data store. 380 00:21:24,363 --> 00:21:27,473 and there's some nice stuff that you can do in SQL world, for instance, to use 381 00:21:27,473 --> 00:21:29,267 like instead of triggers to combine it. 382 00:21:29,267 --> 00:21:32,033 So it just feels like you're working with a single table. 383 00:21:32,283 --> 00:21:35,813 Now it's a little bit additional complexity on something like defining 384 00:21:35,813 --> 00:21:39,573 a client side data model, but what it gives you is it gives you a 385 00:21:39,573 --> 00:21:42,133 very solid model to reason about. 386 00:21:42,163 --> 00:21:46,038 So like, You can go, okay, basically the sync state is always golden. 387 00:21:46,078 --> 00:21:46,798 It's immutable. 388 00:21:46,808 --> 00:21:48,468 Whenever it syncs in, it's correct. 389 00:21:48,818 --> 00:21:53,518 If I have a problem with this local state, that's just, that's like mutable stuff. 390 00:21:53,518 --> 00:21:57,845 Worst case, I can get rid of it, or I can develop more sophisticated strategies for 391 00:21:57,845 --> 00:22:00,445 dealing with rollbacks and edge cases. 392 00:22:00,705 --> 00:22:04,504 So it in a way it can give you a nice developer experience. 393 00:22:04,658 --> 00:22:08,618 with that model, you could choose then whether your writes are, whether you're 394 00:22:08,618 --> 00:22:11,948 writing to the database, detecting changes, and then sending those to 395 00:22:11,948 --> 00:22:15,978 some sort of like replication ingest point, or whether you're still just 396 00:22:15,978 --> 00:22:21,268 basically talking to an API and writing the local optimistic state separately. 397 00:22:21,658 --> 00:22:24,268 So, so at that point you can have, again, you can have, you have this 398 00:22:24,268 --> 00:22:27,421 fundamental model of like, Are you writing directly to the database and 399 00:22:27,421 --> 00:22:28,901 all the syncing happens magically? 400 00:22:29,311 --> 00:22:33,661 Or are you just using that database as a sort of unified, local optimistic store? 401 00:22:34,251 --> 00:22:36,851 So this is the sort of type of like progression of patterns. 402 00:22:36,851 --> 00:22:41,921 And once you start to go through something where you would, for instance, have a 403 00:22:42,111 --> 00:22:46,821 synced state that is mutable, or you are writing directly to the database, 404 00:22:46,881 --> 00:22:49,461 that's really where you start to get a little bit more into the world of like 405 00:22:49,531 --> 00:22:54,921 convergence logic and kind of merge logic and CRDTs and sort of what's commonly 406 00:22:54,921 --> 00:22:57,011 understood as proper local-first systems. 407 00:22:57,971 --> 00:22:59,931 And I think that's the point where almost the complexity of those 408 00:22:59,931 --> 00:23:01,401 systems does become very real. 409 00:23:01,561 --> 00:23:04,931 Like, as you well know, from building LiveStore and as we see from the 410 00:23:04,931 --> 00:23:08,318 kind of, quality of libraries like AutoMerge, Yjs, et cetera. 411 00:23:08,545 --> 00:23:11,985 so that's probably where as a developer, it makes sense to reach for a framework. 412 00:23:12,405 --> 00:23:15,375 And you certainly could reach for a framework for that sort of like. 413 00:23:15,710 --> 00:23:21,250 Combine on read, sync, sync into a mutable kind of persist local mutable state. 414 00:23:21,930 --> 00:23:25,146 But what we find is that it is actually if you want to, it's actually 415 00:23:25,146 --> 00:23:28,476 relatively straightforward to develop yourself, you can reason about it 416 00:23:28,476 --> 00:23:32,186 fairly simply, and so it's not too much extra work to just basically go 417 00:23:32,186 --> 00:23:36,366 as long as you've got that read sync primitive, you can build like a kind of 418 00:23:36,366 --> 00:23:41,716 proper locally persistent, consistent local-first app yourself, basically. 419 00:23:42,246 --> 00:23:44,623 Just using fairly standard front end primitives. 420 00:23:44,956 --> 00:23:45,396 Right. 421 00:23:45,566 --> 00:23:45,886 Okay. 422 00:23:46,226 --> 00:23:50,316 Maybe sharing a few reflections on this, since I like the way how you, 423 00:23:50,460 --> 00:23:54,186 portrayed this sort of spectrum of this different kind of write patterns. 424 00:23:54,450 --> 00:23:58,610 in a interview that I did with Matthew Weidner, I learned a lot there 425 00:23:58,610 --> 00:24:02,825 about the way, how he thinks about different categorizations of like state 426 00:24:02,825 --> 00:24:06,935 management, and particularly when it comes to distributed synchronization. 427 00:24:07,138 --> 00:24:12,098 and I think one pattern that got clear there was that there's either you're 428 00:24:12,098 --> 00:24:16,988 working directly manipulating the state, which is what like Automerge, et 429 00:24:16,988 --> 00:24:21,638 cetera, are de facto doing for how you as a developer interact with the state. 430 00:24:21,638 --> 00:24:25,008 So you have like a document and you manipulate it directly. 431 00:24:25,258 --> 00:24:30,983 You could also apply the same logic of like, you have a Database table, for 432 00:24:30,983 --> 00:24:35,013 example, that's how CR SQLite works, where you have a SQLite table and you 433 00:24:35,113 --> 00:24:41,053 manipulate a row directly and that is being synchronized as the state and 434 00:24:41,053 --> 00:24:46,043 you're ideally modeling this with a way where the state itself converges and 435 00:24:46,063 --> 00:24:48,773 through some mechanisms, typically CRDTs. 436 00:24:49,448 --> 00:24:53,608 But then there's another approach, which might feel a little bit more 437 00:24:53,638 --> 00:24:58,308 work, but it can actually be concealed quite nicely by systems, for example, 438 00:24:58,308 --> 00:25:02,458 like LiveStore, in this case, unbiased, and where you basically separate 439 00:25:02,468 --> 00:25:04,908 out the reads from the writes. 440 00:25:05,403 --> 00:25:10,270 And often enough, you can actually fully, re compute your 441 00:25:10,270 --> 00:25:12,170 read model from the write model. 442 00:25:12,480 --> 00:25:16,500 So, if you then basically express everything that has happened, that 443 00:25:16,500 --> 00:25:20,290 has meaningfully happened for your application as a log of events. 444 00:25:20,510 --> 00:25:24,886 Then you can often kind of like how Redux used to work or still works, you can 445 00:25:24,886 --> 00:25:29,506 fully recompute your view, your read model from all the writes that have happened. 446 00:25:29,736 --> 00:25:33,526 And I think that would work actually really, really well together in tandem 447 00:25:33,726 --> 00:25:39,636 with Electric, where if you're replicating what has happened in your Postgres 448 00:25:39,696 --> 00:25:45,193 database as like a log of historic events, then you can actually fully, recreate 449 00:25:45,375 --> 00:25:49,955 Whatever derived state you're interested in and what is really interesting about 450 00:25:49,975 --> 00:25:54,863 that approach, but that particular write pattern is that it's a lot easier to 451 00:25:54,893 --> 00:25:57,043 model that and reason about that locally. 452 00:25:57,293 --> 00:26:00,673 Did you say like, Hey, I got those events from the server, those 453 00:26:00,673 --> 00:26:03,250 events, I am applying optimistically. 454 00:26:03,580 --> 00:26:09,640 You can encode sort of even a causal order that doesn't really, If someone 455 00:26:09,655 --> 00:26:13,755 is like confused about what does causal order mean, don't worry about it. 456 00:26:13,765 --> 00:26:18,215 Like you can probably at the beginning, keep it simple, but once you layer 457 00:26:18,215 --> 00:26:22,615 on like more and more dependent, optimistic state transitions, this is 458 00:26:22,615 --> 00:26:24,995 where you want to have the information. 459 00:26:25,195 --> 00:26:25,615 Okay. 460 00:26:25,835 --> 00:26:29,195 If I'm doing that, and then the other thing depends on that, that's basically a 461 00:26:29,195 --> 00:26:32,545 causal order and modeling that as events. 462 00:26:32,835 --> 00:26:38,245 I think is a lot simpler and is a way to, to deal with that monstrosity of like, 463 00:26:38,442 --> 00:26:41,272 losing control over your optimistic state. 464 00:26:41,342 --> 00:26:44,782 Since I think one thing that's, that makes optimistic state management 465 00:26:44,792 --> 00:26:49,732 even more tricky is that, like, how are things dependent on each other? 466 00:26:50,122 --> 00:26:53,797 And then also like, when is it assumed to be good. 467 00:26:54,017 --> 00:26:57,257 I think in a world where you use Electric, once you're from the 468 00:26:57,257 --> 00:27:01,047 Electrics server, you've got sort of confirmation, like, Hey, those 469 00:27:01,057 --> 00:27:02,777 things have now happened for real. 470 00:27:02,777 --> 00:27:03,637 You can trust it. 471 00:27:04,000 --> 00:27:07,638 but there's like some latency in between, and the latency might be 472 00:27:07,638 --> 00:27:09,378 increased by many, many factors. 473 00:27:10,013 --> 00:27:15,343 One way could be that you just, you are on a like slow connection or the server 474 00:27:15,343 --> 00:27:19,793 is particularly far away from you and might take a hundred milliseconds, but 475 00:27:19,803 --> 00:27:25,343 another one might be your have a spotty connection and like packages get lost and 476 00:27:25,353 --> 00:27:30,603 it takes a lot longer or you're offline and being offline is just like a form 477 00:27:30,603 --> 00:27:36,403 of like a very high latency form and so all of that, like if you're offline, 478 00:27:36,433 --> 00:27:40,758 if it takes a long long time, and maybe you close your laptop, you reopen it. 479 00:27:41,108 --> 00:27:43,548 Is the optimistic state still there? 480 00:27:43,578 --> 00:27:45,398 Is it actually locally persisted? 481 00:27:45,688 --> 00:27:49,118 So there are many, many more layers that make that more tricky. 482 00:27:49,818 --> 00:27:54,303 But I like the way how you're like, how you split this up into the read 483 00:27:54,363 --> 00:27:56,193 concerns and the write concerns. 484 00:27:56,833 --> 00:28:00,713 And I think this way, it's also very easy to get started with new 485 00:28:00,713 --> 00:28:05,373 apps that might be more read heavy and are based on existing data. 486 00:28:05,733 --> 00:28:09,833 I think this is a very attractive trade off that you say like, Hey, with 487 00:28:09,833 --> 00:28:14,323 that, I can just sink in my existing data and then step by step, depending 488 00:28:14,323 --> 00:28:16,063 on what I need, if I need it at all. 489 00:28:16,188 --> 00:28:19,378 Many apps don't even need to do writes at all, and then you 490 00:28:19,378 --> 00:28:20,848 can just get started easily. 491 00:28:21,724 --> 00:28:25,674 Yeah, I think, I mean, that's explicitly a design goal for us is like, yeah, 492 00:28:25,674 --> 00:28:29,104 if you start off with an existing application and maybe it's using REST 493 00:28:29,104 --> 00:28:32,734 APIs or GraphQL, it's like, well, what do you do to start to move that 494 00:28:32,734 --> 00:28:34,424 towards a local-first architecture? 495 00:28:34,844 --> 00:28:37,624 And exactly, you could just go, okay, well, just, let's just leave the way 496 00:28:37,624 --> 00:28:39,004 that we do writes the same as it is. 497 00:28:39,004 --> 00:28:41,874 And let's move to this model of like syncing in the data 498 00:28:41,934 --> 00:28:43,114 instead of fetching the data. 499 00:28:43,114 --> 00:28:44,704 And that can just be a first step. 500 00:28:45,504 --> 00:28:48,927 And I think, I mean, Across all of these techniques for writes, there 501 00:28:48,927 --> 00:28:52,501 is just something fundamental about keeping the history or the log 502 00:28:52,911 --> 00:28:58,141 around as long as you need it, and then somehow materializing values. 503 00:28:58,401 --> 00:29:01,647 So sort of internally, this is what a CRDT does, right? 504 00:29:01,687 --> 00:29:05,704 it's clever and has a sort of lattice structure for the history, but basically 505 00:29:05,704 --> 00:29:08,687 it keeps the information and allows you to materialize out a value. 506 00:29:09,126 --> 00:29:11,632 if you just have like an event log of writes. 507 00:29:11,642 --> 00:29:14,162 So as you were saying with, with LiveStore, when you have like a 508 00:29:14,162 --> 00:29:17,392 record of all the write operations, you can just process that log. 509 00:29:17,676 --> 00:29:21,336 so I think, you know, you can do it sort of within a data type. 510 00:29:21,836 --> 00:29:25,226 And I think that fits as well for greenfield application where you're trying 511 00:29:25,226 --> 00:29:29,559 to craft, kind of real time or kind of collaboration and concurrency semantics, 512 00:29:29,839 --> 00:29:32,539 but like from our side of coming at it, from the point of saying, right, when 513 00:29:32,539 --> 00:29:35,529 you've got applications that build on Postgres, you already have a data model. 514 00:29:35,774 --> 00:29:39,641 You just sort of layer the same kind of history approach on top by like, keeping 515 00:29:39,641 --> 00:29:44,017 a record of the local writes until you of sure you can compact them and actually 516 00:29:44,017 --> 00:29:49,407 that same principle is exactly how the read path sync works with Electric. 517 00:29:49,614 --> 00:29:56,556 So Postgres logical replication, it just basically, it emits a stream, it's like 518 00:29:56,746 --> 00:30:00,496 transactions that contain write operations and it's basically inserts, updates, 519 00:30:00,496 --> 00:30:01,916 and deletes with a bit of metadata. 520 00:30:02,486 --> 00:30:06,027 And so we end up consuming that and basically writing 521 00:30:06,087 --> 00:30:07,577 out what we call shape logs. 522 00:30:07,607 --> 00:30:10,764 So we have a primitive called a shape, which is how we control the partial 523 00:30:10,764 --> 00:30:14,724 replication, like which data goes to which client and a client can define multiple 524 00:30:14,724 --> 00:30:16,313 shapes, and then you stream them out. 525 00:30:16,993 --> 00:30:21,173 But that shape log comes through our replication protocol as just that 526 00:30:21,173 --> 00:30:23,193 stream of logical update operations. 527 00:30:23,633 --> 00:30:28,001 And so in the client, you can just, you can materialize the data immediately. 528 00:30:28,371 --> 00:30:31,991 So like we provide, for instance, a shape stream primitive in a JavaScript client 529 00:30:32,181 --> 00:30:34,101 that just omits the series of events. 530 00:30:34,291 --> 00:30:37,371 And then we have a shape, which we'll just take care of materializing that 531 00:30:37,628 --> 00:30:39,398 into a kind of map value for you. 532 00:30:39,644 --> 00:30:42,644 but you could do what you want, whatever you wanted with that stream of events. 533 00:30:42,654 --> 00:30:46,094 So if you found that you wanted to keep around a certain history of the 534 00:30:46,094 --> 00:30:49,618 log in order to be able to reconcile some sort of causal dependencies, 535 00:30:49,618 --> 00:30:51,038 that's just totally up to you. 536 00:30:51,088 --> 00:30:54,448 And so, yeah, it's quite interesting that it's almost just the same approach, 537 00:30:54,448 --> 00:30:58,318 which is the general sort of principle for handling concurrency on the 538 00:30:58,318 --> 00:31:02,008 write path is also just exactly what we've ended up consolidating down on 539 00:31:02,008 --> 00:31:04,438 exposing through the read path stream. 540 00:31:04,874 --> 00:31:05,844 That makes a lot of sense. 541 00:31:05,864 --> 00:31:08,639 So, Let's maybe go a little bit more high level. 542 00:31:08,639 --> 00:31:12,569 Again, for the past couple of minutes, we've been talking a lot about like how 543 00:31:12,599 --> 00:31:14,679 Electric happens to work under the hood. 544 00:31:14,679 --> 00:31:17,409 And there's many commonalities with other technologies and 545 00:31:17,419 --> 00:31:19,169 all the way to CRDTs as well. 546 00:31:19,599 --> 00:31:23,619 But going back a little bit towards the perspective of someone who would 547 00:31:23,629 --> 00:31:27,949 be using Electric and build something with Electric and doesn't maybe 548 00:31:28,019 --> 00:31:32,519 peel off all the layers yet, but get started with one of the easier off the 549 00:31:32,519 --> 00:31:34,899 shelf options that Electric provides. 550 00:31:35,159 --> 00:31:39,879 So my understanding is that you have your existing Postgres database. 551 00:31:40,156 --> 00:31:44,686 you already have your like tables, your schema, et cetera, or if it's 552 00:31:44,686 --> 00:31:47,650 a greenfield app, you can design that however you still want. 553 00:31:47,980 --> 00:31:49,820 And then you have your Postgres database. 554 00:31:50,160 --> 00:31:53,770 Electric is that infrastructure component that you put in front 555 00:31:53,770 --> 00:31:58,110 of your Postgres database that has access to your Postgres database. 556 00:31:58,130 --> 00:32:02,240 In fact, it has access to the replication stream of Postgres. 557 00:32:02,240 --> 00:32:04,850 So it knows everything that's going on in that database. 558 00:32:05,310 --> 00:32:10,580 And then your client is talking to the Electric sync engine to 559 00:32:10,636 --> 00:32:12,496 sync in whatever data you need. 560 00:32:12,816 --> 00:32:17,346 And the way that's expressed what your client actually needs is through 561 00:32:17,346 --> 00:32:19,306 this concept that you call shapes. 562 00:32:19,346 --> 00:32:23,481 And my understanding is that a shape basically defines a subset 563 00:32:23,701 --> 00:32:28,601 of data, a subset of a table that you want in your client. 564 00:32:28,768 --> 00:32:32,628 since often like tables are so huge and you just need a particular 565 00:32:32,678 --> 00:32:36,618 subset for your given user, for your given document, whatever. 566 00:32:36,928 --> 00:32:38,158 is that accurate? 567 00:32:38,441 --> 00:32:40,581 Yeah, that's just exactly how it works. 568 00:32:40,591 --> 00:32:40,791 And. 569 00:32:41,135 --> 00:32:43,751 the Electric Sync Engine it's a web service. 570 00:32:44,281 --> 00:32:47,311 It's a Docker container, like technically it's an Elixir application. 571 00:32:47,901 --> 00:32:51,991 And it just connects to your Postgres as a normal Postgres client would. 572 00:32:52,451 --> 00:32:56,351 So you have to run your Postgres with logical replication enabled. 573 00:32:57,051 --> 00:32:59,421 And then we just connect in over a database URL. 574 00:32:59,618 --> 00:33:03,018 And so it's just as if you were like, imagine you're deploying a Heroku app, 575 00:33:03,308 --> 00:33:06,895 and it's sort of Heroku Postgres, and it just provisions a database URL, and your 576 00:33:06,915 --> 00:33:08,715 back end application can connect to it. 577 00:33:08,725 --> 00:33:12,685 So it's the same way that a sort of Rails app would talk to, talk to Postgres. 578 00:33:12,985 --> 00:33:16,811 And then Electric does some stuff internally to of route data into 579 00:33:16,811 --> 00:33:21,131 these shape logs, which are the sort of logs of update operations for each 580 00:33:21,161 --> 00:33:22,971 kind of unit of partial replication. 581 00:33:23,365 --> 00:33:28,955 And then we actually just provide a HTTP API, which is quite key to a whole 582 00:33:28,985 --> 00:33:31,288 bunch of the, affordances of the system. 583 00:33:31,288 --> 00:33:33,728 So I can dive into that if it's interesting. 584 00:33:33,855 --> 00:33:37,180 But then, yeah, you basically have a client, Which pulls data 585 00:33:37,180 --> 00:33:39,070 by just making HTTP requests. 586 00:33:39,226 --> 00:33:44,366 and so HTTP gives you back pressure and the client's in control of 587 00:33:44,366 --> 00:33:48,526 which data it pulls when, and then how you process that stream. 588 00:33:48,916 --> 00:33:51,546 Yeah, we do provide some primitives to make it simple. 589 00:33:51,546 --> 00:33:55,496 Like we give you React hooks to just sort of bind a shape to a state variable, 590 00:33:55,766 --> 00:33:58,621 but Basically, you can do what you like with the data as it streams it. 591 00:33:59,221 --> 00:34:03,801 So, yeah, I would love to learn more about that design decision of choosing HTTP 592 00:34:03,811 --> 00:34:05,921 for that network layer, for that API. 593 00:34:05,931 --> 00:34:10,721 Since I think most people think about local-first, think about real time 594 00:34:10,721 --> 00:34:12,841 syncing, et cetera, that reactivity. 595 00:34:13,131 --> 00:34:17,351 And for most people, I think particularly in the web, the mind goes to web sockets. 596 00:34:17,821 --> 00:34:19,551 So why HTTP? 597 00:34:19,561 --> 00:34:21,381 Wouldn't that be very inefficient? 598 00:34:21,441 --> 00:34:23,001 How does reactivity work? 599 00:34:23,041 --> 00:34:24,081 Can you walk me through that? 600 00:34:25,201 --> 00:34:25,911 Yeah, so. 601 00:34:26,406 --> 00:34:27,006 I mean, exactly. 602 00:34:27,006 --> 00:34:30,840 We, went on that journey with the product where with the earlier, slightly more 603 00:34:30,840 --> 00:34:36,193 ambitious Electric that I was describing, we built out a custom binary WebSocket 604 00:34:36,213 --> 00:34:39,183 protocol to do the replication, and it's just what you sort of immediately 605 00:34:39,193 --> 00:34:41,973 think you're like, let's make it efficient over the wire and obviously 606 00:34:41,973 --> 00:34:44,783 it should be a WebSocket connection because you're just having these sorts 607 00:34:44,783 --> 00:34:48,823 of ongoing data streams, but, So one of the things that happened with the, 608 00:34:48,976 --> 00:34:52,656 focusing of the product strategy was that, Kyle Matthews joined the team. 609 00:34:52,896 --> 00:34:57,276 So Kyle was actually the founder of Gatsby, which is like the React framework. 610 00:34:57,451 --> 00:35:01,561 And through Gatsby, he did a lot of work around basically data 611 00:35:01,561 --> 00:35:03,851 delivery into CDN infrastructure. 612 00:35:04,281 --> 00:35:08,941 And so one of the insights that Kyle brought into the team was if 613 00:35:08,941 --> 00:35:13,821 we re engineered the replication protocol on plain HTTP, and we just 614 00:35:13,821 --> 00:35:16,196 do like plain HTTP, plain JSON. 615 00:35:16,373 --> 00:35:20,003 And we replicate over an old fashioned long polling protocol. 616 00:35:20,303 --> 00:35:24,253 So you just, basically we have a model where the client makes a request to a 617 00:35:24,253 --> 00:35:28,583 shape endpoint, and then we just return the data that the server knows about. 618 00:35:28,623 --> 00:35:31,563 So we'll sort of chunk it up sometimes over multiple requests, but it's 619 00:35:31,563 --> 00:35:34,903 just a standard, like load and load a JSON in a document request. 620 00:35:35,243 --> 00:35:38,123 And then once you get a message to say that the client is up to date 621 00:35:38,123 --> 00:35:41,563 with the server, then you trigger into a long polling mode where basically 622 00:35:41,563 --> 00:35:45,543 the server holds the connection open until any new data arrives. 623 00:35:45,856 --> 00:35:50,116 and yes, you kind of think instinctively like, okay, it's say JSON instead of 624 00:35:50,116 --> 00:35:52,646 binary, so it'll be less efficient and you're having to make these 625 00:35:52,646 --> 00:35:56,386 sort of extra requests that surely they add latency over some sort of 626 00:35:56,386 --> 00:35:58,400 more optimized, WebSocket protocol. 627 00:35:58,990 --> 00:36:02,830 But the key thing is that by doing that, it allows us to deliver the data 628 00:36:02,840 --> 00:36:05,160 through existing CDN infrastructure. 629 00:36:05,730 --> 00:36:10,595 So those initial data loading requests, like typically when you're building 630 00:36:10,595 --> 00:36:15,125 applications on this shape primitive, you can find ways of defining your shapes 631 00:36:15,135 --> 00:36:16,685 so that they're shared across users. 632 00:36:16,925 --> 00:36:21,098 You might have some unique data that's unique to a user, but Like say you have a 633 00:36:21,098 --> 00:36:24,698 project management app and there's various users who are all in the same project, 634 00:36:24,888 --> 00:36:28,178 you could choose to like sync the kind of project data down rather than just 635 00:36:28,178 --> 00:36:29,898 sort of syncing all the user's data down. 636 00:36:30,238 --> 00:36:33,108 And so that way you get shapes being shared across users. 637 00:36:33,408 --> 00:36:37,628 And so the first user to request it hits the Electric service, we 638 00:36:37,628 --> 00:36:41,068 generate these responses, but then they go through Cloudflare or Fastly 639 00:36:41,068 --> 00:36:43,078 or CloudFront or what have you. 640 00:36:43,718 --> 00:36:46,578 And every subsequent request is just served out of like 641 00:36:46,738 --> 00:36:48,238 essentially Nginx or Varnish. 642 00:36:48,578 --> 00:36:50,108 And so it's just super efficient. 643 00:36:50,158 --> 00:36:52,808 All of this infrastructure is just like super battle tested 644 00:36:52,808 --> 00:36:54,278 and as optimized as it can be. 645 00:36:54,718 --> 00:36:56,008 That is very interesting. 646 00:36:56,008 --> 00:37:00,248 It reminds me a little bit of like how modern bundlers, and I think even like 647 00:37:00,378 --> 00:37:06,078 all the way back to Webpack, used to split up larger things into little chunks. 648 00:37:06,318 --> 00:37:08,298 And those chunks would be content hashed. 649 00:37:08,783 --> 00:37:12,980 And that would be then often, be cached by the browser across 650 00:37:12,980 --> 00:37:14,820 different versions of the same app. 651 00:37:15,070 --> 00:37:19,720 In this case, it would be beneficial to the individual user who would reload it. 652 00:37:20,060 --> 00:37:24,120 And also of course, like to other people who visit this, but now you 653 00:37:24,350 --> 00:37:29,180 take the same idea, even further and apply it to data shared across users 654 00:37:29,390 --> 00:37:35,785 by applying the same infrastructure, HTTP servers, CDNs, et cetera, to make, 655 00:37:35,928 --> 00:37:37,578 things cheaper and faster, I guess. 656 00:37:38,198 --> 00:37:41,211 Well, and, and the local browser c or client cache as well. 657 00:37:41,211 --> 00:37:45,081 So you have this sort of shared caching within a CDN layer where you 658 00:37:45,081 --> 00:37:48,441 might have multiple clients, which are like, literally it's a sort of 659 00:37:48,441 --> 00:37:50,121 shared cache in the HTTP cache control. 660 00:37:50,121 --> 00:37:50,991 That makes a lot of sense. 661 00:37:50,991 --> 00:37:53,934 Since like, on a website level, I'm not sure whether you 662 00:37:53,934 --> 00:37:55,544 have clear caching semantics. 663 00:37:55,544 --> 00:37:56,634 I don't think so. 664 00:37:57,004 --> 00:37:59,864 Yeah, you'd have to do some very sort of custom stuff to 665 00:37:59,864 --> 00:38:01,324 sort of achieve the same things. 666 00:38:01,464 --> 00:38:05,564 but also because, so with the browser, when you're loading data, like HTTP 667 00:38:05,564 --> 00:38:08,874 requests with the write cache headers can just be stored in the local file cache. 668 00:38:09,204 --> 00:38:12,400 So one of the really nice things with just, like loading shape data 669 00:38:12,430 --> 00:38:16,920 through the Electric API is you can achieve an offline capable app without 670 00:38:16,940 --> 00:38:20,900 even having to implement any kind of local persistence for the data 671 00:38:20,900 --> 00:38:22,810 that's loaded into the file cache. 672 00:38:23,205 --> 00:38:26,792 So that sort of model, if like say you've gone to a page and you've just 673 00:38:26,802 --> 00:38:30,382 loaded the data through Electric, even if you didn't store the data, if you 674 00:38:30,382 --> 00:38:34,072 navigate back to the same page, the data's just there out of the file cache. 675 00:38:34,282 --> 00:38:37,252 So the application can work offline without even having 676 00:38:37,252 --> 00:38:38,302 any kind of persistence. 677 00:38:38,562 --> 00:38:41,782 So you almost get like, I mean, there's some sort of edge cases on this stuff, 678 00:38:41,782 --> 00:38:44,652 but it's the thing, because you're just working with the standard primitives, 679 00:38:44,682 --> 00:38:47,569 you've just got the integration with the existing tooling and you get a 680 00:38:47,569 --> 00:38:49,609 whole bunch of these things for free. 681 00:38:49,765 --> 00:38:54,425 That is very elegant and I guess that is being unlocked now because like 682 00:38:54,425 --> 00:39:00,015 you embrace the semantics of change of like how the data changes more and by 683 00:39:00,025 --> 00:39:04,105 modeling and this is where it now gets relevant again why everything here is 684 00:39:04,115 --> 00:39:08,195 modeled as a log under the hood since like to the log you just append and so 685 00:39:08,195 --> 00:39:12,505 you can safely cache everything that has happened up until a point in time, 686 00:39:12,685 --> 00:39:16,525 and from there on, you just add things on top, but that doesn't make the stuff 687 00:39:16,525 --> 00:39:18,375 that has happened before less valid. 688 00:39:18,375 --> 00:39:20,015 So you can cache it immutably. 689 00:39:20,275 --> 00:39:21,535 That makes it super fast. 690 00:39:21,535 --> 00:39:25,655 You can cache it everywhere on the edge, on your local device, et cetera. 691 00:39:25,995 --> 00:39:30,975 And that gives you a checkpoint that at least once in a point in time was 692 00:39:31,005 --> 00:39:34,755 valid, and now there might be more stuff that should be applied on top of 693 00:39:34,755 --> 00:39:38,465 it, but that's already a better user experience than not getting anything. 694 00:39:38,899 --> 00:39:41,639 I mean, another thing is like the operational characteristics of the 695 00:39:41,639 --> 00:39:44,159 system, for this type of sync technology. 696 00:39:44,229 --> 00:39:47,489 So, for instance, again, comparing HTTP with WebSockets, like 697 00:39:47,699 --> 00:39:50,969 WebSockets are stateful, and you do just keep things in memory. 698 00:39:51,299 --> 00:39:55,444 And so across, if you look across most real time systems, They have scalability 699 00:39:55,444 --> 00:39:57,964 limits because you will come to the point where if you have, say, 10, 000 700 00:39:57,964 --> 00:40:01,670 concurrent users, it's almost like, you know, it's like the thing of don't have 701 00:40:01,670 --> 00:40:03,380 too many open Postgres connections. 702 00:40:03,420 --> 00:40:07,120 But if you're holding open 10, 000 WebSockets, you may be able to do the 703 00:40:07,120 --> 00:40:11,027 IO efficiently, but you will ultimately be growing that kind of memory and 704 00:40:11,027 --> 00:40:12,177 you'll hit some sort of barrier. 705 00:40:12,505 --> 00:40:15,609 Whereas, with this approach, you can basically offload that 706 00:40:15,609 --> 00:40:17,139 concurrency to the CDN layer. 707 00:40:17,579 --> 00:40:23,192 So, it's not just about, being, basically taking away the query workload of the 708 00:40:23,192 --> 00:40:27,602 cached initial sync requests, but these kind of reverse proxies or CDNs have 709 00:40:27,602 --> 00:40:31,622 a really nice feature called request collapsing or request coalescing, which 710 00:40:31,642 --> 00:40:36,362 means that when they have a cache of requests come in on a URL, if they have 711 00:40:36,522 --> 00:40:40,982 Two clients making a request to the same URL at the same time, they sort of hold 712 00:40:41,022 --> 00:40:44,792 both of them at the cache layer and only send one request onto the origin server. 713 00:40:45,412 --> 00:40:51,362 And so basically we've been able to scale out now to 10 million concurrent clients 714 00:40:51,362 --> 00:40:56,039 receiving real time data out of Electric on top of a single single Postgres. 715 00:40:56,422 --> 00:41:01,357 And there is literally no CPU overhead on the Postgres or the Electric layer. 716 00:41:01,407 --> 00:41:05,137 It's just entirely handled out of the CDN CDN serving. 717 00:41:05,647 --> 00:41:09,390 And so it's sort of remarkable that the combination of the initial data 718 00:41:09,390 --> 00:41:13,150 load caching means that we, like one of our objectives is we want to be 719 00:41:13,460 --> 00:41:17,570 as fast as just querying the database directly for an initial data load 720 00:41:17,930 --> 00:41:21,470 and then orders of magnitude faster for anything that then subsequent 721 00:41:21,480 --> 00:41:25,090 requests coming out of the cache, but also this sort of challenge with. 722 00:41:25,430 --> 00:41:29,120 Almost like the, this thing about saying, okay, you're building an application. 723 00:41:29,570 --> 00:41:32,680 You maybe want some of the user experience or developer experience 724 00:41:32,680 --> 00:41:36,490 affordances of local-first, but if to do that, I need a sync engine and a 725 00:41:36,490 --> 00:41:38,660 sync engine is kind of a complex thing. 726 00:41:39,210 --> 00:41:43,920 And so you end up either going, okay, maybe I'll sort of use an external system. 727 00:41:44,130 --> 00:41:47,990 And then you get like, A siloed real time database in your main database 728 00:41:47,990 --> 00:41:51,470 and you get operational complexity, or you get some sort of system where 729 00:41:51,470 --> 00:41:54,514 you have, yeah, you're basically of stewarding these web sockets and 730 00:41:54,514 --> 00:41:56,284 it's very easy for it to fall over. 731 00:41:56,774 --> 00:42:00,234 And I think actually, like, if you just sort of honestly view that 732 00:42:00,244 --> 00:42:04,374 type of, architectural decision from the lens of like somebody trying to 733 00:42:04,374 --> 00:42:07,484 build a real project, which is their day job, trying to get stuff done. 734 00:42:08,064 --> 00:42:10,854 You're just going to avoid that as much as you can, because like 735 00:42:10,864 --> 00:42:13,804 you'd far rather just like, I just want to serve this with Nginx. 736 00:42:13,804 --> 00:42:14,944 I know how that's going to work. 737 00:42:14,944 --> 00:42:16,544 I'm not going to stay up at night worrying about it. 738 00:42:17,124 --> 00:42:20,414 Whereas I have 10, 000 concurrent users going through some crazy WebSocket stuff. 739 00:42:20,414 --> 00:42:21,804 I'm going to get pager alerts. 740 00:42:22,394 --> 00:42:26,274 And so like the whole approach here with what we're trying to do is to 741 00:42:26,314 --> 00:42:31,324 change that sense that sync is a complex technology that you sort of. 742 00:42:31,664 --> 00:42:34,064 Play with on the weekend and only adopt when you have to. 743 00:42:34,334 --> 00:42:37,544 So going, look, you can actually do sync in such a way that it is 744 00:42:37,544 --> 00:42:41,424 just as simple and standard as normal web service technology. 745 00:42:41,604 --> 00:42:44,664 And then suddenly you can actually unlock the ability for kind of real 746 00:42:44,694 --> 00:42:47,997 projects you know, you can take this stuff into a day job and not, get it 747 00:42:47,997 --> 00:42:49,537 shouted down at the design meeting. 748 00:42:49,537 --> 00:42:52,187 Cause it just feels like too much black box complexity. 749 00:42:52,487 --> 00:42:54,927 You're using the word simple here. 750 00:42:54,977 --> 00:43:00,857 And I think that really speaks to me now, because it's both simple in terms of 751 00:43:01,097 --> 00:43:03,827 architecturally, like, how does data flow? 752 00:43:04,070 --> 00:43:09,624 so I think this is where Electric provides a very simple and I think 753 00:43:09,704 --> 00:43:14,294 easy to use and easy to work with trade off, like, how does data flow, 754 00:43:14,524 --> 00:43:18,704 but then it's also gives a very simple answer of like, how does it scale? 755 00:43:19,229 --> 00:43:23,429 Since you can throw at it like all the innovations and all the hard 756 00:43:23,429 --> 00:43:27,779 work that has now gone into the like our web infrastructure for the last 757 00:43:27,809 --> 00:43:33,059 decades, you can run on the latest and greatest and all the innovations that 758 00:43:33,279 --> 00:43:39,549 Nginx and HAProxy and Cloudflare and like all the work that has into that. 759 00:43:39,549 --> 00:43:44,739 You can just piggyback on top of that without having to innovate on the 760 00:43:44,749 --> 00:43:48,149 networking side as well, since like you, you're really doing the hard work 761 00:43:48,149 --> 00:43:51,079 on the more semantic and data side. 762 00:43:51,419 --> 00:43:54,039 And that's a really, really elegant trade off to me. 763 00:43:54,249 --> 00:43:54,459 Yeah. 764 00:43:54,459 --> 00:43:56,949 And it's, it's fun because like our benchmarking testing at the 765 00:43:56,949 --> 00:43:59,669 moment, like we break CloudFlare before we break Electric. 766 00:43:59,879 --> 00:44:01,819 if something is battle tested, it's CloudFlare. 767 00:44:02,265 --> 00:44:05,095 It again, it carries on because it's not just about this sort of 768 00:44:05,095 --> 00:44:06,625 scalability or operational stuff. 769 00:44:06,625 --> 00:44:10,665 It's also about then how you can achieve, like we talked about the write patterns. 770 00:44:10,665 --> 00:44:12,295 And so this sort of pattern of how do you do writes? 771 00:44:12,295 --> 00:44:14,745 And it's like, well, actually you can do the sync like this, use 772 00:44:14,745 --> 00:44:16,355 your existing API to do writes. 773 00:44:16,785 --> 00:44:18,606 And it can work with your existing stack. 774 00:44:19,006 --> 00:44:22,776 But you have other obvious concerns with this type of architecture, like 775 00:44:22,776 --> 00:44:27,233 say, authentication, authorization, data security, encryption. 776 00:44:27,773 --> 00:44:28,173 But HTTP. 777 00:44:29,413 --> 00:44:32,963 just has proxies and it works with the sort of middleware stack. 778 00:44:33,403 --> 00:44:39,883 And so for us, a shape endpoint as a sync endpoint is just a HTTP resource. 779 00:44:40,343 --> 00:44:43,653 So if you want to just put like an authorization service in front of it, 780 00:44:43,873 --> 00:44:47,263 you just proxy the request through and you like, you have the context from 781 00:44:47,263 --> 00:44:49,783 the user, you can have the context about the shape and you can just 782 00:44:49,783 --> 00:44:51,953 authorize it using your existing stack. 783 00:44:52,463 --> 00:44:53,882 If you want to do encryption, then you can do that. 784 00:44:54,033 --> 00:44:55,463 It's just a stream of messages. 785 00:44:55,503 --> 00:44:58,543 And yeah, a bit like you were saying that, like with Electric, you could 786 00:44:58,543 --> 00:45:02,773 just use it as a transport layer to like, say, route a log of messages. 787 00:45:03,353 --> 00:45:05,142 That can be ciphertext or plaintext. 788 00:45:05,143 --> 00:45:08,453 So you could just like encrypt on device, sync it through. 789 00:45:08,453 --> 00:45:11,316 You can just decrypt whenever you're consuming the stream. 790 00:45:11,506 --> 00:45:13,456 And again, you could do that, like in the client, you could 791 00:45:13,456 --> 00:45:14,956 do that in HTTP middleware. 792 00:45:15,681 --> 00:45:20,618 So a lot of the sort of concerns, which, like certainly our experience of trying 793 00:45:20,618 --> 00:45:24,158 to build a more integrated end to end local-first stack, you know, you go, 794 00:45:24,168 --> 00:45:25,648 okay, we need to, we need to solve this. 795 00:45:25,648 --> 00:45:29,298 I need a security rule system because suddenly there is no API and how am 796 00:45:29,298 --> 00:45:30,698 I going to authorize the data access? 797 00:45:30,698 --> 00:45:33,078 And it's like, we don't need a security rule system. 798 00:45:33,708 --> 00:45:37,357 Because you can just use, you can just use normal API middleware 799 00:45:37,438 --> 00:45:39,498 in front of an HTTP service. 800 00:45:39,498 --> 00:45:42,898 And so you just sort of take that problem out of scope and like the 801 00:45:42,898 --> 00:45:44,488 system doesn't need to do encryption. 802 00:45:44,498 --> 00:45:47,328 It doesn't need to provide like a kind of hooks mechanism or some sort 803 00:45:47,388 --> 00:45:51,418 of framework extensibility because the protocol is extensible and just, 804 00:45:51,428 --> 00:45:55,128 you just have all of this ecosystem of existing tooling built around it. 805 00:45:55,728 --> 00:45:59,298 So it is, I mean, it's been fantastic for us because it, because it 806 00:45:59,308 --> 00:46:00,958 simplifies all of this aspects. 807 00:46:01,418 --> 00:46:03,718 And allows us to go, look, this is how you can achieve, say 808 00:46:03,748 --> 00:46:07,668 authorization with Electric, but again, it pushes it out of scope. 809 00:46:07,678 --> 00:46:11,458 So we get to focus our engineering resources on just doing the core stuff 810 00:46:11,488 --> 00:46:12,978 to deliver on this core proposition. 811 00:46:13,268 --> 00:46:19,028 So which sort of things would you say are particularly tricky from a application 812 00:46:19,028 --> 00:46:23,438 of all perspective with Electric, where it might be not as much of a good fit? 813 00:46:23,991 --> 00:46:30,861 I think, One of the things is that we sync through the database and that has latency. 814 00:46:31,481 --> 00:46:36,505 And so if you're trying to craft a really low latency real time multiplayer 815 00:46:36,715 --> 00:46:41,705 experience, like, or even doing things where in a way it doesn't really 816 00:46:41,705 --> 00:46:46,442 make sense to be, synchronizing that information through the database layer, 817 00:46:46,615 --> 00:46:48,975 then it's maybe not the best solution. 818 00:46:49,046 --> 00:46:54,325 So sort of for like presence features, let's say Infignar, where 819 00:46:54,325 --> 00:46:57,705 you see my mouse cursor moving around, those sort of things. 820 00:46:57,929 --> 00:47:01,699 yes, it would be nice if it was in real time shared across the various 821 00:47:01,699 --> 00:47:05,559 collaborators, but you don't need a persistent trace of that for 822 00:47:05,559 --> 00:47:07,479 eternity in your Postgres database. 823 00:47:07,859 --> 00:47:11,459 So I think a common approach for that as well is just to have like 824 00:47:11,679 --> 00:47:15,659 two kind of different channels for how your data flows, like your, 825 00:47:15,949 --> 00:47:19,569 persisted data that you want to actually keep around as a fixed trail. 826 00:47:19,569 --> 00:47:22,039 Like, did I create this GitHub issue or not? 827 00:47:22,389 --> 00:47:26,169 But like how my mouse cursor has moved around, it's fine that that's 828 00:47:26,189 --> 00:47:30,489 being broadcasted, but if someone opens it an hour later, it's fine 829 00:47:30,489 --> 00:47:31,869 that that person would never know. 830 00:47:32,269 --> 00:47:37,779 So for this sort of use case, it's an overkill basically 831 00:47:38,080 --> 00:47:38,534 to pipe that trough Postgres 832 00:47:38,564 --> 00:47:38,844 Yeah. 833 00:47:39,054 --> 00:47:39,559 And you know, it's. 834 00:47:39,954 --> 00:47:41,494 For us, Postgres is a big qualifier. 835 00:47:41,534 --> 00:47:46,114 It's like, if you, if you want to use Postgres, if you have an existing Postgres 836 00:47:46,154 --> 00:47:51,404 backed system, like Electric shines where like, yeah, you have, you already use 837 00:47:51,404 --> 00:47:54,624 Postgres or you know that you want to be using Postgres, maybe you already have a 838 00:47:54,624 --> 00:47:58,154 bunch of integrations on the data model already, maybe you do have existing API 839 00:47:58,644 --> 00:48:02,334 code, like this is the scenario where we're really trying to say, well, look, 840 00:48:02,344 --> 00:48:07,479 in that scenario, this is a great, pathway to move towards these more advanced 841 00:48:07,479 --> 00:48:11,309 local-first sync based architectures, where, whereas if you look at it from a 842 00:48:11,309 --> 00:48:15,549 sort of more greenfield development point of view, and you're trying to craft a 843 00:48:15,549 --> 00:48:20,089 particular concurrency semantics, say, you would reach for Automerge and you 844 00:48:20,089 --> 00:48:24,362 would get custom data types, which you can craft advanced kind of invariant 845 00:48:24,612 --> 00:48:26,632 support with your kind of data types. 846 00:48:27,272 --> 00:48:30,042 But of course, you know, so that's a slightly different sort of world. 847 00:48:30,042 --> 00:48:35,092 And, and I think so almost probably for sort of a lot of people in the local-first 848 00:48:35,112 --> 00:48:39,805 space dive into CRDTs and so forth, you know, it's really, it's fascinating 849 00:48:39,805 --> 00:48:44,345 to try to sort of craft these sort of optimized, kind of, present style, 850 00:48:44,355 --> 00:48:46,445 immediate real time streaming experiences. 851 00:48:47,340 --> 00:48:52,440 And so whilst we do real time sync, it's almost more about keeping the data fresh 852 00:48:52,780 --> 00:48:56,030 and just sort of making sure that the clients are sort of eventually consistent 853 00:48:56,270 --> 00:49:00,567 rather than making that more sort of game kind of experience where, you 854 00:49:00,567 --> 00:49:03,854 know, where maybe peer to peer matters more or of finding clever hacks to have 855 00:49:03,864 --> 00:49:05,634 very low latency kind of interactions. 856 00:49:06,265 --> 00:49:07,345 That makes a lot of sense. 857 00:49:07,375 --> 00:49:12,195 So now we've talked a lot about Electric and Electric is the name of the company. 858 00:49:12,365 --> 00:49:14,145 It's the name of your main product. 859 00:49:14,355 --> 00:49:17,815 But there's also been a project that I'm not sure whether you 860 00:49:17,865 --> 00:49:21,785 originally created, but it's certainly in your hands at this point. 861 00:49:21,785 --> 00:49:22,975 It's called PGlite. 862 00:49:23,315 --> 00:49:26,105 That made the rounds on Hacker News, etc. 863 00:49:26,225 --> 00:49:29,445 Also through a joint launch with the folks at Superbase. 864 00:49:29,905 --> 00:49:31,025 What is PGlite? 865 00:49:31,045 --> 00:49:32,495 What is that about? 866 00:49:33,102 --> 00:49:37,880 Yeah, so I mean, interestingly with Electric, we started off, building 867 00:49:37,880 --> 00:49:42,300 a stack, which was sinking out of Postgres into SQLite because it 868 00:49:42,300 --> 00:49:45,580 made sense as the sort of main like embeddable relational database. 869 00:49:45,840 --> 00:49:50,377 and I remember, speaking to Nikita, who is the CEO at Neon, the Postgres database 870 00:49:50,377 --> 00:49:57,477 company, and some of his advice from building SingleStore or MemSQL was the 871 00:49:57,517 --> 00:50:01,447 impedance or the mismatch between the two database systems and the data type systems 872 00:50:02,057 --> 00:50:05,897 will continue to just be a source of pain for as long as you build that system. 873 00:50:06,357 --> 00:50:09,614 And so we were just having these conversations about going, how do we 874 00:50:09,614 --> 00:50:11,284 make this Postgres to Postgres sync? 875 00:50:11,724 --> 00:50:15,189 And then, You can just eliminate any mismatch. 876 00:50:15,359 --> 00:50:19,479 You just, you don't even need to do any kind of like serialization of the data. 877 00:50:19,799 --> 00:50:23,269 You can just literally take it exactly as it comes out of like the binary 878 00:50:23,269 --> 00:50:26,119 format that comes through in a query or the replication stream from Postgres, 879 00:50:26,519 --> 00:50:29,289 put that into the client and like, you can have exactly the same data 880 00:50:29,289 --> 00:50:31,199 types and exactly the same extensions. 881 00:50:31,405 --> 00:50:32,935 So this was a sort of motivation for us. 882 00:50:32,935 --> 00:50:36,679 And co founder Stas, the CTO at Neon had done an experiment. 883 00:50:37,002 --> 00:50:41,272 to try and make a more efficient Wasm builder Postgres that could 884 00:50:41,292 --> 00:50:42,672 potentially run in the client. 885 00:50:43,032 --> 00:50:47,772 So previously there'd been some really cool work by Superbase, by Snaplet, a 886 00:50:47,792 --> 00:50:52,227 few teams, which had developed these sorts of VM based, Wasm Postgreses. 887 00:50:52,315 --> 00:50:53,405 But they were pretty big. 888 00:50:53,509 --> 00:50:54,949 they didn't really have persistence. 889 00:50:54,949 --> 00:50:57,539 They weren't, they were sort of more of a kind of proof of concept. 890 00:50:57,902 --> 00:51:02,352 and the approach that Stas took was to do a pure Wasm build and 891 00:51:02,352 --> 00:51:04,282 run Postgres in single user mode. 892 00:51:04,692 --> 00:51:08,532 And that allowed you to basically remove a whole bunch of the concurrency 893 00:51:09,192 --> 00:51:12,772 stuff within Postgres, which allowed us to make a much, much smaller build. 894 00:51:13,602 --> 00:51:15,612 So they shared that repo. 895 00:51:15,832 --> 00:51:18,189 And we sort of, played with it for a little while. 896 00:51:18,189 --> 00:51:20,062 Didn't quite manage to kind of make it work. 897 00:51:20,062 --> 00:51:23,622 And then one of the guys on our team, Sam Willis, just picked it up one week 898 00:51:23,642 --> 00:51:27,132 and put in some concerted efforts and basically managed to pull it together 899 00:51:27,512 --> 00:51:29,902 with persistence as a three meg build. 900 00:51:30,387 --> 00:51:34,880 And it worked, and so suddenly we had this project which was like a three meg like 901 00:51:34,880 --> 00:51:39,000 SQLite for context is like a one meg WASM build, and so Postgres is much kind of 902 00:51:39,010 --> 00:51:41,970 larger system and you think it would be much bigger, but suddenly actually it's 903 00:51:41,980 --> 00:51:46,420 not that far off in terms of the download speed, and it could just run as a fully 904 00:51:46,420 --> 00:51:48,150 featured Postgres inside the browser. 905 00:51:48,387 --> 00:51:50,567 and so we sort of tweeted that out and it's gone a bit crazy. 906 00:51:50,567 --> 00:51:53,607 I think it's like, it's the fastest growing database project ever on GitHub. 907 00:51:54,167 --> 00:51:56,927 It's like 250, 000 downloads a week nowadays. 908 00:51:57,237 --> 00:51:59,067 There's a huge, there's lots and lots of people using it. 909 00:51:59,067 --> 00:52:00,397 Superbase are using it in production. 910 00:52:00,397 --> 00:52:01,837 Google are using it in production. 911 00:52:02,227 --> 00:52:05,990 Lots of people are building tooling around it, like drizzle integrations, et cetera. 912 00:52:06,282 --> 00:52:08,242 And it's the sort of thing that just should exist, right? 913 00:52:08,242 --> 00:52:11,622 There should be a WASM built at Postgres, just being able to have it 914 00:52:11,622 --> 00:52:14,852 like the same database system instead of mapping into an alternative one 915 00:52:15,112 --> 00:52:21,362 has these fundamental advantages, and also a lot of people have just been 916 00:52:21,412 --> 00:52:25,372 coming up with like a whole range of interesting use cases for it as a project. 917 00:52:25,392 --> 00:52:28,767 So some people are interested in running it inside Edgeworkers. 918 00:52:28,907 --> 00:52:31,497 As a sort of data layer that you can hydrate data into 919 00:52:31,507 --> 00:52:32,777 for kind of background jobs. 920 00:52:33,057 --> 00:52:37,117 Some people are interested in running it as just like a development database. 921 00:52:37,337 --> 00:52:38,957 So you can just NPM install Postgres. 922 00:52:38,957 --> 00:52:41,707 And if you're running like an application stack, you don't have to 923 00:52:41,707 --> 00:52:43,617 run Postgres as an external service. 924 00:52:43,807 --> 00:52:45,807 The same thing in your testing environment. 925 00:52:46,307 --> 00:52:48,304 So there's a whole bunch of different use cases. 926 00:52:48,594 --> 00:52:51,814 And in fact, like some of the work, for instance, the Superbase have 927 00:52:51,814 --> 00:52:54,854 done is they built a very cool project called database.build, 928 00:52:55,254 --> 00:52:59,744 which is a sort of AI driven database backed application builder. 929 00:53:00,224 --> 00:53:02,844 So it's sort of AI app builder for building Postgres backed 930 00:53:02,844 --> 00:53:06,944 applications, and it just runs purely on PGlite in the client. 931 00:53:07,554 --> 00:53:09,334 And so that's a demonstration where. 932 00:53:09,697 --> 00:53:13,277 this sort of database infrastructure for running software, you had 933 00:53:13,277 --> 00:53:16,647 centralized databases, and then you had this sort of move to serverless 934 00:53:16,667 --> 00:53:18,517 with separation of compute and storage. 935 00:53:18,967 --> 00:53:21,467 And now you sort of have this model where actually you can run the compute, 936 00:53:21,794 --> 00:53:24,644 with a whole range of different storage patterns in the client. 937 00:53:24,694 --> 00:53:28,124 And you don't even need to deploy any infrastructure on the server. 938 00:53:28,609 --> 00:53:30,499 to run database driven applications. 939 00:53:30,642 --> 00:53:34,352 it really reminds me of that time when JavaScript was 940 00:53:34,382 --> 00:53:35,742 getting more and more serious. 941 00:53:35,742 --> 00:53:40,852 And at some point there was no JS and suddenly you could run the same sort of 942 00:53:40,872 --> 00:53:45,152 JavaScript code that you were running in your browser, now also on the server. 943 00:53:45,222 --> 00:53:47,772 And well, the rest is history, right? 944 00:53:47,882 --> 00:53:50,042 Like that changed the web forever. 945 00:53:50,332 --> 00:53:54,452 It has like changed dramatically how JavaScript just become like 946 00:53:54,452 --> 00:53:59,482 the default full stack foundation for almost every app these days. 947 00:53:59,892 --> 00:54:02,912 And there seemed to be a lot of like similar characteristics. 948 00:54:02,912 --> 00:54:07,102 This time, the other way around, like going from the server into the world, 949 00:54:07,475 --> 00:54:11,652 Node, it was rather the other way around, but, that seems like a huge deal. 950 00:54:11,889 --> 00:54:15,012 Yeah, you know, you sort of step forward and we of see, I guess, some 951 00:54:15,012 --> 00:54:19,332 of these trends in data architecture and just, you know, it can just 952 00:54:19,332 --> 00:54:20,772 be the same database everywhere. 953 00:54:20,822 --> 00:54:23,932 And in a way, it's just sort of almost logically extended to wherever you want. 954 00:54:23,942 --> 00:54:28,022 And you almost like, you can just have this idea of like 955 00:54:28,322 --> 00:54:31,352 declarative configuration of what data should sit where. 956 00:54:31,947 --> 00:54:35,297 AI systems can optimize transfer and placement, and it is just 957 00:54:35,297 --> 00:54:36,897 all the same kind of data types. 958 00:54:37,007 --> 00:54:40,754 and I think, this is sort of where systems are moving to, but also 959 00:54:40,754 --> 00:54:44,414 just like some of these things we've been learning with PGlite, like for 960 00:54:44,414 --> 00:54:48,200 instance, if you're running a system that relies on having say a database 961 00:54:48,200 --> 00:54:51,220 behind your application and say it's a SAS system and you're spinning up some 962 00:54:51,220 --> 00:54:55,282 infrastructure for a client, With PGlite, you don't necessarily need to spin up a 963 00:54:55,282 --> 00:54:57,172 database in order to serve that client. 964 00:54:57,442 --> 00:55:01,432 So if you think about something like the free tier of like SaaS platform like that, 965 00:55:01,872 --> 00:55:03,792 it can just change the economics of it. 966 00:55:04,294 --> 00:55:06,834 it can do that on the server by just allowing you to have 967 00:55:06,834 --> 00:55:08,314 the Postgres in process. 968 00:55:08,324 --> 00:55:10,127 So you're not deploying additional infrastructure. 969 00:55:10,727 --> 00:55:13,407 But also you move it all the way into the client and there just is 970 00:55:13,407 --> 00:55:15,087 no compute kind of running on this. 971 00:55:15,097 --> 00:55:17,467 It just moves even more of the compute onto the client. 972 00:55:18,047 --> 00:55:21,517 And I think it like, it obviously aligns with sort of local-first in 973 00:55:21,517 --> 00:55:24,487 general, but I know some of the stuff we've talked about before around the 974 00:55:24,497 --> 00:55:27,007 concept of like local only first. 975 00:55:27,447 --> 00:55:30,807 And as a developer experience for building software, so one of the 976 00:55:30,817 --> 00:55:35,377 things that LiveStore is specifically designed to support is this ability 977 00:55:35,377 --> 00:55:40,046 to Build an application locally with very fast, feedback and iteration. 978 00:55:40,047 --> 00:55:43,737 And then you progressively add on, say, sync or persistence and 979 00:55:43,737 --> 00:55:45,177 sharing and things when you need to. 980 00:55:45,402 --> 00:55:48,622 And I think this sort of model of being able to build the software on 981 00:55:48,622 --> 00:55:52,478 a database like, PGlite and then go, okay, I've played with this enough. 982 00:55:52,478 --> 00:55:53,558 I want to save my work. 983 00:55:53,598 --> 00:55:57,104 And it's at that point that you write out to blob storage, or you 984 00:55:57,104 --> 00:55:59,714 maybe provision the database to be able to of save the data into. 985 00:56:00,405 --> 00:56:04,128 Yeah, I think you've touched on something really interesting and something really 986 00:56:04,128 --> 00:56:09,328 profound, which I think is kind of two second order effects of local-first. 987 00:56:09,688 --> 00:56:13,578 And so one of them is for the app users directly. 988 00:56:13,598 --> 00:56:19,744 So ideally it should just become so cheap and so easy to offer the full 989 00:56:19,774 --> 00:56:24,598 product experience as sort of like a taste, fully on the client that is 990 00:56:24,598 --> 00:56:26,548 no longer sitting behind a paywall. 991 00:56:26,768 --> 00:56:30,875 But if the product experience generally allows for that, if it's sort of like 992 00:56:30,875 --> 00:56:35,125 a note, note taking tool or something like that, that I should be able to 993 00:56:35,125 --> 00:56:41,093 like fully try out the app, on my device and doing the signup later and 994 00:56:41,183 --> 00:56:43,643 being able to offer that economically. 995 00:56:44,033 --> 00:56:47,343 That is basically with those new technologies, that's no longer 996 00:56:47,343 --> 00:56:49,033 an argument, so you can offer it. 997 00:56:49,623 --> 00:56:54,203 So hopefully that will be a second order effect where software is way easier to 998 00:56:54,283 --> 00:56:59,258 offer, where it's way easier to just try it out from an end user perspective. 999 00:56:59,660 --> 00:57:04,360 But then also from the second point, from an application developer 1000 00:57:04,390 --> 00:57:08,858 perspective, I think it makes a huge difference in terms of complexity. 1001 00:57:08,858 --> 00:57:12,828 How, when you build something, whether it is just a local script 1002 00:57:12,858 --> 00:57:16,708 without any infrastructure, whether you can just run it, has no infra 1003 00:57:16,708 --> 00:57:21,718 dependencies, you can just run it, maybe you run like your Vite dev server. 1004 00:57:22,043 --> 00:57:22,683 And that's it. 1005 00:57:22,683 --> 00:57:25,143 It's self contained and you can move on. 1006 00:57:25,293 --> 00:57:29,013 There's like no Docker thing you need to start, et cetera. 1007 00:57:29,273 --> 00:57:30,983 That's like your starting point. 1008 00:57:31,433 --> 00:57:35,513 And if the barrier to entry there, if like, if that threshold is lower, 1009 00:57:35,513 --> 00:57:39,759 that you can build a fully functional thing just for yourself, just in that 1010 00:57:39,779 --> 00:57:44,809 local session, and you can get started this way, and if you then see like, 1011 00:57:44,829 --> 00:57:48,569 Oh, actually, there's a case here that I want to make this a multiplayer 1012 00:57:48,569 --> 00:57:52,799 experience or a multi tenant experience, then you can take that next step. 1013 00:57:53,079 --> 00:57:56,431 But right now, like, you can't really, leap ahead there. 1014 00:57:56,431 --> 00:58:00,071 You need to start from that multi tenant, that multi player experience, 1015 00:58:00,461 --> 00:58:04,491 and that makes the, the entry point already so much more tricky that many 1016 00:58:04,491 --> 00:58:06,011 projects are never getting started. 1017 00:58:06,646 --> 00:58:10,691 And I think both of those, I think can be second order effects and 1018 00:58:10,691 --> 00:58:16,201 improvements that local-first inspired architectures and software can provide. 1019 00:58:16,251 --> 00:58:18,261 So, I love those observations. 1020 00:58:18,809 --> 00:58:20,159 Yeah, yeah, totally. 1021 00:58:20,159 --> 00:58:23,499 And I mean, I think, for instance, with, it's interesting as well that a 1022 00:58:23,499 --> 00:58:29,129 lot of people do define their database schema using tools like Prisma, Drizzle, 1023 00:58:29,249 --> 00:58:33,159 like Effect Schema is a great example that obviously you're working on. 1024 00:58:33,563 --> 00:58:37,126 the more layers or indirection between where you're, say, iterating on the 1025 00:58:37,126 --> 00:58:40,806 user experience in the interface, and you want to be able to, say, customize 1026 00:58:40,806 --> 00:58:44,630 a data model to adapt to trying to sort of iterate there quickly. 1027 00:58:44,630 --> 00:58:47,110 But if you have to sort of go all the way into some other language, another 1028 00:58:47,110 --> 00:58:50,629 system, it just sort of takes you out of context and slows everything down. 1029 00:58:50,949 --> 00:58:54,326 So that's somehow the ability to like, yeah, apply that sort of schema into 1030 00:58:54,326 --> 00:58:59,200 the local database, not have to sort of work against these sort of different 1031 00:58:59,270 --> 00:59:01,720 legacy layers of the stack in order to actually be able to build out 1032 00:59:01,720 --> 00:59:03,160 software is really transformational. 1033 00:59:03,452 --> 00:59:08,732 So going back to PGlite for a moment, how does PGlite and Electric, Electric 1034 00:59:09,182 --> 00:59:13,352 as a product and Electric as a company, how do those things fit together? 1035 00:59:14,372 --> 00:59:14,782 Yeah. 1036 00:59:14,872 --> 00:59:18,492 I mean, there basically are sort of two main products. 1037 00:59:18,492 --> 00:59:19,432 We have two products. 1038 00:59:19,562 --> 00:59:22,072 They're both open source, Apache licensed. 1039 00:59:22,872 --> 00:59:26,792 One is the Electric Sync Engine, and one is PGlite. 1040 00:59:26,912 --> 00:59:31,757 And so you can use them together, or you can just use them independently, 1041 00:59:31,897 --> 00:59:35,860 so it's not like the Electric system is designed only to sync into PGlite, 1042 00:59:35,880 --> 00:59:38,990 you don't have to have an embedded Postgres to use it Electric, and 1043 00:59:38,990 --> 00:59:41,270 you can use PGlite just standalone. 1044 00:59:41,570 --> 00:59:44,700 There's a range of different mechanisms to do things like data 1045 00:59:44,700 --> 00:59:48,887 loading, data persistence, et cetera, virtual file system layers, 1046 00:59:48,894 --> 00:59:51,404 loading in, unpacking Parquet files. 1047 00:59:51,979 --> 00:59:56,062 But if you do like have an application with this local database and you wanted to 1048 00:59:56,062 --> 00:59:59,492 then be able to sync that data with other users or into your Postgres database, 1049 00:59:59,762 --> 01:00:01,092 then Electric is just a great fit. 1050 01:00:01,182 --> 01:00:03,542 And obviously we make a kind of first class integration. 1051 01:00:04,042 --> 01:00:09,522 So I think for us, I mean, as a, as a company, as a startup, Electric is the 1052 01:00:09,522 --> 01:00:14,072 main product that we aim to build the business around, because in a way that 1053 01:00:14,092 --> 01:00:18,335 type of operational data infrastructure is just slightly more natural to build 1054 01:00:18,335 --> 01:00:21,795 a commercial offering around, like you have to run servers to move the data 1055 01:00:21,795 --> 01:00:24,805 around, we can do that efficiently, it sort of makes sense and adds value. 1056 01:00:25,685 --> 01:00:29,655 Whereas with PGlite as a open source embedded database, it's not 1057 01:00:29,665 --> 01:00:32,835 something that we're aiming to sort of monetize in quite the same way. 1058 01:00:32,855 --> 01:00:37,265 And potentially, maybe it could be upstreamed into Postgres, like, you know, 1059 01:00:37,275 --> 01:00:39,175 there should be a Wasm build to Postgres. 1060 01:00:39,635 --> 01:00:42,704 or, you know, maybe it kind of moves into a, a foundation and sort 1061 01:00:42,704 --> 01:00:47,122 of develops more governance, like certainly already with, PGlite. 1062 01:00:47,140 --> 01:00:50,580 So like Superbase, co sponsored one of the engineering roles with 1063 01:00:50,690 --> 01:00:53,590 us, there's been contributions from a whole bunch of companies. 1064 01:00:53,600 --> 01:00:56,577 So it is already a sort of, wide attempt in terms of the. 1065 01:00:56,817 --> 01:00:59,817 The stakeholders who are sort of stewarding the development of the project. 1066 01:01:00,077 --> 01:01:01,577 That is very cool to see. 1067 01:01:01,637 --> 01:01:06,757 I'm a big fan of those sort of like multi organizational approaches where you 1068 01:01:06,757 --> 01:01:09,367 share the effort of building something. 1069 01:01:09,437 --> 01:01:10,964 And, yeah, I love that. 1070 01:01:11,024 --> 01:01:14,144 I'm very excited to get my own hands on PGlite as well. 1071 01:01:14,454 --> 01:01:18,464 I'm mostly dealing with SQLite these days just because I think it is 1072 01:01:18,464 --> 01:01:23,294 still a tad faster for like, those single threaded embedded use cases. 1073 01:01:23,624 --> 01:01:27,744 But if you need the raw power of Postgres, which often you do, then 1074 01:01:27,744 --> 01:01:31,914 you can just run it in a worker thread and you get the full power of Postgres 1075 01:01:31,934 --> 01:01:33,614 in your local app, which is amazing. 1076 01:01:34,014 --> 01:01:38,224 So maybe rounding out this conversation on something you just touched on, 1077 01:01:38,404 --> 01:01:42,354 which is a potential commercial offering that Electric provides. 1078 01:01:42,710 --> 01:01:44,020 can you share more about that? 1079 01:01:44,030 --> 01:01:47,210 Which problems it intends to solve and where it's currently at? 1080 01:01:47,895 --> 01:01:51,095 Yep, so we're building, a cloud offering, which is basically 1081 01:01:51,095 --> 01:01:53,055 hosting the Electric sync service. 1082 01:01:53,405 --> 01:01:57,125 So like we, we, for instance, we don't host the Postgres database. 1083 01:01:57,135 --> 01:01:58,805 We don't host your application. 1084 01:01:59,095 --> 01:02:03,355 We just sort of host that kind of core sync layer, and then that can integrate 1085 01:02:03,395 --> 01:02:07,639 with other Postgres hosts like Superbase, Neon, et cetera, and kind of other 1086 01:02:07,639 --> 01:02:09,629 platforms for deploying applications. 1087 01:02:09,952 --> 01:02:12,242 that's our sort of first commercial offering. 1088 01:02:12,545 --> 01:02:17,352 And we of see that as like a almost sort of utility data infrastructure 1089 01:02:17,652 --> 01:02:22,312 play, where we've put a lot of effort in being able to run the software 1090 01:02:22,592 --> 01:02:26,422 very resource efficiently, and with sort of flat resource usage, so 1091 01:02:26,422 --> 01:02:30,109 it doesn't you know, scale up with memory with concurrent users, etc. 1092 01:02:30,417 --> 01:02:32,447 So we want to be able to run that very efficiently. 1093 01:02:32,757 --> 01:02:36,580 And so, we, we sort of see that that's kind of, low cost usage based pricing 1094 01:02:36,590 --> 01:02:39,350 based basically on the sort of data flows running through the software. 1095 01:02:39,780 --> 01:02:43,337 I think, you know, monetizing open source software is quite a sort of, 1096 01:02:43,337 --> 01:02:45,884 it's an interesting topic, but it's also sort of, there are a lot of, 1097 01:02:45,943 --> 01:02:47,643 common patterns that are well known. 1098 01:02:47,673 --> 01:02:54,516 And like, ultimately our aim as a company is, We want people building real 1099 01:02:54,516 --> 01:02:58,886 applications with this technology, and we want developers to enjoy doing it 1100 01:02:58,906 --> 01:03:00,996 and become advocates of the technology. 1101 01:03:01,466 --> 01:03:05,873 And then, there is a pathway when, imagine that you're a large company 1102 01:03:05,883 --> 01:03:09,243 and say you have like five projects and they're all using Electric sync. 1103 01:03:09,713 --> 01:03:12,483 It's very common for those sort of larger companies to need 1104 01:03:12,483 --> 01:03:13,583 additional tooling around that. 1105 01:03:13,913 --> 01:03:17,263 So governance, compliance, data locality. 1106 01:03:17,263 --> 01:03:19,333 There's a whole bunch of sort of considerations there. 1107 01:03:19,733 --> 01:03:22,856 So, it's quite common to be able to build out a sort of enterprise offering 1108 01:03:22,856 --> 01:03:24,656 on top of the core open source product. 1109 01:03:25,026 --> 01:03:27,636 And so, you know, there are various routes like that, that we 1110 01:03:27,636 --> 01:03:29,206 could choose to pursue in future. 1111 01:03:29,346 --> 01:03:33,209 and maybe that's how it plays out as we build a cloud, we focus on, making 1112 01:03:33,229 --> 01:03:37,469 this sync engine and these components bulletproof, make sure people are being 1113 01:03:37,469 --> 01:03:39,349 successful building applications on them. 1114 01:03:39,519 --> 01:03:42,973 And then we can look at maybe some sort of, value added tooling to help you 1115 01:03:42,973 --> 01:03:46,506 operate them successfully at scale, or help you operate them within sort of 1116 01:03:46,676 --> 01:03:48,836 larger companies or regulated contexts. 1117 01:03:49,299 --> 01:03:50,359 That makes a lot of sense. 1118 01:03:50,964 --> 01:03:51,444 Great. 1119 01:03:51,514 --> 01:03:55,074 James, is there anything that you would want from the audience? 1120 01:03:55,114 --> 01:03:56,834 Anything that you want to leave them with? 1121 01:03:57,011 --> 01:03:59,801 anything to give a try over the next weekend? 1122 01:03:59,811 --> 01:04:01,758 The holidays are upon us. 1123 01:04:01,871 --> 01:04:03,268 what should people take a look at? 1124 01:04:03,594 --> 01:04:05,813 Yeah, I know that, You may be listening to this at any time in 1125 01:04:05,813 --> 01:04:09,043 future, but, we're recording this in the lead up to kind of December. 1126 01:04:09,083 --> 01:04:12,693 So if you have some time to experiment with tech over the holiday period, 1127 01:04:12,703 --> 01:04:14,063 just take a look at Electric. 1128 01:04:14,186 --> 01:04:15,976 you know, it's ready for production use. 1129 01:04:16,016 --> 01:04:17,246 It's well documented. 1130 01:04:17,486 --> 01:04:19,116 There's a whole bunch of example applications. 1131 01:04:19,126 --> 01:04:21,099 So there's a lot that you can of get stuck into there. 1132 01:04:21,149 --> 01:04:26,443 So please do come along and check it like our website is electric-sql.com. 1133 01:04:26,853 --> 01:04:28,573 we have a Discord community. 1134 01:04:28,573 --> 01:04:30,373 There's about 2000 developers in there. 1135 01:04:30,433 --> 01:04:31,963 So that's linked from the site. 1136 01:04:32,193 --> 01:04:34,341 we're on GitHub at, Electric SQL. 1137 01:04:34,614 --> 01:04:37,174 so you can see the Electric and the PGlite repos there. 1138 01:04:37,647 --> 01:04:39,307 and so those are the kind of the main things. 1139 01:04:39,307 --> 01:04:43,497 And if you're interested, for instance, in building applications, we already 1140 01:04:43,497 --> 01:04:46,517 have a wait list for the new cloud service, and we're starting now to 1141 01:04:46,517 --> 01:04:50,647 work with, some companies to help manually onboard them onto the cloud. 1142 01:04:50,927 --> 01:04:54,264 So if a cloud offering for hosted Electric is important, let us know, 1143 01:04:54,284 --> 01:04:57,474 and there's a pathway there to work with us if you're interested in being 1144 01:04:57,474 --> 01:04:59,084 an early adopter of the cloud product. 1145 01:04:59,554 --> 01:05:02,754 But also just, we spend a whole bunch of time talking to teams 1146 01:05:02,754 --> 01:05:04,194 and people trying to use Electric. 1147 01:05:04,194 --> 01:05:09,124 So our whole goal as a company is to help people be successful building on this. 1148 01:05:09,134 --> 01:05:10,764 And so if you've got questions about. 1149 01:05:11,051 --> 01:05:14,621 how best to approach it, challenges with certain application architecture. 1150 01:05:14,741 --> 01:05:16,941 We're very happy to hop onto a call and chat stuff through. 1151 01:05:16,941 --> 01:05:21,231 So if you come into the Discord channel, say hi and just ask any questions, and 1152 01:05:21,231 --> 01:05:22,721 we're happy to help as much as we can. 1153 01:05:22,974 --> 01:05:23,974 That sounds great. 1154 01:05:24,004 --> 01:05:28,034 Well, I can certainly plus one that anyone who I've interacted with from your 1155 01:05:28,034 --> 01:05:33,744 company has been A, very helpful and B, very, very pleasant to interact with. 1156 01:05:34,204 --> 01:05:38,954 And also at this point, a big thank you to Electric, not just for building what 1157 01:05:38,964 --> 01:05:42,884 you're building, but also for supporting me and helping me build LiveStore. 1158 01:05:42,904 --> 01:05:46,414 You've been sponsoring the project for a little while as well, which I really 1159 01:05:46,414 --> 01:05:51,324 much appreciate, and there's actually a really cool Electric LiveStore syncing 1160 01:05:51,334 --> 01:05:53,304 integration on the horizon as well. 1161 01:05:53,604 --> 01:05:58,921 That might be, some potential topic for a future episode, but I think with 1162 01:05:58,921 --> 01:06:00,701 that, now we've covered a lot of ground. 1163 01:06:00,991 --> 01:06:05,137 James, thank you so much for coming on the podcast, sharing a lot of knowledge 1164 01:06:05,187 --> 01:06:07,267 about Electric and about PGlite. 1165 01:06:07,614 --> 01:06:08,264 thank you so much. 1166 01:06:08,834 --> 01:06:09,034 Yeah. 1167 01:06:09,034 --> 01:06:09,634 Thanks for having me. 1168 01:06:10,837 --> 01:06:13,117 Thank you for listening to the Local First FM podcast. 1169 01:06:13,527 --> 01:06:16,587 If you've enjoyed this episode and haven't done so already, please 1170 01:06:16,597 --> 01:06:18,037 subscribe and leave a review. 1171 01:06:18,407 --> 01:06:20,917 Please also share this episode with your friends and colleagues. 1172 01:06:21,307 --> 01:06:24,537 Spreading the word about this podcast is a great way to support 1173 01:06:24,537 --> 01:06:25,987 it and help me keep it going. 1174 01:06:26,587 --> 01:06:30,767 A special thanks again to Rosicorp and PowerSync for supporting this podcast. 1175 01:06:31,177 --> 01:06:32,227 I'll see you next time