Welcome to Energy Bytes with John Kalfayan and Bobby Neelon. Your essential guide to the intersection of data and energy. This podcast dives deep into the world of energy, shedding light on how data, AI, and technology are revolutionizing this sector. Each episode equips listeners with insights into the most efficient tools and resources, paving the way for a data-driven future in energy. From technical nuances to broader industry trends, Energy Bytes offers an unparalleled perspective on the evolution of the energy industry. Join us as we decode the algorithms of energy, one byte at a time.
Welcome to another episode of Energy Bites. I'm Bobby Neelam and I've got the Rad Dad, John Calvin, over here with me. How's it going? How's it going? How's it going, Bob? And then sitting we got David Yaffe with us from estuary.dev, founder and CEO, and we're really excited to have you here. Thanks for having me. Of course. So, uh, yeah. Like I said, usually we kick this off with maybe a current event or something we saw in the news or on LinkedIn. And I think last night I saw a few people posting about how their, uh, FiveTran bills have, uh, you know, with renewals and everything, are starting to go up 50, 30%um, year over year, and people aren't too pleased about it. So, uh it's becoming a much more saturated uh kind of space, and just uh you know, that obviously going to be pertinent to what we're we're talking about, but curious to get uh your guys' thoughts. I know John, you said you just got a bill, so um Yep. It's, uh, I mean, even when, you know, when you first told me about them and I started playing around and and getting my stuff set up, it's like, "This is really handy," but man, it feels expensive, even though, you know, it's replacing an intern or, you know, a a part-time person for me. But, yeah, it just keeps They keep adding It's like they keep finding new ways to add more data to the existing sources that they can ingest, so that uh my bill is higher, because that's how they do it off of the the MAR. And so, um yeah, as with all things cloud [snorts], this is one of the fun risks of getting locked into a specific service, because everyone has the same general idea of a business model or at least the PE firms do which is, you know, get user growth, hyper user growth really early, regardless of the cost, and then we'll screw everybody once we got them, have them dependent on our stuff, and raise prices when everyone is dropping prices, or, you know, other things like that. So, it's on my short list of things to do is looking into uh some of the alternatives and competitors out there. So, this is a really perfectly timed uh podcast. [snorts] That That is It's a funny first topic for me and I'm I'm very happy to be talking about it, but you know, something that I found was interesting is clouds always lower prices, right? Like GCP, AWS, one of the big things that they do is you're never going to have the awesome storage go up, right? It always goes down and that's just like a fundamental thing. They rely on people storing more data naturally, not like making the unit cost go up and that's a really interesting thing when you look at like vendors in this type of space that are actually increasing their unit cost as there's more competition. It doesn't make much sense to me. And funny enough, we're actually next week we're going to be announcing a decrease in the cost, so you'll probably have heard about that right right after this podcast comes out. That's awesome. Yeah, and it's one of the things I mean like I mean I I I I can't totally crap on Fivetran. I mean I I I picked them and and honestly I I picked them. I knew they were expensive when I picked them, but I mean I, going through I, you know for the point I was at and where I was at with the company, like, they were one of, if not the most, prevalent in the market and, really, at, especially at the time, I think were the best at what they did. I mean I, I kind of crowd sourced it, and I could not find anyone to tell me anything bad about Fivetran and. and by all accounts, they've delivered on everything I needed from them, as far as being reliable, having good customer service and, all and all those things. But again, when I went into it, I knew it was expensive. It was kind of that initial shock like, wait, I'm going to pay you know, potentially twice as much for data ingestion as I'm paying for my warehouse. Um [snorts] but I mean again that that time to delivery, you know, where I could just turn on Fivetran and set up the connections and bing bing bang, data's all all the way across and that part's handled. Now I can focus on the business logic, but now with folks like yourself and Portable and some of these other folks around, I mean like it's it's becoming much more of a commodity and and people are doing it well. Um so it's it's it's going to be an interesting space to follow. Yeah, and that's [clears throat] the one of the risks that people don't do it well, right? Like you You something, you get burned on ingestion and it turns out that's actually part of your production recording. And you can't really make another mistake. So maybe that's one of the reasons that Fivetran's been able to raise their prices. And there's there are some legitimate options out there that they might not be great for certain types of replication. They've also bought one of the better systems out there. HVR was really good for Yeah. for very high scale replication. So There's two reasons there I think at least. And good options are definitely working. Yeah. I mean from my perspective they were I would say the most established one and honestly out of the gate they just had they had the most connectors at the end of the day, right? Like if I don't have to write a custom API or figure out how to write a custom API script in your system, if I could just click at an API key or off you know, off oh off authenticate directly and then pick which tables like that's exactly the user experience. I feel like that they were trying they're selling, right? Is you know, low code streamline integrations. And so that that was ultimately one of the reasons I went with them was because they had everything I needed. But as with all things, time changes things, and so I'm glad there's healthy competition in the space, and I'm very curious to kind of see what you guys are doing and hear, hear a little bit more about, you know, what you guys do and and what kind of separates you and differentiates you guys. Yeah. I'm happy to give a little bit of the background on myself. I was actually mechanical engineer. Um so was thinking about going into a space like like energy. I I ended up ended up at aviation for a little bit working at a I was going to say you were aerospace at one point, yeah. Yeah, exactly. Um and I really liked it but it just it for me long term didn't feel like the right thing. So I ended up getting into tech and working at a couple of tech type firms. Um and so we built That was pretty humble. As an ex-Googler I think that's a [laughter] You worked at some pretty good companies, so that's that's awesome. I'm probably going to pick pick and prod at those along the way as well because my my biggest thing that I get frustrated with our industry and is it's like they look at problems like they're brand new and no one else faces them and it's like IoT, oh how are we going to deal with all these distributed assets and all this data coming in and it's like guys, this is not a unique thing. And maybe the bounding conditions that we have are unique, but the problem in and of itself is not a unique problem and there are other industries and people that are probably either solved it or working on it as we speak and we need to look outside of what we do on a daily basis to see what those things are do you know, what's going on. So the company that I I worked for was um it was we were a business called Invite Media. We sold to Google back in 2010. Became DoubleClick Bid Manager. I left there. Um and so when I left, we had scaled to doing 16 million requests per second. Um [snorts] so like that's pretty intense scale to be able to actually listen to and hold an auction each time looking at like hundreds of variables for each of those each second and do that in an efficient way where you're making money. You're making like you know, hundreds of a cent on each of these auctions. So you can't really spend too much money doing it. So that type of tech I'm sure it it goes into um an oil and gas type problem, too, right? Like similar similar concept maybe not quite as high scale, but I was going to say, that's the thing; like we, will I guarantee you there's a dozen big data talks this year across the industry. But we don't, other than our seismic data (and I think Bobby and I kind of agree on this), is seismic data is the only true, like, big data that we deal with in in the energy space today. It's just so high frequency. It's We worked at a company. We what we were collecting what six what 600 hertz data? Yeah. 5 600. You know, from the field, but that was a very rare occurrence. And most of it was noise that you didn't even need. Like, you just kick it out. Like They Everyone thinks they want 1-second data till you give it to them and they're like, "Actually, I can't open this in my Excel sheet. So, can you, you know, filter it down or or whatever?" And it's like, "So, you don't actually want 1-second data." It's like Yeah, and and the pressure value didn't change, you know, even a PSI over the last 12 hours, but you know, it's okay. Yeah, no, that's that's a whole'nother can, right? Is Is a lot of these tests and and, you know, science things that we do, you're measuring derivatives over very long periods of time that are changing very minutely. And so, it's like, yeah, you you're looking at trends, not individual points. But so, even then, anyway, that's a whole 'nother conversation. But yeah, we really don't have, like, a true, you know, compared to a Netflix, or yeah, like a live ad platform, like, like on Google, where you've got millions and millions of people hitting it every second. Yeah, yeah, neither the velocity nor the volume. Like, I mean, again, cuz again, you're in you're talking about velocity in the sense of and you're also using it to make decisions in real time. I mean, [laughter] right? I mean, it's not just collecting data. There's rare. I mean, I think we're getting to the point where there are applications like weather drilling or completions where people are trying to do autonomous drilling maybe in the next 5, 10, 15 years but as far as really needing to process a ton of data that fast, I mean, we're not there yet. But again, I think bringing this all the way back, people have solved this problem, you know, for different domains. It's a matter of thinking of identifying where those have been solved and where where we can integrate those. Yeah, I'm curious. In that situation, today, because now we're you're past all your I I I don't know. At least I hope they have changed the architecture since you've been there, but I also don't hold my breath on that with any of the big cloud companies. Um But like today, I would think of something like a Kafka broker for something to do something like that, right? But I don't even think Kafka wasn't around back then, right? Yeah, it didn't exist, right? So, that that was a problem, right? We had to figure out We have to build things with off-the-shelf technologies and so we we didn't have anything like that. We built systems for pure stuff like that without you know, like basically Kafka before Kafka type things like that [laughter]. We're using messaging queues and doing stuff like that. However, the second company that I started afterwards was this this one called Arbor and we started that one back in 2014. So, back then Kafka was kind of starting to become and we chose not to use it and to build our own streaming system that was kind of like Kafka. That system was called Gazelle and the reason that my co-founder chose to do that was he thought that Kafka was operationally very challenging to run. Um And it was at the time. I mean, I was going to say That was the consensus. 2017, 2018 and it's like you had to do the whole Zookeeper thing and it was right at least when Confluent was starting to make it a little easier, but early on it was a bear. Yeah, if you want to pay Confluent now, it's easy, but even if you if you just want to go host it yourself, you better have an extra engineer at least to throw at it and make sure that it's like healthy. So, it's not cheap, right? To to run it and and we've been running this this system called Gazelle now for about oh, 2014, 10 years. [snorts] And so that's what our streaming company's based on. We sold that company back in 2016 to a public company called Libratus and ended up staying and this thing is now it's a lot easier to run streaming. What if we didn't just try to go run this against Kafka and turn it into like another streaming system, but we up-leveled it to the point where it looked more like Fivetran and you could create data flows that were truly real-time just with you know, a UI. And so that's what we did. All right, so let's unpack that cuz that that got me excited instead of batch instances. Um Yeah, so tell me tell me a little bit more cuz there we don't necessarily have the volume, but we do have a lot of situations You know, the money or all the cost in energy operations is tied up in the operation itself, right? So, time is truly money when you're, you know, delaying something in the field, whether you're drilling or fracking or flowing the well back, whatever it may be. And so, having that real-time data is incredibly important or especially moving forward for a lot of these folks who are trying to, you know, optimize their operations in as near real-time as they can get, which if we won't even go down the definition of what real-time means, but near to the second, um and so, you know, my at least my Fivetran setup, which I, you know, I don't even know what tier I'm on anymore, is I think every 15 minutes it it does its batches or whatever, but I'm very curious to hear from y'all's end you know, y'all it's on y'all allow streaming or can do real-time streaming replication. Yeah, depending on your destination and and how you how are you using it? Can you hear that like a dog? Yeah, it's all good. Okay. Uh my dog just got crazy. So, depending on how you're you're using it and we can we can stream on millisecond latency, right? End to end because it is truly a streaming service. So, you capture from a source and of course this depends on what your source is, right? You can't do this with a with an API that only publishes data once an hour, but with a database we can capture data in millisecond latency. Um we can package it up and then even run transformations on it. So, we we do have our own transformations. That's another big difference between something like us versus Fivetran. And then and push it into a destination all end to end millisecond latency, so sub-second latency. So, of course that's kind of a bit complex because um there are two ways of pushing to some destinations. You you go either like stream data only stream every single event or you can create a fact table and upsert the data into it. So, we we manage both options. Uh When you do the latter one, it's your beholden to the data warehouse that you're using or the database you're using. Databases can still be in a second latency. Data warehouses usually have seconds at least of latency at that point, but we're not talking about 15 minutes.
Yeah, that's it. No. Yeah, that that was pretty cool. So, what I think that also I talked about like the warehouse and stuff that you're you're loading into brings up another aspect of cost that people often forget. And I'd I'd be curious if you're saying whether it's or say if we're going to focus on Snowflake
so cuz again there there's a cost with ingesting too. Like, like you know, you have to have a compute warehouse on to ingest data into your Snowflake warehouse; is how efficient is, you know, that platform, like say whether it's Fivetran, Estuary, whoever? How efficient is their code at inserting data into Snowflake, you know, using COPY commands or, you know, Snowpark streaming or whatever? I mean I think I've seen whether it's you all or a handful of other people have started leveraging say Snowpark streaming and can insert data much more cost efficiently and timely than some of your competitors. So, curious you know to hear your thoughts on on that side of the the equation too that there's a secondary cost associated just with the warehouse you're you're ingesting into. Yeah, I think it and that secondary cost it's funny. Over and over, we hear that that cost is more expensive than the actual queries that someone's running on their warehouse, which is super surprising. But being in the space for so long, you kind of realize how much effort there is in actually, in actually loading a warehouse in a way that's easy to query, right? So, you kind of a few different options for doing this. Snowpark streaming is a really good option because it enables you to just stream data in and that doesn't wake up the warehouse. There's no real loading cost. It's very low at least from a loading cost perspective. But, you do incur a little bit additional loading cost on on when you query the data. Right? The problem with something like Snowflake streaming is you get one row for every insert, update, and deletion event. So, if you want to query that data, then there's a little bit of burden on the querier to either transform it into a fact table that can be queried easily the way you're expecting, or to structure your queries in a way that's a little bit more cost cost intensive. So, there is a little bit of extra effort on that. We make that available as an option, of course. Um we also make it for people who just want to have their data structured in in a way that they can directly query it. We make it available to just do upserts into the warehouse, too. So, both of those are pretty valid options. Um and I it really depends on the use case. If you If you care about sub second, if you care about saving money, probably streaming is streaming ingestion is a great way to do it. Um if you just want it to be super easy, then a lot of our clients actually choose for us to just upsert. Even within that, like upserting level, um there's there's a huge variation on how you can do it, right? How you can stage files, how you can copy them into it. And we hear that some implementations are are really inefficient. So, those can be extremely expensive to actually push data into the destination system. I won't name vendors cuz I I try not to do that, but we've heard that from a couple of places.
Yeah, but so you're talking there about like you writing directly to a fact table. So, is that a function of some of those transformations y'all do in flight as well? I mean, to to have it structured into like a fact table instead of you know, replicating it, say, exactly as it is at the source or You can do that. Um when I say the fact table, I'm actually talking about replicating exactly like it is at the source. Okay. But, we we we do offer the ability to like just, you know, write a SQL query, take a subset of the data, um keep it up to date as as more of like a a limited fact table, too. Uh yeah. Yeah, exactly.
Yeah, I mean, I think that kind of hits on what John was saying before about, you know, we don't necessarily have to get into what what have what does real time mean, but it again it means different things to different people. I mean, you talk to some accountants and real time could be that I get this report every month, you know, when I need it, you know. But then you got, you know [snorts], I remember we had like an electrical he was an electrical engineer but you know, a guy worked on like PLCs and stuff like that, and for him real time was like microsecond or nanosecond, you know, cuz that's how fast things happen, you know, electrically. Um so it's uh yeah, you've really got to like it I mean it comes down to that business side then understanding the business and what the needs are um to define timeliness or real time or any of that, so. I worked at a company a while ago, which I will also not name, um that it um they struggled to get their sales compensation order in the quarter that they this their sales people were incentivized. So they weren't able to report from last quarter's sales. Um and that's just to tell you how much you made for the previous quarter and setting next targets till like three quarters of the way through the next quarter. I mean, And that was, you know, like that's one month would have been real time for them for sure. It would have been an an amazing achievement, but what we talk about is like yellow second latency, so like tens or hundreds of milliseconds. Really the biggest thing for us is wire time, right? Like when we operated as a cloud service, so data going over the wire usually adds 100 milliseconds of latency in our world. What are y'all What cloud are y'all Are y'all multi-cloud? Are you dedicated to one or Good question. So we are just releasing our multi-cloud deployment. And that's going to be any cloud. Even even like small players. We're using a a service called Pulumi. I'm not sure if you're familiar with it, but we really like it. It and basically what it We're GCP, right? We started in GCP, but we can release those private data flying in AWS, Azure Vulture, you know, whatever um you're you're looking at and and in the region that that a customer sits on, and so that it doesn't have to traverse, you know, regions and all that stuff. Um and that was really important for our customers just cuz they don't they want to make sure that their data is not being lightened during gold with everyone's everyone else's and that we could potentially even deploy into their DPC. Now, so All right, and you said so you said Palumi? It's Palumi, yeah. That's it. Refresh me on Palumi. Is that the one that is it like a almost like a Terraform competitor or That's exactly right, yeah. They just make it really easy for you to like create rules and recipes of like what what I need to accomplish this task and then they'll translate it across all clouds. Okay. So now like awesome. All right, so you've got the multi-cloud coming out. So with that, is that something that you know, if I spun it up you know, whenever that's available, like is it kind of like Snowflake where it pops up and like where do you want to deploy this and I choose Azure or Central you know, blah blah blah. Like will be like that or do I have to work with you all and like then you guys are able to set up an environment you know, in those clouds like how is it is it going to be pretty seamless to that degree or I'm just curious how that engagement looks with you all. Yeah, so day one, when we first release it, you're going to talk to me and then I'll talk to my co-founder and then he'll click some buttons on his computer and I'll say, "Oh crap, this isn't working," and then, you know, hopefully like 20 minutes later it will work. Yeah, you know, hopefully you know, day day 20 or day 50, it will be dropped out in the UI where you can just select it. Sweet. All right, so yeah. I mean, so let's talk about both sides of, you know, you know, S 2 I sits in the middle, you know, but like let's talk about both sides. I mean, what are some of the most common um sources and most common like destinations or sinks? Uh a lot of the sources are databases that we see. Okay. You know, this this can be anything from um Postgres to Oracle to SQL Server, you know, the the main major ones. A lot of NoSQL databases, too, we tend to see. Especially after the Rockset acquisition, we see a lot of DynamoDB right now. Yikes. Yeah. Um it's it's actually good for us, though. Um one of the features that we we have is that we can take NoSQL documents and we can look at all of the an entire collection of NoSQL documents and figure out a schema for it so that we can make it play nicely with um with like a a Snowflake or a warehouse or something like that. So, um that that tends to be pretty pretty nice for actually using that NoSQL data, turning it into value, not having to eat JSON parsing and all that stuff. It's no fun in Yeah, 100%. any language, really. Um but then on the destination side, I I was surprised. I was looking at it the other day and of course warehouses are, you know, one of the biggest ones that we see, but a lot of databases. So, when someone wants to do something truly real-time, um they they tend to push data to a database. Um but the thing I'm most excited about right now for destinations is we are actually just releasing our Kafka compatibility layer. So, you can get Kafka from us well. So, you know, instead of having to So, that means that we can now load a system that can talk to Kafka. So, we're about to have a partnership announcement with a bunch of major real-time analytics companies like ClickHouse and StarTree and and those types of companies that Nice. Yeah. So, that's a very natural fit for our business. Well, that's very cool. Um yeah, what about I mean, I guess one of the I mean, John and I mess around with and then I guess I don't know. We didn't end up using it, I guess, like we had a bunch of iterations on this red velvet project we were working on, but uh have you have you guys messed around with like Postgres like Timescale DB or feeding into that and Yeah. We have customers that use it. We have a TimescaleDB connector. It's a It's a pretty cool one. Yeah, you can definitely use that. Yeah. Yeah, cuz that I mean, that that really helps probably with the ingestion side. I don't like you can you can insert a lot more data that way and then it's also nice for querying downstream for for people as opposed to a traditional OLTP or even data warehouse, so. Yeah, I've been surprised by how well Postgres even works for if if you structure your queries right and you structure your tables right, Postgres works for real-time analytics. We currently power our own real-time reporting in our products using a Postgres database actually. Okay, nice. And uh, yeah, and you know, like the key there is making sure that you're you're not putting tons and tons of very, very wide data that, you know, has a high [clears throat] dimensionality and stuff like that. But as long as you're structuring it in a way that's going to scale with your customer base, it's totally fine. Yeah, and that something, I I, I forget what why I was thinking about it even, like, today. But, um, you know, there there's a guy in oil and gas that John and I are pretty good friends with, Jeff Hughes, and he's always banging the, you know, the desk that all you need is SQL Server. Um. But yeah, I think sometimes where you don't really know what you want, like you said, you get this this high dimensionality and you get, you know, wider tables and bigger like those can be hard for OLTP. But like when, when you're writing your own application or you've got a lot of control over the data that's going into it and the schemas, I mean, when you can write indexes because you're the SME and domain expert on that data that's going in there, I mean, yes, those those can absolutely be very viable options for, you know, very large amounts of data even. Um so it's you know, it's kind of an interesting thing, but I think sometimes where I guess with this whole ELT movement is like I don't know what I want from my you know, data. I just, but I know I can put it all in the Snowflake and store it there cheaply. And then, if I ever need it, then you know, it'll have the power to query out what I, what I need. But when you're building more applications, you know, I'm highly normalizing this data and it's going to be mainly fairly skinny, and you know, it can handle a lot of rows. I mean, I think there's there's a it's very viable to to do things in that way. Yeah. Funny, every business that I've built historically, we we've ended up having to scale data in in some way, right? Like like the last one of our we we used in the teaser box and the teaser is I think it was a million-dollar box that we had to buy just to to run recording for a while and that didn't even scale eventually. So, it's nice to have um this type of option for for reporting just pushing it to a cloud data warehouse and hoping that it it it scales for you and it works really well. But, I think there's still an art to it and a science to it where you don't want to just shove everything there and then run production reporting off of it. There's If you're running the same query over and over and over or a very similar one, you're probably wasting money if you're at a S3 on data warehouse to to serve that as a recording life for you. Yeah, that's funny. Yeah, hi John. No, I was just going to say it's funny cuz that is literally our setup at with Fivetran right now is it's hey, this data is valuable. I don't want to lose it cuz it's coming from, you know, a lot of social channels that like to delete your data after or restrict access to your historic data and all the fun [ __ ] that people who shall not be named like to do Google. Um but, they uh Facebook also or meta whatever the hell they're called now. But, they uh, so yeah, for me, it's one of those things. Well, and like, you don't even realize it, but you go in and you look at, like, Google Analytics. is like the most overwhelming thing to try and understand from a data just if you don't have a data table or a database or anything in front of you and it's just, "Here's the AP, here's the documentation, and here's everything." There's so much in there. And so then it becomes well, do I need all of this [ __ ] Or do I just choose a few of these tables and then it's like you don't know what you need until you start querying it and then it's just this waterfall effect. So, but you know that you at some point like for us at some point we'll have someone who that's all they're going to be doing is all of our data and analytics and stuff. And so, it's uh yeah, it's it's a weird kind of dichotomy cuz I know it's it's not optimized. I know it's not structured how it should be. I'm not even using the DBT transformations on them at the moment, but it it just knowing that it's it's going to be there, right? Like the as an engineer, I'd rather have the data not need it than not have it and need it, of course, which of course I'll be the first one to complain about other engineers wanting that, but um yeah, it's it's a it's a tricky problem. That's what One of the things I was going to ask you, David, is do y'all also have connectors to like API sources like a Google Analytics or, you know, a Meta or any of those like third-party type tools outside of databases? We do. Yeah, we do. Um and we can connect using like a batch load or a streaming mode. So, a lot of those two of those providers don't really offer streaming data. Yeah, there's a few that do like Salesforce and HubSpot both have streaming endpoints where you can take data in real time. But, a lot of companies were having to, you know, run write connectors that are pulling big reports of data and then aggregating it. We've kind of made the system so that it can handle both of those things. Um for your use cases, one interesting thing that I would think about is just storage tiers, right? Like do you want that data all in Snowflake or would you rather just have it in like S3 or Google Cloud Storage or Azure, you know, Azure Blob Storage, whichever you like. going to BigQuery right now. Yeah, and But, I don't know how BigQuery charges for that, but I think it's pretty reasonable, probably. It's pretty Yeah, it's pretty Yeah, I'm also not I'm I'm not hitting it a bunch or really using it all that much other than just for storage at this point. And I you know, like yes, I had and when I first started, I had HubSpot turned on and then a few months into Fivetran, I was like, "Oh, okay. That's where 90%of my data each month comes from and it's mostly junk and I don't need it. So, I'm just going to turn HubSpot off on that one altogether because we're not using it." Um but, that's what I'm talking about, right? Like, you know, HubSpot is a effectively a CRM, but there is just so much data in there about, you know, all the user activities and then all the integrations through GA and all this other stuff that you just start looking at the data, you're like, "What, what? I mean, I understand the app needs this data, but why is it even exposed to the consumer in most cases?" Because I can't even envision a world where someone would need some of this stuff, but [snorts] yeah, that's a tricky Oh, yeah, that's the thing, right? Like, as an engineer, I know it's better to capture more and not need it than not to capture the one stupid thing that I end up needing 2 months from now, you know. Yeah. No, it's it's incredible how much HubSpot actually has all the integrations it can do. I feel like this space, just the how at least all these SaaS systems at this point integrate with each other requires like a college degree that no one has ever gotten. [laughter] Yeah. Well, that's why I think there's a huge market for what you guys are doing, right? Is it's like, "Hey, I understand APIs. I understand data tables and relationships between these tables, but there's so much [ __ ] in here that I just need I don't need to. And the other big problem with most of them, which is why I'm all for services like yours, is that they always change either constantly changing their their authentication stuff or their API or the new you know, there's new data tables that you have to set up a new API like, "No, I don't want to mess with that on a daily basis."Like, that that turns When you have enough of those, that compounds to where it's a full-time job just maintaining and managing it. Just the connections, not the actual data, right? Like. And so, that's that's why I think there's so much value in it; because now we finally have these SaaS software, we have APIs, and you know, the snow Snowflake, Parquet, and all this stuff where people can finally start sharing this data across without having to be a data scientist to go, you know, just handle it and manage it. Yeah. Um so, I definitely agree with you. Um I think, you know, we were talking about the the cost of ingestion. One of the hidden cost of ingestion is just that. Like APIs change all the time. There's not There's no like one integration with any service. I don't care who you are. HubSpot has four APIs, I think, that you can pull data from. Right? [laughter] Like It depends what you want. It depends how you use them. You can get whatever you want, but a lot of them are wrong for certain things. You just really have to spend the time to get it right. Uh but databases are the same way or when you're actually extracting data from a database, you can do it in one of I don't know, five different ways from Oracle. Um and and so they provide things like API layers that you can extract from like LogMiner. But then you can also go all the way down and and look at the the underlying representation of the data within the database itself. And that's, you know, no one publishes how that how that is there. And so when you talk about integrations to extract data from any of these things, they're just not There's not one-size-fits-all, right? Like it really depends on your use case, your scale, your latency requirements, and the technology you choose kind of has to depend on that. So, you know, all of that together, I'm curious to get your thoughts on zero ETL. It's great. No, it's a good thing. It's there. Yeah. No, I mean
Again, I think it's it's it's an it's a almost a paradox because you're selling something that people are paying for, but it's like that's not where the value is. But it's a very hard problem to solve. Like the value is downstream of just moving the data from one spot to the next. Like it's people transforming the data and reporting it and being able to make decisions on it. But you have You know, oftentimes you have to move the data in some way, shape, or form out of all these different systems and into a centralized spot to to analyze it. Um so, anything that makes that easier and again, within reason for a price, makes all the sense. I think that's probably where Snowflake probably got a ton of traction and is still getting traction. You know, when people have a Snowflake data share, there's not many people still like an offer is zero ETL option like that that would just works and that takes three clicks of a button. Um and I don't have to babysit pipelines anymore. Yeah, well even Bobby and I have had that conversation about APIs, right? Like I'm originally as an engineer, I'm like, oh, these are like APIs are fantastic. I can pull all this data from all these places and then it's like well, [ __ ] Now when I've got a bunch of APIs, I've got to go read all the documentation and I've got to maintain any changes that they make to the documentation if they make them to the documentation, which most of the time they don't. They just push the change and then at some point the documentation gets updated. Um but it's just like a complete can where you're just, you know, the instead, just give me access to how you have it structured and let me just copy it and then go do whatever I need to from there. And so just that, the whole I remember when Bobby first told me about that, I was like, holy [ __ ] that makes a lot of sense and APIs are going to die because of this because it is so much easier, right? Instead of me having even just like the act of it, right? It's not that big of a deal. I can pass, you know, I do this all the time now. I pass a output examples j-or API call response into chat GPT with the API documentation tell me to write the API call. But before, you know, you have to go through and make sure you have every that every field, pull out the right names, and all this stuff and like it just becomes, again, it's this compound effect where if it's just one, not that big of a deal. But when you have dozens of them or hundreds of them, that is literally [clears throat] becoming a full-time job of someone that you're paying, normally, a decent amount of money that shouldn't be dealing with kind of menial [ __ ] like this, so to speak, that you can kind of code out of existence, or you guys kind of do, right? And so, it's a...it's one of those things where I feel like...and Bobby is the manager, so he probably has a little bit easier of a time just supplying things. But I'd be very curious to, like you know, be a fly on the wall in of those conversations with with different operators and client you know, just people that want to employ these services and then just to hear all the the responses from as to why this is they shouldn't spend money on this or whatever and it's like the people who are making those decisions have never had to do it. So they have no clue how much time how much it's just a true time suck at the end of the day the maintenance and management of all of it. A lot of what I see is that when people, usually the operators, are choosing to use something like this if, they're locked into the decision. And a lot of the times, the times that something off the shelf isn't chosen to be used is by a big engineering team that just has a lot of resources and wants to solve every problem themselves. And that tends to be you know, really common that especially if you're at a huge enterprise company, right? Like a massive public day traded company that has thousand engineers you can throw one or two engineers at solving your data integration problems and not really think about it because it's really not a not a huge problem for you. Um but I think it really depends on on the business and also the sensitivity of your data and all those things. Yeah. The only reasons that I've seen are around sensitivity of data or if some someone just decided to go build something. [laughter] Yeah, right. Well, even then though like if you build it Bobby and I have this conversation all the time. We've got there's a couple operators that are notorious for like bringing a new technology have them explain how it works and then they go and try and develop them themselves. And but it's one of the things it's like as we were talking about before we got on you go through this the industry goes through these downturns and it's like the guy that built that gets laid off. Then what do you do? Right? Like tools like this where they're SAS where you can move people in and out and it requires a very low level of technical kind of skill if if you need it to be that way. Uh makes so much more sense is way more scalable and And probably cheaper in [laughter] the long run, right? Yeah. I mean, again, I think it's back to that like the value is not necessarily in that that layer, but Right. going to pay a lot more for a data engineer who can build anything close to what Estuary does than I'm going to but I can find a lot of people that can write some SQL and understand the business and they're going to be a lot more affordable and probably wear a few different hats at a smaller company, but you know, the guy that can, you know, write some, you know, you know, not even just Python, but, you know, probably Go or Rust or, you know, something more uh low-level than that is going to be a lot more expensive resource and they're not even the person adding the value that the business needs. Right. I just think too like what a miserable job that that's all you're doing is managing and updating APIs all Like that honestly sounds like a terrible job that I personally would hate. So, having a a software that does it for me like but that's I mean, tying this back, like that's why in my opinion, that's what technology should do. It should automate the [ __ ] that humans are bad at and don't like doing. Hard stop, right? Like or make it easier, make it faster, whatever. And so like that's where AI and these tools make so much sense. Yeah, I sure would be nice if APIs were all documented and they were [laughter] What is this in case fit well together and it was easy for you to plug into a machine to understand what the API looks and just translate it and ingest it automatically. Or plug in GenAI and it come up with like, um, you know, this you land all that stuff; but that is not the current what state of the industry, and you can get like a 60% solution using those types of things. But that last 40%is just as hard no matter whether you have [snorts] GenAI or off-the-shelf tool stuff off-the-shelf stuff, right? Yeah. Unfortunately, we're still at a place that is, you know, it's not any easier due to all the recent technological advancements. They do that stuff and it's just painful. Yeah. Well, it's funny cuz on the AI or on the language model side at least, right? That's a universal problem that whoever solves it first is going to make a [ __ ] ton of money because everyone we talk to is like,"Hey, I want to be able to query all of this numerical, you know, database data in my language model."I'm like,"Well, it's not how they work."[laughter] Like, it's it's not just like a a normal database. They're not made for numbers, all this fun stuff. But, at the same time, then you've got the gen AI side where it's like, I will probably not have to learn SQL because ChatGPT can write SQL very well for me, right? So, it's like it knows how to write the SQL query. Someone will eventually figure out how to connect the two, and that's going to be wild once once that starts happening. True. Have you ever gotten between the SQL query that's like subtly wrong though in a way that, you know, you'd actually have to have a a master's in CS to figure out why it's wrong? That that's the scariest part. No, absolutely. Well, that, or you need to be the subject matter expert that knows that the data is wrong by looking at it; and it could be, you know, some again, something very nuanced that only the SME would would understand, you know, that that's that's wrong in the results, or it's off by, you know, some significant digit. Yeah. Yeah, I always take them with a grain of salt, right? Like, I [laughter] never forget the time that I was asking it to write me an API call, and it gave me the script, but it didn't include a statement for the API key. I'm like, "Hey, aren't you missing the API key?" It's like,"Oh, yeah, you're right."And then it updates it and adds it. Like, you we're not we're not fully automated there yet, but you you still got to be smarter than your than your language model results. Yeah, yeah, definitely. Yeah, definitely. So, David, um earlier you mentioned the don't like I I I would have you considered I writing it to, you know, S3 buckets or um you know, any of those object stores. Um I want to get your thoughts just on the state of the world a lot around like with these open formats of like parquet, delta, you know, iceberg, and so on. And like, I mean, do we really see kind of a agnostic storage layer, you know, being the predominant force going forward? Or do you think that it ends up boomerang back to, you know, kind of what Snowflake or even traditional databases have been where it's a more proprietary um storage format? Um cuz I I I I feel like it almost sounds too good to be true that oh yeah, there's all this interoperability and I can query this layer with Fabric or DuckDB or Snowflake or Databricks. It doesn't really matter cuz then at that point does it really matter? Those all those engines are going to converge and be good enough. Um Google or Microsoft will buy it and yeah, that's another thing, right? Yeah, um, it was actually a leading question that I didn't I forgot to lead into. But the reason that I asked the question was cuz we, um, one of the big differences between us and any other ELT tool out there is that you can actually connect your own cloud storage as an intermediary and then store the data in your own cloud storage. And before you ship it off anyone else, you can have this copy of it and that enables several things, you know, like one example is that you can push to multiple destinations since we have an intermediary layer. Um and you don't have to just do one-to-one. Another is that you can actually create transformations that are based on that as well. But so that was that's kind of our view on it. Uh I think Iceberg is another really really good view of that so you can also just push it directly to Iceberg and Iceberg can maintain a history of your data over time, which is, you know, a really nice means of doing it. And we do have clients that are doing that right now so um Iceberg is it I would just say that Snowflake was probably our number one destination and Iceberg is really starting to take over from that. As time goes on, people want to start even switching from even if they're using Snowflake, they're starting to switch to first pushing data to Iceberg and then having Snowflake query that data as opposed to directly. Just so that they future-proof their data, they can ensure that like if they have another use case in the future, they they can um just seamlessly execute that use case. If Snowflake decides to raise their prices by, you know, some egregious amount, or there's another better technology out there which is totally a possibility at some point, they they're equipped to be able to do that. The other thing that it does is it lowers your ingestion costs, right? It actually has the capability to um since you're pushing data directly to cloud storage, you're not loading you don't actually have to load a a warehouse. You lose a lot of those ingestion costs. No, that's really interesting. Yeah, cuz I mean I was reading the other day um, cuz someone was asking, cuz I think Snowflake has the Fabric um partnership now, and I think someone was asking about, actually one of our former uh podcast guest, JD Long, was asking around about it, like about the performance of Snowflake and kind of like almost like the responsibilities of each side of of the partnership. Um, and about, you know, being able to query one leg from Snowflake and how, you know, how that all works but I think I came across, kind of researching it myself, then was that the, at least, Snowflake-managed Iceberg is they basically get equivalent performance, um, to their native format, which is like, at that point, it's like, why wouldn't I store it, you know, externally? And even if it even in like the you know, the next tier back of like an externally-managed Iceberg, as long as it's managed and not just a wild west, um it seems like they get probably good enough performance. Um so it's like just having that um I I again, to your point, you're not paying for the compute to load it, and then you're also getting similar if not equal query performance. It's It starts to seem like kind of a no-brainer. Yeah, I I 100%agree and you know, everyone wants to own their own data. This is the first step, right? Like it feels like a really good first step to make. Yeah. Now, what do you see in as far as like cuz I mean obviously it's kind of bizarre and I I was I almost wonder if they did it just to spite Snowflake, but like Databricks purchasing Tabular uh who was the maintainer they were the maintainers of Iceberg, correct? Or they um that I work basically. Yeah, yeah, they they were the initial people who created it and so at the $2 billion acquisition happened during the Snowflake Summit, right? Like all that stuff happened together and I think it was because there was a bidding war and and Snowflake was bidding on it and so was Databricks and Databricks just came apparently 4x over the top or something like that that's Oh, wow. what the rumor mill says just to get the thing done so they could announce it at Snowflake Summit. Um I I think it's um well, it's a huge waste of money for marketing. Right? Like [snorts] you know, in no world has spending $2 billion on $1 million in revenue ever been a beneficial thing. But [clears throat]
it it definitely um shows that they're serious. Um De-Delta Lake has always um at least for the last year been able to extract data from uh open source formats, right? Delta Lake has it supports open source formats and so it at some level it's not doing too much except for saying," [snorts]Hey everyone, we're really serious about Iceberg and we really want to make sure that you know, you take us seriously when when you're looking at that and we are open source and stuff." But in my in my opinion in my opinion it wasn't like Delta Lake was is a competitor or a competitive format to Iceberg or is am I misunderstanding the um the gist? Yeah, so Delta um Delta I forget what it's called, but um they supported I believe they supported Iceberg before. And like a year ago they supported Iceberg. But Delta format is one format um and then and and that's the thing that they created, but there's um a an engine that supports for that has support for multiple data lake formats that Databricks supports already and has for like the last 3 years. Good. Yeah, cuz I don't know why I thought they were, cuz I was like basically either saying,"Hey, you know, that Delta Lake is not as good as, you know, as Iceberg,"or B, I think it was just I mean, either that or they were they were worried about the viability of Iceberg and letting Snowflake, you know, own that and and be able to drive that. Um would be a be a threat to their to their business, but um cuz I mean, I feel like Databricks and Snowflake are converging on the same platform but from different directions. You know, I mean, Snowflake started off as a data warehouse and is adding all the data engineering and, you know, their Spark version of Spark and uh data science. And then, obviously, those are the roots of Databricks and then they've moved into the uh data warehousing um side of the world. But I think cuz even what Tabular is a is a catalog as well, right? Isn't Wasn't that a big big deal with it? Um and that's what Databricks already have with the Unity catalog. So, it was just an interesting I almost duplicate, you know, technologies to your point like to go spend $2 million on something that you kind of sort of already had. Yeah, it's um at least they're serious when they when they do something. That's uh that's an impressive thing. But they've been acquiring a lot, right? They've also they've acquired um in the ingestion side of things. They bought a bought a company called Arcion over a year ago. Um so, that's, you know, another interesting one that I'm seeing that we're going to make them as much as just a native Is that the genesis of this recent announcement of like say the SQL Server replication and stuff that they have natively that that's basically them weaving the Arcion stuff into into Databricks? Yeah, that's been a year coming, right? Like I'm I'm surprised they didn't announce it a little earlier, but you know, these things are hard, so. Tech debt. [laughter] Yeah, integrating a company That sounds like the most miserable thing I could possibly imagine is trying to integrate two completely different code bases. I've done it three times. [laughter] Through three different acquisitions. It's never fun. I mean, that's why I going to say Bobby's getting ready ready to do it. [laughter] Yeah, yeah, we're we're going through it, but but I mean, that's kind of almost to John's what a part of your your LinkedIn headline of that startup masochist. Mhm. Yeah, that's definitely in the the masochist category for sure. No, I mean, I was manually tagging metadata to all of our sources this week. So, it it's it's a real thing. [laughter] It's one of those where it's like you you got to do it when you got to do it, but thinking about doing it if you don't have to do it sounds terrible. [laughter] All the power to you, though, cuz I know what that feels like, right? Like, turns out every problem is your problem when you start a startup. Yeah, no, we're doing For bad [clears throat]. So, Dave, can you tell us more about the transformation layer in Estuary? And then like is there any even maybe open where like people are able to source control any of that stuff at all? I don't know if it's a thing or not or if it's just kind of native to the platform. And then just how do you all play you see even with the a DBT or those types of tools or what do you see in that in that space? I'm going to get even more specific, Dave, since you've got the the Google background. Why can y'all if So, this is one of the reasons I was looking at DBT, right? Or Fivetran plus DBT was because one of the big data sources for us being on the media podcast side is GA all the podcast platforms all the social channels. But, of course, all of them have completely different terminology for the same [ __ ] thing or new things that other ones don't have. And so, this whole normalization layer ETL layer is important. But, what I want to know is like is there a way or maybe you guys have it, maybe somebody else does. But do you know of anybody that has a truly normalized "Hey, here's all of the social media sources and we're going to normalize those into one thing that you can use across all of them"instead of having to do it yourself? Because I think that there's that would be a very good money maker if someone can figure that out. Cuz it's a huge problem. talk about that, but I never seen an actual product that, you know, emerged and and just solved that problem. It's a great idea, right? Like, there's so many businesses that kind of sit on the edges of that that you think would have done it already [clears throat], right? I used to work for a company called LiveRamp, and LiveRamp one of their core value adds is taking an offline ID and connecting it to an online ID system and like profiles across the internet. Why not do that across social media platforms? So, you can like have a unified view of your data across social media platforms. Someone should be able to do something like that. And you're you're definitely going to ask for something that complex or ask for just the mapping of names, right? Like, that's a single file that someone could produce. So, um I'll keep an eye out and I'll let you know, but so far nothing came up. To answer Bobby's question, we do a bunch of stuff in this space. We have our own transformations. We also work with your your warehouse, and so we can do things like fire off DBT jobs using a DBT job trigger. We can do all that type of stuff. Um but that's not really novel. That's, you know, just the same things whatever everyone else is doing. It's great for analytics. Um it's not great for like real-time streaming data, and a lot of our customers use our transformations when they care about like really low latency or in-pipeline mapping data um so that they're joining two things or they're they're mapping fields or they're changing the name of something before ever enters your data warehouse. Uh so, that's what people use us for. We have the capability to write SQL or TypeScript. Um you know, TypeScript's a really weird choice. Don't ask me. Well, that was going to be There's plenty people that know enough JavaScript to get by. Yeah, it's it's interesting industries. It turns out that like a lot of people in crypto use use that, which is random. We didn't know that when we made it. But we're going to have Python, too. In any case, the transformations that you can do in our system, think about them as a little bit more complex, a little bit less simple to set up. But super powerful and they're literally millisecond latency. I can join two streams with millisecond latency and get and get a result out that's that's a fully joined up thing. It looks more like a materialized view. It's really good for filtering. It's good for aggregating, for summing, for doing stuff like that. And so if you you have to invest a little bit more time, but it's powerful if you care about that little latency. I want to make a statement that every frac company out there should be looking at these guys and normalizing their data before it gets into the stupid database instead of naming it, you know, FR1. You can standardize your naming conventions, which will solve so many of your problems. And then you can also share the data across to other people or the operator or whatever because, it like that's ultimately so David, you know, we were talking about the just the industry and kind of the energy tech stuff before; but, you know, the one of the biggest issues we really have on the data side is that the oil and gas company, the operator, owns the data. They generate next to none of it. So, the data is always generated by somebody else. So, it's a service company and they all have their own naming conventions, formats, structures, time, you know, frequency, all this stuff. And then they get So, all of them send it to Bobby essentially and then Bobby has to just figure it out. Um you know, they Bobby has Bobby You can put rules and stuff around how it's structured and all of that, but generally speaking, every but there's not a lot of normalization around it. And so, tools like this or even like the Snowflake parquet stuff, right? Like that's really starting to take off in our industry just because there is so much data that is valuable when multiple people have access to it. But they still want to own the root of the data, right? And so, having these things is so can streamline so much of this stuff and overcome a lot of these problems that I feel like most people have been making excuses for historically around why we don't share data and why we don't integrate data and all this stuff. It's like these tools exist, they're very useful, they're very functional, and we should be using them. Are things like Snowflake data shares something that companies in your space would look at? Yeah, I mean and some people are, the more progressive people are. I mean, again, we had Brian Weaver on from Liberty Frac and, and they're probably the most advanced, you know, frac company out there, but as far as technology downstream of the actual fracs...but, you mean they, they have a Snowflake data share that they can offer, um, to companies, but otherwise it's still shipping around CSVs and emails. And again, even from one stage to the next, the order of the columns could be different, or someone swapped a channel and this channel actually isn't that anymore. I mean, it's just it's a mess.
Yeah, it's CSV files. The way of the world still in 2024 and stuff. Well, I mean I'm I'm I'm glad if I get a CSV. It's usually like some bizarre formatted Excel or PDF, but Or an ASCII. Yeah.
Um, so, I mean, like, with that, I mean, I know y'all talked maybe ahead of us getting started here; but, and when we talked before, David, did you say do y'all have like a, a handful of energy clients? Or, um, I mean, you may not be able to name them, or maybe you can, but just curious: what some of the use cases look like that you're aware of. Most of them what we see is is like database change data capture. So it's it's really like taking their data and getting into Snowflake for a company in energy. Um so that's that's really it so far that we're seeing. Sometimes a few transformations. Usually what we see with customers is that they start off doing the basic thing and then they're like they start realizing oh well, we can actually add real-time transformations to this. This is interesting for a new use case that we have and our our relationship tends to expand based on that. Nice. One other thing that I've seen is that like companies have usually choose two two sides of a business, right? One is an operational data side of the business and the second is an analytics side. We usually start on one of the other and then get roped into the other side. And that's kind of like my favorite way of running the relationship if we can expand into the other side of the business where like the first team that we bring in doesn't care about any of our local value ads. They they bring us in because they're like oh, we have some HubSpot data that we want in our Snowflake. And then later on um someone on their team mentions in a meeting like oh, we want real-time you know, a real-time view of this this data and do something with it. And so they're like oh, wait, I think that during the demo they showed me we could do that. So that's that's like my favorite.
That's awesome. Um yeah, I mean that's I mean even for us it's it's the biggest thing has been have 89%of it's how do I get data out of SQL Server? Yeah, and that's where all the pretty much every major [laughter] um software vendor in oil and gas uses SQL Server. So um it's just good chance that all the things that most people care about come out of there and then it's like if you can connect to the office um suite, you know, through like SharePoint and OneDrive, then you've captured another, you know, 90% of it. And then if you can go through and catch stuff off of a file share file share and SFTP, then you know, you're about 100%of the way there. So I was just going to throw out the FTP Bobby, I'm glad you covered it. [laughter] Don't forget it. Awesome. Well, then I think we're we're pretty good in the oil and gas space. Yeah. Um I think one last thing and then we'll probably kick over into probably the um kind of speed round whatever, but um what are your thoughts are? I mean yeah, I hear a lot about um damn it, now that term's escaping me. Data virtualization. Um have you seen that, you know, kind of in practice? I mean obviously kind of it skirts what you guys do, but you know, what have been problems you've seen that people have with that and probably why they end up coming to you all. Um and using something what that is first to us all.
So maybe Yeah, go ahead. Um we we you know, I think a lot of people if they're trying to do that, they're you kind of have to be able to combine data from a bunch of different sources in one, right? Like that's the the principle of it. You're you're basically taking data from disparate sources and combining them all in one and being able to use that in a single in a single place. It's almost like one big table, right? Am I getting that right? I think, yeah, I mean I, I mean, I guess I've been a little, you know, kind of hazy on on it in general, just cuz like I feel like you're So, cuz so, see, one of the reasons that, say, for us that I want all my data coming into a Snowflake or a separate data warehouse, even if it was a separate SQL server from all the different systems, is that I want that taking on the analytical burden, letting those transactional systems do what they do. Um, and I think I'm just curious like how, like say the, the data virtualization works where, like say, a Trino or something like that, where I mean it's when you're doing that like it's isn't it essentially running queries against all the different systems and combining them in kind of a middle layer? So you're still putting that, really, that analytical burden back on those source systems, which is kind of reason not to do it in the first place, but. That's exactly right. So yeah, like a Trino that's exactly what they're doing. They're using like a usually like an ODBC driver to actually issue queries their source systems which is kind of like a big no-no in production systems. Um you don't really want to like production reporting Postgres database or SQL Server database and and you know, impact production and have someone be able to hit that query with who knows what. Um So and lose a transaction while they're doing it or something like that. Um So I think that's one of the risks there. You can do it in a way that's a little bit safer by having like read replicas and only hitting read replicas or making sure that you're um but but that doesn't really scale, right? The whole point of of doing change data capture or offloading um reads from a database is that that way you can um you know throw a much bigger and purpose-built system for that at it and you know not impact production at all. Um so, you know, I think we see that. There are some really good businesses that are that are built on that. Like I think Starburst is a good example of a business that they they are the company that they can pull out this stuff of Trino. Um and they they make it so it's it's easy to do data virtualization across a lot of different sources. But even in those businesses, you know, like we have a partnership with Starburst, for instance, in which when their when their clients don't want to extract data or they don't want to pull [clears throat] data and issue queries directly to the source tables, they can extract it, putting it into Starburst Galaxy, which is their cloud offering product. Actually centralizing the data so they can query it there, too. So, both methods are like kind of good for different different options. Um and I think, you know, there there it's not like you're going to just see one or the other. But I'm curious to get your thoughts. Yeah, I mean I've heard people talk about it cuz then that cuz it almost goes to a zero ETL type thing in a way, right? I guess it's that virtualization layer that's doing the the transformation and the and the providing the data out and you don't really have to move the data around. So, you kind of eliminate that data moving cost. But I again I think it's there's no such thing as a free lunch, right? You know, you someone's got to uh pay pay the pay the piper. Um And I mean I guess I guess one last thing then like speak to how Estuary like handles querying like hitting how you all can hit production databases and do it with minimizing the impacts. Cuz I don't I mean I have some people think about like I still got to get, you know, if when I first want to seed my database, I still got to load 1.5 billion rows out of this table into into Snowflake. How do I get it out of there and how how can Estuary do do without crashing prod? Yeah, that's a good question. So, we have a whole method to do this, right? Like this is uh is something we spent a lot of a lot of time on. Um there's kind of two two things that you have to be thinking about when you're extracting data from a database, ongoing rights and ongoing reads. And then also backfilling all of your historical data, right? So, those are two separate concerns. There's a lot of tactics to do it. We've chosen to do it in a the backfills are incremental. So, we figure out a way of querying the database in its natural order. The natural order is important so you don't have to order anything. Ordering is an expensive operation. So, we query the database in its natural order in incremental chunks. And then we're able to at the same time read the write-ahead log so or the redo log or whatever the database supports and be able to see like all the changes that are happening. And once we get, you know, to the exact place that we started reading the write-ahead log, that's a trick to do that. It's not easy. We stop reading the database. We stitch the two together and we have the our complete view. So, that's that's exactly what we did um for this problem. Um it's not simple, but like the key things there are incrementally doing it so making sure you're not trying to do a huge backfill query at the same time. Um doing it in the natural order of the database so that you're not having to like impact the database when you're doing that backfill query. And then also being able to atomically stitch the write-ahead log together with with your historical data. Nice. Yeah, cuz that that you know, even though they were fighting that was I mean they did a pretty good job of it, too. I'll you know, give credit where credit's due, but it's just but I think they would even talk about like how they're much more friendly to databases than some of their competitors say even at the time. You know, where some of these competitors like, yeah, it technically does what you want it to do, but you know, your your sysadmin or the people you know, your accountants are going to hate you for a day and a half while it reads out of the ERP. Um Yeah, and some of the you know, when we when we when we query that suite when we do our backfill on that suite, there is no kind way to to do that, right? Like you're going to have to cause some impact on that suite to do it just cuz that's the way the system's designed. But in general, no one really cares cuz they just need to acquire it anyways. Yeah. Cool, Johnny, you want to kick off the speed round? Yeah, I've got I've got one right off the bat because you're a Googler and you worked on the ad side. So, what's your favorite kind of maybe well-known or unknown Google tool or GCP tool that that people should be that or that you enjoyed using or liked or found value in. The answer can also be none. Um I would probably say like internal Google tools that I thought were cool that we used that, you know, like were interesting. So, there was one called Trips that someone tried to actually some Google person tried to create a business out of. I think they might have They might not have bought it. I haven't tracked it, but that basically was just like this really great way of tracking your travel within Google and then crediting you and also rewarding you for like choosing tickets that weren't out that expensive. So, what they did was they looked at like the historical average for prices between those two destinations. If you're below it, they'd actually give that to you and you'd be able to use it towards your next ticket. Oh, that's cool. Um Yeah, and so it was it was a really great way of doing it. It made travel like turn into a game where you it would mostly force you into not taking business class, but then you did [clears throat], you felt really good about it. [laughter] It's like the unlimited vacation policy where it's like, oh, well, can I take vacation even though I have an unlimited vacation? How much vacation can I take? Yeah, that's cool that they like they gamified it. That's smart. Yeah, it was really neat. All right, so outside of Ben Rogajan, who's your favorite social media follow? Good question. Um Wow, I I've been saying So, Ethan and Aaron worked in my last company. Okay. Nice. Because of that I have to say Ethan and Aaron. He was our team um and you know, I've known him for a long time. So, you guys have some history you know, and they're now playing in the same space kind of. Yeah, yeah. Um we we actually, you know, have partnership too where if you want a long tail SaaS uh SaaS connector, we're going to recommend them. If you want a database, they're probably going to recommend us. Okay, nice. That's good that's good to think of cuz I mean it Give him credit where credit is due. I I had a 30-minute call with him and he asked about like two database connectors that I, you know, wanted that he maybe he didn't already have and by the time I was done with the call, he had a prototype up that I could log in and try to [snorts] do it. It was the most bizarre thing I've ever seen. Like I mean, I don't know how they've got it structured in the back end but I mean, he found like the API endpoints or I sent him the documentation. He said,"Yeah, try this out."And like they weren't perfect yet, but they freaking worked with it and it was like I didn't get off the call and I had something in my hand. It was it was kind of bizarre. Yeah, and he's good at that. That's what they focused all of their energy on, right? So, yeah. And I think they had a whole system where, you know, it can figure out the type of law that you're using and all the other things that it probably influence. Once you've done a hundred of those, the next hundred isn't and you've done it intelligently. Yeah. The next hundred isn't as hard. So, am I assuming that people find you at a Lowkey Data Happy Hour in in New York then or They can, yeah. I got I'm actually about to have a my first child in the next month, so Oh, okay. probably won't find me in a little bit. Yeah. Congratulations. after. Yeah, congrats. Appreciate it. I'm not going to sleep for a while, but we'll see. Yeah. [laughter] You're acting like you you've been sleeping. You've been at like three or four startups. That's You're right [laughter]. I've learned not to sleep so perfect. Oh, that's the masochist part, right? Like that's what I'm saying. I got my master's and then we were like,"Hey, maybe we can start trying to have a kid."And of course, basically the first time halfway through the master's, we have my daughter, which was supposed to be born after I was graduated. And so, it's like,"Oh, yeah, working at a growth company with a master's and a child. Why why the hell not? You know, it's I think it's just inherent in people that like to build stuff. Just like like to be busy. Um what about either I'll give you a little option here. Either book or library. Python library. Open source library. It doesn't have to be Python. I definitely go with the book probably, I think. Um I just was reading a lot of fiction recently. Um I So I read a book called So I used to be in this golden age of Arctic exploration. Are you familiar with it? It's you know, a period in that the late 1800s and early 1900s in which like the British Empire just like and all of the empires tried to Map the world. Map the world, yeah. Explore the Northwest Passage and find the North Pole and all that stuff. So there's a book called The Terror that is a fictional book about around the same time period that I just finished and that was It was good. I would recommend it. Nice. You got one more, Bobs. Uh yeah, we're I mean well, you're about to have a baby, so you probably won't go on one for a little while, but where's your favorite place to go on vacation? What or type of Yeah, vacation, beach, mountains, you know, whatever else. I'm I'm more in like the adventure style of vacation. My wife is probably more in the beach side, so we end up I end up splitting the baby a little bit, so to speak. But we tend to go Telluride, Colorado a good amount. I really love that the place. It's just a beautiful place in the US. I love that area. We ski there. We I go hiking. I just got back last week, actually. Oh, beautiful. Yeah, I've heard great things. Yeah, I would recommend it for sure. Awesome. Well, that hour flew by. Dave, we appreciate you joining us. Where can How do people find you? Either personally or professionally? Yeah, so our website is Estuary. S T U A R Y dot gang. You can find me at the same website. That's And you know, we'd love to hear from you. We have a we have slack. We have you know, you can contact me with any any medium basically and I will be there. I will respond. Nice. Bobby, you know what next week is? Uh, Calgary Energy Tech Night? Nope. What am I missing? Oh, the Hogs. Kickoff, baby. College football starts next week. They're they're they're still undefeated, John. We're undefeated, baby. This is the season, [laughter] Bobby. We're undefeated. It's all good. Hofstra has been undefeated for the last uh 15 years now, so cuz we haven't played a game. Hopefully we beat our Thursday night cupcake uh opener next week, but as of the time of this uh podcast dropping, the the Hogs are undefeated and are on track to uh win the national title this year, so I'm rooting with you. Fun stuff, but again, thanks David. We appreciate it. Guys, if you like uh what we're doing, what you heard, go follow David. Um please leave us a a like, give us a comment. Please make sure to subscribe to the channel. Also, uh jump on Collide. We're getting ready to uh debut the the new language model here in a couple weeks. So, uh make sure you jump on there. We also just had a couple AMAs and live streams that are have been really good. So, check those out. We appreciate you guys. We'll see you all next time. Thanks.