Subscribe
Share
Share
Embed
Tractable is a podcast for engineering leaders to talk about the hardest technical problems their orgs are tackling — whether that's scaling products to deal with increased demand, racing towards releases, or pivoting the technical stack to better cater to a new landscape of challenges. Each tractable podcast is an in-depth exploration of how the core technology underlying the world's fastest growing companies is built and iterated on.
Tractable is hosted by Kshitij Grover, co-founder and CTO at Orb. Orb is the modern pricing platform which solves your billing needs, from seats to consumption and everything in between.
Kshitij Grover [00:00:04]:
Hey, everyone,I'm Kshitij, welcome to another episode of Tractable. I'm the cofounder and CTO and Orb. And today, I have Neil Patel joining me, who's the cofounder and CEO of Axiom. Axiom is a new type of logging tool built with modern cloud primitives, meaning it's serverless and works really well even at massive scale. Axiom is used by a ton of great companies like Prisma, M1 Finance, and Cal.com. I'm super into the logging space, and I think there's gonna be a bunch of technical stuff to talk about.
Kshitij Grover [00:00:32]:
So we're really excited to have you here, Neil.
Neil Jagdish Patel [00:00:34]:
Hey, great to be on. Thanks for having me.
Kshitij Grover [00:00:37]:
Of course. Let's get into it. I want to give you a chance to talk a little bit about the general product shape of Axiom. So why don't you give us and the audience, kind of an explanation of what Axiom is and what the kind of use cases it serves?
Neil Jagdish Patel [00:00:53]:
Axiom is essentially a store for 100% of your event data across many different needs. So I think you head on logging, observability as well, we are known for more and more security, product analytics, and other insights which you'd like to gain across these kinds of data is where we find a home. The reason for that is is we've basically built a data store from scratch which is as you mentioned, using kind of cloud native technologies like serverless, object storage, etcetera. But really, it's tuned specifically for events. And so that means any kind of key value pairing, which one of them has time. Instead of worrying about, for instance, data that would be correct for, let's say, Snowflake or Postgres or anything like that, this is hyper specialized for events, and therefore, we can do some really neat stuff and allow some core use cases on top.
Neil Jagdish Patel [00:01:47]:
One of them is being able to do things that you would generally do in a logging tool or event management tool in terms of querying, being able to manipulate those during queries. So schema on read, those kinds of things, dashboarding, monitoring, etca. And then also with that, we also have the ability to do the pipelining side. So actually processing of events and having them output somewhere else. And so it's mixing those two things together in one place.
Kshitij Grover [00:02:15]:
That makes sense, and when I go to your website, Axiom, the first thing I see on the homepage is this idea of stop sampling, observe results.
Kshitij Grover [00:02:24]:
So talk a little bit about that product philosophy and why that's so important to I'm assuming, especially your target audience, but maybe more broadly like that's almost a reactionary message, so speak a little bit about it.
Neil Jagdish Patel [00:02:39]:
I think we're still the only company that actually asks people to stop sampling instead of giving all the different ways you can do sampling. We always felt even in the products we built previously, anytime you were losing data, and especially when it was to do with cost of all reasons, you're putting yourself in a position where you essentially have like Swiss cheese of history, essentially, like tons of holes, tons of gaps. And we always felt like you'd wanna lean more heavily onto the ability to do analytics insights, those kinds of things, forecasting more of the ML side, etcetera. And this is before the kind of AI stuff took off. But I'm just talking about stuff that was available, but maybe not easy for people to use or developers to engage with. We felt like coming at it with an angle where essentially, if you keep everything, you have history. If history is there, you can learn from it and do something interesting in the future.
Neil Jagdish Patel [00:03:37]:
And so we went about just trying to solve that problem and probably hit every issue on the way there. But as we got to somewhere where we can actually turn around and say, "Hey, stop sampling, and it's not gonna completely bankrupt you."
Kshitij Grover [00:03:52]:
Makes a lot of sense, one thing I'm wondering about is this logging and events and metrics, there are a lot of different terms here, and these products are built for a lot of different audiences. My background, I worked at Asana. Asana made heavy use of Datadog and had some logging and metrics set up there, but also made heavy use of Interana, which I believe maybe is now rebranded but it was originally inspired by Scuba. And talk a little bit about who Axiom is built for. Is it the engineer that is piping request logs? Is it someone in biz ops, or even a different audience? How do you think about that?
Neil Jagdish Patel [00:04:31]:
So that's a great question. I think the way we see it is and this may be because we have databases in mind too much, is essentially those are specializations on top of the actual store. And so what happens is when you're a startup and you're sending data to segment and then Mixpanel, whether you're sending data to Datadog, or whether you're generating data from, let's say, Zendesk or something or plain or et cetera. The problem that you have is you're basically creating and then dispatching into silos. And those silos might be specialized on the actual data, but what you've done is essentially sent everything to there, and then you pay them monthly to lease back that data from them. And so the thinking on our end is it there's it doesn't have to Axiom doesn't have to be the answer where the biz ops person comes in or a data scientist comes in and uses it. It's more in terms of for those specializations, can you at least have one source of truth where
Kshitij Grover [00:05:31]:
Yeah.
Neil Jagdish Patel [00:05:32]:
The ownership is with the organization that generated the event, and then there can be specializations on top, which actually go and do they solve a problem for you. And so there's a bunch of startups who are building on top of Axiom at the moment, and I'm hoping for a summer release schedule, so I can shout about them a little bit more. But, essentially, what they've decided is they wanna offer that to their customers where they're saying, hey. We're gonna go solve this security problem or finance problem or whatever, but we're gonna actually hold your data in Axiom because you have direct access to it, and you may wanna do something else with it.
Kshitij Grover [00:06:09]:
That's really interesting, and I'm actually curious about the the product thesis there. So to reflect that back at you, it sounds like Axiom's job is to be a really good source of truth, really good data holder. And it sounds like what you're saying is it's not as important to maybe in the current phase of the company to directly address all of the different workflow needs. You think that might be a better place to leverage integrations or partnerships? It sounds like the kind of core IP and the technical innovation of the company is in really fast access to this data and durable storage of the data. Is that right?
Neil Jagdish Patel [00:06:45]:
So that's one large part of it. And then the other side of it is what we call Flows, which is essentially if you think about the data store essentially bringing data at rest, essentially bringing it to rest, Flows brings it back into motion again. And so, if you're thinking about the building blocks of any company who actually has an event store as its base, these are all the things you need to go and achieve. For instance, being able to build models or be able to export into the third party so that can do something, or even simply to run long running reports and things like that. And so it's that mixture of storage and such with flows to bring it back up into that motion piece and do something with it. And so it feels like more like for us where we're building a platform, and that platform has various kind of tooling, which then you can bring together. Now as a user you can bring it together for the things you're trying to solve. Absolutely.
Neil Jagdish Patel [00:07:49]:
And that's obviously the first customer and the first people that we wanna make happy. But then there could be a security product that's trying to build the next gen SIM, and they're looking at Axiom and saying, but you already speak events, and so why don't I just use you and focus on the actual what the users would care about?
Kshitij Grover [00:08:09]:
That's interesting. And I like that because it seems like a central part of that story is that Axiom is still directly accessible to you in this sort of really nice user interface, which maybe is different than just selling a pure time series database. It's a product, not a database. And of course, maybe the the core underlying technology is a database. But it's not something where it's completely wrapped by another product. It's more like you plug it into a bunch of different layers.
Neil Jagdish Patel [00:08:42]:
And that's how those these partnerships, that's what they will show more clearly in exposing because this hasn't you can kinda do it with something like Snowflake, et cetera, but it hasn't really come to the masses in a way which actually is easy to understand and see where the value is. But we really think the value is every startup should actually have a complete history of everything that they've done because that's gonna be useful for the future and many reasons now as well. But it's also this idea where it's one writer, many readers. And taking that philosophy and putting it onto the storage layer where we feel there's more happening and more kinds of tooling you use, which is actually using event data than there is the other kinds of data. And therefore, probably gonna get more value out of this over time.
Kshitij Grover [00:09:34]:
I wanna back up for a second and just talk about the sort of data that people are putting in the Axiom and maybe even you can explain what the different steps you need to start using Axiom are. So how do you get data in, What sort of data does that typically look like? And then what can you do with it? You already touched on this a little bit, but I think that that's a that's good explanation to have. So then we can talk a little bit about the technical internals.
Neil Jagdish Patel [00:09:59]:
We have various methods of ingest, and so some of them are, you could classify them for a specific use case. So for instance, we support Elastic's bulk API. We support, other kind of ndjson, those kinds of APIs where if you have a log shipper or event shipper, it's because you're using Logstash or Cribble or something like that. You can just hook it up to Axiom and say, from now on, send everything there. And so the kind of data you'd be getting is from Kubernetes. You'd be grabbing data. From firewalls, you'd be getting application data, trace data. All those kinds of things can land inside of Axiom.
Neil Jagdish Patel [00:10:36]:
Axiom doesn't know what it is. It just sees that it's essentially adjacent and it has a time stamp. And so that in the middle, you almost, it loses that knowledge of what's happening. So when it comes in, we know, and when we do something within the UI, we know. But, otherwise, in the middle, what the data store sees is just a whole bunch of key values. And essentially, those can be thousands and thousands of them. So super, very wide events. And, essentially, the idea being is you emitted some kind of record, and you want that to be as feature-full as possible of all of the state of what was happening.
Neil Jagdish Patel [00:11:18]:
And so, yeah, the kind of observability streams and logging streams are one of them. The other side is, of course, security data. So a lot of flow logs and things like that. But then also syncing in every Stripe webhook that's existed for your system for your company that comes in for replicating all the segment events or the mix panel events, Google Analytics. So, essentially, what happens is mostly people come in because they see us as logging and that's the quickest and probably the pain that they had which brought them to us. And they're happy. And then a week or two later, you talk to them, and they've grown the number of datasets, and you say, hey.
Neil Jagdish Patel [00:11:58]:
So what have you been doing? And it says, oh, no. We're just sending anything that whenever it happens, we're just sending it over. And it's super cool to see because they're just seeing value where maybe that data would have been lost in a queue or in Kafka or something and gone forever after it was processed. Now they have a history. They're learning from what's actually happening, which is valuable.
Kshitij Grover [00:12:18]:
And one thing that, we're happy users of Axiom at Orb, and one thing that really stuck out to us, which is worth mentioning, is in addition to not having to sample, this idea of a white event is really freeing from a logging perspective. I think the fact that other providers penalize you for cardinality is really tough because it requires you to reason about what things am I going to care about, and also let me make sure that I'm conservative in the sort of tag explosion that might happen in the future. The reason why one thing that's been great about Axiom is this idea that you can log something like the exact API path resource ID and all, and you don't have to think about the fact that this is gonna be a big cardinality, dimension and instead, you can query over all of that.
Kshitij Grover [00:13:10]:
And I just I imagine that's a big reason why people can't use other tools even if they wanted to for something like this.
Neil Jagdish Patel [00:13:17]:
The way we think about that is for you to send something to Axiom should require the minimum as minimal thought as possible. So it's schemaless, so there's no there's nothing to manage in terms of telling us before you send us anything. High cardinality or it just doesn't matter what the cardinality is essentially. So that's great. And then the idea of essentially overthinking the shape of the event before you send it. We have customers who accidentally put SQL queries as the key of the event, and that's not great.
Neil Jagdish Patel [00:13:53]:
Outside of that, overthinking it on the send side would just add more layers before you're actually doing something. The idea is just not having to think about it too much. And then what we did instead was really focus hard on the query side. So the schema on read side, but also just the tooling you need to extract value from messy events. And so it's a combination of you may be sending highly structured data, but we're all human, and the reality is inside of that highly structured, there's probably something very unstructured. And, how do you bring those worlds together in a flexible way?
Kshitij Grover [00:14:29]:
Yeah. And it's a little bit counterintuitive because I think that the temptation is on the technical side to think that there's some trade off somewhere. If you're giving a lot of flexibility at right time, and maybe 10 years ago or something the way you would have architected this or thought about this is, okay, but you have to have some stream processing job that is constantly doing these roll ups in stream.
Kshitij Grover [00:14:51]:
And at the end of the day, you're not actually storing all of the data. You're just storing some rolled up pre aggregated version of it. And what that meant was you had to, as you're saying, think about what you cared about in the data as you were sending it, or at least as you were configuring it. And so the temptation to me feels okay. If Axiom is giving me all of this ability to log whatever I want, change the scheme over time, not predeclare anything, I must be losing something in the query layer. But as it turns out, Axiom's query layer is both very expressive and also very fast. So where is that gap? How can you do both really fast and really flexible writes, but also really flexible reads? Is that just a function of how things have evolved over the last decade? Where where is that coming from?
Neil Jagdish Patel [00:15:38]:
I think that's a good point. The reality is the gap is always closing between being able to be hyper flexible and or hyper efficient. And I think we're on the right side of that fence because I think it's easier to go from being very flexible and then start to shave off the milliseconds until you get to where you want to be, versus starting from a very strict schema heavy position, and then trying to move across to being flexible. It also comes down to this query language and things like that. And so there's a bunch of stuff that's related. The reality is you wouldn't expect to useAxiom for something where you wanted the query to return within a 100 milliseconds right now. The whole idea being that our architecture is meant for large scale with the minimal lift from somebody before they hit a query.
Neil Jagdish Patel [00:16:33]:
There's no warehouses to create before you run a query
Neil Jagdish Patel [00:16:37]:
There's no sizing or anything like that. You just hit run query, and the right thing happens. And what that means is, though, is the flexibility. You have to lose a few 100 milliseconds to make sure that all happens versus what you could probably hyper optimize on something like ClickHouse if you knew everything up front. I think you'd probably would have both depending on what your output is on the other side. But with in Axiom's case, you'd be more in terms of for everything you care about between monitors, dashboards, and just ad hoc querying, everything is within a time frame where you're happy and you can move forward. But that said, we are having some there is some stuff that we're working on to make. And, again, we see it as slightly different.
Neil Jagdish Patel [00:17:23]:
We're like, "How much of the data do you want under a 100 milliseconds?" Maybe it's just the last hour or the last, 30 days or something. And so there's things we can do to make that make that happen, essentially. So that's the more exciting things to land over the summer.
Kshitij Grover [00:17:37]:
That makes a lot of sense. And I think that's consistent with I feel like an industry wide shift we're seeing on the technical side, which is you have products like WarpStream where they're questioning, how much what performance guarantee do you really need from from your account structure versus the economics of it and maybe the simplicity of the architecture. And I think it's just not clear that we have made this trade off in a super principled way. And so I imagine that a lot of people, they're not looking at their logs very often. They don't care about the difference between their dashboard loading and 10 millisecond queries versus 200.
Neil Jagdish Patel [00:18:14]:
And then they also that's not the bottleneck anyways. They have their minimums 15 or 10 second refresh anyway. You don't know that it's happening realistically. But I think WarpStream's a great example because it's people will build up Kafka clusters. They'll do all the work around it, everything, and they're like, "From in to out is a 100 milliseconds or less."
Neil Jagdish Patel [00:18:35]:
I asked, "Why?" These are all background analytics jobs. Like, what are you doing? And why does it take this team scaling this massive thing and essentially wealth transfer from VCs to AWS to make it happen? But that doesn't make sense. So I think the problem that they're solving and the way they're solving is absolutely right because they're just saying there's a better architecture for that problem.
Kshitij Grover [00:18:58]:
One thing you mentioned early on was this idea that no matter what use case the data is serving, the commonality is it's maybe semi structured but has some time column or or some time series aspect to it. Can you talk a little bit about what are the common technical things you need to solve when you're building a generic time series database? What are the different dimensions of technical challenge? And then maybe we can talk about, how has, Axiom started to to attack each of those individually. But I'm just curious, what are the key kind of technical concerns?
Neil Jagdish Patel [00:19:34]:
I think you'd always break it down. And in my head, I've always broken that down to ingest storage and such. And taking in that much data in an efficient way where you don't know what's coming. So just because of the data set, you don't know a 100% what the event is gonna look like. So optimizations are very, let's say, you have to be an optimistic person, essentially. To write optimizations for something which you're not sure of. Robustness on the ingest pipeline, backup on the ingest pipeline, being able to get that written to something as quickly and efficiently as possible. So I think ingest is super important.
Neil Jagdish Patel [00:20:14]:
And it's actually one of the key value where we found efficiency and Axiom's architecture, the value of the Axiom's architecture can provide is around ingest, actually, and it's not. In other ones are technologies people recognize, but the sheer amount of work we did on ingest is one of the things. The other thing is, of course, storage. What do you need to store the data? Is it is it object store? Are you saying it's object store, but actually it requires some containers to be running or some VMs to be running to keep the data before it's available? What does that look like? And then finally, on the search side or query side, again where is where are searches taking place? Like, how do you construct them? What is the, planning looking like? What is distribution looking as for scale and things like that? So those would be the key 3 key things that you would be looking at.
Kshitij Grover [00:21:13]:
Let's talk a little bit about each of those areas. On the ingest side, I know that as you mentioned, Axiom makes it really easy to ingest from different data sources. The higher order bit being you shouldn't have to think a lot about restructuring the data, if at all about restructuring the data. Is the technical problem there, like, an n + 1 problem where you're having to think about compatibility carefully? Or is it more of a performance and durability once the data is piping in that part of the injection stack?
Neil Jagdish Patel [00:21:49]:
I think most of the time goes into performance, and durability because it's just that's people's events. You're asking them to trust you with that. And so certain patterns emerge for a reason in terms of, let's say, the way Elastic would do ingest. You have separate VMs. Ingest comes in or a request comes in. It needs to be acknowledged by every VM. Recovery is completely banked on the fact that it is actually acknowledged. Because if something goes down, that's the only way, etcetera, etcetera.
Neil Jagdish Patel [00:22:24]:
And so those layers, the cross node connectivity and networking, all of the middle stage of that was some data stores basically turned around and said, actually, it's not our problem. You have to run Kafka in front of us. And if you run Kafka in front of us, we will just be a consumer, and therefore, Kafka will make beautiful batches, and we will basically pull them in. And if we go wrong, it's on Kafka, then we can go grab it from Kafka again. And so that was 4 years ago or 2 years ago, and there's a bunch of data stores that still do that. And then our approach was like, you wanna build in the Kafkaness, but into something which isn't doing with coordination free, so it doesn't need constant acknowledgments from everywhere. But it makes it durable quicker, so you're not keeping things in memory in the same way that you would in in some of the other solutions? And what guarantees are you getting from Kafka, and how does it we don't wanna replicate it, and that's WarpStream's job now. But what can we do for us to make it super easy to do? And so that's where the time and focus went.
Kshitij Grover [00:23:37]:
It sounds like there's a question there of how much responsibility do you take and absorb and how much do you offload to a different thing sitting in front of your solution. But I imagine the more responsibility you offload, the less you get to be in control of the developer ergonomics too, if you're you're enforcing something, then then now all of a sudden, in in order to interact with Axiom, you have to do x, y, and z, which kind of, undermines the whole not having to think about it, which I imagine is still an a very important principle.
Neil Jagdish Patel [00:24:08]:
And, honestly, coming back to not having to think about it, it's insane how many developers know the pricing pages of the tools that they use for logging and
Neil Jagdish Patel [00:24:18]:
Tracing and stuff. They know them off by heart. And one way that I describe it sometimes is there's decisions made at the keyboard before someone even types, which the company will never know is actually happening. Because they're just like, I'm not even gonna get into that. How do you make that low end then? By adding these pieces or giving up responsibility into those, you are just ex I think, yeah, you're putting the weight on the wrong side.
Kshitij Grover [00:24:46]:
Now let's move on to the the second part of this, which is storage. Obviously, this is a key part of building a time series database. I don't believe I think this is something you've talked publicly about, but the fact that you all use S3 as one of these, storage layers. Tell us a little bit about that storage architecture and maybe what trade offs that implies for the product or just how much mileage you've gotten out of using something like S3.
Neil Jagdish Patel [00:25:14]:
We did storage and search on hard mode. So storage was I'll start with that it's only object store. So Axiom works on different clouds, but I use AWS because that's where our cloud is, essentially. So S3 and only S3. So we have one storage layer, and that's it. And so all of your data in Axiom, there's no actual there's no such thing as, like, warm, cold, archived, or warehouse data in Axiom.
Neil Jagdish Patel [00:25:40]:
Everything is hot all the time. So you can basically do a query and check January five years ago or January now. It's all the same to Axiom.
Kshitij Grover [00:25:48]:
Yeah.
Neil Jagdish Patel [00:25:48]:
And so that's like that SSD type of between when you we had hard drives and SSDs that way. But, no, there are no SSDs in the process of Axiom, essentially. Everything events are pulled into what we call blocks, and those blocks are essentially like these hyper optimized batches of events where we've almost prebuilt some caches as well as we've done various kinds of encodings, compression, and the format itself is ours as well. So this is what we've been chipping away at for four and a half years and making better and better since we decided to have this architecture. So we are sensitive enough to how good the S3 team are doing because we can tell when things are not a 100%. But it's one of those things where we were betting on that as the only way forward. Because it it has to be and this architecture exists in different ways. But what we were betting on more than that was you can't take an off the shelf kind of format, and have that as the way you actually put all the events in and do that.
Neil Jagdish Patel [00:26:56]:
And that's two reasons. One is the right side where you want the highest compression that you can get and as much work done up front as possible. Because on the query side, on the read side, we had pushed all and we'll talk about that, but we pushed all the querying to serverless functions. And so you don't have this ability to have fast SSD reading to make something happen. You have to do it from there. That storage layer is critical, and the way we brought that out has been, we're always on it.
Neil Jagdish Patel [00:27:27]:
We're always trying new things out and always trying to make it as efficient as possible.
Kshitij Grover [00:27:32]:
And what this sounds to me similar to and and so I'm familiar with, for example, Druid. And Druid has this concept of a segment. And a segment, I think, sounds similar to what you're describing, which is this heavily, maybe, index and compacted version of events on something like S3. Is it true that you probably have some metadata store somewhere that's not S3, and then some all the actual event data is mapped from that metadata store to some file in S3. Is that roughly right?
Neil Jagdish Patel [00:28:00]:
So we have, we call it the registry, but it's like a catalog essentially of where those blocks are a little bit of what they represent and things. But the way of tuning it is essentially what can you put in the registry versus into the headers of an object industry versus the object itself. Interesting. And then you come into different clouds can do different things. So if you're on Azure, you may be suffering for some other bits and pieces, but their blob service has some of the suffering for some other bits and pieces, but their blob service has some really incredible technology around the object store. Essentially, they have index metadata on things that, like, what could you do there? Do you even need a registry and stuff? So there's a ton of stuff we played around with and stuff until we came to that.
Kshitij Grover [00:28:45]:
That makes sense. And one of the things that I'm wondering is, you know, you mentioned this idea of it's all the same tier. Meaning there's no quote, unquote "hot tier" on SSD. How much does S3's innovation here matter in the sense of S3 introduces ExpressOne's own? And and I, I there's this kind of idea that you can shuffle around what S3 tier you're on without really changing the fundamental architecture. And maybe you pass some of that cost down to the customer if they need that. But really, you don't have to change much as a product.
Kshitij Grover [00:29:19]:
You just have to tell S3, "Hey, after 30 days, move it away from Express One Zone. Is that realistic?" Does that create too many kind of oddities around guarantees? Specifically, the express one zone, you are changing the guarantees on data durability. How do you think about that?
Neil Jagdish Patel [00:29:36]:
So I think in a perfect world, we would essentially have you'd only be using, a zone like x express, and then we would just assume that AWS will move it over whatever. I think more what we do is essentially layer, and so it's more in terms of caching and things like that. I think that just comes down to, as you said, for us what is it? 11 nines or 15 nines or whatever S3 is. Plus the fact that you can do cross region replication and things like that. So we set that up in the best possible way, and then every layer on top is essentially more in terms of it can exist or it can't exist. It doesn't really matter. The main thing is that we know that the other side has been covered, essentially.
Neil Jagdish Patel [00:30:19]:
But we have Bring Your Own Bucket, which means you can add your S3 to Axiom, essentially, and we can read and write from your own. We abuse the crap out of that all the time as we'll be connecting it over to Mineo. It'd be connected over to some of these new object stores. And we're just trying things out to see what are those guarantees. Are they better? Are they faster? Could they be integrated into Axiom's processing? What benefits could a customer get out of that? And then I think that's what I'm saying more towards the summer side is some of this shouldn't be inside of Axiom with us making the decision. I think some of this, we should be exposing to the users and saying, you're building on top of us. You know the right thing that you wanna get done.
Neil Jagdish Patel [00:31:02]:
Click this button, and it will happen, essentially.
Kshitij Grover [00:31:06]:
That's really fascinating. It sounds like directionally moving towards a little bit more of a plug and play model, where you're really trying to isolate. Here's our core technical competency, but, hey, you as the consumer or the user can make the right technical trade off surrounding that. Okay. Let's talk about query, which is I imagine where a lot of the fun stuff is. So I don't want to undersell this point because I find it very impressive.
Kshitij Grover [00:31:30]:
Axiom can query 100 of millions of rows in a matter of maybe it's not single digit milliseconds, but tens of or hundreds of milliseconds, so really quite fast. Yeah. How have you built up to that? Are there any significant unlocks that you realized over time that really helped get to where you are now?
Neil Jagdish Patel [00:31:49]:
We couldn't really we didn't release our public cloud until S3 and Lambda were talking to were talking to each other in a good way, basically. So we decided early on that if we got the data in, we got it stored, but then you still needed to run VMs to query it, that would be a failure, essentially. We have deep experience around scaling elastic and things like that. We knew that wasn't the way either. And so we knew we essentially came up and said, we're gonna make serverless query working work.
Neil Jagdish Patel [00:32:24]:
And that means when you press a button, whatever's the right amount of resources to use, we will create them, use them, drop them, and you'll get the answer. And so that required a few kind of things. A- all our data is on S3. Serverless queries Lambda. So Lambda S3 connectivity, that interconnect had to be at a level where we could pull enough, as you say, if it's 100 of millions of rows, billions of rows, you could be scanning high terabytes of data or more. And now these days, now people, they're setting it up in, okay, go scan a path of byte.
Kshitij Grover [00:33:01]:
Yeah.
Neil Jagdish Patel [00:33:01]:
The interconnects need to be great. We definitely were building this before they were great. And so we know what it looks like on the other side, but they are pretty good now. And so that was one of the things. The other side was, that query planning and execution. The reality is if you're querying a month or a year, it's it's not on. All one is going up. It's gonna be more like log because it's all about essentially being able to merge of all the results that come through, essentially, which can take effort.
Neil Jagdish Patel [00:33:35]:
How do you make that better? How do you make sure that someone just wanting to do a year's and, what was the count for the year? And they just said it on the thing, and now they go, how do you make that better? And so, essentially, we were building this out. And then what we realized was yes, there's work you do on the query side. Our query language API lets you do all sorts of stuff, like pass running regex, passing a call, lots of different things that you can do. But, actually, the work you do up front is gonna basically set off everything that happens afterwards. So then it came back to the block. Essentially, that block format needed to do the right thing for this kind of event queries where you're just streaming out the raw events or parts of the raw events, or analytical queries where you're running 5 aggregations at once, and you wanna just peek into the bytes you need to peek into to get that data and move on, and then obviously build the caches and things like that.
Neil Jagdish Patel [00:34:35]:
I love that that's how it works. It'll be everything we built, flows, everything work on essentially what you could be calling super efficient batch processing. That's actually what's happening. But it's just there was those stumbling blocks initially where we were trying to go faster than AWS were ready. Then we waited. But now it's like when they make performance improvements, we can see them, which is amazing. As they make Lambda, you can choose the sizes you do, whatever.
Neil Jagdish Patel [00:35:08]:
Axiom's doing that all the time. It's choosing, right, do you need a little bitty Lambda because you just need this much memory, or shall we go give you the heavy ones because you need to do x y z? And no one clicking that button has any idea of that's what's happening, and I love that. Because the entire idea that you need to know, the kind of how large your query will be is ridiculous to me.
Kshitij Grover [00:35:32]:
That's really interesting. Tell me if I'm thinking about this the wrong way, but there are certain tools that have done a lot of optimization at the storage block segment, whatever you wanna call it layer, which it sounds like Axiom has. And then there are certain tools like DuckDB or even ClickHouse that just have concentrated on an extremely performant query engine. Let's say once the data's in memory. They can just do these queries the fastest. What I'm hearing from you is for the Axiom use case, the former matters a lot more than the latter. Is that right? Or have you also and do you intend to put a lot of work into the actual query engine even if you have optimized the storage layer as much as possible? How do you think about those 2 different parts of the problem?
Neil Jagdish Patel [00:36:16]:
So I think you just find us on that journey at the moment because it's now we focus much more on the query side than we where than we traditionally did. Traditionally, we were just ringing that block for everything we could get out of it. And even today, we'll still try things out, and it's easy for us to do that. But mostly now, we do focus on the query planning and execution. People want joins, they want subqueries, lookups, all these things. And so for that to be delivered, and you do need to spend more time on the execution there. So that's where we are now. But I think we that block format is something which is like it does we do tune it and change it, but definitely feel like we've been touching it less now than we had done in previous years.
Kshitij Grover [00:37:05]:
Got it. Actually, maybe let's talk about what is not quite the final piece, but one of the edges of this, which is what you mentioned as your query language. So most companies, I think about one surface of these problems like the storage or the query execution or even the ingest. But you all have also decided to build your own query language. And the question is why and what benefits has this brought you? And just guide us through that as an additional piece of innovation that you all have been leading.
Neil Jagdish Patel [00:37:41]:
Definitely times where I think that why the hell did we do this? But there is a reason, essentially so our query language APL it's a super subset of Microsoft's KQL, which is the Kusto Query Language. So Kusto, for those who don't know it, is this kind of world store that Microsoft built in Azure. And it can do all sorts of different things with events, and including you can build graphs out of them, geo, all these different things. And they were looking for a query language, and, essentially, Splunk was that first pipelined query lang SPL and really nice to use, etc., but dated. And so KQL is like a modernized version of SPL.
Neil Jagdish Patel [00:38:26]:
And when we were in the market for a query language, we knew it wouldn't be SQL, and there's reasons for that in terms of just when you don't have a schema, when you wanna do tons of text things, run functions, whatever. That SQL just doesn't feel right. But we were trying not to completely build a brand new language. So we looked at the KQL spec, and then API is essentially our implementation where we've picked and we we've chosen what makes sense. And then we just added all the stuff. Axiom can do that, and drop this off because this kind of thing. But for us, it was pipelining is great, for when I think of event streams, I think of things moving from left to right, and I'm picking and choosing from them.
Neil Jagdish Patel [00:39:08]:
And they drop to the next level, and then I do it again or I aggregate them, whatever. That is literally what you would write into APL to make it happen. So I love that idea. The other side of it was we can do multiple aggregations at once and things like that. And so we expanded it with support for that. More options, more functions, and all of those things that you would care about. But the real thing was we knew we would get into the processing part of this kind of event problem that exists in every company. And so as we graduated to the processing site with flows and things, we actually wanted to add more operators, more verbs for the thing that you're actually doing.
Neil Jagdish Patel [00:39:48]:
And so we wanted an extension point for, okay. If tomorrow you're running functions, we wanna add function calling. We wanna add this, that, whatever. And so I think up until now, we just tried to make it solid as possible and cover as many bases as possible for the needs. And again now we're at this core point where we're now extending it and it becomes a language where when you read the query, you're essentially just saying English in exactly the way that the query process is gonna process. And I think that's really valuable.
Kshitij Grover [00:40:20]:
This sounds like a lot to be working on at the same time. So I'm curious how are you thinking about where you push the boundaries the most in what sequence? So you just talked about these 4 very heavy things, ingest storage, query, and then the query language itself. What is your mental model or framework for that?
Neil Jagdish Patel [00:40:44]:
It is moving left to right in a weird way. So ingest and storage was the most important things that we had to nail, and we did that initially and early. Then storage and querying, so the blocks and how we do querying was the next thing. And getting query up to a point where as these complexities grow because the organization the size of the organizations grow and what they wanna get done, is different, We can mash them and make sure that everything they want is always just on the horizon of what we do. And so rather than worry too much about the kind of bread and butter parts of querying, now it really is about, okay, being able to execute something and make sure that we can join it with 2 other things and or union or whatever it is and make sure it comes down. So I think that's where we're at now. Flows becomes the next kind of thing where you see a peak of it very soon. And then that peak will have a very specific use case, which we all nail first.
Neil Jagdish Patel [00:41:42]:
But then you'll see that becoming what we want it to become in terms of almost like a heartbeat for events inside your org and allowing you to not have to rely on these kind of off the shelf tech, which wasn't really built for this problem, but absolutely solved the issue. No doubt about it. But now maybe people who've been obsessing about this can give you something which gets you from A to B much, much quicker. And obviously, company grows, people grow, whatever. But, generally, we're moving more to the middle and the right hand side of things now where it's, less make query exceptional. Then let's get flows out there and make sure that people can get real value on day 1. But then let's make flows exceptional. And then what you'll see Axiom be is like a very, I think, a super unique place for what you do with event.
Neil Jagdish Patel [00:42:35]:
And it doesn't matter whether you're a dev and you just wanna know what's happening with traces or logs, or whether you're actually connecting it to your product, like so many startups have already done, or whether you're just like, "Actually, what the hell happens with our support cases?" What can I do with that to make or whatever? You can just use Axiom in a more flexible way.
Kshitij Grover [00:42:57]:
So that's related to the last question I wanted to ask you, and I feel like you've talked about lots of things here. But what is it that you're most excited about? It sounds like there's the summer launch coming up. It sounds like this is both on the technical innovation as well as some of these partnership angles. Which part are you most looking forward to?
Neil Jagdish Patel [00:43:16]:
Nothing actually makes sense if we couldn't put it into everyone's hands. And so the efficiency of Axiom is still the most exciting thing. And the fact that it's getting more efficient while doing more things is crazy to me. But it's the architecture is coming together in an awesome way. So absolutely crazy excited about the fact that very soon you'll be able to run pieces of code against Axiom, like on a event pipeline or be able to connect it back out into not just something like s 3 or for export or Splunk or whatever, but also other tooling which you're using. That silo busting that you can do, because you can put Axiom in front of the tool that you're already using and actually have Axiom have months or years worth of data and that tool have 30 days. That's super cool.
Neil Jagdish Patel [00:44:05]:
And I think it's gonna be awesome. But the reality of it is all of that would be great. People do it, but they it's only the top companies with the deepest pockets that can do it. And so the exciting thing is we can bring it to everyone, and that's super cool. And just continues to get more and more efficient, which is really cool as well. We don't we still don't qualify for any of the AWS credits, which is insane. Or discounts. It's wild.
Kshitij Grover [00:44:33]:
I appreciate that. And I love the continued focus on the customer experience as the kind of core thing that makes Axiom great. And, thank you for taking the time. This was a lot of fun, and I appreciate you walking me through that technical journey.
Neil Jagdish Patel [00:44:46]:
Not at all. Thanks for having me on. It's it's cool to talk about it.
Kshitij Grover [00:44:50]:
Of course. Thanks, Neil.
Neil Jagdish Patel [00:44:52]:
Cheers.