Subscribe
Share
Share
Embed
Join Kostas and Nitay as they speak with amazingly smart people who are building the next generation of technology, from hardware to cloud compute.
Tech on the Rocks is for people who are curious about the foundations of the tech industry.
Recorded primarily from our offices and homes, but one day we hope to record in a bar somewhere.
Cheers!
Nitay (00:02.343)
David, it's pleasure having you today with us. Thanks for joining us. Why don't we start by giving us kind of a brief introduction about yourself.
David Mytton (00:10.552)
Hi everyone, I'm David Mytton I'm the founder and CEO of Arcjet, which is a security as code startup. helping developers build security related functionality like bot detection and attack protection into their applications. It's an SDK, so it integrates right into your code, which means you can change the rules dynamically based on what your code knows, like who's logged in, and then use the security.
decision signals to change the logic of your application. I also founded console.dev, which is a weekly DevTools newsletter. Every week I write three really short reviews about the most interesting DevTools because I couldn't find anywhere online as a somewhat experienced engineer to know what I should pay attention to. And so decided to start that. We're now at 30,000 subscribers and I've just been playing with DevTools for quite a few years now.
Nitay (01:06.865)
That's amazing. Awesome. Probably there's a lot of people here listening that would happily do that all day long, getting to play with DevTools and write about them. Tell us a bit, backing up maybe even before, and before we get into each of those two, what got you into each of these things, into console.dev and then leading to creating Arcjet what's the background precursor to those things?
David Mytton (01:26.51)
Well, I started my first company in 2009, which is a cloud monitoring product similar to Datadog and New Relic. that was over eight or nine years, we grew it to 1.5 million in revenue. It was a bootstrap business and we did a small seed round before an acquisition in 2018. I joined the acquirer.
which is an edge security platform and was responsible for the launch of their edge compute product, containers running at the edge basically like fly.io these days. And then at the end of 2019, I left, took a bit of a break and went into academia briefly. I did a master's in environmental technology and focused on energy science and then started to bring my cloud computing and software experience into energy and sustainable energy systems.
particularly around the energy consumption of computing and cloud computing in particular. It was interesting because most of the people at the time were doing it from the opposite direction, coming from science and applying computing. And so I thought I could bring an interesting direction kind of doing it the other way around.
Nitay (02:35.409)
That sounds fascinating. So what was the like particular research or things that you were looking at there?
David Mytton (02:41.72)
So the environmental impact of computing kind of is a very broad topic and most people will think about it in relation to the energy system, to how much energy is involved in data centers and using your computer. And then the associated environmental impact of that is the carbon footprint of manufacturing and then the usage of computing. But there are also other impacts as water consumption, there's the mining involved with the actual manufacturing process.
But my area of research, which then led into a PhD or part-time PhD at Oxford, was around the environmental impact of cloud computing specifically. And the idea really being how can we use more computing over time, but reduce the environmental impact because there's inevitable growth with all the services that we're using. More data centers are being built. More services are going online. There's more software.
you're not going to get people to reduce their usage. So how do we flip the equation and reduce the environmental impact from that increasing consumption? That was kind of a side project that I got into. then there was some challenges with academia and decided to get back into tech and startups properly, which is what then led to the creation of console. And through all the research I was doing, we to DevTools, testing them out, finding out what works, what doesn't work. I came across the idea of...
security for developers, particularly in production. Rather than when you're writing code, there's a lot of tooling to help you write more secure code. But when you go into production, there just wasn't anything that was at the same level that developers are used to today with the likes of Vassal and PlanetScale and TailsScale. These kind of next generation dev tools with an amazing developer experience, an amazing user experience, that just didn't exist in security in production, runtime security, which is what led to ArcJet.
Nitay (04:37.095)
Very cool. what was the, cause going back to your previous company, as you said, was also doing kind of edge, edge things and security and so forth. What was kind of the state of the market and the landscape then versus now when you started ArcGEN, what you're seeing to your point with like Vercell and others in the field.
David Mytton (04:52.792)
So depends how far back you go. think if you go back to 2009, when I started Server Density, a cloud monitoring startup, Datadog didn't exist then. So they came a year later. And so the state of the art was maybe one or two different open source projects, which had existed for many years. And so the user experience was poor and it was very difficult to configure. You had to run your own monitoring somewhere. The idea of cloud infrastructure just didn't really exist.
AWS was around, but I only had a few services and this was a very new way of delivering infrastructure, infrastructure as a service essentially. And so we built that out and tried to pioneer this idea of cloud computing, bringing observability tools to operations teams and DevOps and kind of rode the wave of DevOps as a philosophy, but also cloud computing and more workloads moving into the cloud, which allowed us to provide
provide the monitoring that people were looking for.
Kostas (05:58.001)
All right, so you've been working with observability for a while. And before we go into what you are doing today, I'd like to ask you, what do you see as the future there? Observability is one of these things that kind of like keep building, but almost always feels that we still haven't figured it out, at least to me. And I'd love to hear from you why this is the case.
what is missing there and what are the opportunities like for someone who is thinking of starting something in this space.
David Mytton (06:35.118)
So mean in observability specifically, kind of the idea of metrics, logs, traces, is that the kind of area you're thinking?
Kostas (06:43.485)
That's what I'm thinking, but if you definitely have much detailed and broader context around that, so please, if you feel like this description with these three pillars is not enough, please add more to that because that would be awesome to hear about.
David Mytton (07:03.734)
If we ignore the idea of AI for the moment, which is a big thing to ignore, observability is still probably the hardest challenge in software engineering. Just after you've built the system, understanding what it's doing when it's in production is incredibly difficult. And so we've come up with these concepts, metrics, logs, and traces to try and give us an understanding of what's happening. But even with those, can't really see every single action the system is taking.
because there's often too much volume. And so you start sampling data. You don't record all of the traces, not at any kind of reasonable scale anyway. And so you have this challenge of not seeing everything that's going on in the system and trying to use multiple heuristics to understand what is generally happening. And then when you have a problem, kind of drill down into what's actually going on. And the real challenge there is just the volume of data that's being generated. If you have a single request coming into a system, you're generating.
huge amounts of logs, huge amounts of metrics, and the trace itself can be very complicated. And that's even before you get into distributed systems and going to third-party APIs and cloud providers and into databases and all sorts of different layers of the tech stack. And then when you add AI in, it becomes even more challenging because we don't really know what's going on behind the scenes in the same way that we would do with a traditional software application. And so bringing these all together to answer the question, what happened?
Maybe you've got a request ID or a transaction ID or something like that. It's actually a very, very difficult problem.
Nitay (08:39.207)
How much do you think it's a, like, where does the problem reside? Meaning it could it be, the problem that we just don't have the right solution yet. So to your point, like the logs aren't good enough or you don't have full visibility or, or like dealing with the inundation of data, you said, or is it that the problem resides in further back from that, how are architecting the systems themselves? It's just not lending itself to be, you know, ever to be great and observable. Or is it more like even like a people process kind of like problem? where, where do you think the
root of it comes from.
David Mytton (09:11.032)
Probably all three, right? It's the design of the system is how you get the data out of it. And then it's how people think about the system. And when you start with a product, it's realistic that one person can have the entire system in their head and they can map it in their head and understand all the components. But that doesn't last very long. As soon as you add more people to the team, you build more features, you have more complicated infrastructure. Or even if you're just dealing with distributed systems problems.
Nitay (09:13.139)
You
David Mytton (09:39.136)
understanding what's happening at any given point is really difficult. And so we come up with these abstractions, the metrics that hopefully will alert us when something is wrong and then get the right people involved who know where to look to debug the system. And we're getting to the point, perhaps not quite, but we're getting to the point where the aviation industry is where it's very rare that it's a single thing that causes a particular problem. Like it's very rare that a single failure will cause an airline to crash.
There's going to be a cascade of multiple things, multiple failures. And it's really the same in these complex systems. think it's really interesting to read through the post-mortem analysis from a cloud provider like Google, or really any software provider to understand all the different things that had to go wrong for the service to go down and why they're therefore writing about it. And what lessons you can draw from that. aren't, a lot of them aren't generalizable anymore.
Nitay (10:36.453)
What's the, do you have like one tool and this will tie into your console that that stuff maybe you have like one observability tool or one thing that's like on your wishlist of like, man, I wish that tool could do X or I wish I had maybe even for your own company. Like I wish the observability already just solved problem Y.
David Mytton (10:53.582)
The one that everyone goes to is Datadog, right? It's the most advanced, has the most integrated feature set. And so they charge the most money for it. And that's where it starts to become a problem. You look at your Datadog bill and it's absolutely insane. And so then people start ripping apart different pieces and trying to reinvent it. And the great thing about Datadog is it's all fully integrated. That's why they can charge the money is because developers can go in and
put in a trace ID and then you can see all the logs and you can see all the metrics. You can see the container data, the server data, all the network information, the processes, all these things that build up the picture of your application. And the alternatives still just aren't as good. At least when you're looking at the whole system. But there are some really interesting components, like the traces side of things. OpenTelemetry has become really good there and some really interesting tools like Honeycomb.
that are innovating in really interesting ways, particularly around the data store. They built a custom data store just because of the volume of data and how you're querying the parameters on a particular trace is very different from how you're querying logs, for instance. And so you need a custom data store there. But breaking out these pieces, you'd lose the overall image of the whole system, the overall map. And so that becomes the real problem.
It's kind of connecting all these different pieces to understand what's actually happening.
Kostas (12:25.019)
So that's very interesting. You're about like the different, how different, handling the data between like logs and traces and metrics is, and to the point where companies need to go and build like dedicated data stores for that, right? Do you think that it's even possible to somehow unify these three pillars under one system? Even if you have, okay, like built the different data stores, right? But at the end of the day,
you need to have like a unified view over these things. Like it's not only one of them is going to suffice to fix your problem. So is it even possible like to do that? Or you think that the future at the end will be dedicated vendors to each one of these and something, I don't know, like let's call it control plane for observability or something that sits on top of that and somehow makes things work all together.
David Mytton (13:27.266)
And you can even add to that, does the system change by the fact that you're observing it? What overhead does it add? You often see that in logging, depending on how optimized the logging library is, can add significant overhead to the overall processing. I think open telemetry has helped a lot there because of this split between logs metrics and traces. And the fact that you can direct those into different systems that allows you to have a single
framework, is open telemetry and a single SDK that you bring into your application to instrument it. But then you can send the data out to different providers. that is working, that seems to be working quite well, but it depends on the maturity of those different connectors within the hotel libraries. And I found, for instance, we write a lot of API code in Go. The metrics are great. The traces are great, but the logs
aren't as advanced as the other two and connecting them together is what you really want to do because when you see a log line, you want to see the associated trace that generated it and the other way around and simple things like just being able to link a single log line to a particular trace is not as straightforward. It has improved over the last year or two and you can now do that, but the different components, it's an open source project that developed at different cadences by different people.
And so you can see this disconnect appearing with different versions of the library, which makes it a lot more challenging, particularly if you're trying to replace something like Datadog, which in theory can instrument everything for you automatically.
Nitay (15:07.825)
That's really interesting because that's something I've thought about as well. It's just that like, think it's almost a fallacy to think I should try to split up these different pieces of data. Right. Because it's, it's, think where things diverge or maybe not diverge, but where there's opportunity, let's say to disrupt and kind of come in. It's different use cases, not different data, because I would guess that a lot of the different use cases might use the same underlying data.
And you see this a lot in the database world. Enterprises famously have many copies of the same data. You have a transactional store, have an analytics store, you have some other database that's ERP, have whatever. Stuff's copied everywhere. Why? Because most use cases don't need just one piece of data. They need a whole bunch. It's probably like 80 % of your enterprise data or whatever it is. But in a particular way, with a particular packaging, with a particular product, and so forth. And so...
I think it's a really interesting point you make because my guess is that like, one of the things open from a chicken, maybe hopefully.
Again, this to establish is these like open formats and open easy exchange, similar to what like Parquet and S3 object stores have done for the data world, but for the observability world to make it really easy, but to have all the data, unique logs, metrics, et cetera, et cetera, all in one kind of concrete standard form, but that can feed different products. Whereas right now, because they're so integrated, they're so well tied to Datadog that it kind of owns the whole thing. And it's hard to split apart.
David Mytton (16:37.046)
Yeah, I think that's right. And the underlying storage mechanism for each of those is different as well, because the way you store and query logs is very different from the way you would store and query metrics. And Datadog has this reputation for being very expensive. And an easy way to do that is to give them more money is to create a custom metric and send to them on a very regular basis. And for some reason that becomes very expensive. And I'm assuming it's because of how they...
Nitay (16:44.723)
Mm-hmm.
David Mytton (17:05.964)
they have to store that and their own cost comes into it, not in their profit margin. But these are all very different challenges. Storing metrics versus logs is a very different problem. So you get these interesting new databases like Clickhouse, which completely flips as a common data store. And it's being used for observability to very easily store huge volumes of data, just because the way that you're querying it is different from the relational database.
Kostas (17:35.857)
Yeah, although I have a question for you there because like one of the things that you said at the beginning is that you have your logs, you have your metrics, you have your traces and at the end of the day you have to join all these things like together. And I'm using the word join here on purpose, right? Because you also mentioned ClickHouse and ClickHouse and all the systems like ClickHouse, right? One of the trade-offs there is that
you can't, you don't have the flexibility to join the same way that you do in like in an OLAP system. Right. So again, goes back to every solution feels like solves the problem by doing a trade off that leaves something to be desired in observability. Right. So we still, how do we solve then this problem? Right. Are we going again to have two systems there?
kind of typical, let's dump everything, let's say on S3, do the heavy lifting of joining, rejoining the data, preparing the data, da da da, whatever, and then push everything into ClickHouse. And then we query the data from there without having to do that. But that means that first, we still have duplicate data that we were saying, right? We also have two very different systems that need to be operated with very different skill sets needed there.
And whenever you change, you mutate state, it means that you might always need to go back and make changes again, because something might be missing there, right? So the problem is actually pretty hard. It doesn't sound like at the beginning, but it is. So do you think that even with something like ClickHouse out there, like the problem at the end is solved, or we still, have more to do in order like to be, Hey, like let's call this problem solved.
David Mytton (19:31.478)
Right. I think there's some really interesting innovation happening around object storage and using that as a layer for many different applications. There's a startup called WarpStream, which was recently acquired by Confluent, which was building a Kafka alternative on top of S3 compatible storage. And there are trade-offs for that. And certainly the write latency on S3 is quite high, unless you're using something like the S3 Express zone. But even then it's still...
It's still kind of 20 to a hundred milliseconds to write something into the object storage, which might be fine for your use case. But it depends on what you're doing. I think with that, you can offload all of that really difficult work, the object storage to Amazon or to some other provider that provides an S3 compatible API. And I'm seeing other really interesting product innovation happening there. And there's another database called QuickWit, which also uses S3 as a
as the data store. And I think there are lots of these use cases where the storage side is the most challenging things to run and the most expensive thing to run. And if you can push off to S3 with its tiered storage and you can take use of, make use of that, then you can reduce the operational burden. Just give it to Amazon or to someone else who's doing it and significantly reduce the cost. And if you can make that data store basically a commodity, then you can innovate on top of it.
I don't think that's going to be really interesting pattern.
Kostas (21:02.035)
That's awesome. Okay, one last question about observability because we have also like to talk about what we are building today. So one of the things that you mentioned was also AI, right? And I think with AI, when it comes to observability, are two interesting sides there. One is observability of AI and what that means. Like how do we observe something that is so different as a piece of software compared to what we've built like so far, right?
But there's also the how we can leverage AI in solving problems for observability. Right. And I would say that we see a lot of investments going into that kind of, it almost feels to me that especially like VCs have been, we've been pouring money into solving the storage layer for way too long. It becomes, looks very hard. I don't know how many more PhDs from Stanford we can throw on this problem.
Maybe we can have some better returns by focusing on using AI and focusing more on the higher layer, let's say, of observability, like supporting the user, finding the signal out of the noise and these kinds of things. So what's your take there for both? Both the observability of this new artifact, the software artifact, which is the module itself and operating it, and also how AI
being unpredictable, again, like you mentioned sampling, right? And now we're going to an extreme version of that, which is by definition something that's stochastic in how it replies, can help in solving some of the long lasting problems in observability.
David Mytton (22:48.438)
If we think about what computers are good at, it's large scale automated repetitive tasks. And I think that's going to be the same with AI. With observability and AI, the use cases are very similar to security and AI, where you have a huge volume of data and you want to spot something very specific, or you want to be informed about something unusual or something that differs from a particular pattern.
The experiments I've been doing with AI and the request data that we see at Artjet is to look at huge volumes of web requests and get it to first summarize what's happening in a particular time window, and then pick out interesting characteristics from that. And it's been able to spot certain patterns of attacks. When I give it data where I know there's an attack and I know where it is, I'm kind of testing the AI. I can get it to pick out things that I might've missed or to get it to summarize interesting data.
And that's really the same challenge that security engineers are dealing with is very similar to what Preservability and DevOps teams are dealing with, which is a huge volume of data and too many alerts. And you need to be able to triage and figure out what the humans should be paying attention to. And I think that's where the AI can augment our abilities by highlighting and prioritizing what humans should then spend their valuable time on.
I think that is probably going to be where we see an initial set of really, really helpful use cases on both sides. What's happening with observability data and then also what's happening with the security data and helping those teams with the huge workload that they already have.
Nitay (24:30.899)
So shifting gears then to that topic, how do you get the AI guardrails, if you will? Like how do you get it to behave the way you want? How much customization does that end up actually taking? You one of the things we hear and see often as I'm sure you've seen too is it's very easy to get a demo up and running that looks all shiny and pretty, but much harder to get it actually work at production and scale. What's been your experience there?
David Mytton (24:54.828)
Yeah, definitely. And also the context window is really relevant because if you, particularly with web requests where just ignoring the body, just looking at all the headers of every single request, you've very quickly going to hit the context window limits of an AI. And when you then go to something like Google's Gemini with a much larger context window, you end up just paying loads of money to analyze it. And that's no longer feasible when you have millions of requests that you're having to process.
So think that's going to be the initial challenge going from the early models, which have very small context windows, but are cheap or free. You can run it on your own hardware or your own systems. To the models that have a huge context window, but are very expensive. And what I've seen happen in other use cases for analyzing things like code analysis, pull request analysis, looking for malicious actors within software releases.
is to run the code through multiple layers of LLMs and have the really cheap, not as intelligent LLMs do the initial analysis and try and pick things out. And then when they find something unusual or flag something, that thing goes to a more sophisticated LLM that can do a deeper analysis. And you could, you can see this layer going deeper and deeper, like three or four layers until you can then pass it onto a human with much more useful context. And then that can be verified and
and presented to the user as a result.
Nitay (26:26.353)
Gotcha. That makes a lot of sense. that's, that's kind of well-known architectural pattern, right? I mean, search engines use this like add options, famously do this kind of thing where when you have kind of a large data set, you filter it down, filter it down using kind of broad heuristics. then you're finding, make the heuristics more more fine. Great. One of the things, you know, having worked on many of those types of systems before, one of the things you often find is this question of like,
If at the end result is not what you wanted, where in the chain did it go wrong? Right? Where did it get filtered out? Which one should be corrected? And then you have this fine tune balance of like, okay, well, my first pass is returning, I don't know, 200 results. So it's just filtering down 80 % of things. it be 90 %? But then I have to, and then I miss more and again, more kind of, you know, false negative, false positives kind of thing. or should I dial it back to 50%, but then I'm going to pay more in that my later, later layers. How have you found people kind of like tweet?
those parameters if you will.
David Mytton (27:22.254)
I think it's just brute force at the moment, trying to understand what's happening and make the manual tweaks. I think this is where the forefront of the tooling is starting to appear is observability of those systems, understanding what they're actually doing. Because the big challenge is they're not deterministic. So you run the same thing twice and then you get a different result. And that makes it very difficult to understand what has actually happened. The LLM itself can't.
I can't tell you. I don't know. Have you seen any interesting approaches to solving that?
Nitay (27:56.849)
It's a great question. One of the things I'm seeing a lot of...
to your point is exactly because it's kind of this like a little bit of art, not feels like sometimes not a hundred percent science. I've seen more and more people chaining LLMs where the thing doing the judging is another LLM. And that gives you some sense of confidence. It does work. It does actually work. I want to be clear and I'm not trying to make it sound like your magic because essentially you have, mean, like this has been shown where there's certain types of prompting and fine tuning that you can do where you essentially constrain the reach
of exploration that the LLM does. So I'll give you a concrete example. One of the things that's well known with these, even open AI or whatever, if you go into it and you just ask it like, hey, explain to me how thermonuclear energy works, whatever. Versus you tell it like, hey, you're a nuclear engineer. Explain to me as a nuclear engineer, what are the key innovations, cetera, and you constrain it to be, okay, I'm a nuclear engineer. Now knowing that, here's my space of answers I can give you.
And so I've seen a lot of people kind of play with these approaches where they tell one LLM, like, you are a judge. Your only job is to say this is good or this is bad. I want you to absolutely remove all notion of false positives. I don't care about false negatives. And you like kind of tweak it to say, like, you must be absolutely certain this is a security issue or whatever in order to make it pass through or vice versa. Like you tune it for whatever you're optimizing for.
And then you have these LLMs that act in this kind multi-LLM workflow that you end up building. So that's one of the things I've seen a lot of folks doing. Is it perfect? Certainly not. And it kind of lends itself back to the same question of, who's judging the judgerous? Kind of thing.
David Mytton (29:41.858)
Yeah, definitely. And do you think, I suppose that's prompt engineering, right? So do you think that is a role that's going to be around in the future? Or do you think it's going to, that can become sophisticated enough that you don't need to do that?
Nitay (29:56.647)
That's a very good question. I think you'll always need some level of it. Whether prompt engineering stays like a job description or a role, I don't know. But...
Even today, seeing, you know, if you look at some of these companies like, famously like scale that AI, right? A lot of what they do, or at least where they started from is basically human annotation. Right. And so I think, I think we will always need some level of human curation, both at the early stages of building the system, obviously, meaning like, you know, where the AI co-pilot helping you gen code or whatever.
but all the way through to the kind of tuning the knobs and understanding that the system's doing what you want it to do, as well as having like the escape hatches and so forth. So yes, I do see there being a future for a need of that, whether it stays quote unquote prompt engineering and like that is the thing, I don't know, right? Like it may evolve that, that may be only a point in time kind of thing, or it may be more like, know,
agent engineer or something, right? Like, like may like morph into some other kind of like role or some other day to day kind of thing. Yeah. What do you think?
David Mytton (31:13.482)
Yeah, I was recently took an Uber from San Francisco airport downtown and you can't currently get a Waymo from the airport. And the driver was telling me that he used to work for Waymo. He was driving the cars before they became, before they were available for, for general pickup for, for people to get a taxi. And he, he was training the AI driver. And then once they hit a certain level, he was made, he was laid off and he went back to Uber.
because that was where he could get a job. so that's a really interesting use case of a human giving all the initial data up to the point where the AI can do the job just as well. And then the humans no longer needed. But then a couple of times later, a couple of days later, I was in a Waymo in downtown San Francisco, going on a journey where I could take the Waymo and it got stuck because a car made this weird, weird maneuver and there was a bus and it, just couldn't move. And so it was just like, sorry, you can't do anything. And it called the support.
And then they had to reverse the car and get it around the bus. And then the AI took over again. So I think there's this interesting dichotomy between the initial driver who trained it and then he got laid off, but it still relies on humans in certain edge cases, which may, I don't know how long that will last, whether they will ever be able to fully run autonomously. It's pretty good these days, but there always going to be those use cases, those edge cases that they just can't deal with.
Nitay (32:41.681)
Yeah, and I think the reality is that the entire market becomes the edge cases very quickly, right? Because you solve the general cases quickly and easily. And it's amazing when you do, but the entire market becomes edge cases. And that actually kind of, in a way, almost brings me back to what Archit's doing, what you guys, because in a way, your entire business is edge cases, no?
David Mytton (33:01.762)
Yeah, I suppose so the security side of software. And at the moment we deal with most of the standard security things, bot protection, but bot protection, you can't get a hundred percent. You can never get a hundred percent because it's an arms race. You invent some kind of new detection for an automated client. then whoever is getting value from that, that bot will find some kind of work around to pretend to be a human.
So you're always going to need humans to try and detect these different methods. And it's the same with security in general, because for every vulnerability, we invent a new mitigation, we implement it, becomes best practice, and then lots of people implement it. And so the way around that is to find some novel method of interacting with the application in a way that it wasn't designed. So you're right. These are all edge cases that you have to deal with ultimately.
Kostas (33:55.251)
David, tell us a little bit more about the bots because everyone has heard the term. There are many different bots out there from like Twitter bots, sorry, X bots now to the bots that Google is actually using to go and read your content and actually do something useful for you. But tell us a little bit more about what the bot is and then when...
security gets into the picture and why we need security.
David Mytton (34:31.596)
Bot detection is all about detecting automated activity on your application or your website. And you'll typically categorize bots into good bots or bad bots. And there are some general definitions, but the difference between them also depends on what your use case is. So a good bot could be Google crawling your website. It's indexing it because you want people to be able to discover your content when they're making a search. So you want Google to do that.
Google's a good bot because they identify themselves. They follow the robots.txt file and you can understand how they're going to access your site. They've written their crawler in a way that's not going to take your site down and you're getting a benefit ultimately because it's then indexed. But Google is now also crawling for AI and they're using your content as part of their corpus for their AI training and maybe you don't mind. so Google is still a good bot.
But if you don't want your content to appear or to be used for training, now Google has become a bad bot. Now in Google's case, you can still determine that it's Google because they'll tell you and you can just say, well, I want to block Google. And they will respect that. The challenge comes when you have bad bots who are trying to scrape your content, even though you don't want them to, or who are trying to submit forms to spam you, or maybe try fraudulent credit card transactions to get a list of cards that are
available for fraudsters to use. And those bad bots are really difficult to deal with because they don't tell you who they are because otherwise you just block them. And they pretend to be humans in very interesting ways. One of the easiest ways to detect a bot is because the IP address that's coming from is a cloud provider like AWS because humans don't put their internet traffic through AWS usually. But there are ways around that. You can run a proxy and so you could
put a proxy on a residential ISP and pay people to run your little software. Maybe you're giving them points or something, or they're filling out surveys in return for your personal data and you're installing a Chrome extension. And what you don't realize is it's actually proxying their traffic through your network. And there are even huge banks of mobile phones that sit in people's houses that are then connected to the mobile network and people...
David Mytton (36:53.334)
would normally associate that cellular traffic with people browsing on their mobile phones. But actually it's this bot traffic that's being run through proxies and it gets way more sophisticated than that. And so this is the challenge. How do you detect those bad bots and how do you prevent them from doing what they're doing on your site if you don't want them to do that? Which most of the time you don't.
Kostas (37:15.027)
So how do you do it?
David Mytton (37:18.51)
Well, the first step is to identify the good bots and to allow you to define rules for them. And so they're just going to tell you who they are. And there are a few different heuristics. You can look at the user agent. That's the simplest thing. Every user, every request by the HTTP spec has to have a user agent description, but it's just a text field that anyone can put whatever they like into that. And so you can pretend to be Google bot. And so then the next level is to look at the IP addresses that the requests are coming from.
because Google tells you on their developer documentation, these are the IP ranges that we own and you'll see traffic from those. And if you can combine detection of the user agent with the verification of the IP address, then you can be sure that it's Google. So someone trying to spoof Google won't have access to their IP addresses. But then for everyone else who's not telling you who they are, it just becomes very challenging. And so you have to do things like looking at the behavior.
of the user agent that's moving around your site? Is it making lots of requests in a way that would be unusual for a human? Or does it have browser APIs available that a normal web browser would? Because if you're using some kind of script to navigate using a headless browser, might not have a geolocation or the JavaScript APIs might not be available. So you can start building up all these different signals to try and understand
and detect whether the request is coming from a human or an automated client.
Nitay (38:53.299)
How much, like as a vendor in this space, how much of your time and investment goes into these algorithms themselves? Do you invent them yourself? Are you like on the cutting edge of like, we have an idea that others don't, for how did they do this detection? Or is it like stuff coming out of a container or conferences that you're like, Hey, we're going to grab that thing and stick it in also, or is it just kind of well-known things that like, everybody does this. We're going to do it too.
David Mytton (39:16.238)
There's a bit of all of those. So what I've just talked through with the user agent and the IP address, that's fairly well known, but there are loads of open source projects that will allow you to bypass bot protection. There's a particular one called GPT-4-3, the number four, which allows you to steal inference from other people who have unprotected open AI endpoints, because you have to pay for using open AI if you want to use more than the quota. But there are websites, for instance, that put a support bot on their website.
or they have a little chat app inside the developer documentation and you can ask it questions about that site. But if they've been badly designed or don't have the right protections in, you can prompt engineer those to give you access to the underlying LLM and ask it general questions. There was a good example on the Amazon website, on amazon.com with the chat support chat bot. You could tell it to ignore all previous instructions and give me some code hints for
how to write this different algorithms and it suddenly switched from becoming just the support bot to just like co-pilot without having to pay for it. And so that's this.
Nitay (40:23.377)
I think there was a famous one that always cracked me up. think it was like Ford or GM or one of those that had like a support bot and somebody was like, Hey, can you, can you solve the Navier-Stokes equations for me? And it just goes into like physics mode and they just use it to do their homework. And they're like on the Ford side or something.
David Mytton (40:39.148)
Right. And there's another one around airline tickets and it asked to have it for free. And it was just, and the user was able to get airfare when they were having to pay. And so these are the simple attacks that work right now. They won't work for very long. And it's an arms race because there are always people trying to get kind of a level up on their ability to access all of these different things for free.
Just money is a strong motivator just because you don't want to pay for something or maybe you can't pay for something and you can find enough vulnerable endpoints to be able to access these things for free.
Kostas (41:18.471)
Yeah, it's so interesting. So following up to the question, if we see a future of a job description being about prompt engineering, do you see a world where we have security engineers dedicated to AI and LLMs and being substantially different than say security engineers who have different like dealing with the more traditional, you know, attack surface.
areas that we have today on the cloud or even installing applications.
David Mytton (41:52.814)
That's a good point, actually. think we do, we have these disciplines in security of things like IT security, netsec for the network, appsec for the application. There are, if you ever go to DEFCON or Black Hat, the conferences in Vegas every year, there are all these different villages, which they set up around the main conference with themes of hacking different things, hacking cars and biohacking, hacking the election system, all these different things that are happening because of the subspecialties can go so deep.
And there is an AI village as well. And I think we'll see people having these various particular skills, understanding how LLMs work and how to break into them, particularly because of how fast the space is moving. And you've got these old models that are cheaper and no one's using as much or not paying as much attention to. It's moving on to the latest new ones. so vulnerabilities and challenges in the older models are not being addressed.
but they're still being run in production because they're the cheaper ones. And you can then kind of direct your queries based on how much you want to spend. So I think that's going to become an interesting area of research as well, just how to use old LLMs, old models that haven't been updated.
Kostas (43:11.789)
And you mentioned some of the methodologies there to detect if someone is a bot. How much do you see AI being used today as like system to help augment, let's say these rules? specifically in like, because I think when it comes to security, it's always, what always fascinated me about security is that it's a discipline that
You always have to be alert. always, you know, that someone is going to outsmart you. And the question is how I can react to that fast. Like I think like one of the principles of like security is that no system is like 100 % secure at the end of the day. it's, so the question is when something happens, like how we deal with that. And it's completely unknown because we don't know what these people came up with. Right. So how do you see AI being helpful to augment this process there?
I mean, you all the different disciplines that we have like the security, but let's, let's talk more about like the bots, like the things that you are actually obsessed right now and working hard to deliver value there.
David Mytton (44:24.622)
Well before AI was called AI, we had this idea of machine learning and machine learning is a perfect approach and has been used for a long time to try and detect bots because really you're looking at huge volumes of network traffic and web traffic and you can understand just by labeling which traffic is human, which is not based on getting feedback and also what that activity is actually ultimately going to end up doing.
And so machine learning has been used to do this. And there are some really interesting blog posts from CloudFlare about how they've done it because the amount of traffic they see through their network means they've been able to build up these really interesting models over time and improve them to detect bots with even when you have very few signals and things like the user agent match good bots. And there are other systems that would otherwise have not been able to detect them.
So some good blog posts from Cloudflare about how they've done it. I think when we talk about AI today, that often means LLMs or some kind of language model. And that's quite a different challenge because of the speed of inference and the cost of inference. Often when you're detecting a bot, you want to do it in real time. You want to block the user or you want to allow them. And if it's a real user, you start impacting their quality of experience within 50 to 100 milliseconds.
After that, you start seeing real problems with users browsing off the site and going elsewhere and your search quality ranking decreasing because your site is just slow. And so the challenge you have is making a decision as quickly as you can to either block it or to allow the traffic. And I think that's where we need to see some improvements in the current models. It's just the speed of inference to able to use those new approaches versus just doing large ML training jobs and then applying it to every request.
Nitay (46:24.061)
How do you as a startup going by the description you just gave, how do you compete with that? Given like it seems like what you're describing is like there's this natural data mode effect that a cloud for there has a network layer or an open AI or anthropic would have at the bot layer knowing understanding the text that's coming through. Like how do you come into this market? How do you compete with that effect?
David Mytton (46:44.942)
So we compete by differentiating somewhere else. So Cloudflare's bot protection is probably the best in class at the moment. But the way that we think about Artjet is that we'll get there over time. But right now, we're better at the developer experience and integrating security closer into your application. Because Cloudflare exists as a network firewall, essentially. You route all of your traffic through them. They apply your rules that you've defined or that you've allowed them to define.
And then they either block the traffic and your app has no idea that it's been blocked or they allow the traffic. And again, your app doesn't know why it's been allowed. It just sees that the request has come through. In contrast to how we're doing at Artjet with the SDK that's integrated directly into your code, you can use the full context of the request to understand how to define the rules, like setting quotas for different users.
maybe disabling bot protection because the user's logged in and you know who they are. So you don't need to even run any of those rules. And then when we have completed our analysis, we can return all that information to you as part of the SDK. So you can change the logic of your application because certain rules, like if you were, let's say you're hosting a Next.js application on Vassal, it's a JavaScript application. That's one of the first, first tech stacks that we support in the whole JavaScript ecosystem.
You don't need to apply rules that are only relevant to WordPress, to a PHP application. So you can remove entire classes of analysis that add processing time and overhead and management requirements just because we know that you're running a JavaScript application on a particular provider. A network level firewall just doesn't understand that. It will just provide general generic protections and they have to apply to every application that it could be protecting. And that's a really good use case for
DDoS protection, that's the kind of traffic that you just don't want to see ever, the volumetric traffic. But really everything else, you want your application to be able to take a decision because it has more context, it knows more about what your application is, the tech stack you're using, the potential vulnerabilities you might have, the route that it's handling, who's logged in, the context, everything all comes together to making that decision. And so...
David Mytton (49:01.49)
we think we can do that better by building into your application rather than offloading it to a firewall running somewhere.
Nitay (49:08.263)
That's super interesting. I'll come back to the developer side in a second. One more question I have, which is you've mentioned speed and latency and doing things incrementally and not massive workloads a few times now. So what are the latencies that matter? What's the scale that we're talking about? What difference does it make?
David Mytton (49:26.318)
So for our job, we try and take a decision within 25 milliseconds. That's our P95 response time. The P50 is about 10 milliseconds. And we do that by trying to do as much as possible locally. So we take the decision entirely within your environment. There's no agent required. And we do that because as part of our SDK, we are bundling a WebAssembly module, which provides us with a secure sandbox that can execute our analysis code at near native speeds.
So for every request that comes into your application, we're able to intercept it, push it through this analysis module, come to a decision, and then return that back to your application. And so the overhead of using Artjet is very, very low. so we spend a lot of time trying to minimize that overhead so that installing Artjet doesn't slow your application down at all.
Nitay (50:18.705)
That's very cool. Yeah, that's impressive engineering for sure. Tell us a bit about, so you touched on kind of integrating the developer into the flow and developers historically at least don't really care about security, most of them. So how do you like make them care or how have you found or is the market shifting and now they do care?
Obviously, you have a lot of experience kind of getting DevTools right. So what does it take to make a DevTool, a security DevTool even more so that developers will love and use and care about?
David Mytton (50:51.286)
Yeah, developers have this reputation of not caring about security and that can be true for some, I think really it's just, that's lower down on their list of priorities because the incentives are different. Really developers are there to build customer features and to solve bugs and problems. Security is, it really only comes up when the security team has found a problem and is forcing the developers to do something or to use a particular tool or
when customers are asking for some kind of compliance feature or some security audit or certain security features. And again, it's probably the product team trying to force the developers to do things. So developers have this adversarial relationship with security because they often just have to do it and they have no choice. Developers are also thinking about particular problems rather than security solutions. They're thinking someone's abusing our API.
or we're getting those to spam through a signup form, or there's fraudulent transactions going through our e-commerce flow. How do we solve that? And that's why the Artjet product has been designed around a set of core primitives like bot detection and rate limiting, because they're designed to solve particular problems like bots scraping your website or someone submitting your signup form automatically too many times. And so you combine these primitives to solve these security problems and it's in code.
as opposed to all the other security products today, which are not designed for developers. Therefore DevOps or security engineers, and they're just not in code. And security as code is the same thing as infrastructure as code. It's the developers taking a bit more responsibility and building those things into their application, ideally earlier, earlier in the development process. Because if they don't do it earlier, they're to be forced to do it later and probably have to rewrite or refactor a lot of the code they've already been working on.
Kostas (52:49.299)
David, I have question because you said something very interesting about how you deliver the software in order to...
the latency penalty there. Pretty much embed the engine to the system of the user, which obviously is great, but there is a trade-off there, which is now you have a new version and you have to go to all these users that they have embedded this and update it, right? And one of the promises of
SaaS and the cloud is that you know what you don't have to worry about that. It's not like back in the days where you know a new version of your software would come you get a new CD-ROM you have to go like install it on your computer and do like all these things right which is great but obviously there are some issues there when you care about extreme latency for example low latency.
So how do you deal with that? How do you deploy the software? How do you manage the life cycle of the deployment? What happens if, let's say, you break their system, right? I mean, okay, at the end of the day, still you're writing software, matter how good you're trying to be with your CI, CD and like your tasting and QA and everything, something might go wrong. As you said, like systems are like super, super complicated and that's why we have observability. So how do you have observability to...
your system now that it's pretty much distributed on the extreme case of edge, right? So I'll tell you little bit more about that because that's like super fascinating.
David Mytton (54:34.706)
So first thing is Huckjet will fail open by default. And so if we're down or there's a problem, then your app doesn't stop. You can configure that because failing open by default is also a potential security hole. so depending on your level of sensitivity for the particular endpoint, you can change that. You can handle the error however you like. There are quite a few interesting benefits of having security in code and having your rules in code because you get all the benefits of using a code
control system like Git, you can make rule changes through pull requests. You can test them locally. You can push them out to staging before they go to production. These are all things that just don't exist today in the current security systems. Typically you'll test in production because you can't run whatever security system you're using on your laptop. And it's often a click ops type product where you log into a dashboard and you're changing settings by pressing buttons. And then it's suddenly changing your production environment.
And the question is, well, has it broken anything? You'd have to manually test it or you have broken something and all your users are complaining. This becomes a real problem. But like you said, it's in code. So are the developers going to update the dependencies? What happens if you need to roll out a new rule very quickly and you can't wait for your build process and then your deployment process, which might not be as fast as you'd like it to be. And so that's where we have an API that comes in and
This is one of the downsides of WebAssembly is entirely sandboxed. And so there's no shared state. And so if you want to do things like tracking a user across multiple requests, you have to store that state somewhere. In serverless environments, like where developers are deploying today, there is nowhere to store that shared state. So tracking a rate limit across requests is impossible because the serverless function is going to be recycled at some unknown point. It might live.
warm waiting for requests for a few minutes and then it might shut down. When it restarts, it doesn't know anything about what just happened. So we have an API that runs very close to your application. We run it for you. It's in every AWS region and it's designed to be no more than one to two milliseconds away from your application. So that when the SDK needs to track state or it needs to augment a decision, or you need to apply rules on top of the rules you've already defined in code, you can do that through our API.
Kostas (57:05.651)
All right. There's some very interesting engineering going on there. I think we need at least one more episode just to go through that stuff, to be honest. You have to promise that you will be back. We have to talk about these things. One last question from me, and then I'll give the microphone back to Nitai because we're also close to the end here. I was hearing you talking and telling about...
how security is not priority for developers and how you try to change that by creating tools and abstractions that are more natural to them and they can use as part of their workflows. But there's also education that has to happen. These people need to either learn a new SDK or just slightly, let's say, change the way they think about things.
many times product development when it comes to infrastructure and building tooling for developers is a lot about that. How we can kind of educate developers out there, even in some cases, they don't know that they are getting educated right now. So how do you deal with that? Because resistance to change and let's say
big code bases and being comfortable with what you already know and all these things. At the end of the day, it's probably one of the biggest modes that, let's say, established companies have. So it's an interesting thing to hear about creators like you, how you deal with that and how you try to work around these issues with developers.
David Mytton (58:57.89)
This is the real challenge with security and software engineering, because unless the developers inherently care about it and the incentives are aligned for them to care about it, then security is always going to be sidelined. Startups, they don't care about security. All they care about is getting to the first set of customers so that they can prove that this thing that they've built is actually going to work. But they start caring about it when their customers are asking for it or when they have to go through some compliance requirements.
And this is potentially, this is where the innovation in software and technologies come from, just the lack of regulation. And it's an anomaly and there's no other critical infrastructure in the world that has no regulation. Anyone can just go into it with no qualifications and start building systems that millions of people rely on. And we're starting to see this idea of liability come in from certain regulatory bodies.
where the software manufacturer has certain requirements that they have to build into their products to make sure they can be updated and that they deal with security vulnerabilities quickly. And they follow certain best practices with coding to make sure that classes of vulnerabilities like memory safety, for example, there's no reason why anyone should be building anything new in C today because there's no memory safety in it. These kinds of principles should be applied or can be applied at the regulatory layer.
But regulation has its challenges because there are always unintended consequences. And so just getting developers to care about things just because it makes their life better, I think is the right way to approach it, to encourage them so they don't have to think about it. And I see interesting parallels with sustainability. There's often this angle of use less and don't fly as much and don't use your computer as much. think trying to...
change human behavior is basically impossible and people won't do it. And so you have to actually change the system so that even though people might fly more, they use more computing or they do certain things more. This goes back to the kind of motivation of my environmental researchers. We want people to use more of these services and systems, but in doing so, we want to reduce the environmental impact, not through individual action, but because the system itself has changed in a way that the developer or the individual doesn't need to think about it.
David Mytton (01:01:17.324)
the system has changed under them and they have no idea that it's happened.
Nitay (01:01:22.259)
I love how we brought this back around full circle. I'm a strong, strong agreement and belief in what you just said, Rich, is that because I think it's very, very hard to change human behavior no matter how much you incentivize or motivate or regulate it. I think we've seen this in many different aspects of life and space and so forth.
and, I do feel like the right approach to your point is more to incentivize and motivate new innovation to solve those underlying problems and have that innovation drive this, secular market shift demand where you just are almost like.
to an extreme almost ostracize if you don't do it that way. If you have a website today that doesn't have SSL, it's like, what are you doing? It's so easy these days to get a certificate and boom, make it secure. If you have all these kind of basic things that have just become standard practice.
As opposed to a mandated practice, which to your point has so many potential hurdles of like regulatory capture and actually stifling innovation. Like I would argue that the fact that we have very little innovation is actually in a way why tech has been so successful. Now it's a fine balance because to your point, you can get, you know, your crowd strike movements and so on, right? Like to the point of, of something embedded causing issues. yeah, it's a, I think it's a very interesting topic. since we are here kind of a wrapping up soon, just last, last topic I wanted to touch on.
one question. The console that dev stuff, you've obviously reviewed and looked at many, many, different dev tools. I'd love to hear you our audiences kind of like, makes a great dev tool and what makes one that's like meh? Like when you're reviewing, how far, how long does it take you? How many minutes before you're like, this one's going to be amazing versus you're like, like, man, I have to spend another hour on this. Like what makes the difference?
David Mytton (01:03:07.808)
I can generally tell within a couple of seconds whether it's going to be good or not. And it comes down to the documentation. That's the service level analysis. But then the second thing is always once you get past the quick start guide, how is the rest of the product experience? Because the quick start is probably the thing that the team has spent the most time on just because you want to get users activated. But it's also where most products fall down because once you try and do anything more sophisticated,
the documentation's wrong or the APIs aren't there or the capability doesn't exist. And that kind of differentiates the really easy to build developer tools from ones which can become huge businesses. And it's a pain for developers because developers want to write code. They don't want to write documentation, but you've got to consider them the same thing. And the documentation is the product and it has to be updated in parallel with the product and it has to be incredibly detailed.
and cover all use cases and it has to be up to date 100 % of the time, which is a, it's difficult to do. But it is what distinguishes the good ones from the bad ones.
Nitay (01:04:17.439)
Are you starting to see like auto-generated documentation? Are you starting to see AI helping out there or is it manually tuned like expertly written expert engineering docs? And is it like technical writers or is it like just an engineer explaining their API does the best?
David Mytton (01:04:34.606)
I think technical writers come in once you have a big enough team and they can stay up to date. But what I have seen is documentation written for LLMs. If you've used something like Cursor and you just give it the URL to a documentation and ask it to implement this API, it's then going to the website, reading the documentation in real time and trying to give you the answer. And traditional documentation for humans is not the most optimum way for an LLM to explore because typically you'd
Nitay (01:04:54.258)
Mm-hmm.
David Mytton (01:05:04.374)
You split them out into different sections and you might have diagrams and different code with different dropdowns, depending on which language you're using. And the AI just can't use any of that stuff. And so I've seen single page documentation with every single section properly structured with headers and things, but designed for an LLM to consume rather than a human.
Nitay (01:05:25.811)
That's fascinating. All right. We'll probably have to pause it there, but that's super fascinating stuff. it's very interesting to hear that one of the things that makes the biggest difference in DevTools is great documentation. So that's a great insight. We will definitely have to do another episode, dive deeper into console.dev and all the interesting stuff you mentioned around Wasm and all the other cool stuff you guys are doing in ArcGed. So thank you very much for joining us. It's been a pleasure.
David Mytton (01:05:50.552)
Thanks having me.