Subscribe
Share
Share
Embed
Developers building a software business on our own terms.
Ben: Josh, it has been way too long since we recorded our last episode.
Josh: It's been, what, like a month?
Ben: Yeah, about a month. Time flies. But every time I looked up to think about recording, I was like, Oh, I want to get this other thing done for insights.
Josh: We're just too busy. Yeah. So at some point you gotta just, stop talking and start shipping.
Ben: Right, right. Well, John kind of called me out. John did make a kind of call me out on, on Mastodon the other day because I was like, yeah, I'm thinking about writing this blog post. He's all stop thinking and start writing. I'm like, yeah
Josh: that. That's a pretty good quote. I
Ben: It is pretty
Josh: to, clip that one.
Ben: Yeah. Yeah, I appreciated the encouraging thoughts there.
It's just for me It's hey, I'm busy and it take but it takes I don't know Maybe it takes a lot time a lot of time for me to Do a really good blog posts, I got to pull things together, do a lot of research and because my blog posts tend to be like code heavy and example heavy, I got to make sure all that stuff is right.
You know, I don't want to put something out there that's, got a typo or worse that does something that, doesn't do what I say it does. So anyway, I don't know. I'm just making excuses, but,yeah, I need to get on that blog post.
Josh: It's good advice. Yeah, I like I'm a big fan of like just staged editing like the first You know, the first draft is just the first draft and it can be terrible. It can be completely rewritten most likely in the second draft But it can help to get the thoughts out on paper And I would also like to have a better Develop a better writing habit so that, you don't, it would be nice if you didn't have to sit down and, spend like three days.
Writing the perfect blog post. If you had a habit of working on it a little bit,block out an hour in the mornings to, to write and, gradually, polish it up over time until it's ready to [00:02:00] publish and just have that as a constant process, that's ongoing.
Ben: that sounds like a good idea. But when you're talking, I thought, well, I guess I could drop in snippets. of stuff that I'm working on while I'm working on it to help accumulate that post over time, rather than trying to grab it all at once, you know?
Josh: Yeah. Yeah. That's a little bit like the,like the field stone method of, Collecting things as you read, consume content, retweet or whatever. and then use that to, influence your writing or construct your writing. Yeah,
Ben: has excellent posts all the time, and it seems like he can just knock them out in five minutes. Like, you know, he does this tweet stream, and it's like, oh, well, I just made that a blog post. And it's that's a fantastically good blog post.
Josh: So I don't know. He's got some secret that I should get, we should get a, we should get him on here for a writing episode. That's, that is something that we're, internalizing as a writing habit for, like, as a team,building that muscle. So we could probably use it as advice.
Ben: I love the Evil Martians blog and, they always have great stuff. And one of the [00:03:00] things, I don't remember when this was or who it was. It was probably a year or two ago now. but they were, a new employee on the team and part of their blog post was like,it's a thing here at Evil Martians where we all write blog posts.
So here's my first one. And then they dove into whatever they're talking about. And I thought that's really cool. being at a place that encourages that kind of activity. So. You know, we talked this week about doing that amongst us. And I think that's going to be a good thing. I think it just, I know that whenever I write about whatever I just did, then I, it kind of gels it more and I'm also really bad at documentation.
So that helps document things.
Josh: Yeah. Yeah. There's all sorts of benefits and, I think even more so as it becomes easier to write the, just generic blog posts,you can have chat GPT write a tutorial, like a generic tutorial.I've been thinking a lot about what's the future of like content marketing when it's so easy to just generate a bunch of.
Um, and I think that's the thing that like, AI is never going to be able to replicate is like the things you learn as a [00:04:00] team, doing things together, and solving like hard engineering problems that are unique to specific domains. it can make up a story I'm sure about a hypothetical, engineering problem, but it can't actually like, you know, do the work and communicate what you learned.
To your specific audience. So I think that's the path forward for us. and all along, I think that, that just produces better, more interesting content. I think that's why everyone loves the evil Martians blog. And,it's why people have loved our blog in the past as well.
Ben: Well, uh, you know, I, I really enjoyed your blog posts recently about Ahoy! and using Insights to track that Ahoy! data. And I suspect that we'll see a number of Insights related articles on our blog coming up pretty soon as we come up with more use cases for Insights because, it's just too much fun to use.
So why wouldn't we write about it?
Josh: that's my plan is I'm, I've been having fun just exploring, just like Ruby things lately, just to keep up on latest developments, new projects. and I think. A fun exercise is to like, [00:05:00] as a part of that, figure out how can we like integrate insights into this from like a monitoring standpoint?
Just because you can, there's so many different, like it can monitor basically any kind of event. And there's so many different events out there and different libraries and things that could be instrumented. so it's a good excuse to go like tinker with new projects that you just want to play with.
And then you can record a screencast or write a blog post about what you learned,approach it as I'm learning, like a learning in public sort of thing. Uh, so like we have a whole month's worth of things, to talk about, I guess we should say, um, insights is, has reached general availability.
Ben: Yoo hoo!
Josh: We did it. It's we shipped it.
Yeah, we finally shipped it. And it was somewhat anti climatic because we used the feature flag to just, you know, we just enabled it for everybody. it's been there all along.
Ben: It's, yeah, it's been there for a long time, but it was, but on the flip side, despite the less climatic version of the deploy, it was pretty neat because it wasn't, a really stressful deploy.
It wasn't, uh, you know, there [00:06:00] wasn't a whole lot of drama. It's just yeah, it's still working. And, we found a few more bugs, just because more people were using it and there are more, corner cases and things we forgot about, but there wasn't like this huge. like a stress filled, six hour long deploy window kind of thing.
It was just like, yep, it's out there. Great. which is pretty exciting,
Josh: Yeah, and we didn't do,we haven't done like a big, like marketing push yet. Like we haven't done a huge company announcement or whatever, but, our customers, as they use honey badger daily, they're going to start to see, insights in the UI and they can go check it out.
And, and then the rest of the launches is coming in March.
Ben: Right. So we've made some progress towards our goal of a hundred customers spending a hundred bucks a month. we haven't reached our goal yet, but we do have customers who are sending data into Insights and they are passing the thresholds at which they would be charged. so yeah, so we're on the right track,
Now that we actually have a sales page on the website that says what it is, and it's not labeled as preview anymore. Now I'm feeling a lot more comfortable about going to [00:07:00] people and saying, Hey, check out this thing. And, you know, you can actually see the pricing and yeah, it's, I think that'll make, my job of selling it a little easier.
Josh: yeah, I think that, the flywheel has, started um, spinning, and, as long as people keep using it, the money will start to come in.
Ben: Yeah, I was thinking about, we had our marketing meeting yesterday and I was looking at the new revenue for the month of February and, I don't know where I saw this recently on Twitter or somewhere. but talking about like every additional dollar you bring in revenue, you know, sticks around, right?
It's going to stick around for a while. whatever, based on whatever your turn rate is and ours is very low. So every dollar that we add is just going to be. compounding, right? Month over month. So I am just totally excited about having this expansion revenue opportunity in Insights.
Josh: Yeah, me too. and I'm really excited for the future. Just I don't know, we've been building a lot and, I think that's what I've been lacking in recent years is just the building.so yeah, it's been a lot of fun. who knew business crises could,make us excited about [00:08:00] work and software again.
Never let a good crisis go without, go to waste, right?
so should we talk, we did a lot in February and, I don't know if we can get through every single thing, but we could talk a little bit about like, um, kind of tie up loose ends on some of the projects we'd discussed on past episodes, and then maybe talk about what we've done leading up to the launch of insights and then we can talk about what's next, what's next for March and, and beyond if that works.
Ben: Yeah, totally.
Josh: Cool. Well, the first thing I'm really, interested in is I want to know the results of your science experiments. we had talked about launching, or we, we switched our, search, back end, for, for error search from Elasticsearch to Clickhouse. And you were pretty optimistic on the, uh, like the speed and.
Like cost improvements,running like ClickHouse is a search index. So I'm curious,what were the results of that? And you were using the science gem, which lets us what like test, do benchmarks and test the results based on different, sample groups of people using the same feature [00:09:00] with the, like with the different search backends.
How'd that go?
Ben: so yeah, we took a two prong approach. I'll get back to the science thing in a sec. So the other thing we did is we put it on a feature flag, right? So some people have been using Clickhouse as their search longer than the rest of our customers. we did fully deploy it, last week or two weeks ago.
I can't remember now. again, just enabling that feature flag for everybody. so A. on the one hand, there was no, impact that we noticed that was a bad, like no bad search results coming back from humans actually using it. So that was good. All right. So on the science side.the cool thing about the science gym is that not only does it tell you what their performance differences are between your, what you're using now, the control and what you want to use the candidate, but it also gives you an opportunity to compare the results, right?
And you can, you might need to massage those results because maybe the new thing returns results in a slightly different format than the old results. And so a couple of rounds of that I had to go in and so I started when I started using science, I just compared them like the raw results and it was obvious very quickly that A hundred percent mismatch.
I was like, Oh, and then I went in Oh, I see. we're using, string keys here [00:10:00] versus symbol keys there or whatever. but that was really helpful. Like there was this one spot that I totally would have missed, which was, the timestamps coming back from Clickhouse were in, were coming back as a time class, because the library, the active record or whatever is, doing that transform for you.
but the, stuff coming back from Elasticsearch was just strings. Right. And so we had, we did have one place in our UI, which was depending on that being a string and it broke, with, the ClickHouse results. So, you know, finding little things like that behind a feature flag was fantastic, but also seeing it in the science results when it compared the two results was very good.
All right. So the, so step one was I had to get all of the, get all the results to match and found a couple of little interesting, behavioral differences between Elasticsearch and Clickhouse. And so that took a few rounds of work. never really actually got it to a hundred percent in all the cases.
So we were testing like five paths through our code, I think. And, uh, you know, I guess as. Bit of explanation, like Wi Fi paths. We have a lot of different places that use the search index in our UI. So there's like the [00:11:00] notices timeline, there's notices list, there's the actual search that you type in. So there's, four or five places like that, that were going to the search index.
And so I wanted to test all those paths. And, there were some that were like. if you've ever compared to accounting systems,you'll often notice that like the numbers just don't match, right? ah, like the time buckets are different or just the, when the data arrives can be different.
So it puts it in a different bucket, you know, so there's that. So you can't always get a hundred percent match, but I got it pretty close. Like in, in some cases we had a hundred percent for every execution and then some others, it was, less than a hundred percent, but I eyeballed those.
I reviewed them manually and yeah, they look
Josh: Well, and these are things that no, no one's going to notice because no one is comparing the results side by side, but you,
Ben: Thankfully. Yeah. but on the performance side, it was really exciting. So if you take out the outliers, so that the, the gym that we use to the lab tech gym that we use to actually evaluate the science experience, it gives you this cool little chart, that shows you what the returns were on how fast it was, or if you have bad case, how slower [00:12:00] it was.
And in our case, if you leave out the outliers, we were getting a 30 times, performance improvement, right? And in our case, that means that responses were coming back from the search index either anywhere from 100 milliseconds faster to a full second faster. And, when you're talking about UI responsiveness, like slicing off a second response time is a huge deal.
So yeah, it was pretty exciting.
Josh: That is amazing. Nice work.
Ben: Thanks. Yeah.
Josh: Yeah. Well, I haven't noticed any, I haven't noticed a difference since it's been, since it's been deployed. So
Ben: we had one customer who sent in a support request about how the search wasn't behaving like it was before. And, it was a minor issue. So the way we're doing the, now you basically have to provide the asterisk if you want to do a substring search. Whereas before Elasticsearch just assumed that you wanted to do that sort of thing.
so aside from that little nit, um, yeah, it's been great.
Josh: very cool. And I guess we should, just real quick, we should address, because everyone who [00:13:00] knows anything about ClickHouse, and using it in this way is going to ask, what about updates? so I know that we've encountered some,Rough edges around updating data in ClickHouse.
what's our approach there.
Ben: Yeah,we ran into this problem because we allow you to merge, errors together, right? And so now there's, if you're searching across what was two errors and now it's one error, what happens to all that old source data? And so we actually had to Build like a layer on top of the search that can translate.
We keep track of those moves and then we just massage the data as it's coming back out and it's like, Oh, it moved. Okay. And we'll switch place that data with this data. So they don't actually update the old data in the search index. We just mask it and, it works pretty well.
Josh: Yeah. So in this case, the answer is don't do
Ben: Don't do updates.
Josh: we found a way around
Ben: Yeah, that's really the answer. Uh, and in later versions of ClickHouse, the delete story is better. So now you can actually delete data and a way that won't kill your cluster. but still updates. not so good.
Josh: yeah, cool. that's good to [00:14:00] know. nice. that's one thing down and also that's some cost savings for us being able to turn off the elastic search cluster eventually. so that'll be nice. Cool. so on the product side, I think we talked about some of the things we were planning to do, on the last episode, in order to launch insights.
So we can maybe run through some of these pretty quick. we created a billing UI, so we have, the Stripe integration finally. and, that turned out actually to be,not as hard as I imagined, Roel worked on it and, we ended up basically just, billing it as an add on.
we just have a, I have an additional set of plans that you can choose in the UI, in addition to your current honey badger plan. And, and that's basically it. and I think we even have a couple of customers that have Started to choose some plans in there, so that's a good sign.
Ben: Yeah, we also did a default on that. So if someone doesn't sign up for any plan, because, you know, we have a bunch of customers who exist today who, don't have this plan, because it's brand new, we have a default allotment. right now it's set to 50 megabytes a day. we went back and forth on that.
We're like, do we give everyone this free [00:15:00] plan or do we just, and we decided not to do that and just give everyone the free allotment if they don't have a plan.
Josh: yeah, I was doing the math on that last night because someone on Twitter had mentioned that they were looking for a paper trail alternative. Now the paper trail is being absorbed by I think SolarWinds observability platform and is going away. And, this person on Twitter said that they were spending, uh, like 7 a month for one, one and a half gigs, of ingest per month.
and I did the math and about 50 megabytes a day is about that, with seven day, retention. But the good news is that we are giving that away for free. So can, use the Honeybadger developer tier.
Ben: That, that may not be a complete coincidence.
Josh: Right. I see your evil plan.
I did spend a lot of time thinking about pricey and comparing it to alternatives. Yeah. Yeah. I really like how the pricing turned out. so yeah, on the pricing side, as a part of the billing, we figured out like what we're going to charge. And I think if you're not sending a lot of volume, it's a little more, but once you reach the mid range of the plans that we [00:16:00] chose, it's, what is it roughly around a dollar, like a dollar per
Ben: yeah, something like that.
Josh: we're not, counting per month. it's ingest per day, which is different from some of the other tools out there.
Ben: yeah, we did decide to go with a daily quota rather than a monthly quota. there aren't many logging systems that go that way. There are a couple, but the reason we did that is, well, one, it's kind of an experiment. We want to see how that goes, but two also from a, from an operational standpoint, like if we have to handle someone, 100 gigs of data, let's say from someone in a day, that's very different than handling 100 gigs.
Of data from someone over a month, right? We have to have a much different, scaling kind of thing for that. And so we wanted to account for that. Like we wanted to have that kind of, cost structure reflected in our pricing. And then also what I think one of the, when the ongoing issues that we have with honey badger is, we have people who hit their monthly quota limit.
You know, at some point during the month, maybe it's, week three or whatever, and they're like, oh, yeah, we had this, they had this big burst of traffic and that used up all of our [00:17:00] quota. can we get a waiver, for this month? And we almost always do that because, yeah, we totally understand, right?
You know, that happens. but, yeah. if we can avoid even having to have that interaction, because like, yeah, it's fine. Like it'll go over and the next day it'll be fine. then that saves time and frustration for everybody. So, um, that's, that's kind of those motivations and I'm really curious to see how it goes.
I hope people like it and I hope it works out well.
Josh: Yeah. so the way it works is that if you go over your daily quota, you're not going to get cut off immediately. but because it's being tracked daily, we can say if you've exceeded your quota for like a certain percentage of the days out of the month, like you should probably upgrade to the next plan.
And that's the point where we can recommend that to people or even do it automatically or something like that.
Ben: Yup. Yeah, so we have that tracking now, and that's part of what we've been building over the past couple weeks as well, make sure we have that in place, and we have it posting to our Slack channel, so we can just keep an eye on it for now, right now we're not really enforcing any quotas, and so we'll see, if we see the same project or the same account show up in [00:18:00] Slack every day, oh, they went past it today, and today, and today, then we'll reach out and say, okay, you should upgrade, but I think, when it comes to logging, you don't really want to cut people off.
Okay. All right. again,like with Honeybadger, like we have on our website, like when you're having a bad day, the last thing you want is for your tool to stop reporting your data for you, so yeah, we're trying to be like super flexible on this and, do what makes sense and what's easier for people.
Josh: yeah, that makes sense. so we did build the throttling, and the ability to, enforce quotas. and we, initially you were thinking, we could probably, launch without that. and we changed our minds. Why did we change our minds?
Ben: we changed our minds because one, one of our test customers decided to send us a whole bunch of data and caused us some operational issues. And so they're like, I guess we cannot launch without having some way to control this. We gotta be able to turn people off.
Josh: Yeah. Yeah. So we had a couple opportunities in February to realize that we need to have some controls around the scaling story. I think we had probably more, incidents than usual, would you say in February?
Ben: Yeah. Yeah. I think [00:19:00] so. And, you know, cause we, we built this whole new pipeline and, I guess before we say that, it's obviously this did not impact our production error pipeline. Cause that's completely separate. that's solid. But this is brand new. This is a brand new pipeline.
We're using new code, like we're doing a lot of work in go for the first time. so yeah, so there were some. That was some growing pains and some learning experiences. And, sometimes when our Kafka cluster was, begging for mercy because we were way, way, way behind the offset, you know, the backlog was super high.
but I think those are really good learning experiences and,we've got, we're much, we're in a much better place. We we're, uh, three weeks ago.
Josh: Yeah. Well, I mean, this is totally expected. This is like what, this is what should be happening and this is what we should be doing. no one said this is going to be easy and dealing with this amount of data and especially as we, have more to deal with, we're going to reach the limits of the system and have to do performance tuning and, optimizations and all that.
luckily for us, we're, smart software developers who can solve these problems and like challenges.
Ben: For sure. And [00:20:00] that's the benefit of having the preview period too, right? It's A, not everyone is using it. It's a small, it's a small set of people using it. And then B, like if things go badly, you can be like, Oh, sorry, things got broken, And it's not going to cost someone their job, right?
Cause they know they're just, it's experimental. It's it's going to, it's going to change. It's going to have some problems.
Josh: Yeah, someone was talking about that recently. It might have been Jason Cohen. But about how, like large companies, who launched products like this, like they're dealing with scale from day one and so they have to do things much differently than like a small company who can like gradually scale up, like a product.
Cause if you are, if you're already at scale, like you can't just throw out, some scrappy,We'll fix it in production type. Type thing, but that's a benefit for smaller companies that can move faster.
And iterate a little quicker.
Yeah. So, uh, one thing I would say that though, is like, it has felt like we've gotten a little bit bigger because we couldn't just launch this without any kind of scaling protection. We're a little bit in
Ben: yeah, we're kind of in between, or we're not at the point where you just throw it out [00:21:00] there, but we're not at the point where it has to, take a year to launch something either.
But, yeah, it's definitely took longer because we had, we knew we had to build in some capacity for scale from the beginning.
Josh: Yeah, we might actually be suffering from that problem a little bit, come to think of it, but also, I mean, that's why we chose to like do the longer rollout and alpha with a limited set of people because it gave us that ability to gradually scale,and solve some of those like performance issues.
Yeah.
Ben: we had some time also in February to build the, update the honey badger gem. So I think, Jan did that for us and wrapped up that work in January. And we really were able to test it out over this past month with some of our own personal projects. And from day one of general availability, we have the Ruby gem supporting, you can now call honey badger dot event, and you can send any data insights that you want.
And then Pangradios a few months ago, I think he, he beat us to the punch. he added that to the JavaScript library. So you can do that from a console or from your client side or node code as well. You can use the log event method in our JS library to send events. So, [00:22:00] uh, eventually we will add this to Python and Laravel and so on.
but we made a decision that we didn't want to hold up. The launch of insights by requiring us to have all the client library supported, out at day zero. And we decided well, we'll focus on some of the more basically go in order of usage, Ruby, JavaScript, then probably Laravel next and then Django after that.
Because, it takes time, right? To write some code. You don't want to, Put something out there that's going to be embedded in someone's application. That's going to mess things up, when they start trying to send a bunch of logs, right? so instead what we've told people on our instructions is, rely on vector, which is, it's a very lightweight, very rock solid, agent that can run, that can basically look at your log files.
And then send them to us, to our API. So worst case scenario, if you're running on a, on an app, that's not supported yet by our client libraries, you can run vector and it can grab your log files and send them over to us. So it's been cool to see the different customers we've had using the different platforms.
We've had [00:23:00] someone using the gem. We've had someone using the JavaScript library. We've had a bunch of people using the vector. We even had a person who's what about CloudWatch? I want to send CloudWatch logs. And I had a little side quest where I went and.got dirty and figured out how to get, that going, which was a little bit of an adventure.
so it's fun seeing all these different, oh, and someone using fluent, which is something I never touched before. So that's, it's just kind of fun to see all that happening. Yeah.
Josh: yeah. Oh, hotels coming up and, a lot, all the different platforms and integrations. but yeah, I think with the client libraries, in particular,we're starting with like the lightest touch possible by just basically adding a method to each client library, which is honey badger event.
And it sends an event to honey badger. and of course it does batches and does some performance things and underneath to make it, perform well. but the next thing I want to do, like, I'm already wanting. like full rails instrumentation for this, because,we can basically send all the active support events and, and then we can start doing some [00:24:00] APM style performance monitoring things.
and I'm really excited for that. So, um, I think we're going to be diving into instrumentation pretty soon. and I, and also maybe Otel or open telemetry as part of that. that's something that's one thing people are using that automatically instruments, a lot of different frameworks already. But I don't know about you.
I'd like to have, some like just simple, straightforward instrumentation built into our gem that makes it an easy, like, you know, three minute install and you don't have to think about it.
Ben: Well, because you asked, I. Happen to have an answer for that. I released a new gem. was it last week? Yeah, it was last week. first time I've released a new gem in a very long time. It might be the first one since Faker, but I can't remember. but it was the Yebeda Honeybadger Insights gem.
So basically if you're familiar with the Yebeda gem or suite of gems, I guess I should say,it's an instrumentation. Set of gyms for Rails and you throw it in there and it's got instrumentation for sidekick, for Puma, for all sorts of [00:25:00] things. and then it has a bunch of reporters, so you can send them to, a lot of people use Prometheus.
That's the official blessed one from Evil Martians. And, but there's also like a cloud watch one. And so I add an exporter for insights. And so if you're already using Y Beta, I guess that's how you pronounce it, I don't even know. then. You can just add the beta honey badger insights, Jim, I have to link in the show notes.
and then you can just have all the instrumentation automatically showing up and insights. It's great. I play with, I, I started a new rails app for the first time, probably in how many years and set it up and it was like, literally drop it in, just add water and go kind of thing.
It was fantastic.
Josh: That is so cool. Okay. I'm going to have to go, I'm going to have to go try that, add it to one of my, one of my hobby rails apps and play around with that. I'm excited now.
Ben: Yeah, and Otel there's a suite of stuff for all the languages when it comes to Otel So we definitely want to yeah lean on the what's out already out there and support all the popular formats rather than trying to build A whole big
instrumentation thing ourselves.
Josh: I do love that trend. Like the trend [00:26:00] of everything, just interoperability and open instrumentation, like it just solves, I don't know, just back in when we were initially building Honeybadger client libraries and. I remember commiserating with like the New Relic client team when I would run into them at conferences about, we all had the same problems and we were all building the same instrumentation and it was just, it was a pain.
and it's nice that there's standardized instrumentation coming out that, uh, that kind of everyone can plug into.
Ben: Totally.
Josh: Cool. so back to insights, we got our landing page and pricing pages updated on the marketing side. And, got that all shipped. And I think, all that's left is to like actually start telling customers about it, emailing people. We got to write a blog post. Do some of the actual, marketing things that companies have to do.
I don't know. I just want to be building, but,we should probably do some marketing here,to tell people about this.
Ben: some people might say, well, you know, you should have had that stuff ready, for launch day, right? You should have had that email ready to go and that blog post ready to go. [00:27:00] we actually decided intentionally not to do that. We wanted to have a little bit of a slower drip out there because we wanted, again, to ramp up kind of slowly and have customers discover it and make it a more organic process.
So what we did do is we did put a UI. Um, and then we also have a little notification that shows, cause there's a new, UI element for a new tab, that you can see when you're in your projects list. And so we put a little blue dot there, this thing is new and they put a little message in the sidebar that's dismissible that says, Hey, you should check out this new thing.
And then when you go into that new thing, there's a little message there. It says what it is. We decided to skip that whole like tour thing where you got to, go through seven steps to actually get to the use of the project. So we wanted people to dive in as soon as possible. but. We wanted to do that kind of easy onboarding first and get people like starting to use insights before we did this big push.
We didn't have, so we didn't have, yeah, hundreds of thousands of people showing up on day one. so yeah, I guess the launch that was a little lag, but,I'm liking how it went. the thing that really got me was like on Friday, last Friday when, Like the dust had settled from the launch.
And, I was like, [00:28:00] okay, now I'm ready to really promote this thing. Now we've got to get that blog post, that email out there. Cause I, I was before then I was nervous. I was like,is it going to work? Is it going to fall over? but then once I started seeing customers starting to just get into it and use it, I was like, yeah, this is totally ready.
Let's do this.
Josh: Yeah. Yeah. With a SAS company, you don't really need to do huge product launches. like you can, and that's like a common thing to do, but, like there's all kinds of things that we can do over time. Like we're going to be running ads and we're going to be building up more of a content marketing pipeline and we're going to be building like drip email campaigns and all those sorts of things, basically like the flywheel marketing flywheel type projects.
Ben: So I got a question for you. we've been working on this project for a long time and we had. A pretty long, trial or preview period. So looking back, is there anything that you would do differently on how we built this thing?
Josh: I think we, we built it for a really long time, like you said, and I wish we would have shipped it sooner. I don't know if we could have, I mean, it [00:29:00] has been a lot of work and we've been doing a lot of other things in the meantime.
but I think like, probably like we spent a lot of time, Last year, working on just like more general, I don't know. we did a lot of things, but I think we could have pushed a little harder to be honest,to get it actually shipped. And,there's a lot of reasons we didn't, and we were still like a lot of this, like strategy, like we were still learning, like we were still learning last year, like what this thing is, because initially we were planning to launch this as a separate product.
And actually, now that I say that, I think that was probably the mistake, was deciding to do this as a separate product instead of just building it into a honey badger from day one. I think we would have gotten to market a lot quicker if we had done the latter.
Yeah, I was thinking the same thing like having that pivot. We spent several months going down that path of the separate product. And, I think I still want to get back to that someday because I think there are things that we can do with this code base and, what [00:30:00] we have that are outside of HoneyBadger could be interesting.
Ben: but yeah, I, having gone down that path and now change course and bring it into HoneyBadger, I think about, how extractions really are. So powerful, right? When you build something and then you see a pattern and you extract that into something new, like I did with rails kits back in the day, you know, after having built three different billing systems and rails apps and did the same thing over and over again, extracting that out into a product.
And, rails itself being an extraction from base camp, I think it would have been, it would have been smoother, faster. It might've been uglier and rougher, you know, because extractions, you have to clean things up, right? That's how it's part of it, cause you learn as you go. but yeah, if we could rewind, that's one thing I think I would do differently, butmaybe we'll get around to that second product that uses this backend and that'll be awesome too.
Josh: Yeah, I don't know. I think there's a lot of potential there for that type of product, but the problem for us is that, like in terms of revenue, like we have existing [00:31:00] customers and existing revenue and existing product, an existing market. And, and we know how to reach that market.
so why not bring it to what we have our first and then go after the new or whatever larger market is at some point in the future, once we've learned,what it can do For our existing market. and just like having this in honey badger, like the more we've,built on top of it, it's been a light bulb moment that like there, it enables so much more that we can do, to make the rest of the product, better and add new features and, do deeper, like.
Cross integrations between different, like different product features, like uptime monitoring and error tracking and observability. They all tie together. And, the more we can integrate them, like it'll just, it'll make adoption better. It makes the product better. And,
Ben: Yeah.having been playing with insights for the past few weeks. I'm just, I'm so excited about the. Things that it could open up in the product over the next year, things that we can build on that we have this incredible [00:32:00] foundation now, that, I guess as a counterpoint to what we just said about doing it inside of honey badger first, I think one of the benefits of doing it outside of honey badger first was that, like Kevin's goal was really to make it very agnostic, right?
Not be focused on one use case. And I think he did an amazing job. He was a, completely successful at that because now we have this foundation where we can do all kinds of cool things that unlocks a lot of potential that would have been much more difficult to do without. so maybe it was worth it in the end.
Josh: it's a toss up. And I think mainly just I like the way that we built it. I just, I think that from day one, we should have been thinking of our honey badger customers and audience and not like a new green field, like the mass observability market. Um, but on the other hand, like you said, like thinking about.
Monitoring and like what are the current trends? like let's say honey badger pre insights was like,V1. and we've always talked about wanting to do honey badger V2. I feel like we're kind of getting there now, by bringing this [00:33:00] new, modern observability, approach into the product and then gradually building the rest of the product around it.
yeah, I feel like we're bursting with ideas right now about how, like how we can now revamp, the rest of the product. So, um, maybe, it's a little harder for us to like actually do the, you know, just rebuild. Honey badger from the ground up,like base camp might've done with, and do the V2.
but this was kind of a backwards way to, to get some of those benefits, get some of those fresh ideas into the app, that are drastically different from, like the, the existing paradigm.
Ben: And without disrupting what's already awesome about it. Cause people can still use honey badger. Like they used it two days ago. And not change a thing, They could completely ignore insights, but we can behind the scenes, we're doing all kinds of cool stuff that I think will bring even the old experience, you know, new polish and new features that will be based on the insights foundation.
Josh: Yeah.
Ben: Yeah.
You know. I'm [00:34:00] excited to, and, we've gotten, we tried, I guess we started in 2022 on the shape up process, where we were doing the six week cycles and we got away from that as we got really deep into the building on, insights, because it was just such a huge project and we couldn't find a good way that.
Felt like a good fit for doing this, you know, breaking it up into those kinds of cycles. But now I think we're at a point where we can get back to the cycles. I'm excited about that. And the only problem is like picking the next cycle. there's so many options of things we could do.
I'm, I don't know.
Josh: yeah, there's a lot of work to do. but that's a good problem to have. I guess we'll have to strike the right balance between what the product needs and what we want to do. because yeah, I want to have fun while we're doing this.
but I also want to build the things that we need.
Ben: Yeah.
Josh: Cool. So what's next? I guess I think like we're going to be switching into marketing mode here. for a little while, we want to actually do that public launch now.write the blog posts and the emails, start [00:35:00] some advertising,keep producing, Content do some, we, I think we have a lot of team engineering blog posts that we want to work on, that should be really good.
I'm really excited for some of the content that we can put out there and share about some of the things we've learned while building all of this, and launching it.
Ben: I'm really interested in doing some outreach to existing customers. I think we have a lot of customers who could use this and who probably aren't, paying attention to what we're doing on a daily basis. Cause they're, you know, they're paying attention to their jobs and their own stuff.
So I think we have a lot of. educational institutions and we have some large enterprises who are using Honeybadger that I think we could get in there and say, Hey, you'd love us for exceptions. what are you doing for structured logging? And, check us out. I've tried to do some outreach in the past few years ago, tried to do that and it didn't work out so well.
And so I'm gearing myself up like, okay, I got the energy. I'm excited about the product. And I'm Ready to try it again. So I'm hopeful that I don't have to come back in six months and say, Oh, that was a failure, you know? Um, but at the worst, if I get some people on the phone and we're talking about honey badger, the product, and they tell [00:36:00] me what they, you know, just talking to customers is a great thing, right?
So, uh, that's something that I've neglected a lot in the past. And so I'm looking forward to getting more in the groove on that.
Josh: yeah,
it's always good to be talking to customers. So, um,it definitely can't hurt. Um, but yeah, so we've, got about, uh, a little less than a month, until March 31st. So, now's the time the product is out there. the people can pay us for it. So now we just got to get the customers in the door and, uh, and I think Like you said, that's, that has already begun, because we're starting to see usage.
Ben: And, I think usage is where, the, expansion revenue comes from. So I'm excited to get our first dollar in the door for this. Yeah. Fun times.
Josh: Cool. Well, uh, this has been FounderQuest, you can find us on founderquestpodcast. com and if you can give us a, a review in the,Podcast app of your, preference. That would be great. We really appreciate those. if you can write something, but definitely give us, like [00:37:00] five stars or whatever.
And, uh, we'll, we'll catch you again next time.
Ben: See ya.
Outro_1: Founder Quest is a weekly podcast by the founders of Honey Badger Zero Instrumentation, 360 degree coverage of errors, outages, and service degradations for your web apps. If you have a web app, you need it available@honeybadger.io. One more from the founders. Go to founder quest podcast.com. That's one word where you can access our huge back catalog of episodes.
Founder Quest is available on iTunes, Spotify, and other purveyors of fine podcasts. We'll see you next [00:38:00] week!